Monitor

Purpose

This system hosts an Icinga 2 instance to centrally monitor the services in the CAcert network (especially for security updates and certificate expiry).

Note

To access the system you need a client certificate where the first email address in the Subject Distinguished Name field is a cacert.org address. Subject Alternative Names are not checked.

If you are the administrator of a service please ask the monitor admins to add your system to the monitoring configuration and add you as system contact to allow for notifications and tasks like service outage acknowledgement, adding notes, rescheduling checks or setting downtimes for your service.

Administration

System Administration

Application Administration

Application

Administrator(s)

Icinga 2

Jan Dittberner

Contact

Additional People

No additional people have sudo access on that machine.

Basics

Physical Location

This system is located in an LXC container on physical machine Infra02.

Logical Location

IP Internet:

213.154.225.230

IP Intranet:

172.16.2.18

IP Internal:

10.0.0.18

IPv6:

2001:7b8:616:162:2::18

MAC address:

00:ff:73:b3:17:43 (eth0)

See also

See Network

Monitoring

internal checks:

Monitoring checks for monitor.infra.cacert.org

DNS

Name

Type

Content

monitor.cacert.org.

IN CNAME

infrastructure.cacert.org

See also

See Wiki page SystemAdministration/Procedures/DNSChanges

Operating System

  • Debian GNU/Linux 11 Bullseye

Applicable Documentation

This is it :-)

See also

Setup package update monitoring for a new container

Services

Listening services

Port

Service

Origin

Purpose

22/tcp

ssh

ANY

admin console access

25/tcp

smtp

local

mail delivery to local MTA

80/tcp

http

ANY

Redirect to https

443/tcp

https

ANY

Icingaweb 2 frontend

5665/tcp

icinga2

monitor

remote monitoring service

5432/tcp

pgsql

local

PostgreSQL database for IDO

8000/tcp

git-hook

internal

HTTP endpoint for git-pull-hook

Note

The ssh port is reachable via NAT on infrastructure.cacert.org:11822

Running services

Service

Usage

Start mechanism

Apache httpd

Webserver for Icingaweb 2

systemd unit apache2.service

cron

job scheduler

systemd unit cron.service

dbus-daemon

System message bus daemon

systemd unit dbus.service

git-pull-hook

Custom Python3 hook to pull git changes from the cacert-icinga2-conf_d repository

systemd unit icinga2-git-pull-hook.service

Icinga2

Icinga2 monitoring daemon

systemd unit icinga2.service

openssh server

ssh daemon for remote administration

systemd unit ssh.service

Postfix

SMTP server for local mail submission

systemd unit postfix.service

PostgreSQL

PostgreSQL database server

systemd unit postgresql.service

Puppet agent

configuration management agent

systemd unit puppet.service

rsyslog

syslog daemon

systemd unit rsyslog.service

Databases

RDBMS

Name

Used for

PostgreSQL

icinga2

Icinga 2 performance and alerting data

PostgreSQL

icingaweb2

Icingaweb 2 group and user preference data

Connected Systems

  • Extmon

  • Git for triggering the git-pull-hook on newly pushed commits to the cacert-icinga2-conf_d repository

Outbound network connections

Security

SSH host keys

Algorithm

Fingerprints

RSA

SHA256:8iOQQGmuqi4OrF2Qkqt9665w8G7Dwl6U9J8bFfYz7V0, MD5:df:98:f5:ea:05:c1:47:52:97:58:8f:42:55:d6:d9:b6

DSA

SHA256:Sh/3OWrodFWc8ZbVTV1/aJDbpt5ztGrwSSWLECTNrOI, MD5:07:2b:10:b1:6d:79:35:0f:83:aa:fc:ba:d6:2f:51:dc

ECDSA

SHA256:GWvYqhQUt9INh/7VRVu6Z2YORoy/YzgBxNBmX+ZvMsk, MD5:48:46:b1:5a:4e:05:64:8a:c3:76:33:77:20:91:14:70

ED25519

SHA256:L5roC867bvxDJ0ckbhIQOt2A9Nh1RQBVuIJFWwrPLG0, MD5:10:94:56:09:5b:a2:28:ab:11:e0:0f:6e:e4:0c:38:bb

See also

SSH host key list

Non-distribution packages and modifications

The Puppet agent package and a few dependencies are installed from the official Puppet APT repository because the versions in Debian are too old to use modern Puppet features.

Risk assessments on critical packages

Icinga 2 and Icingaweb 2 are well maintained community projects with a good security track record.

Apache httpd has a good reputation and is a low risk package.

The system uses third party packages with a good security track record and regular updates. The attack surface is small due to the tightly restricted access to the system. The puppet agent is not exposed for access from outside the system.

Critical Configuration items

The system configuration is managed via Puppet profiles. There should be no configuration items outside of the Puppet repository.

Keys and X.509 certificates

All keys and certificates are managed in the file hieradata/nodes/monitor.yaml in the CAcert Git repository cacert-puppet.

  • Certificate for CN monitor.cacert.org, see details in the certificate list

    • certificate in file /etc/ssl/public/monitor.cacert.org.chain.pem

    • private key in file /etc/ssl/private/monitor.cacert.org.key.pem

  • /etc/ssl/public/monitor.cacert.org_client_cas.pem CAcert.org Class 1 and Class 3 CA certificates (allowed CA certificates for client certificates)

  • /var/local/ssl/crls/

See also

CRL fetch job

The script /etc/cron.hourly/update-crls is used to fetch CRLs once per hour.

Apache httpd configuration

The HTTP and HTTPS VirtualHost configuration is defined in /etc/apache2/sites-available/icinga-nossl and /etc/apache2/sites-available/icinga the HTTP VirtualHost redirects to the HTTPS VirtualHost.

Icinga configuration

The Icinga 2 configuration is stored in the /etc/icinga2/ directory. The /etc/icinga2/conf.d/ directory is managed in CAcert Git repository cacert-icinga2-conf_d repository which has a post-receive hook to trigger updates of the Icinga 2 configuration and performs a graceful reload when configuration has changed.

Tasks

Changes

Planned

System Future

  • No plans

Additional documentation

See also

References

Wiki page for this system

Wiki page SystemAdministration/Systems/Monitor