Beholder¶
Purpose¶
The beholder system is the new monitoring system for CAcert based on Icinga2 with Icingaweb and Icingadb. The system will replace Extmon and Monitor.
Application Links¶
Administration¶
System Administration¶
Primary: Jan Dittberner
Secondary: None
Application Administation¶
Application |
Administrator(s) |
|---|---|
Icinga, Icingaweb, Icingadb |
Contact¶
Additional People¶
No additional people have access to the system yet.
Dirk Astrath has access to the Icingaweb2 web interface.
Basics¶
Physical Location¶
The system is a virtual machine on a Hetzner ARM64 system. The machine has 2 ARM64 cores, 4 GiB of RAM and 40 GiB of SSD backed disk space. Console access is available via the Hetzner Cloud console.
Logical Location¶
- IP Internet:
- IPv6:
- MAC address:
96:00:02:73:0e:7c(eth0)
See also
See Network
Monitoring¶
- internal checks:
- external checks:
Monitoring checks for beholder-from-infra02, Monitoring checks for beholder-from-infra03
DNS¶
Name |
Type |
Content |
|---|---|---|
beholder.cacert.org. |
IN A |
49.13.65.97 |
beholder.cacert.org. |
IN AAAA |
2a01:4f8:c17:7495::1 |
See also
Operating System¶
Debian GNU/Linux 12 Bookworm
Services¶
Listening services¶
Port |
Service |
Origin |
Purpose |
|---|---|---|---|
22/tcp |
ssh |
ANY |
admin console access |
25/tcp |
smtp |
local |
mail delivery |
80/tcp |
http |
ANY |
redirect to https |
443/tcp |
https |
ANY |
Icingaweb |
2003/tcp, 2004/tcp, 7002/tcp |
carbon-cache |
local |
Graphite API |
5432/tcp |
PostgreSQL |
local |
PostgreSQL database |
5665/tcp |
icinga2 |
213.154.225.224/27, 2001:7b8:616:162::/64 |
Icinga2 API |
6379/tcp |
redis |
local |
Redis Cache |
8443/tcp |
json2file |
Webhook for Icinga2 configuration updates |
Access to services with specific origin IP addresses is limited via local
firewall rules implemented via nftables rules in /etc/nftables.conf.
Running services¶
Service |
Usage |
Start mechanism |
|---|---|---|
Apache httpd |
Webserver for Icingaweb2 |
systemd unit |
atd |
task scheduler |
systemd unit |
carbon-cache |
metric collection for Graphite |
systemd unit |
cron |
job scheduler |
systemd unit |
dbus-daemon |
System message bus |
systemd unit |
icinga2 |
Icinga2 monitoring agent |
systemd unit |
icingadb |
IcingaDB API |
systemd unit |
incron |
inotify cron daemon to start actions based on filesystem events |
systemd unit |
json2file-go |
Webhook that accepts json formatted data and writes it to files |
systemd unit |
openssh server |
ssh daemon for remote administration |
systemd unit |
Postfix |
SMTP server for local mail submission |
systemd unit |
PostgreSQL |
PostgreSQL database server for Icingaweb2and IcingaDB |
systemd unit |
QEMU guest agent |
Agent for the QEMU/KVM virtualization |
systemd unit |
Redis |
In memory cache/database |
systemd unit |
Databases¶
RDBMS |
Name |
Used for |
|---|---|---|
PostgreSQL |
graphitedb |
Graphite data |
PostgreSQL |
icingadb |
IcingaDB data |
PostgreSQL |
icingaweb_db |
Icingaweb2 configuration data |
Connected systems¶
Infra02sat and Infra03sat monitoring satellites
Outbound network connections¶
DNS (53) to Hetzner cloud DNS resolvers
mirror.hetzner.com and deb.debian.org Debian mirrors
matrix.tchncs.de for Matrix notifications
Security¶
The beholder system uses server certificates from letsencrypt. It accepts client certificates issued by the CAcert class3 root and allows access for certificates that belong to the cacert.org domain. Further access restrictions are setup in Icingaweb2 itself.
SSH host keys¶
Algorithm |
Fingerprints |
|---|---|
RSA |
|
DSA |
- |
ECDSA |
|
ED25519 |
|
See also
Dedicated user roles¶
None
Non-distribution packages and modifications¶
None
Risk assessments on critical packages¶
The primary application is icinga2 and its surrounding applications (icingadb, icingaweb2, graphite). All applications come from security supported Debian packages. Icinga2 uses mTLS for strong authentication. Access is further restricted by nftables firewall rules.
Critical Configuration items¶
The system is NOT managed by Puppet to avoid a dependency on the availability of the Puppet system.
Configuration for Apache 2¶
The Apache 2 web server is configured by files in /etc/apache2.
Configuration for graphite¶
Graphite and Carbon (the time series database for graphite) are configured in
/etc/graphite and /etc/carbon.
Configuration for Icinga2¶
Icinga2 is configured by files in /etc/icinga2. The monitoring
configuration is managed in a Git repository at
https://code.cacert.org/cacert/icinga-zones. All changes to the monitoring
configuration must be done via commits to this repository.
The repository is cloned to /etc/icinga2/zones.d. The repository has a
configured Webhook that triggers json2file to write to
/var/spool/json2file-go/icinga_zones. The directory is monitored by
incron. If a new file is created in the spool directory incron calls
/usr/local/sbin/update-icinga2-zones.d which pulls changes from the Git
repository and takes care of sanity checks and a reload of the Icinga2 service.
Credentials and other system specific configuration is stored in
/etc/icinga2/constants.conf.
Configuration for icingadb¶
Icingadb is configured in /etc/icingadb/config.yml.
Configuration for incron¶
/etc/incron.d/icinga-zones defines the incron configuration to call the
update script.
Configuration for json2file-go¶
/etc/json2file-go contains configuration files and a README for
json2file-go.
Configuration for redis¶
The redis cache is configured in /etc/redis/redis.conf
Tasks¶
Create PKI ticket for new monitored system¶
To add new systems to the monitoring setup you need to create a PKI ticket that can be used to get a client certificate:
sudo -u nagios icinga2 pki ticket --cn <common_name>
Changes¶
Planned¶
None
System Future¶
Additional documentation¶
https://icinga.com/docs/ Documentation for Icinga 2, Icinga Web, Icinga DB, Icinga DB Web
