Add the Prometheus Alert Manager UI

Video Lecture

Add the Prometheus Alert Manager UI Add the Prometheus Alert Manager UI

Description

When installing the Prometheus Alertmanager using apt install prometheus-alertmanager, the version at the time of creating this document, didn't include a user interface. The message on the default UI says "The Debian package of the alertmanager does not include a web application."

The prebuilt binary version that you can download from GitHub does contain a UI.

In this lesson I will set up the prometheus-alertmanager from binary which can be downloaded from GitHub.

Stop the existing Prometheus Alertmanager

sudo service prometheus-alertmanager stop

Download the binary with the same version number. My existing Alertmanager is version 0.15.3 so I will download the prebuilt equivalent binary using wget.

wget https://github.com/prometheus/alertmanager/releases/download/v0.15.3/alertmanager-0.15.3.linux-amd64.tar.gz

Untar it

tar xvfz alertmanager-0.15.3.linux-amd64.tar.gz

CD into the new folder

cd alertmanager-0.15.3.linux-amd64/

Copy the new alertmanager binary to /usr/local/bin/alertmanager

cp alertmanager /usr/local/bin/alertmanager

CD into /usr/local/bin/alertmanager and inspect.

cd /usr/local/bin/alertmanager
ls -lh

Now create a new specific service configuration for it.

sudo nano /etc/systemd/system/alertmanager.service

Copy into it, these lines

[Unit]
Description=Prometheus Alert Manager Service
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/bin/alertmanager \
        --config.file=/etc/prometheus/alertmanager.yml \
        --web.external-url=https://example.com/alertmanager \
        --web.route-prefix=/

[Install]
WantedBy=multi-user.target

Note the values set in the ExecStart parameter.

  • I am pointing the new binary in the folder /usr/local/bin/alertmanager
  • It is using the existing config file at /etc/prometheus/alertmanager.yml
  • I have set the web.external-url to the url I set up in nginx with the ssl certificate.

Save and exit from nano

Start the new service called alertmanager. Note that this service is not the same as the service prometheus-alertmanager that we stropped at the beginning.

sudo service alertmanager start
sudo service alertmanager status

If, like me, you have a iptables firewall rule blocking external access to port 9093, you won't be able to see the alert manager UI remotely.

So, since I already have Nginx setup, with basic auth on my Prometheus server, I will create a new location in the Nginx config specifically for this binary version of the alertmanager.

sudo nano /etc/nginx/sites-enabled/prometheus
    ...
    location /alertmanager/ {
        proxy_pass           http://localhost:9093/;
    }
    ...

Save and exit from nano

Check the Nginx config is ok

nginx -t

Restart

sudo service nginx restart
sudo service nginx status

Now visit http://[your domain name]/alertmanager

Also, note that the link to the Alertmanager in any new email alerts will be correctly set due to the setting --web.external-url in the service configuration.

Disclaimer

I am using Alertmanager 0.15.3 with Prometheus 2.15.2 on Ubuntu 20.04. The process to get a user interface for the Prometheus Alertmanager is very likely to change between different versions of the software. It could become more easy, or more difficult.

To get the versions of the Prometheus software

amtool --version
promtool --version

If you decide not to use this version that contains the ui, you can stop the service and restart the original prometheus-alertmanager again.

sudo service alertmanager stop
sudo service prometheus-alertmanager start
sudo service prometheus-alertmanager status

Changing the Email Template

You may decide that changing the email template for the alert manager may be easier. You can modify the file called default.tmpl that is located at /usr/share/prometheus/alertmanager. It is a template file. Documentation on this is scarce but if you are familiar with templating and html you may be able to reverse engineer it. If you make any changes test them one at a time by causing an alert to send. It is very easy to break it. If you break the default.tmpl then you can replace it with the file at https://raw.githubusercontent.com/prometheus/alertmanager/master/template/default.tmpl