Skip to content

Configure Prometheus Alertmanager to Send Email Alerts

Video Lecture

Configure Alertmanager to Send Alerts from Prometheus Configure Alertmanager to Send Alerts from Prometheus

Description

We now configure the Prometheus Alertmanager process to send emails when the alerting rules fire and resolve.

You will need an SMTP server. Many corporations will provide an SMTP service for their staff. This same service can be used to send emails via the Prometheus Alertmanager.

You will need an email account in your email provider, prometheus@your-domain.tld perhaps may be good. You will also need an accessible SMTP server address, e.g., mail.example.com and a port, usually 25 or 465.

In this video, I already have a domain that I purchased from Namecheap. For an extra $1 a month, I can get an email service add-on that I can use to send and receive emails. I will configure the Prometheus Alertmanager to use this email service.

If you have a corporate email service that you already connect using your work mobile phone, then the set-up process and settings will be very similar. You will be able to use Prometheus Alertmanager to send emails using your work email address.

So, to get started setting up your Email provider in Prometheus Alertmanager, you will need the SMTP server address, SMTP server port, a valid email address and password that will be used to send emails through your email service.

Edit Prometheus Alertmanager Configuration

CD into the folder containing the Alertmanager config file called alertmanager.yml

cd /etc/prometheus

Backup the original configuration

cp alertmanager.yml alertmanager_orig.yml

Open alertmanager.yml

sudo nano alertmanager.yml

And replace the full contents with this below, and also edit the parameters to match your settings.

route:
    receiver: smtp
receivers:
    - name: smtp
      email_configs:
          - smarthost: mail.your-email-service-domain:port
            hello: your-email-service-domain
            to: 'to@example.com'
            from: 'from@example.com'
            auth_username: 'from@example.com'
            auth_password: 'your-password'
            send_resolved: true
            require_tls: false

Check your configuration with the supplied amtool

amtool check-config alertmanager.yml

If all is ok, restart the Prometheus Alertmanager service.

sudo service prometheus-alertmanager restart
sudo service prometheus-alertmanager status

We can view the updated Alertmanager configuration.

amtool config

You can also tail the Prometheus Alertmanager logs using the command

journalctl -u prometheus-alertmanager.service -f

In the default email alerts, you get a link at the bottom that points to the source metric that is firing/resolved.

This URL will not be configured by default. To configure it, we can change the Prometheus default parameter --web.external-url

I am using an Nginx reverse proxy, where I configured SSL and a domain, so I can set it to use the external URL to https://prometheus.sbcode.net.

sudo nano /etc/default/prometheus

Add --web.external-url to the ARGS= variable. E.g.,

ARGS="--web.external-url=https://example.com"

If you have existing settings in the ARGS parameter, then separate them using a space.

E.g.,

ARGS="--web.enable-admin-api --web.external-url=https://example.com"

We have made a change to the Prometheus configuration, so we should check it using promtool.

promtool check config /etc/prometheus/prometheus.yml

If everything is ok, then we can restart and check status.

sudo service prometheus restart
sudo service prometheus status

Changing the Email Template

You may want to change the email template used by the alert manager.

You can modify the file called default.tmpl that is located at /usr/share/prometheus/alertmanager.

It is a template file. Documentation on this is scarce but if you are familiar with templating and HTML you may be able to reverse engineer it.

If you make any changes, test them one at a time by causing an alert to send. It is very easy to break it.

If you break the default.tmpl then you can replace it with the file at https://raw.githubusercontent.com/prometheus/alertmanager/master/template/default.tmpl

Troubleshooting

You can view the Alertmanager logs using journalctl

journalctl -u prometheus-alertmanager.service -f

You can test if your domain name has an SMTP service by using the host command.

E.g.,

host your-domain.tld

Should show a reference to mail service that may attempt to process incoming SMTP provided that it is set up correctly.

E.g.,

your-domain.tld has address ###.###.###.###
your-domain.tld mail is handled by 10 mx1.your-email-providers-domain.tld.

You can also test if your SMTP service will allow connections from your server by using telnet.

The port numbers to try are usually port 25 or 465 but can be different. Your email service provider will provide you with that information.

E.g.,

telnet mx1.your-email-providers-domain.tld 25

It should connect waiting for more information. If otherwise you will see an error similar to

telnet: Unable to connect to remote host: Connection refused

You may need to whitelist your Prometheus Alertmanager IP address/domain, or consult your email providers documentation for the correct port to use.

Alertmanager Configuration

Namecheap

Comments