Skip to content

Alerting Rules

Video Lecture

Alerting Rules Alerting Rules

 (Pay Per View)

You can use PayPal to purchase a one time viewing of this video for $1.49 USD.

Pay Per View Terms

  • One viewing session of this video will cost the equivalent of $1.49 USD in your currency.
  • After successful purchase, the video will automatically start playing.
  • You can pause, replay and go fullscreen as many times as needed in one single session for up to an hour.
  • Do not refresh the browser since it will invalidate the session.
  • If you want longer-term access to all videos, consider purchasing full access through Udemy or YouTube Memberships instead.
  • This Pay Per View option does not permit downloading this video for later viewing or sharing.
  • All videos are Copyright © 2019-2025 Sean Bradley, all rights reserved.

Description

Alerting rules are created in Prometheus very similar to how you create recording rules. We can use the same prometheus_rules.yml or, if you wish, create a different file but remember to add the reference to it in the rule_files section in prometheus.yml

We will create a new group named alert_rules.

And add the script below.

  - name: alert_rules
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
            severity: critical
        annotations:
            summary: 'Instance {{ $labels.instance }} down'
            description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minute.'

Save it and test it with the promtool

./promtool check rules prometheus_rules.yml

If everything is ok, then restart Prometheus.

sudo service prometheus restart
sudo service prometheus status

Next we create another rule that uses one of the Recording rules we created in the previous lesson, node_filesystem_free_percent

      - alert: DiskSpaceFree10Percent
        expr: node_filesystem_free_percent <= 10
        labels:
            severity: warning
        annotations:
            summary: 'Instance {{ $labels.instance }} has 10% or less Free disk space'
            description: '{{ $labels.instance }} has only {{ $value }}% or less free.'

Troubleshooting

If when you use promtool, you see an error similar to

prometheus_rules.yml: yaml: line #: did not find expected key

Then it is possible you have unexpected indentations in your YML.

Also, do not use tabs for indentation, use spaces.

The YML format is extremely white-case sensitive.

Alerting Rules

Alerting Rule Syntax

Prometheus 2.31 and Ubuntu 22.04 Notes

In Prometheus 2.31, the Alerts tab allows you to toggle the visibility of rules in each alerting state by checking the tick box.

There are no considerable differences to be aware of as shown and discussed in the video in case you decide to install Prometheus 2.31.2 on Ubuntu 22.04 LTS.