Skip to content

Alerting Rules

Video Lecture

Alerting Rules Alerting Rules

Description

Alerting rules are created in Prometheus very similar to how you create recording rules. We can use the same prometheus_rules.yml or, if you wish, create a different file but remember to add the reference to it in the rule_files section in prometheus.yml

We will create a new group named alert_rules.

And add the script below.

  - name: alert_rules
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
            severity: critical
        annotations:
            summary: 'Instance {{ $labels.instance }} down'
            description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minute.'

Save it and test it with the promtool

./promtool check rules prometheus_rules.yml

If everything is ok, then restart Prometheus.

sudo service prometheus restart
sudo service prometheus status

Next we create another rule that uses one of the Recording rules we created in the previous lesson, node_filesystem_free_percent

      - alert: DiskSpaceFree10Percent
        expr: node_filesystem_free_percent <= 10
        labels:
            severity: warning
        annotations:
            summary: 'Instance {{ $labels.instance }} has 10% or less Free disk space'
            description: '{{ $labels.instance }} has only {{ $value }}% or less free.'

Troubleshooting

If when you use promtool, you see an error similar to

prometheus_rules.yml: yaml: line #: did not find expected key

Then it is possible you have unexpected indentations in your YML.

Also, do not use tabs for indentation, use spaces.

The YML format is extremely white-case sensitive.

Alerting Rules

Alerting Rule Syntax

Prometheus 2.31 and Ubuntu 22.04 Notes

In Prometheus 2.31, the Alerts tab allows you to toggle the visibility of rules in each alerting state by checking the tick box.

There are no considerable differences to be aware of as shown and discussed in the video in case you decide to install Prometheus 2.31.2 on Ubuntu 22.04 LTS.

Comments