Skip to content

Recording Rules

Video Lecture

Recording Rules Recording Rules

 (Pay Per View)

You can use PayPal to purchase a one time viewing of this video for $1.49 USD.

Pay Per View Terms

  • One viewing session of this video will cost the equivalent of $1.49 USD in your currency.
  • After successful purchase, the video will automatically start playing.
  • You can pause, replay and go fullscreen as many times as needed in one single session for up to an hour.
  • Do not refresh the browser since it will invalidate the session.
  • If you want longer-term access to all videos, consider purchasing full access through Udemy or YouTube Memberships instead.
  • This Pay Per View option does not permit downloading this video for later viewing or sharing.
  • All videos are Copyright © 2019-2025 Sean Bradley, all rights reserved.

Description

Time series queries can quickly become quite complicated to remember and type using the Expression Browser in the default Prometheus User Interface.

Example query

100 - (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)

Rather than remembering and typing this query every time, we can create a recording rule that will run at a chosen interval and make the data available as a time series.

'Example Prometheus Time Series Graph

Recording Rule Example 1

CD into the /usr/local/bin/prometheus folder

cd /etc/prometheus

Create a new file called prometheus_rules.yml

sudo nano prometheus_rules.yml

Add our test expression as a recording rule

groups:
  - name: custom_rules
    rules:
      - record: node_memory_MemFree_percent
        expr: 100 - (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)

Save it, and we can now verify the syntax is ok.

Prometheus now comes with a tool called Promtool which you can use to check your rules files and other things.

We will check our rules file is ok.

promtool check rules prometheus_rules.yml

The response should contain SUCCESS otherwise there was a problem with your prometheus_rules.yml file.

Now let's add the prometheus_rules.yml reference to the prometheus.yml rule_files section.

rule_files:
  - "prometheus_rules.yml"

And restart the Prometheus service.

sudo service prometheus restart
sudo service prometheus status

Refresh the Prometheus user interface and check the dropdown

'node memory free percent' now appearing in the dropdown

Recording Rule Example 2

Let's do another more complicated example.

Update our prometheus_rules.yml file with,

groups:
  - name: custom_rules
    rules:
      - record: node_memory_MemFree_percent
        expr: 100 - (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)

      - record: node_filesystem_free_percent
        expr: 100 * node_filesystem_free_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}

Check it with the promtool

promtool check rules prometheus_rules.yml

If all is ok, restart the Prometheus service.

sudo service prometheus restart
sudo service prometheus status

Refresh the Prometheus user interface and check the drop-down.

'node filesystem free percent' now appearing in the dropdown

Recording Rules

PromQL Operators

Metric Names

Prometheus 2.31 and Ubuntu 22.04 Notes

There are no considerable differences to be aware of as shown and discussed in the video in case you decide to install Prometheus 2.31.2 on Ubuntu 22.04 LTS.