Recording Rules
Video Lecture
Description
Time series queries can quickly become quite complicated to remember and type using the Expression Browser in the default Prometheus User Interface.
Example query
100 - (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)
Rather than remembering and typing this query every time, we can create a recording rule that will run at a chosen interval and make the data available as a time series.
Recording Rule Example 1
CD into the /usr/local/bin/prometheus folder
cd /etc/prometheus
Create a new file called prometheus_rules.yml
sudo nano prometheus_rules.yml
Add our test expression as a recording rule
groups:
- name: custom_rules
rules:
- record: node_memory_MemFree_percent
expr: 100 - (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)
Save it and we can now verify the syntax is ok.
Prometheus now comes with a tool called Promtool which you can use to check your rules files and other things.
We will check our rules file is ok.
./promtool check rules prometheus_rules.yml
The response should contain SUCCESS otherwise there was a problem with your prometheus_rules.yml file.
Now lets add the prometheus_rules.yml reference to the prometheus.yml rule_files section.
rule_files: - "prometheus_rules.yml"
and restart the prometheus service.
sudo service prometheus restart sudo service prometheus status
Refresh the Prometheus user interface and check the dropdown
Recording Rule Example 2
Lets do another more complicated example.
Update our prometheus_rules.yml file with,
groups:
- name: custom_rules
rules:
- record: node_memory_MemFree_percent
expr: 100 - (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)
- record: node_filesystem_free_percent
expr: 100 * node_filesystem_free_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}
Check it with the promtool
./promtool check rules prometheus_rules.yml
If all is ok, restart the prometheus service.
sudo service prometheus restart sudo service prometheus status
Refresh the Prometheus user interface and check the drop down.
