Paperback 
Setup Grafana Metrics Prometheus Dashboard
Video Lecture
Description
Depending on your Grafana and Prometheus versions, the pre-built Grafana Metrics dashboard may partly work or not at all.
In this video, I will show the steps that I used to get it to work.
Install the Grafana Metrics dashboard from the Prometheus Data Source ⇾ Dashboards tab.
The Prometheus service, since it is local will, retrieve Grafana stats from the URL http://127.0.0.1:3000/metrics
We need to add a scrape target into the Prometheus configuration for a job called grafana
sudo nano /etc/prometheus/prometheus.yml
Scroll down and add a new job name grafana to the scrape_configs section.
# Sample config for Prometheus.
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
monitor: 'example'
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets: ['localhost:9093']
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s
scrape_timeout: 5s
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9090']
- job_name: node
# If prometheus-node-exporter is installed, grab stats about the local
# machine by default.
static_configs:
- targets: ['localhost:9100']
- job_name: grafana
static_configs:
- targets: ['localhost:3000']
Save and restart the Prometheus service
sudo service prometheus restart
sudo service prometheus status
Go back into the Grafana UI, Open the Explore tab, select the Prometheus data source and filter by job and there should be a new job named grafana
Now going back into the imported Grafana Metrics dashboard, you will begin to see some data after a few minutes.
Due to the versions of Grafana and the Prometheus data source plugin used when creating this video, I had particular problems with some queries.
I needed to change these visualizations.
| Title | Old Query | New Query |
|---|---|---|
| http status codes | sum by (statuscode) (irate(http_request_total{job='grafana'}[5m])) |
sum by (status_code) (irate(grafana_http_request_duration_seconds_sum{job='grafana'}[5m])) |
| Prometheus alerts | sum (ALERTS) |
sum (grafana_alerting_active_alerts) |
| Most used handlers | sort(topk(8, sum by (handler) ((http_request_total{job="grafana"}))) |
sort(topk(8, sum by (handler) (grafana_http_request_duration_seconds_sum{job="grafana"}))) |
Grafana Metrics Endpoint
Grafana will return metrics data by default.
You can verify or change the settings in the grafana.ini file.
sudo nano /etc/grafana/grafana.ini
Scroll down and find the metrics settings and ensure they are set as
# Metrics available at HTTP API Url /metrics
[metrics]
# Disable / Enable internal metrics
enabled = true
# Disable total stats (stat_totals_*) metrics to be generated
disable_total_stats = false
Optionally, you can also set the authentication settings of the metrics endpoint.
basic_auth_username =
basic_auth_password =
If you changed the grafana.ini, then you will need to restart the Grafana service.
sudo service grafana-server restart
sudo service grafana-server status
The Grafana metrics will be visible at URL http://localhost:3000/metrics.
Since my /metrics endpoint could be accessed remotely, I also created a new Nginx location in my site config to deny access to the /metrics URL path. Note that this does not prevent calling the /metrics endpoint locally, just externally via the Nginx reverse proxy. So the local prometheus service is still able to call the Grafana metrics endpoint.
sudo nano /etc/nginx/sites-enabled/YOUR-DOMAIN-NAME.conf
...
location /metrics {
deny all;
}
...
After changing any Nginx configs, verify the configs are ok using,
nginx -t
and the restart, and check status.
sudo service nginx restart
sudo service nginx status
Grafana 9 and Ubuntu 22.04 Notes
There are no considerable differences to be aware of as shown and discussed in the video.