Setup Grafana Metrics Prometheus Dashboard

Video Lecture

Description

Depending on your Grafana and Prometheus versions, the pre-built Grafana Metrics dashboard may partly work or not at all.

In this video, I will show the steps that I used to get it to work.

Install the Grafana Metrics dashboard from the Prometheus Data Source ⇾ Dashboards tab.

The Prometheus service, since it is local will, retrieve Grafana stats from the URL http://127.0.0.1:3000/metrics

We need to add a scrape target into the Prometheus configuration for a job called grafana

sudo nano /etc/prometheus/prometheus.yml

Scroll down and add a new job name grafana to the scrape_configs section.

# Sample config for Prometheus.

global:
    scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
    evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
    # scrape_timeout is set to the global default (10s).

    # Attach these labels to any time series or alerts when communicating with
    # external systems (federation, remote storage, Alertmanager).
    external_labels:
        monitor: 'example'

# Alertmanager configuration
alerting:
    alertmanagers:
        - static_configs:
              - targets: ['localhost:9093']

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
    # - "first_rules.yml"
    # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
    # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
    - job_name: 'prometheus'

      # Override the global default and scrape targets from this job every 5 seconds.
      scrape_interval: 5s
      scrape_timeout: 5s

      # metrics_path defaults to '/metrics'
      # scheme defaults to 'http'.

      static_configs:
          - targets: ['localhost:9090']

    - job_name: node
      # If prometheus-node-exporter is installed, grab stats about the local
      # machine by default.
      static_configs:
          - targets: ['localhost:9100']

    - job_name: grafana
      static_configs:
          - targets: ['localhost:3000']

Save and restart the Prometheus service

sudo service prometheus restart
sudo service prometheus status

Go back into the Grafana UI, Open the Explore tab, select the Prometheus data source and filter by job and there should be a new job named grafana

Now going back into the imported Grafana Metrics dashboard, you will begin to see some data after a few minutes.

Due to the versions of Grafana and the Prometheus data source plugin used when creating this video, I had particular problems with some queries.

I needed to change these visualizations.

Title	Old Query	New Query
http status codes	sum by (statuscode) (irate(http_request_total{job='grafana'}[5m]))	sum by (status_code) (irate(grafana_http_request_duration_seconds_sum{job='grafana'}[5m]))
Prometheus alerts	sum (ALERTS)	sum (grafana_alerting_active_alerts)
Most used handlers	sort(topk(8, sum by (handler) ((http_request_total{job="grafana"})))	sort(topk(8, sum by (handler) (grafana_http_request_duration_seconds_sum{job="grafana"})))

Grafana Metrics Endpoint

Grafana will return metrics data by default.

You can verify or change the settings in the grafana.ini file.

sudo nano /etc/grafana/grafana.ini

Scroll down and find the metrics settings and ensure they are set as

# Metrics available at HTTP API Url /metrics
[metrics]
# Disable / Enable internal metrics
enabled             = true
# Disable total stats (stat_totals_*) metrics to be generated
disable_total_stats = false

Optionally, you can also set the authentication settings of the metrics endpoint.

basic_auth_username =
basic_auth_password =

If you changed the grafana.ini, then you will need to restart the Grafana service.

sudo service grafana-server restart
sudo service grafana-server status

The Grafana metrics will be visible at URL http://localhost:3000/metrics.

Since my /metrics endpoint could be accessed remotely, I also created a new Nginx location in my site config to deny access to the /metrics URL path. Note that this does not prevent calling the /metrics endpoint locally, just externally via the Nginx reverse proxy. So the local prometheus service is still able to call the Grafana metrics endpoint.

sudo nano /etc/nginx/sites-enabled/YOUR-DOMAIN-NAME.conf

---
location /metrics {
deny all;
}

After changing any Nginx configs, verify the configs are ok using,

nginx -t

and the restart, and check status.

sudo service nginx restart
sudo service nginx status

Grafana 10 and Ubuntu 22.04 Notes

Explore → Prometheus → Metrics browser

In Grafana 10, when exploring the new Prometheus data source, the default view is the new Builder UI. My video shows the Code UI, where it shows the Metrics browser option.

You can view the Code UI by pressing the option at the right of the screen.

Comments