Setup Grafana Metrics Prometheus Dashboard

Video Lecture

Setup Grafana Metrics Prometheus Dashboard Setup Grafana Metrics Prometheus Dashboard

Description

Depending on your Grafana and Prometheus versions, the pre built Grafana Metrics dashboard may partly work or not at all.

In this video, I will show the steps that I used to get it to work.

Install the Grafana Metrics dashboard from the Prometheus Datasource --> Dashboards tab.

The Prometheus service, since it is local will, retrieve Grafana stats from the url http://127.0.0.0.1:3000/metrics

Grafana will return metrics data by default.

You can verify or change the settings in the grafana.ini file.

sudo nano /etc/grafana/grafana.ini

Scroll down and find the metrics settings and ensure they are set as

# Metrics available at HTTP API Url /metrics
[metrics]
# Disable / Enable internal metrics
enabled             = true
# Disable total stats (stat_totals_*) metrics to be generated
disable_total_stats = false

Optionally, you can also set the authentication settings of the metrics endpoint.

basic_auth_username =
basic_auth_password =

If you changed the grafana.ini, then you will need to restart the Grafana service.

sudo service grafana-server restart
sudo service grafana-server status

The Grafana metrics will be visible at url http://localhost:3000/metrics.

Next,

Add a new scrape target to the Prometheus configuration.

sudo nano /etc/prometheus/prometheus.yml

Scroll down and add a new job to the scrape_configs node.

...
scrape_configs:
...
  - job_name: grafana

    scrape_interval: 15s
    scrape_timeout: 5s

    static_configs:
      - targets: ['localhost:3000']

Save and restart the Prometheus service

sudo service prometheus restart
sudo service prometheus status

Go back into Grafana, and there is likely to be some issues with the visualizations.

In the video, I fix each problem in turn and demonstrate my problem solving process.

Note that depending on the versions of Grafana and Prometheus, you may see different results. My problem solving process should hopefully help you to resolve your particular issues.

Since my /metrics endpoint could be accessed remotely, I also created a new Nginx location in my site config to deny access to the /metrics url path. Note that this does not prevent calling the /metrics endpoint locally, just externally via the Nginx reverse proxy. So the local prometheus service is still able to call the Grafana metrics endpoint.

sudo nano /etc/nginx/sites-enabled/YOUR-DOMAIN-NAME.conf
...
    location /metrics {
        deny all;
    }
...

After changing any Nginx configs, verify the configs are ok using,

nginx -t

and the restart, and check status.

sudo service nginx restart
sudo service nginx status