Skip to content


 Zabbix
 Grafana
 Prometheus
 Threejs and TypeScript
 SocketIO and TypeScript
 Blender Topological Earth
 Sweet Home 3D
 Design Patterns Python
 Design Patterns TypeScript
   
 Course Coupon Codes
Three.js and TypeScript
Kindle Edition
$6.99 $9.99 Paperback 
$22.99 $29.99




Design Patterns in TypeScript
Kindle Edition
$6.99 $9.99 Paperback
$11.99 $19.99




Design Patterns in Python
Kindle Edition
$6.99 $9.99 Paperback
$11.99 $19.99




Setup Grafana Metrics Prometheus Dashboard

Video Lecture

Setup Grafana Metrics Prometheus Dashboard Setup Grafana Metrics Prometheus Dashboard Setup Grafana Metrics Prometheus Dashboard

Description

Depending on your Grafana and Prometheus versions, the pre-built Grafana Metrics dashboard may partly work or not at all.

In this video, I will show the steps that I used to get it to work.

Install the Grafana Metrics dashboard from the Prometheus Data SourceDashboards tab.

The Prometheus service, since it is local will, retrieve Grafana stats from the URL http://127.0.0.1:3000/metrics

We need to add a scrape target into the Prometheus configuration for a job called grafana

sudo nano /etc/prometheus/prometheus.yml

Scroll down and add a new job name grafana to the scrape_configs section.

# Sample config for Prometheus.

global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'example'

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['localhost:9093']

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    scrape_timeout: 5s

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9090']

  - job_name: node
    # If prometheus-node-exporter is installed, grab stats about the local
    # machine by default.
    static_configs:
      - targets: ['localhost:9100']

  - job_name: grafana
    static_configs:
      - targets: ['localhost:3000']

Save and restart the Prometheus service

sudo service prometheus restart
sudo service prometheus status

Go back into the Grafana UI, Open the Explore tab, select the Prometheus data source and filter by job and there should be a new job named grafana

Now going back into the imported Grafana Metrics dashboard, you will begin to see some data after a few minutes.

Due to the versions of Grafana and the Prometheus data source plugin used when creating this video, I had particular problems with some queries.

I needed to change these visualizations.

Title Old Query New Query
http status codes sum by (statuscode) (irate(http_request_total{job='grafana'}[5m])) sum by (status_code) (irate(grafana_http_request_duration_seconds_sum{job='grafana'}[5m]))
Prometheus alerts sum (ALERTS) sum (grafana_alerting_active_alerts)
Most used handlers sort(topk(8, sum by (handler) ((http_request_total{job="grafana"}))) sort(topk(8, sum by (handler) (grafana_http_request_duration_seconds_sum{job="grafana"})))

Grafana Metrics Endpoint

Grafana will return metrics data by default.

You can verify or change the settings in the grafana.ini file.

sudo nano /etc/grafana/grafana.ini

Scroll down and find the metrics settings and ensure they are set as

# Metrics available at HTTP API Url /metrics
[metrics]
# Disable / Enable internal metrics
enabled             = true
# Disable total stats (stat_totals_*) metrics to be generated
disable_total_stats = false

Optionally, you can also set the authentication settings of the metrics endpoint.

basic_auth_username =
basic_auth_password =

If you changed the grafana.ini, then you will need to restart the Grafana service.

sudo service grafana-server restart
sudo service grafana-server status

The Grafana metrics will be visible at URL http://localhost:3000/metrics.

Since my /metrics endpoint could be accessed remotely, I also created a new Nginx location in my site config to deny access to the /metrics URL path. Note that this does not prevent calling the /metrics endpoint locally, just externally via the Nginx reverse proxy. So the local prometheus service is still able to call the Grafana metrics endpoint.

sudo nano /etc/nginx/sites-enabled/YOUR-DOMAIN-NAME.conf
...
location /metrics {
  deny all;
}
...

After changing any Nginx configs, verify the configs are ok using,

nginx -t

and the restart, and check status.

sudo service nginx restart
sudo service nginx status

Grafana 9 and Ubuntu 22.04 Notes

There are no considerable differences to be aware of as shown and discussed in the video.