Skip to content

Read Nginx Logs with Promtail

Video Lecture

Read Nginx Logs with Promtail Read Nginx Logs with Promtail

Description

We will add to our Promtail scrape configs, the ability to read the Nginx logs.

#
cd /usr/local/bin/

We need to add a new job_name to our existing Promtail scrape_configs in the promtail-local-config file.

#
nano promtail-local-config.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log
      stream: stdout
      host: Grafana-Server

- job_name: nginx
  static_configs:
  - targets:
      - localhost
    labels:
      job: nginx
      __path__: /var/log/nginx/*log
      host: Grafana-Server

Restart the Promtail service and check its status.

#
#
service promtail restart
service promtail status

Using the Loki Pattern Parser

Since Loki v2.3.0, we can dynamically create new labels at query time by using a pattern parser in the LogQL query.

E.g., we can split up the contents of an Nginx log line into several more components that we can then use as labels to query further.

{job="nginx"} | pattern `<_> - - <_> "<method> <_> <_>" <status> <_> "<_>" "<_>"`

The above query, passes the pattern over the results of the nginx log stream and add an extra two extra labels for method and status. It is similar to using a regex pattern to extra portions of a string, but faster.

Nginx log lines consist of many values split by spaces. E.g.,

62.197.135.164 - - [10/Oct/2024:08:59:50 +0000] "GET /api/dashboards/home HTTP/1.1" 200 1437 "https://grafana.sbcode.net/?orgId=1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0"
62.197.135.164 - - [10/Oct/2024:08:59:50 +0000] "GET /api/dashboards/home HTTP/1.1" 200 1437 "https://grafana.sbcode.net/?orgId=1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0"
62.197.135.164 - - [10/Oct/2024:08:59:50 +0000] "GET /public/fonts/inter/Inter-Regular.woff2 HTTP/1.1" 200 108488 "https://grafana.sbcode.net/?orgId=1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0"

You can extract many values from the above sample if required

  • remote_addr
  • remote_user
  • time_local
  • method
  • request
  • protocol
  • status
  • body_bytes_sent
  • http_referer
  • http_user_agent

A pattern to extract remote_addr and time_local from the above sample would be,

{job="nginx"} | pattern `<remote_addr> - - <time_local> "<_> <_> <_>" <_> <_> "<_>" "<_>"`

It is possible to extract all the values into labels at the same time, but unless you are explicitly using them, then it is not advisable since it requires more resources to run.

Sample Nginx Dashboard

Grafana v11.2.2

{
    "annotations": {
        "list": [
            {
                "builtIn": 1,
                "datasource": {
                    "type": "grafana",
                    "uid": "-- Grafana --"
                },
                "enable": true,
                "hide": true,
                "iconColor": "rgba(0, 211, 255, 1)",
                "name": "Annotations & Alerts",
                "type": "dashboard"
            }
        ]
    },
    "editable": true,
    "fiscalYearStartMonth": 0,
    "graphTooltip": 0,
    "id": 17,
    "links": [],
    "panels": [
        {
            "datasource": {
                "default": false,
                "type": "loki",
                "uid": "ddzqvpejw660wa"
            },
            "fieldConfig": {
                "defaults": {
                    "color": {
                        "mode": "palette-classic"
                    },
                    "custom": {
                        "axisBorderShow": false,
                        "axisCenteredZero": false,
                        "axisColorMode": "text",
                        "axisLabel": "",
                        "axisPlacement": "auto",
                        "barAlignment": 0,
                        "barWidthFactor": 0.6,
                        "drawStyle": "line",
                        "fillOpacity": 0,
                        "gradientMode": "none",
                        "hideFrom": {
                            "legend": false,
                            "tooltip": false,
                            "viz": false
                        },
                        "insertNulls": false,
                        "lineInterpolation": "linear",
                        "lineWidth": 1,
                        "pointSize": 5,
                        "scaleDistribution": {
                            "type": "linear"
                        },
                        "showPoints": "auto",
                        "spanNulls": false,
                        "stacking": {
                            "group": "A",
                            "mode": "none"
                        },
                        "thresholdsStyle": {
                            "mode": "off"
                        }
                    },
                    "mappings": [],
                    "thresholds": {
                        "mode": "absolute",
                        "steps": [
                            {
                                "color": "green",
                                "value": null
                            },
                            {
                                "color": "red",
                                "value": 80
                            }
                        ]
                    }
                },
                "overrides": []
            },
            "gridPos": {
                "h": 11,
                "w": 18,
                "x": 0,
                "y": 0
            },
            "id": 1,
            "options": {
                "legend": {
                    "calcs": [],
                    "displayMode": "list",
                    "placement": "bottom",
                    "showLegend": true
                },
                "tooltip": {
                    "mode": "single",
                    "sort": "none"
                }
            },
            "targets": [
                {
                    "datasource": {
                        "type": "loki",
                        "uid": "ddzqvpejw660wa"
                    },
                    "editorMode": "code",
                    "expr": "sum by (status) (count_over_time({job=\"nginx\"} | pattern `<_> - - <_> \"<method> <_> <_>\" <status> <_>` [1m])) ",
                    "legendFormat": "{{status}}",
                    "queryType": "range",
                    "refId": "A"
                }
            ],
            "title": "Panel Title",
            "type": "timeseries"
        },
        {
            "datasource": {
                "default": false,
                "type": "loki",
                "uid": "ddzqvpejw660wa"
            },
            "fieldConfig": {
                "defaults": {
                    "color": {
                        "mode": "thresholds"
                    },
                    "mappings": [],
                    "thresholds": {
                        "mode": "absolute",
                        "steps": [
                            {
                                "color": "green",
                                "value": null
                            },
                            {
                                "color": "red",
                                "value": 80
                            }
                        ]
                    }
                },
                "overrides": []
            },
            "gridPos": {
                "h": 11,
                "w": 6,
                "x": 18,
                "y": 0
            },
            "id": 2,
            "options": {
                "displayMode": "lcd",
                "maxVizHeight": 300,
                "minVizHeight": 16,
                "minVizWidth": 8,
                "namePlacement": "auto",
                "orientation": "horizontal",
                "reduceOptions": {
                    "calcs": ["lastNotNull"],
                    "fields": "",
                    "values": false
                },
                "showUnfilled": true,
                "sizing": "auto",
                "valueMode": "color"
            },
            "pluginVersion": "11.2.2",
            "targets": [
                {
                    "datasource": {
                        "type": "loki",
                        "uid": "ddzqvpejw660wa"
                    },
                    "editorMode": "code",
                    "expr": "sum(count_over_time({job=\"nginx\"} | pattern `<remote_addr> - - ` [$__range])) by (remote_addr)",
                    "legendFormat": "{{remote_addr}}",
                    "queryType": "range",
                    "refId": "A"
                }
            ],
            "title": "Panel Title",
            "type": "bargauge"
        },
        {
            "datasource": {
                "default": false,
                "type": "loki",
                "uid": "ddzqvpejw660wa"
            },
            "gridPos": {
                "h": 15,
                "w": 24,
                "x": 0,
                "y": 11
            },
            "id": 3,
            "options": {
                "dedupStrategy": "none",
                "enableLogDetails": true,
                "prettifyLogMessage": false,
                "showCommonLabels": false,
                "showLabels": false,
                "showTime": false,
                "sortOrder": "Descending",
                "wrapLogMessage": false
            },
            "targets": [
                {
                    "datasource": {
                        "type": "loki",
                        "uid": "ddzqvpejw660wa"
                    },
                    "editorMode": "code",
                    "expr": "{job=\"nginx\"}",
                    "queryType": "range",
                    "refId": "A"
                }
            ],
            "title": "Panel Title",
            "type": "logs"
        }
    ],
    "schemaVersion": 39,
    "tags": [],
    "templating": {
        "list": []
    },
    "time": {
        "from": "now-15m",
        "to": "now"
    },
    "timepicker": {},
    "timezone": "browser",
    "title": "Nginx",
    "uid": "ce0fsbsyozn5sa",
    "version": 1,
    "weekStart": ""
}

Troubleshooting

Spaces versus Tabs

YML files are whitespace sensitive. Many errors restarting Promtail can be attributed to incorrect indentation.

E.g., you might see the error, found a tab character that violates indentation.

Double check all indentations in the YML are spaces and not tabs.

Permission Denied

You may see the error "permission denied". Ensure that your Promtail user is in the same group that can read the log files listed in your scope configs __path__ setting.

E.g., log files in Linux systems can usually be read by users in the adm group.

You can add your promtail user to the adm group by running

#
usermod -a -G adm promtail

Origin Not Allowed

Since Grafana 8.4, you may get the error "origin not allowed".

To fix this, edit your Grafana servers Nginx configuration to include the proxy_set_header in the location proxy pass.

E.g.,

...
    location / {
        proxy_set_header Host $http_host;
        proxy_pass http://localhost:3000/;
    }
...

Tail Promtail

On Linux, you can check Promtail logs using the command,

#
journalctl -u promtail

Comments