Skip to content

Read Nginx Logs with Promtail

Video Lecture

Read Nginx Logs with Promtail Read Nginx Logs with Promtail Read Nginx Logs with Promtail

Description

We will add to our Promtail scrape configs, the ability to read the Nginx access and error logs.

We need to add a new job_name to our existing Promtail scrape_configs in the config_promtail.yml file.

server:
  http_listen_port: 9080
  grpc_listen_port: 9097

positions:
  filename: /tmp/positions.yaml

clients:
  - url: 'http://localhost:3100/loki/api/v1/push'

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*log
          host: grafana

  - job_name: nginx
    static_configs:
      - targets:
          - localhost
        labels:
          job: nginx
          __path__: /var/log/nginx/*log
          host: grafana

Restart the Promtail service and check its status.

sudo service promtail restart
sudo service promtail status

Using the Loki Pattern Parser

Since Loki v2.3.0, we can dynamically create new labels at query time by using a pattern parser in the LogQL query.

E.g., we can split up the contents of an Nginx log line into several more components that we can then use as labels to query further.

{job="nginx"} | pattern `<_> - - <_> "<method> <_> <_>" <status> <_> <_> "<_>" <_>`

The above query, passes the pattern over the results of the nginx log stream and add an extra two extra labels for method and status. It is similar to using a regex pattern to extra portions of a string, but faster.

Nginx log lines consist of many values split by spaces. E.g.,

206.189.7.141 - - [29/Nov/2021:08:22:54 +0000] "POST /loki/loki/api/v1/push HTTP/1.1" 204 0 "-" "promtail/"
213.205.198.138 - - [29/Nov/2021:08:23:54 +0000] "GET /public/build/grafanaPlugin.9293a56f182a84c40c07.js HTTP/1.1" 200 11042 "https://grafana.sbcode.net/?orgId=1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/1.2 (KHTML, like Gecko) Chrome/1.2.3.4 Safari/537.36 Edg/1.2.3.4"
213.205.198.138 - - [29/Nov/2021:08:23:55 +0000] "GET /api/search?limit=30&starred=true HTTP/1.1" 200 2 "https://grafana.sbcode.net/?orgId=1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/1.2 (KHTML, like Gecko) Chrome/1.2.3.4 Safari/537.36 Edg/1.2.3.4"

You can extract many values from the above sample if required

  • remote_addr
  • remote_user
  • time_local
  • method
  • request
  • protocol
  • status
  • body_bytes_sent
  • http_referer
  • http_user_agent

A pattern to extract remote_addr and time_local from the above sample would be,

{job="nginx"} | pattern `<remote_addr> - - <time_local> "<_> <_> <_>" <_> <_> <_> "<_>" <_>`

It is possible to extract all the values into labels at the same time, but unless you are explicitly using them, then it is not advisable since it requires more resources to run.

Sample Nginx Dashboard

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "datasource",
          "uid": "grafana"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "target": {
          "limit": 100,
          "matchAny": false,
          "tags": [],
          "type": "dashboard"
        },
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": 11,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "datasource": {
        "type": "loki",
        "uid": "dguRbs74z"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 10,
        "w": 16,
        "x": 0,
        "y": 0
      },
      "id": 2,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "loki",
            "uid": "dguRbs74z"
          },
          "expr": "sum by (status) (count_over_time({job=\"nginx\"} | pattern `<_> - - <_> \"<method> <_> <_>\" <status> <_> <_> \"<_>\" <_>`[1m])) ",
          "refId": "A"
        }
      ],
      "title": "Panel Title",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "loki",
        "uid": "dguRbs74z"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 16,
        "w": 8,
        "x": 16,
        "y": 0
      },
      "id": 6,
      "options": {
        "displayMode": "lcd",
        "minVizHeight": 10,
        "minVizWidth": 0,
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showUnfilled": true,
        "text": {}
      },
      "pluginVersion": "9.1.6",
      "targets": [
        {
          "datasource": {
            "type": "loki",
            "uid": "dguRbs74z"
          },
          "expr": "sum(count_over_time({job=\"nginx\",filename=\"/var/log/nginx/access.log\"} | pattern `<remote_addr> - -`[$__range])) by (remote_addr)",
          "refId": "A"
        }
      ],
      "title": "Panel Title",
      "transformations": [
        {
          "id": "labelsToFields",
          "options": {
            "valueLabel": "remote_addr"
          }
        }
      ],
      "type": "bargauge"
    },
    {
      "datasource": {
        "type": "loki",
        "uid": "dguRbs74z"
      },
      "gridPos": {
        "h": 6,
        "w": 16,
        "x": 0,
        "y": 10
      },
      "id": 4,
      "options": {
        "dedupStrategy": "none",
        "enableLogDetails": true,
        "prettifyLogMessage": false,
        "showCommonLabels": false,
        "showLabels": false,
        "showTime": false,
        "sortOrder": "Descending",
        "wrapLogMessage": false
      },
      "targets": [
        {
          "datasource": {
            "type": "loki",
            "uid": "dguRbs74z"
          },
          "expr": "{job=\"nginx\"}",
          "refId": "A"
        }
      ],
      "title": "Panel Title",
      "type": "logs"
    }
  ],
  "refresh": "5s",
  "schemaVersion": 37,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-5m",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "",
  "title": "Nginx Example",
  "uid": "TAQUDUp7z",
  "version": 1,
  "weekStart": ""
}

Troubleshooting

Spaces versus Tabs

YML files are whitespace sensitive. Many errors restarting Promtail can be attributed to incorrect indentation.

E.g., you might see the error, "found a tab character that violates indentation"

Double check all indentations in the YML are spaces and not tabs.

Permission Denied

You may see the error "permission denied". Ensure that your Promtail user is in the same group that can read the log files listed in your scope configs __path__ setting.

E.g., log files in Linux systems can usually be read by users in the adm group.

You can add your promtail user to the adm group by running

sudo usermod -a -G adm promtail

Origin Not Allowed

Since Grafana 8.4, you may get the error "origin not allowed".

To fix this, edit your Grafana servers Nginx configuration to include the host header in the location proxy pass.

E.g.,

...

    location / {
        proxy_set_header Host $http_host; //copy and paste this line
        proxy_pass           http://localhost:3000/;
    }

...

Restart Nginx

sudo service nginx restart

Tail Promtail

On Linux, you can check the syslog for any Promtail related entries by using the command,

tail -f /var/log/syslog | grep promtail

Grafana 9 and Ubuntu 22.04 Notes

There are no considerable differences to be aware of as shown and discussed in the video.