Skip to content

Read Nginx Logs with Alloy

Video Lecture

Read Nginx Logs with Alloy Read Nginx Logs with Alloy

Description

We will add to our Alloy config, the ability to read Nginx logs.

#
cd /usr/local/bin/

We need to add a new file_match target to our existing config.alloy file.

#
nano config.alloy

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
local.file_match "system" {
        path_targets = [{
                __address__ = "localhost",
                __path__    = "/var/log/*log",
                job         = "varlogs",
                stream      = "stdout",
                host        = "grafana",
        }]
}

loki.source.file "system" {
        targets               = local.file_match.system.targets
        forward_to            = [loki.write.default.receiver]
        legacy_positions_file = "/tmp/positions.yaml"
}

local.file_match "nginx" {
        path_targets = [{
                __address__ = "localhost",
                __path__    = "/var/log/nginx/*log",
                host        = "grafana",
                job         = "nginx",
        }]
}

loki.source.file "nginx" {
        targets               = local.file_match.nginx.targets
        forward_to            = [loki.write.default.receiver]
        legacy_positions_file = "/tmp/positions.yaml"
}

loki.write "default" {
        endpoint {
                url = "http://localhost:3100/loki/api/v1/push"
        }
        external_labels = {}
}

Restart the Alloy service and check its status.

#
#
service alloy restart
service alloy status

Using the Loki Pattern Parser

Since Loki v2.3.0, we can dynamically create new labels at query time by using a pattern parser in the LogQL query.

E.g., we can split up the contents of an Nginx log line into several more components that we can then use as labels to query further.

{job="nginx"} | pattern `<_> - - <_> "<method> <_> <_>" <status> <_> "<_>" "<_>"`

The above query, passes the pattern over the results of the nginx log stream and add an extra two extra labels for method and status. It is similar to using a regex pattern to extra portions of a string, but faster.

Nginx log lines consist of many values split by spaces. E.g.,

62.197.135.164 - - [10/Oct/2024:08:59:50 +0000] "GET /api/dashboards/home HTTP/1.1" 200 1437 "https://grafana.sbcode.net/?orgId=1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0"
62.197.135.164 - - [10/Oct/2024:08:59:50 +0000] "GET /api/dashboards/home HTTP/1.1" 200 1437 "https://grafana.sbcode.net/?orgId=1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0"
62.197.135.164 - - [10/Oct/2024:08:59:50 +0000] "GET /public/fonts/inter/Inter-Regular.woff2 HTTP/1.1" 200 108488 "https://grafana.sbcode.net/?orgId=1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0"

You can extract many values from the above sample if required

  • remote_addr
  • remote_user
  • time_local
  • method
  • request
  • protocol
  • status
  • body_bytes_sent
  • http_referer
  • http_user_agent

A pattern to extract remote_addr and time_local from the above sample would be,

{job="nginx"} | pattern `<remote_addr> - - <time_local> "<_> <_> <_>" <_> <_> "<_>" "<_>"`

It is possible to extract all the values into labels at the same time, but unless you are explicitly using them, then it is not advisable since it requires more resources to run.

Sample Nginx Dashboard

Visualisations

  • Time Series

    sum by (status) (count_over_time({job="nginx"} | pattern `<_> - - <_> "<_> <_> <_>" <status> ` [1m]))

  • Log

    {job="nginx"}

  • Bar Gauge

    sum(count_over_time({job="nginx"} | pattern `<remote_addr> ` [$__auto])) by (remote_addr)

JSON

Grafana v12.1.1

{
    "annotations": {
        "list": [
            {
                "builtIn": 1,
                "datasource": {
                    "type": "grafana",
                    "uid": "-- Grafana --"
                },
                "enable": true,
                "hide": true,
                "iconColor": "rgba(0, 211, 255, 1)",
                "name": "Annotations & Alerts",
                "type": "dashboard"
            }
        ]
    },
    "editable": true,
    "fiscalYearStartMonth": 0,
    "graphTooltip": 0,
    "id": 5,
    "links": [],
    "panels": [
        {
            "datasource": {
                "type": "loki",
                "uid": "dewndg75qv37kd"
            },
            "fieldConfig": {
                "defaults": {
                    "color": {
                        "mode": "palette-classic"
                    },
                    "custom": {
                        "axisBorderShow": false,
                        "axisCenteredZero": false,
                        "axisColorMode": "text",
                        "axisLabel": "",
                        "axisPlacement": "auto",
                        "barAlignment": 0,
                        "barWidthFactor": 0.6,
                        "drawStyle": "line",
                        "fillOpacity": 0,
                        "gradientMode": "none",
                        "hideFrom": {
                            "legend": false,
                            "tooltip": false,
                            "viz": false
                        },
                        "insertNulls": false,
                        "lineInterpolation": "linear",
                        "lineWidth": 1,
                        "pointSize": 5,
                        "scaleDistribution": {
                            "type": "linear"
                        },
                        "showPoints": "auto",
                        "spanNulls": false,
                        "stacking": {
                            "group": "A",
                            "mode": "none"
                        },
                        "thresholdsStyle": {
                            "mode": "off"
                        }
                    },
                    "mappings": [],
                    "thresholds": {
                        "mode": "absolute",
                        "steps": [
                            {
                                "color": "green",
                                "value": 0
                            },
                            {
                                "color": "red",
                                "value": 80
                            }
                        ]
                    }
                },
                "overrides": []
            },
            "gridPos": {
                "h": 8,
                "w": 15,
                "x": 0,
                "y": 0
            },
            "id": 1,
            "options": {
                "legend": {
                    "calcs": [],
                    "displayMode": "list",
                    "placement": "bottom",
                    "showLegend": true
                },
                "tooltip": {
                    "hideZeros": false,
                    "mode": "single",
                    "sort": "none"
                }
            },
            "pluginVersion": "12.1.1",
            "targets": [
                {
                    "datasource": {
                        "type": "loki",
                        "uid": "dewndg75qv37kd"
                    },
                    "direction": "backward",
                    "editorMode": "code",
                    "expr": "sum by (status) (count_over_time({job=\"nginx\"} | pattern `<_> - - <_> \"<_> <_> <_>\" <status> ` [1m]))",
                    "legendFormat": "{{status}}",
                    "queryType": "range",
                    "refId": "A"
                }
            ],
            "title": "New panel",
            "type": "timeseries"
        },
        {
            "datasource": {
                "type": "loki",
                "uid": "dewndg75qv37kd"
            },
            "fieldConfig": {
                "defaults": {
                    "color": {
                        "mode": "thresholds"
                    },
                    "mappings": [],
                    "thresholds": {
                        "mode": "absolute",
                        "steps": [
                            {
                                "color": "green",
                                "value": 0
                            },
                            {
                                "color": "red",
                                "value": 80
                            }
                        ]
                    }
                },
                "overrides": []
            },
            "gridPos": {
                "h": 8,
                "w": 9,
                "x": 15,
                "y": 0
            },
            "id": 3,
            "options": {
                "displayMode": "lcd",
                "legend": {
                    "calcs": [],
                    "displayMode": "list",
                    "placement": "bottom",
                    "showLegend": false
                },
                "maxVizHeight": 300,
                "minVizHeight": 16,
                "minVizWidth": 8,
                "namePlacement": "auto",
                "orientation": "horizontal",
                "reduceOptions": {
                    "calcs": ["lastNotNull"],
                    "fields": "",
                    "values": false
                },
                "showUnfilled": true,
                "sizing": "auto",
                "valueMode": "color"
            },
            "pluginVersion": "12.1.1",
            "targets": [
                {
                    "datasource": {
                        "type": "loki",
                        "uid": "dewndg75qv37kd"
                    },
                    "direction": "backward",
                    "editorMode": "code",
                    "expr": "sum(count_over_time({job=\"nginx\"} | pattern `<remote_addr> ` [$__auto])) by (remote_addr)",
                    "legendFormat": "{{remote_addr}}",
                    "queryType": "range",
                    "refId": "A"
                }
            ],
            "title": "New panel",
            "type": "bargauge"
        },
        {
            "datasource": {
                "type": "loki",
                "uid": "dewndg75qv37kd"
            },
            "fieldConfig": {
                "defaults": {},
                "overrides": []
            },
            "gridPos": {
                "h": 11,
                "w": 24,
                "x": 0,
                "y": 8
            },
            "id": 2,
            "options": {
                "dedupStrategy": "none",
                "enableInfiniteScrolling": false,
                "enableLogDetails": true,
                "prettifyLogMessage": false,
                "showCommonLabels": false,
                "showLabels": false,
                "showTime": false,
                "sortOrder": "Descending",
                "wrapLogMessage": false
            },
            "pluginVersion": "12.1.1",
            "targets": [
                {
                    "datasource": {
                        "type": "loki",
                        "uid": "dewndg75qv37kd"
                    },
                    "direction": "backward",
                    "editorMode": "code",
                    "expr": "{job=\"nginx\"}",
                    "queryType": "range",
                    "refId": "A"
                }
            ],
            "title": "New panel",
            "type": "logs"
        }
    ],
    "preload": false,
    "schemaVersion": 41,
    "tags": [],
    "templating": {
        "list": []
    },
    "time": {
        "from": "now-5m",
        "to": "now"
    },
    "timepicker": {},
    "timezone": "browser",
    "title": "Nginx",
    "version": 1
}

Troubleshooting

Permission Denied

You may see the error "permission denied". Ensure that your Alloy user is in the same group that can read the log files listed in your path_targets section.

E.g., log files in Linux systems can usually be read by users in the adm group.

You can add your alloy user to the adm group by running

#
usermod -a -G adm alloy

Origin Not Allowed

Since Grafana 8.4, you may get the error "origin not allowed".

To fix this, edit your Grafana servers Nginx configuration to include the proxy_set_header in the location proxy pass.

E.g.,

...
    location / {
        proxy_set_header Host $http_host;
        proxy_pass http://localhost:3000/;
    }
...

Tail Alloy

On Linux, you can check Alloy logs using the command,

#
journalctl -u alloy