Read Nginx Logs with Promtail
Video Lecture
Description
We will add to our Promtail scrape configs, the ability to read the Nginx access and error logs.
cd /usr/local/bin/
We need to add a new job_name to our existing Promtail scrape_configs in the config_promtail.yml file.
sudo nano config_promtail.yml
server:
http_listen_port: 9080
grpc_listen_port: 9097
positions:
filename: /tmp/positions.yaml
clients:
- url: 'http://localhost:3100/loki/api/v1/push'
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*log
host: grafana
- job_name: nginx
static_configs:
- targets:
- localhost
labels:
job: nginx
__path__: /var/log/nginx/*log
host: grafana
Restart the Promtail service and check its status.
sudo service promtail restart
sudo service promtail status
Using the Loki Pattern Parser
Since Loki v2.3.0, we can dynamically create new labels at query time by using a pattern parser in the LogQL query.
E.g., we can split up the contents of an Nginx log line into several more components that we can then use as labels to query further.
{job="nginx"} | pattern `<_> - - <_> "<method> <_> <_>" <status> <_> <_> "<_>" <_>`
The above query, passes the pattern over the results of the nginx log stream and add an extra two extra labels for method and status. It is similar to using a regex pattern to extra portions of a string, but faster.
Nginx log lines consist of many values split by spaces. E.g.,
206.189.7.141 - - [29/Nov/2021:08:22:54 +0000] "POST /loki/loki/api/v1/push HTTP/1.1" 204 0 "-" "promtail/"
213.205.198.138 - - [29/Nov/2021:08:23:54 +0000] "GET /public/build/grafanaPlugin.9293a56f182a84c40c07.js HTTP/1.1" 200 11042 "https://grafana.sbcode.net/?orgId=1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/1.2 (KHTML, like Gecko) Chrome/1.2.3.4 Safari/537.36 Edg/1.2.3.4"
213.205.198.138 - - [29/Nov/2021:08:23:55 +0000] "GET /api/search?limit=30&starred=true HTTP/1.1" 200 2 "https://grafana.sbcode.net/?orgId=1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/1.2 (KHTML, like Gecko) Chrome/1.2.3.4 Safari/537.36 Edg/1.2.3.4"
You can extract many values from the above sample if required
remote_addrremote_usertime_localmethodrequestprotocolstatusbody_bytes_senthttp_refererhttp_user_agent
A pattern to extract remote_addr and time_local from the above sample would be,
{job="nginx"} | pattern `<remote_addr> - - <time_local> "<_> <_> <_>" <_> <_> <_> "<_>" <_>`
It is possible to extract all the values into labels at the same time, but unless you are explicitly using them, then it is not advisable since it requires more resources to run.
Sample Nginx Dashboard
Updated for Grafana 10.0.0
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": 9,
"links": [],
"liveNow": false,
"panels": [
{
"datasource": {
"type": "loki",
"uid": "c66ad50c-de8e-4767-ac8c-dcfe063948d1"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 10,
"w": 17,
"x": 0,
"y": 0
},
"id": 1,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "loki",
"uid": "c66ad50c-de8e-4767-ac8c-dcfe063948d1"
},
"editorMode": "code",
"expr": "sum(count_over_time({job=\"nginx\"} | pattern `<_> - - <_> \"<method> <_> <_>\" <status> <_> <_> \"<_>\" <_>` [1m])) by (status)",
"key": "Q-b1aadd0b-ce49-4595-9f52-c7addbb47c92-0",
"queryType": "range",
"refId": "A"
}
],
"title": "New Panel",
"type": "timeseries"
},
{
"datasource": {
"type": "loki",
"uid": "c66ad50c-de8e-4767-ac8c-dcfe063948d1"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 22,
"w": 6,
"x": 17,
"y": 0
},
"id": 3,
"options": {
"displayMode": "lcd",
"minVizHeight": 10,
"minVizWidth": 0,
"orientation": "horizontal",
"reduceOptions": {
"calcs": ["lastNotNull"],
"fields": "",
"values": false
},
"showUnfilled": true,
"valueMode": "color"
},
"pluginVersion": "10.0.0",
"targets": [
{
"datasource": {
"type": "loki",
"uid": "c66ad50c-de8e-4767-ac8c-dcfe063948d1"
},
"editorMode": "code",
"expr": "sum(count_over_time({job=\"nginx\"} | pattern `<remote_addr> - - ` [$__range])) by (remote_addr)",
"queryType": "range",
"refId": "A"
}
],
"title": "Panel Title",
"type": "bargauge"
},
{
"datasource": {
"type": "loki",
"uid": "c66ad50c-de8e-4767-ac8c-dcfe063948d1"
},
"gridPos": {
"h": 12,
"w": 17,
"x": 0,
"y": 10
},
"id": 2,
"options": {
"dedupStrategy": "none",
"enableLogDetails": true,
"prettifyLogMessage": false,
"showCommonLabels": false,
"showLabels": false,
"showTime": false,
"sortOrder": "Descending",
"wrapLogMessage": false
},
"targets": [
{
"datasource": {
"type": "loki",
"uid": "c66ad50c-de8e-4767-ac8c-dcfe063948d1"
},
"editorMode": "code",
"expr": "{job=\"nginx\"}",
"queryType": "range",
"refId": "A"
}
],
"title": "Panel Title",
"type": "logs"
}
],
"refresh": false,
"schemaVersion": 38,
"style": "dark",
"tags": [],
"templating": {
"list": []
},
"time": {
"from": "now-30m",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "Nginx Example",
"uid": "bdfddcc9-7bfb-4a48-b606-0c273806d474",
"version": 6,
"weekStart": ""
}
Troubleshooting
Spaces versus Tabs
YML files are whitespace sensitive. Many errors restarting Promtail can be attributed to incorrect indentation.
E.g., you might see the error, "found a tab character that violates indentation"
Double check all indentations in the YML are spaces and not tabs.
Permission Denied
You may see the error "permission denied". Ensure that your Promtail user is in the same group that can read the log files listed in your scope configs __path__ setting.
E.g., log files in Linux systems can usually be read by users in the adm group.
You can add your promtail user to the adm group by running
sudo usermod -a -G adm promtail
Origin Not Allowed
Since Grafana 8.4, you may get the error "origin not allowed".
To fix this, edit your Grafana servers Nginx configuration to include the host header in the location proxy pass.
E.g.,
...
location / {
proxy_set_header Host $http_host; //copy and paste this line
proxy_pass http://localhost:3000/;
}
...
Restart Nginx
sudo service nginx restart
Tail Promtail
On Linux, you can check the syslog for any Promtail related entries by using the command,
tail -f /var/log/syslog | grep promtail
Grafana 10 and Ubuntu 22.04 Notes
There are no considerable differences to be aware of as shown and discussed in the video.