Read Nginx Logs with Promtail
Video Lecture
Description
We will add to our Promtail scrape configs, the ability to read the Nginx access and error logs.
We need to add a new job_name to our existing Promtail config_promtail.yml
...
scrape_configs:
- job_name: journal
...
...
- job_name: nginx
entry_parser: raw
static_configs:
- targets:
- localhost
labels:
job: nginx
host: sbcode
__path__: /var/log/nginx/*log
Restart the Promtail service and check its status.
sudo service promtail restart sudo service promtail status
If we look into the existing systemd-journal job that we already set up in the previous lessons, we will start to see some new errors concerning Promtail.
The promtail user, used by the Promtail service doesn't have the permissions to read the Nginx logs.
So we can add it to the appropriate group.
If I type
ls -lh /var/log/nginx/
then I can see that the user group adm has read permissions.
I will add the the promtail user to the user group adm so that it can read the log file.
sudo usermod -a -G adm promtail
I can now list the groups that promtail is part of.
groups promtail
If I go back into Grafana, I will see the new nginx job inside the Explore panel.
This is pretty good now, but we can make it better.
I want to be able to filter by the status code and other http properties.
We can do a regex in the log files as Promtail scans them and add labels to the matches.
For this we use pipeline_stages
So update you whole nginx scrape config section with this below.
...
- job_name: nginx
entry_parser: raw
static_configs:
- targets:
- localhost
labels:
job: nginx
host: sbcode
__path__: /var/log/nginx/*log
pipeline_stages:
- match:
selector: '{job="nginx"}'
stages:
- regex:
expression: '^(?P<remote_addr>[\w\.]+) - (?P<remote_user>[^ ]*) \[(?P<time_local>.*)\] "(?P<method>[^ ]*) (?P<request>[^ ]*) (?P<protocol>[^ ]*)" (?P<status>[\d]+) (?P<body_bytes_sent>[\d]+) "(?P<http_referer>[^"]*)" "(?P<http_user_agent>[^"]*)"?'
- labels:
#remote_addr:
#remote_user:
#time_local:
method:
#request:
#protocol:
status:
#body_bytes_sent:
#http_referer:
#http_user_agent:
The regular expression creates new labels for remote_addr, remote_user, time_local, method, request, protocol, status, body_bytes_sent, http_referer and http_user_agent
In my labels, I have chosen only to only show method and status. You can uncomment the other values that the regex has collected in case you want to also see them in Grafana.
Now I can create a dashboard for Nginx status codes using this sample query.
sum(count_over_time ({job="nginx",host="sbcode"}[1m])) by (status)
Troubleshooting
YML files are whitespace sensitive. Many errors restarting promtail can be attributed to incorrect indentation.
Eg, you might see the error, "found a tab character that violates indentation"
Double check all indentations in the yml are spaces and not tabs.
Also,
You may see the error "permission denied". Ensure that your promtail user is in the same group that can read the log files listed in your scrope configs __path__ setting.