Read Nginx Logs with Promtail
We will add to our Promtail scrape configs, the ability to read the Nginx access and error logs.
We need to add a new
job_name to our existing Promtail
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Restart the Promtail service and check its status.
If we look into the existing
systemd-journal job that we already set up in the previous lessons, we will start to see some new errors concerning Promtail.
promtail user, used by the Promtail service doesn't have the permissions to read the Nginx logs.
So we can add it to the appropriate group.
If I type
admhas read permissions.
I will add the the
promtail user to the user group
adm so that it can read the log file.
I can now list the groups that promtail is part of.
If I go back into Grafana, I will see the new
nginx job inside the Explore panel.
This is pretty good now, but we can make it better.
I want to be able to filter by the
status code and other http properties.
We can do a regex in the log files as Promtail scans them and add labels to the matches.
For this we use
So update you whole nginx scrape config section with this below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
The regular expression creates new labels for
In my labels, I have chosen only to only show
status. You can uncomment the other values that the regex has collected in case you want to also see them in Grafana.
Now I can create a dashboard for Nginx status codes using this sample query.
YML files are whitespace sensitive. Many errors restarting promtail can be attributed to incorrect indentation.
Eg, you might see the error, "found a tab character that violates indentation"
Double check all indentations in the yml are spaces and not tabs.
You may see the error "permission denied". Ensure that your promtail user is in the same group that can read the log files listed in your scrope configs