Read Nginx Logs with Promtail
Video Lecture
Description
We will add to our Promtail scrape configs, the ability to read the Nginx access and error logs.
We need to add a new job_name
to our existing Promtail config_promtail.yml
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
|
Restart the Promtail service and check its status.
1 2 |
|
If we look into the existing systemd-journal
job that we already set up in the previous lessons, we will start to see some new errors concerning Promtail.
The promtail
user, used by the Promtail service doesn't have the permissions to read the Nginx logs.
So we can add it to the appropriate group.
If I type
1 |
|
adm
has read permissions.
I will add the the promtail
user to the user group adm
so that it can read the log file.
1 |
|
I can now list the groups that promtail is part of.
1 |
|
If I go back into Grafana, I will see the new nginx
job inside the Explore panel.
This is pretty good now, but we can make it better.
I want to be able to filter by the status
code and other http properties.
We can do a regex in the log files as Promtail scans them and add labels to the matches.
For this we use pipeline_stages
So update you whole nginx scrape config section with this below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
|
The regular expression creates new labels for remote_addr
, remote_user
, time_local
, method
, request
, protocol
, status
, body_bytes_sent
, http_referer
and http_user_agent
In my labels, I have chosen only to only show method
and status
. You can uncomment the other values that the regex has collected in case you want to also see them in Grafana.
Now I can create a dashboard for Nginx status codes using this sample query.
1 |
|
Troubleshooting
YML files are whitespace sensitive. Many errors restarting promtail can be attributed to incorrect indentation.
Eg, you might see the error, "found a tab character that violates indentation"
Double check all indentations in the yml are spaces and not tabs.
Also,
You may see the error "permission denied". Ensure that your promtail user is in the same group that can read the log files listed in your scrope configs __path__
setting.