Log File Monitoring - Nginx Proxy HTTP Status Codes
Video Lecture
Description
Monitoring Log Files - HTTP Status Codes of a Nginx Proxy
If you don't have an existing Nginx server that you can read the logs for, then you can install one.
Debian/Ubuntu
sudo apt install nginx
CentOS 8
sudo dnf install nginx
The file I monitor is located at /var/log/nginx/access.log
The default Zabbix user that the Zabbix agent user uses, does not have read access to many log files on the system.
You can usually add the zabbix user to a group to solve this problem.
The nginx access.log file can be read by the www-data or adm groups on Ubuntu, so I add the Zabbix user to the adm group.
To find out which groups a log file can be read by, for example, I typed,
$ ls -lh /var/log/nginx/
This tells me that the access.log file can be read by www-data and adm groups.
Then I check which groups the user Zabbix is part of,
groups zabbix
If it's not part of either group already, I then add it,
sudo usermod -a -G adm zabbix
and check again to confirm.
groups zabbix
After changing the Zabbix users permissions, you will need to restart the Zabbix agent.
sudo service zabbix-agent restart
I can read the most recent log file entries by typing
tail -f /var/log/nginx/access.log
You can also check this command works when using the Zabbix user,
sudo -H -u zabbix bash -c 'tail -f /var/log/nginx/access.log'
I then created an item for the host, with settings
| Property | Value |
|---|---|
| Name | HTTPÂ Status Codes |
| Type | Zabbix (active) |
| Key | log[/var/log/nginx/access.log,"^(\S+) (\S+) (\S+) \[([\w:\/]+\s[+\-]\d{4})\] \"(\S+)\s?(\S+)?\s?(\S+)?\" (\d{3}|-) (\d+|-)\s?\"?([^\"]*)\"?\s?\"?([^\"]*)\"",,,skip,\8,,,] |
| Type of Information | numeric (unsigned) |
| Update Interval | 1m |
The regex value that I copy into https://regex101.com is
^(\S+) (\S+) (\S+) \[([\w:\/]+\s[+\-]\d{4})\] \"(\S+)\s?(\S+)?\s?(\S+)?\" (\d{3}|-) (\d+|-)\s?\"?([^\"]*)\"?\s?\"?([^\"]*)\"
This regex can separate the values for both Nginx and Apache access logs.
The regex splits each row of the log into several groups.
The HTTPÂ Status code is in the 8th group.
I can also create triggers to notify on
- 101 Switching Protocols
- 301 Moved Permanently
- 302 Redirect
- 304 not modified
- 400 Bad Request
- 401 Unauthorized
- 403 Forbidden
- 404 Not found
- 405 Method Not Allowed
- 500 Server Error
In this video I demonstrate creating triggers for HTTP 5XX errors and use count to detect 10 or more HTTP 404 Errors in 10 minutes.
How and whether you decide to trigger on HTTP status codes is up to you. The video just provides examples for you to follow.
Note
If you installed Zabbix like I did on Ubuntu 20.04 with the default Apache server, there also are Apache logs at /var/log/apache2/ on the Zabbix server that you can use for this lesson.
You can tail them in the command prompt using
tail -f /var/log/apache2/access.log
This is a good source of http logs of you don't already have an active webserver somewhere that you can use, or you don't want to set one up.
Apache and Nginx logs follow the same format so my regex in the video will still work.