Skip to content


 Zabbix
 Grafana
 Prometheus
 Threejs and TypeScript
 SocketIO and TypeScript
 Blender 3D Topological Earth
 Sweet Home 3D
 Design Patterns in Python
   
 Course Coupon Codes
Design Patterns In Python (Book)
     
     

Read Nginx Logs with Promtail

Video Lecture

Read Nginx Logs with Promtail Read Nginx Logs with Promtail Read Nginx Logs with Promtail

Description

We will add to our Promtail scrape configs, the ability to read the Nginx access and error logs.

We need to add a new job_name to our existing Promtail config_promtail.yml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
...
scrape_configs:
  - job_name: journal
...
...
  - job_name: nginx
    entry_parser: raw
    static_configs:
      - targets:
          - localhost
        labels:
          job: nginx
          host: sbcode
          __path__: /var/log/nginx/*log

Restart the Promtail service and check its status. 

1
2
sudo service promtail restart
sudo service promtail status

If we look into the existing systemd-journal job that we already set up in the previous lessons, we will start to see some new errors concerning Promtail.

The promtail user, used by the Promtail service doesn't have the permissions to read the Nginx logs.

So we can add it to the appropriate group.

If I type 

1
ls -lh /var/log/nginx/
then I can see that the user group adm has read permissions.

I will add the the promtail user to the user group adm so that it can read the log file.

1
sudo usermod -a -G adm promtail

I can now list the groups that promtail is part of. 

1
groups promtail

If I go back into Grafana, I will see the new nginx job inside the Explore panel.

This is pretty good now, but we can make it better.

I want to be able to filter by the status code and other http properties.

We can do a regex in the log files as Promtail scans them and add labels to the matches.

For this we use pipeline_stages

So update you whole nginx scrape config section with this below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
...
  - job_name: nginx
    entry_parser: raw
    static_configs:
      - targets:
          - localhost
        labels:
          job: nginx
          host: sbcode
          __path__: /var/log/nginx/*log
    pipeline_stages:
    - match:
        selector: '{job="nginx"}'
        stages:
        - regex:
            expression: '^(?P<remote_addr>[\w\.]+) - (?P<remote_user>[^ ]*) \[(?P<time_local>.*)\] "(?P<method>[^ ]*) (?P<request>[^ ]*) (?P<protocol>[^ ]*)" (?P<status>[\d]+) (?P<body_bytes_sent>[\d]+) "(?P<http_referer>[^"]*)" "(?P<http_user_agent>[^"]*)"?'
        - labels:
            #remote_addr:
            #remote_user:
            #time_local:
            method:
            #request:
            #protocol:
            status:
            #body_bytes_sent:
            #http_referer:
            #http_user_agent:

The regular expression creates new labels for remote_addr, remote_user, time_local, method, request, protocol, status, body_bytes_sent, http_referer and http_user_agent

In my labels, I have chosen only to only show method and status. You can uncomment the other values that the regex has collected in case you want to also see them in Grafana.

Now I can create a dashboard for Nginx status codes using this sample query.

1
sum(count_over_time ({job="nginx",host="sbcode"}[1m])) by (status)

Troubleshooting

YML files are whitespace sensitive. Many errors restarting promtail can be attributed to incorrect indentation.

Eg, you might see the error, "found a tab character that violates indentation"

Double check all indentations in the yml are spaces and not tabs.

Also,

You may see the error "permission denied". Ensure that your promtail user is in the same group that can read the log files listed in your scrope configs __path__ setting.

Promtail Pipelines

Regex 101