Skip to content

Install a Second Promtail Service

Video Lecture

Install a Second Promtail Service Install a Second Promtail Service

Description

We can install a Promtail service on other servers, and point them to an existing Loki service already running on a different server.

If you have multiple Promtail services distributed around your network, and all pushing data to one main Loki service, then there are a few more considerations.

Download and Install Promtail Binary

I will install a Promtail service on the same MySQL server that we installed in the MySQL section.

Follow all the same instructions on the page Install Promtail Binary as a Service.

When adding the promtail user, don't forget to add it to the adm group so that it can read the log files in the /var/log/ folder.

Also, so that we can query each server independently in Grafana, we should add extra labels to our Promtail configurations. E.g., adding a label for host is a good option. This will allow us to run log stream selectors specific to the host.

Example config,

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://<IP address or domain name of remote loki service>:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log
      stream: stdout
      host: mysql

You may need to consider firewall rules to allow this new Promtail to connect to the existing Loki service running on your Grafana server.

Use the journalctl tool help find any errors.

#
journalctl -u promtail.service

Encryption

If your Promtail is connecting to the Loki service across the public network then we can use our existing Nginx proxy and SSL certificate to encrypt the messages as they travel across the network.

You can add a new proxy_pass rule to the existing Grafana server Nginx configuration.

nano /etc/nginx/sites-enabled/YOUR-DOMAIN-NAME.conf

And added the new location configuration.

...
    location /loki/ {
        proxy_set_header Host $http_host;
        proxy_pass http://localhost:3100/;
    }
...

After making changes to the Nginx configuration, test it using

#
nginx -t

If all ok, restart and check status.

#
#
service nginx restart
service nginx status

Next,

Update the Promtail client configuration clients property to use the new location created in the Nginx proxy.

...
clients:
  - url: https://<Domain name of your Grafana server>/loki/loki/api/v1/push
...

See video for much more concise instructions.

Comments