Skip to content

Install a Second Promtail Service

Video Lecture

Install a Second Promtail Service Install a Second Promtail Service

Description

We can install a Promtail service on other servers, and point them to an existing Loki service already running on a different server. If you have multiple Promtail services distributed around your network, and all pushing data to one main Loki service, then there are a few more considerations.

Download and Install Promtail Binary

I will install a Promtail service on my MySQL server that we installed in the MySQL section.

Follow all the same instructions on the page Install Promtail Binary as a Service. When adding the promtail user, don't forget to add it to the adm group so that it can read the log files in the /var/log/ folder.

Also, so that we can query each server independently in Grafana, we should add extra labels to our Promtail configurations. E.g., adding a label for host name is a good option. This will allow us to run log stream selectors depending on the host label.

My Promtail config looks like this. I've manually set the grpc_listen_port, set the URL that Promtail should push data to, and added a host label.

server:
  http_listen_port: 9080
  grpc_listen_port: 9097

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://<IP address or domain name of your loki service>:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*log
          host: mysql

I have also added the host label to my other Promtail service running on my Grafana server, but set it as host: grafana

Remember to restart and check the Promtail service if you change any configurations.

sudo service promtail start
sudo service promtail status

Note

It can take a minute or two when restarting Promtail.

Firewall

Note

IPTables is a very precise science, to avoid problems setting up the exercises in the course, you can ignore these steps. However, if working in a production environment, ensure your firewalls are managed.

If your setup matches mine, and you are using IPTables to manage your firewall, then Promtail won't be able to send data to the Loki service running on the Grafana server.

I can insert a firewall rule into iptables to allow a specific IP address to access port 3100.

iptables -I INPUT 3 -p tcp -s <IP Address of server sending data> --dport 3100 -j ACCEPT

Note that your iptables configuration may be different from mine, so check the line number that you want to insert to.

Use the command below to list IP tables rules and showing line numbers.

iptables -L --line-numbers

The Promtail on MySQL should new be able to get data into Loki running in the Grafana server. We should be able to read it through the Loki data source we've setup in Grafana user interface.

Encryption

Now, it is important to note that log files can contain very sensitive information. My current setup that I've just demonstrated is not encrypting the data as it is sent across the public network to my Grafana server. Normally you would set up a virtual private network for internal communications between your servers, but in case you don't have this option, we can use the existing Nginx proxy that we installed in the section Reverse Proxy Grafana with Nginx.

I will add a new location record to the existing Nginx configuration so that we can utilize the existing SSL certificate that we installed in the Add SSL section.

On my Grafana server, I have edited my Nginx configuration.

sudo nano /etc/nginx/sites-enabled/YOUR-DOMAIN-NAME.conf

And added the new location configuration.

...
    location /loki/ {
        allow  ###.###.###.###;
        deny all;
        proxy_pass           http://localhost:3100/;
    }
...

Note

The name of your configuration file and the IP address you are allowing will be different from what I show in the video and have written above.

After making changes to the Nginx configuration, test it using

nginx -t

If all ok, restart and check status.

sudo service nginx restart
sudo service nginx status

Next,

I need to update the Promtail client configuration on my MySQL server since the address of the Loki service that it is pushing to, has now changed.

I update the clients property to use the new location that I created in the Nginx proxy.

E.g. Note that your domain name or IP will be different from mine,

...
clients:
  - url: https://grafana.sbcode.net/loki/loki/api/v1/push
...

Now, since I no longer need the iptables rule on my Grafana server to allow the MySQL server to connect to port 3100 directly, I can delete it.

iptables -D INPUT 3

I delete rule at INPUT line 3, since that is where I inserted it earlier. Note that your iptables configuration may be different from mine so check the line number that you want to delete.

With both the Nginx allow and deny rules set, and ensuring the Loki endpoint can only be accessed externally via https, than I have made sure that the data being sent to my Loki service is encrypted and comes from an allowed source.

Warning

If using a Promtail service, or Loki service across the network, then it is important that you consider who can access it, or whether it needs to be encrypted since the transmitted data is likely to contain sensitive information about your server and other services.

Grafana 10 and Ubuntu 22.04 Notes

Read the notes at Install-Promtail-Service

Plus, if you've used IPTables on your server where you are installing this extra Promtail service, then ensure you back up the IPTables rules first before rebooting.

Install Iptables-Persistent if not already.

sudo apt install iptables-persistent

This will save your settings into two files called,

/etc/iptables/rules.v4

/etc/iptables/rules.v6

If you already had Iptables-Persistent installed, then just run

iptables-save > /etc/iptables/rules.v4

iptables-save > /etc/iptables/rules.v6