Skip to content


 Zabbix
 Grafana
 Prometheus
 Threejs and TypeScript
 SocketIO and TypeScript
 Blender Topological Earth
 Sweet Home 3D
 Design Patterns Python
 Design Patterns TypeScript
   
 Course Coupon Codes
Three.js and TypeScript
Kindle Edition
$9.99 $14.99 Paperback 
$29.99 $34.99




Design Patterns in TypeScript
Kindle Edition
$9.99 $14.99 Paperback
$19.99 $24.99




Design Patterns in Python
Kindle Edition
$9.99 $14.99 Paperback
$19.99 $24.99




Install a Second Promtail Service

Video Lecture

Install a Second Promtail Service Install a Second Promtail Service Install a Second Promtail Service

Description

We can install a Promtail service on other servers, and point them to an existing Loki service already running on a different server. If you have multiple Promtail services distributed around your network, and all pushing data to one main Loki service, then there are a few more considerations.

Download and Install Promtail Binary

I will install a Promtail service on my MySQL server that we installed in the MySQL section.

Follow all the same instructions on the page Install Promtail Binary as a Service. When adding the promtail user, don't forget to add it to the adm group so that it can read the log files in the /var/log/ folder.

Also, so that we can query each server independently in Grafana, we should add an extra labels to our Promtail configurations. E.g., adding a label for host name is a good option. This will allow us to run log stream selectors depending on the host label.

My Promtail config looks like this. I've manually set the grpc_listen_port, set the url that Promtail should push data to, and added a host label.

server:
  http_listen_port: 9080
  grpc_listen_port: 9097

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://<IP address or domain name of your loki service>:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*log
          host: mysql

I have also added the host label to my other Promtail service running on my Grafana server, but set it as host: grafana

Remember to restart and check the Promtail service if you change any configurations.

sudo service promtail start
sudo service promtail status

If your setup matches mine, and you are using iptables to manage your firewall, then Promtail won't be able to send its data to the Loki service running on the Grafana server.

I can insert a firewall rule into iptables to allow a specific IP address to access port 3100.

iptables -I INPUT 3 -p tcp -s <IP Address of server to allow> --dport 3100 -j ACCEPT

Note that your iptables configuration may be different than mine so check the line number that you want to insert to.

Use the command below to list IP tables rules and showing line numbers.

iptables -L --line-numbers

The Promtail on MySQL should new be able to get data into Loki running in the Grafana server. We should be able to read it through the Loki data source we've setup in Grafana user interface.

Now, it is important to note that log files can contain very sensitive information. My current setup that I've just demonstrated is not encrypting the data as it is sent across the public network to my Grafana server. Normally you would set up a virtual private network for internal communications between your servers, but in case you don't have this option, we can use the existing Nginx proxy that we installed in the section Reverse Proxy Grafana with Nginx.

I will add a new location record to the existing Nginx configuration so that we can utilise the existing SSL certificate that we installed in the Add SSL section.

On my Grafana server, I have edited my Nginx configuration.

sudo nano /etc/nginx/sites-enabled/YOUR-DOMAIN-NAME.conf

I adding the new location configuration.

...
    location /loki/ {
        allow  ###.###.###.###;
        deny all;
        proxy_pass           http://localhost:3100/;
    }
...

Note

The name of your configuration file and the IP address you are allowing will be different than what I show in the video and have written above.

After making changes to an Nginx configuration, test it using

nginx -t

If all ok, restart and check status.

sudo service nginx restart
sudo service nginx status

Next,

I need to update the Promtail client configuration on my MySQL server since the address of the Loki service that it is pushing to has now changed.

I update the clients property to use the new location that I created in the Nginx proxy.

E.g. Note that your domain name or ip will be different than mine,

...
clients:
  - url: https://grafana.sbcode.net/loki/loki/api/v1/push
...

Now, since I no longer need the iptables rule on my Grafana server to allow the MySQL server to connect to port 3100 directly, I can delete it.

iptables -D INPUT 3

I delete rule at INPUT line 3, since that is where I inserted it earlier. Note that your iptables configuration may be different than mine so check the line number that you want to delete.

With both the Nginx allow and deny rules set, and ensuring the Loki endpoint can only be accessed externally via https, than I have made sure that the data being sent to my Loki service is encrypted and comes from an allowed source.

Warning

If using a Promtail service, or Loki service across the network, then it is important that you consider who can access it, or whether it needs to be encrypted since the transmitted data is likely to contain sensitive information about your server and other services.