Paperback 
Install a Second Promtail Service
Video Lecture
Description
We can install a Promtail service on other servers, and point them to an existing Loki service already running on a different server. If you have multiple Promtail services distributed around your network, and all pushing data to one main Loki service, then there are a few more considerations.
Download and Install Promtail Binary
I will install a Promtail service on my MySQL server that we installed in the MySQL section.
Follow all the same instructions on the page Install Promtail Binary as a Service. When adding the promtail user, don't forget to add it to the adm group so that it can read the log files in the /var/log/ folder.
Also, so that we can query each server independently in Grafana, we should add extra labels to our Promtail configurations. E.g., adding a label for host name is a good option. This will allow us to run log stream selectors depending on the host label.
My Promtail config looks like this. I've manually set the grpc_listen_port, set the URL that Promtail should push data to, and added a host label.
server:
http_listen_port: 9080
grpc_listen_port: 9097
positions:
filename: /tmp/positions.yaml
clients:
- url: http://<IP address or domain name of your loki service>:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*log
host: mysql
I have also added the host label to my other Promtail service running on my Grafana server, but set it as host: grafana
Remember to restart and check the Promtail service if you change any configurations.
sudo service promtail start
sudo service promtail status
If your setup matches mine, and you are using iptables to manage your firewall, then Promtail won't be able to send its data to the Loki service running on the Grafana server.
I can insert a firewall rule into iptables to allow a specific IP address to access port 3100.
iptables -I INPUT 3 -p tcp -s <IP Address of server to allow> --dport 3100 -j ACCEPT
Note that your iptables configuration may be different from mine, so check the line number that you want to insert to.
Use the command below to list IP tables rules and showing line numbers.
iptables -L --line-numbers
The Promtail on MySQL should new be able to get data into Loki running in the Grafana server. We should be able to read it through the Loki data source we've setup in Grafana user interface.
Now, it is important to note that log files can contain very sensitive information. My current setup that I've just demonstrated is not encrypting the data as it is sent across the public network to my Grafana server. Normally you would set up a virtual private network for internal communications between your servers, but in case you don't have this option, we can use the existing Nginx proxy that we installed in the section Reverse Proxy Grafana with Nginx.
I will add a new location record to the existing Nginx configuration so that we can utilize the existing SSL certificate that we installed in the Add SSL section.
On my Grafana server, I have edited my Nginx configuration.
sudo nano /etc/nginx/sites-enabled/YOUR-DOMAIN-NAME.conf
And added the new location configuration.
...
location /loki/ {
allow ###.###.###.###;
deny all;
proxy_pass http://localhost:3100/;
}
...
Note
The name of your configuration file and the IP address you are allowing will be different from what I show in the video and have written above.
After making changes to the Nginx configuration, test it using
nginx -t
If all ok, restart and check status.
sudo service nginx restart
sudo service nginx status
Next,
I need to update the Promtail client configuration on my MySQL server since the address of the Loki service that it is pushing to, has now changed.
I update the clients property to use the new location that I created in the Nginx proxy.
E.g. Note that your domain name or IP will be different from mine,
...
clients:
- url: https://grafana.sbcode.net/loki/loki/api/v1/push
...
Now, since I no longer need the iptables rule on my Grafana server to allow the MySQL server to connect to port 3100 directly, I can delete it.
iptables -D INPUT 3
I delete rule at INPUT line 3, since that is where I inserted it earlier. Note that your iptables configuration may be different from mine so check the line number that you want to delete.
With both the Nginx allow and deny rules set, and ensuring the Loki endpoint can only be accessed externally via https, than I have made sure that the data being sent to my Loki service is encrypted and comes from an allowed source.
Warning
If using a Promtail service, or Loki service across the network, then it is important that you consider who can access it, or whether it needs to be encrypted since the transmitted data is likely to contain sensitive information about your server and other services.
<!-- My new server will not be able to push data to the main Loki service unless I specifically allow it some how. I have many options,
I can
- Add a firewall rule to allow port 3100 from the IP or CIDR of my new server where I installed Promtail.
- Keep port 3100 blocked externally, but create an extra
locationpath in my existing Nginx proxy config that redirects to the main Loki service running locally on my Grafana server - Create a new Nginx config specifically for my Loki service that redirects to my main Loki service running on the server, organize a specific domain or subdomain for it, and add an ssl certificate.
- Install another Loki service locally, point the new Promtail to it, and create a new Loki data source in Grafana.
- Many other options
I chose option 2,
I created a new location in my Grafana Nginx configuration that redirects to the internal endpoint of the Loki service. I also deny all and allow specific ip addresses. Replace ###.###.###.### with the IP or CIDR of your other server(s).
sudo nano /etc/nginx/sites-enabled/YOUR-DOMAIN-NAME.conf
...
location /loki/ {
allow ###.###.###.###;
deny all;
proxy_pass http://localhost:3100/;
}
...
After making changes to an Nginx config, test it using
nginx -t
If all ok, restart
sudo service nginx restart
Next,
I need to update the Promtail clients configuration on my config-promtail.yml. Since the loki service is on another server, setting it to http://127.0.0.1:3100 will not work.
Instead, I use the new location I set up in my Nginx proxy being https://grafana.sbcode.net/loki/.
Eg,
clients:
- url: https://grafana.sbcode.net/loki/loki/api/v1/push
``` -->
## Grafana 9 and Ubuntu 22.04 Notes
Read the notes at [Install-Promtail-Service](install-promtail-service/#grafana-9-and-ubuntu-2204-notes)
Plus, if you've used IPTables on your server where you are installing this extra Promtail service, then ensure you back up the IPTables rules first before rebooting.
Install Iptables-Persistent if not already.
This will save your settings into two files called,
/etc/iptables/rules.v4
/etc/iptables/rules.v6
If you already had Iptables-Persistent installed, then just run
iptables-save > /etc/iptables/rules.v4
iptables-save > /etc/iptables/rules.v6