Install a Second Promtail Service
We can install a Promtail service on other servers, and point them to an existing Loki service already running on a different server. If you have multiple Promtail services distributed around your network, and all pushing data to one main Loki service, then there are a few more considerations.
Download and Install Promtail Binary
Follow all the same instructions on the page Install Promtail Binary as a Service
We then need to make sure that the
job label in your Promtail configuration
scrape_configs is unique from the perspective of the Loki service that it will be pushing to. If not, your data for you servers will all appear under the same job label. eg
systemd-journal and you won't be able to easily distinguish which server the log data is actually referring to.
In my config for my sbcode server, I used the job label
eg, in my
1 2 3 4 5 6 7 8 9 10 11
Since my Loki service is running on my Grafana server, and it is on the internet, I blocked port 3100 for external requests on my Grafana server in the previous section where I installed Loki.
In my Promtail configuration
clients section, I will need to set the url of the Loki service on my Grafana server.
My new server will not be able to push data to the main Loki service unless I specifically allow it some how. I have many options,
- Add a firewall rule to allow port 3100 from the IP or CIDR of my new server where I installed Promtail.
- Keep port 3100 blocked externally, but create an extra
locationpath in my existing Nginx proxy config that redirects to the main Loki service running locally on my Grafana server
- Create a new Nginx config specifically for my Loki service that redirects to my main Loki service running on the server, organize a specific domain or subdomain for it, and add an ssl certificate.
- Install another Loki service locally, point the new Promtail to it, and create a new Loki data source in Grafana.
- Many other options
I chose option 2,
I created a new location in my Grafana Nginx configuration that redirects to the internal endpoint of the Loki service. I also
deny all and
allow specific ip addresses. Replace
###.###.###.### with the IP or CIDR of your other server(s).
1 2 3 4 5 6 7
After making changes to an Nginx config, test it using
If all ok, restart
I need to update the Promtail clients configuration on my
config-promtail.yml. Since the loki service is on another server, setting it to
http://127.0.0.1:3100 will not work.
Instead, I use the new location I set up in my Nginx proxy being
With both the Nginx
deny rules set, and ensuring the Loki endpoint can only be accessed externally via
https, than I have made sure that the data being sent to my Loki service is encrypted and comes from an allowed source.
If using a Promtail service, or Loki service across the network, then it is important that you consider who can access it, or whether it needs to be encrypted since the transmitted data is likely to contain sensitive information about your server and other services.