Install a Second Promtail Service
Video Lecture
Description
We can install a Promtail service on other servers, and point them to an existing Loki service already running on a different server. If you have multiple Promtail services distributed around your network, and all pushing data to one main Loki service, then there are a few more considerations.
Download and Install Promtail Binary
Follow all the same instructions on the page Install Promtail Binary as a Service
We then need to make sure that the job
label in your Promtail configuration scrape_configs
is unique from the perspective of the Loki service that it will be pushing to. If not, your data for you servers will all appear under the same job label. eg systemd-journal
and you won't be able to easily distinguish which server the log data is actually referring to.
In my config for my sbcode server, I used the job label sbcode/systemd-journal
eg, in my config-promtail.yml
1 2 3 4 5 6 7 8 9 10 11 |
|
Next,
Since my Loki service is running on my Grafana server, and it is on the internet, I blocked port 3100 for external requests on my Grafana server in the previous section where I installed Loki.
In my Promtail configuration clients
section, I will need to set the url of the Loki service on my Grafana server.
My new server will not be able to push data to the main Loki service unless I specifically allow it some how. I have many options,
I can
- Add a firewall rule to allow port 3100 from the IP or CIDR of my new server where I installed Promtail.
- Keep port 3100 blocked externally, but create an extra
location
path in my existing Nginx proxy config that redirects to the main Loki service running locally on my Grafana server - Create a new Nginx config specifically for my Loki service that redirects to my main Loki service running on the server, organize a specific domain or subdomain for it, and add an ssl certificate.
- Install another Loki service locally, point the new Promtail to it, and create a new Loki data source in Grafana.
- Many other options
I chose option 2,
I created a new location in my Grafana Nginx configuration that redirects to the internal endpoint of the Loki service. I also deny all
and allow
specific ip addresses. Replace ###.###.###.###
with the IP or CIDR of your other server(s).
1 |
|
1 2 3 4 5 6 7 |
|
After making changes to an Nginx config, test it using
1 |
|
If all ok, restart
1 |
|
Next,
I need to update the Promtail clients configuration on my config-promtail.yml
. Since the loki service is on another server, setting it to http://127.0.0.1:3100
will not work.
Instead, I use the new location I set up in my Nginx proxy being https://grafana.sbcode.net/loki/
.
Eg,
1 2 |
|
With both the Nginx allow
and deny
rules set, and ensuring the Loki endpoint can only be accessed externally via https
, than I have made sure that the data being sent to my Loki service is encrypted and comes from an allowed source.
Warning
If using a Promtail service, or Loki service across the network, then it is important that you consider who can access it, or whether it needs to be encrypted since the transmitted data is likely to contain sensitive information about your server and other services.