Install a Second Promtail Service

Video Lecture

Install a Second Promtail Service Install a Second Promtail Service Install a Second Promtail Service

Description

We can install a Promtail service on other servers, and point them to an existing Loki service already running on a different server. If you have multiple Promtail services distributed around your network, and all pushing data to one main Loki service, then there are a few more considerations.

Download and Install Promtail Binary

Follow all the same instructions on the page Install Promtail Binary as a Service

We then need to make sure that the job label in your Promtail configuration scrape_configs is unique from the perspective of the Loki service that it will be pushing to. If not, your data for you servers will all appear under the same job label. eg systemd-journal and you won't be able to easily distinguish which server the log data is actually referring to.

In my config for my sbcode server, I used the job label sbcode/systemd-journal

eg, in my config-promtail.yml

...
scrape_configs:
  - job_name: journal
    journal:
      max_age: 12h
      labels:
        job: sbcode/systemd-journal
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'
...

Next,

Since my Loki service is running on my Grafana server, and it is on the internet, I blocked port 3100 for external requests on my Grafana server in the previous section where I installed Loki.

In my Promtail configuration clients section, I will need to set the url of the Loki service on my Grafana server.

My new server will not be able to push data to the main Loki service unless I specifically allow it some how. I have many options,

I can

  1. Add a firewall rule to allow port 3100 from the IP or CIDR of my new server where I installed Promtail.
  2. Keep port 3100 blocked externally, but create an extra location path in my existing Nginx proxy config that redirects to the main Loki service running locally on my Grafana server
  3. Create a new Nginx config specifically for my Loki service that redirects to my main Loki service running on the server, organize a specific domain or subdomain for it, and add an ssl certificate.
  4. Install another Loki service locally, point the new Promtail to it, and create a new Loki data source in Grafana.
  5. Many other options

I chose option 2,

I created a new location in my Grafana Nginx configuration that redirects to the internal endpoint of the Loki service. I also deny all and allow specific ip addresses. Replace ###.###.###.### with the IP or CIDR of your other server(s).

sudo nano /etc/nginx/sites-enabled/YOUR-DOMAIN-NAME.conf

...
    location /loki/ {        
        allow  ###.###.###.###;
        deny all;
        proxy_pass           http://localhost:3100/;
    }
...

After making changes to an Nginx config, test it using

nginx -t

If all ok, restart

sudo service nginx restart

Next,

I need to update the Promtail clients configuration on my config-promtail.yml. Since the loki service is on another server, setting it to http://127.0.0.1:3100 will not work.

Instead, I use the new location I set up in my Nginx proxy being https://grafana.sbcode.net/loki/.

Eg, 

clients:
  - url: https://grafana.sbcode.net/loki/loki/api/v1/push

With both the Nginx allow and deny rules set, and ensuring the Loki endpoint can only be accessed externally via https, than I have made sure that the data being sent to my Loki service is encrypted and comes from an allowed source.

Warning

If using a Promtail service, or Loki service across the network, then it is important that you consider who can access it, or whether it needs to be encrypted since the transmitted data is likely to contain sensitive information about your server and other services.