Skip to content

Install a Second Alloy Agent

Video Lecture

Install a Second Alloy Service Install a Second Alloy Service

Description

We can install a Alloy service on other servers, and point them to an existing Loki service already running on a different server.

If you have multiple Alloy services distributed around your network, and all pushing data to one main Loki service, then there are a few more considerations.

Download and Install Alloy Binary

I will install the Alloy service on the same MySQL server that we installed in the MySQL section.

Follow all the same instructions on the page Install Alloy Binary as a Service.

When adding the alloy user, don't forget to add it to the adm group so that it can read the log files in the /var/log/ folder.

Also, so that we can query each server independently in Grafana, we should add extra labels to our Alloy configurations. E.g., adding a label for host is a good option. This will allow us to run log stream selectors specific to the host.

Example config,

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
local.file_match "system" {
        path_targets = [{
                __address__ = "localhost",
                __path__    = "/var/log/*log",
                host        = "mysql",
                job         = "varlogs",
                stream      = "stdout",
        }]
}

loki.source.file "system" {
        targets               = local.file_match.system.targets
        forward_to            = [loki.write.default.receiver]
        legacy_positions_file = "/tmp/positions.yaml"
}

loki.write "default" {
        endpoint {
                url = "http://<IP address or domain name of remote loki service>:3100/loki/api/v1/push"
        }
        external_labels = {}
}

You may need to consider firewall rules to allow this new Alloy to connect to the existing Loki service running on your Grafana server.

Use the journalctl tool help find any errors.

#
journalctl -u alloy.service

Encryption

If your Alloy is connecting to the Loki service across the public network then we can use our existing Nginx proxy and SSL certificate to encrypt the messages as they travel across the network.

You can add a new proxy_pass rule to the existing Grafana server Nginx configuration.

nano /etc/nginx/sites-enabled/YOUR-DOMAIN-NAME.conf

And added the new location configuration.

...
    location /loki/ {
        proxy_set_header Host $http_host;
        proxy_pass http://localhost:3100/;
    }
...

After making changes to the Nginx configuration, test it using

#
nginx -t

If all ok, restart and check status.

#
#
service nginx restart
service nginx status

Next,

Update the Alloy client configuration endpoint property to use the new location created in the Nginx proxy.

...
loki.write "default" {
        endpoint {
                url = "https://<Domain name of your Grafana server>/loki/loki/api/v1/push"
        }
        external_labels = {}
}

See video for much more concise instructions.