Skip to content

Install Loki Binary and Start as a Service

Video Lecture

Install Loki Binary and Start as a Service Install Loki Binary and Start as a Service

Download and Install Loki Binary

To keep this as simple as possible, we will install the Loki binary as a service on our existing Grafana server.

To check the latest version of Grafana Loki, visit the Loki releases page. https://github.com/grafana/loki/releases/

cd /usr/local/bin

curl -O -L "https://github.com/grafana/loki/releases/download/v2.4.1/loki-linux-amd64.zip"

A new Ubuntu is unlikely to have unzip pre-installed, so install it using the command,

apt install unzip

Now unzip the downloaded file,

unzip "loki-linux-amd64.zip"

And allow execute permission on the Loki binary

chmod a+x "loki-linux-amd64"

Create the Loki config

Now create the Loki config file.

sudo nano config-loki.yml

And add this text.

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

This default configuration was copied from https://raw.githubusercontent.com/grafana/loki/master/cmd/loki/loki-local-config.yaml. There may be changes to this config depending on any future updates to Loki.

Configure Loki to run as a service

Now we will configure Loki to run as a service so that it stays running in the background.

Create a user specifically for the Loki service

sudo useradd --system loki

Create a file called loki.service

sudo nano /etc/systemd/system/loki.service

Add the script and save

[Unit]
Description=Loki service
After=network.target

[Service]
Type=simple
User=loki
ExecStart=/usr/local/bin/loki-linux-amd64 -config.file /usr/local/bin/config-loki.yml

[Install]
WantedBy=multi-user.target

Now start and check the service is running.

sudo service loki start
sudo service loki status

We can now leave the new Loki service running.

If you ever need to stop the new Loki service, then type

sudo service loki stop
sudo service loki status

Warning

If you reboot your server, the Loki Service may not restart automatically.

You can set the Loki service to auto restart after reboot by entering,

sudo systemctl enable loki.service

Note it may take a minute to stop.

Configure Firewall

Note

IPTables is a very precise science, to avoid problems setting up the exercises in the course, you can ignore these steps. However, if working in a production environment, ensure your firewalls are managed.

When your Loki server is running, it may be accessible remotely on port 3100. If you only want localhost to be able to connect, then type

iptables -A INPUT -p tcp -s localhost --dport 3100 -j ACCEPT
iptables -A INPUT -p tcp --dport 3100 -j DROP
iptables -L

After blocking port 3100 for external requests, you can verify that local request are still possible by using,

curl "127.0.0.1:3100/metrics"

Also, Loki exposes port 9096 for gRPC communications. This port may also be accessible across the internet. To close it using iptables, then use,

iptables -A INPUT -p tcp -s <your grafana servers domain name or ip address> --dport 9096 -j ACCEPT
iptables -A INPUT -p tcp -s localhost --dport 9096 -j ACCEPT
iptables -A INPUT -p tcp --dport 9096 -j DROP
iptables -L

Warning

iptables settings will be lost in case of system reboot. You will need to reapply them manually,

or

install iptables-persistent

sudo apt install iptables-persistent

This will save your settings into two files called,

/etc/iptables/rules.v4

/etc/iptables/rules.v6

Any changes you make to the iptables configuration won't be auto saved to these persistent files, so if you want to update these files with any changes, then use the commands,

iptables-save > /etc/iptables/rules.v4

iptables-save > /etc/iptables/rules.v6

Troubleshooting

Permission Denied, Internal Server Error

If you get any of these errors

  • Loki: Internal Server Error. 500. open /tmp/loki/index/index_2697: permission denied
  • "failed to flush user" "open /tmp/loki/chunks/...etc : permission denied"
  • Loki: Internal Server Error. 500. Internal Server Error

You should check the owner of the folders configured in the storage_config section of the the config-loki.yml to match the name of the user configured in the loki.service script above.

My user is loki, and the folders begins with /tmp/loki so I recursively set the owner.

chown -R loki:loki /tmp/loki

You may need to restart the Loki service and checking again its status.

If when connecting to Loki using the Grafana data source configuration, you see the error "Loki: Bad Gateway. 502. Bad Gateway", this will happen if the Loki service is not running, your have entered the wrong URL, or ports are blocked by a firewall. The default Loki install uses both ports 3100 for HTTP and 9096 for gRPC. See my iptables rules above for my explicit rules which I've setup on my Grafana server that is also hosting the Loki service.

Data source connected, but no labels received

When connecting to your Loki data source for the first time, you may see the error,

  • Data source connected, but no labels received. Verify that Loki and Promtail is configured properly.

Note how it says, Data source connected at the beginning of the warning. Recent versions of Loki, despite having a successful connection, will still indicate that you have no labels because you don't have Promtail sending any data to it yet. Promtail is set up and discussed in the next lesson.

Unable to connect with Loki. Please check the server logs for more details

To view the Grafana server logs,

tail /var/log/grafana/grafana.log

To filter by errors,

tail /var/log/grafana/grafana.log | grep level=error

IPTables Cheat Sheet

Grafana 10 and Ubuntu 22.04 Notes

Loki v2.8.2

Since recording this video, Loki is now at version 2.8.2. You can install that version instead if you wish.

CD into the /usr/local/bin folder.

cd /usr/local/bin

Download the required version of Loki.

curl -O -L "https://github.com/grafana/loki/releases/download/v2.8.2/loki-linux-amd64.zip"

You will most likely need to install unzip first.

apt install unzip

If you get a warning about the kernel then follow the prompts accepting the defaults, and press Tab to highlight the OK option and then reboot your server.

reboot now

After rebooting, you may need to restart your Grafana Server.

sudo service grafana-server start

To ensure Grafana automatically starts after any reboot, you can enter

sudo systemctl enable grafana-server.service

CD back into the /usr/local/bin folder.

cd /usr/local/bin

Unzip the downloaded file.

unzip "loki-linux-amd64.zip"

Now go back and continue from Create The Loki Config

Explore → Loki Data Source.

In Grafana 10, when exploring the new Loki data source, the default view is the new Builder UI. My video shows the Code UI, where it indicates that there are presently no logs found.

You can view the Code UI by pressing the [Code] button at the right of the screen.

Code UI Button

Comments