Monitor SSH Session Count with Telegraf inputs.exec

Description

The Telegraf inputs.exec can be used to run commands.

In this example, I will setup monitoring of the SSH session counts on my server that is running the Telegraf agent. This will be a good example that you can adapt to run commands that will suit your own needs.

SSH onto the server where the Telegraf agent is running.

From the bash command, I can see how many ssh sessions are runnning by using the command,

ps aux | grep [s]shd:

Create a new file called sshd-count.sh in the /usr/local/bin folder.

nano sshd-count.sh
#!/bin/sh
count=$(ps aux | grep -c [s]shd:)
echo $count

Save the file, and then give it execute permissions.

chmod a+x sshd-count.sh

Test the file works,

/usr/local/bin/sshd-count.sh

It should return a number such as 1 or 2 etc.

Now lets create the input in the telegraf.conf

sudo nano /etc/telegraf/telegraf.conf

And add this input

[[inputs.exec]]
  commands = ["/usr/local/bin/sshd-count.sh"]
  name_override = "ssh_sessions"
  timeout = "5s"
  data_format = "value"
  data_type = "integer"
  interval = "1m"

Now test the new input works by testing the exec input filter.

telegraf --test --input-filter=exec

It should return something like

ssh_sessions,host=127.0.0.1 value=1i 1589646245000000000

Restart Telegraf and check its status is active (running)

sudo service telegraf restart
sudo service telegraf status

Now open the Grafana UI, and visit the Explore tab and copy this query from the screen grab below.

SSHD Counts Explore

If you can see data, then you can now create a dashboard for it and create an alert for it if you want.

SSHD Counts Graph

If you also have installed Logs Panel, Loki and Promtail on your server and setup your Loki Data Source, as described in the previous lessons, then you can then create Annotation queries and link the Logs Panel with your SSHD Graph.

Your Loki Datasourec Query Log Label can be {unit="ssh.service"}

Loki SSHD Service Query

And you could have a dashboard like this. SSH Dashboard With Graph and Log Panel

I also created an Annotation query which looked for the term "invalid", but also case insensitive. {unit="ssh.service"} (?i)invalid

Note you may want to set your SyslogFacility log level in your sshd_config

sudo nano /etc/ssh/sshd_config

Uncomment SyslogFacility

...
# Logging
SyslogFacility AUTH
...
sudo service sshd restart