Monitor SSH Session Count with Telegraf inputs.exec
Description
The Telegraf inputs.exec can be used to run commands.
In this example, I will setup monitoring of the SSH session counts on my server that is running the Telegraf agent. This will be a good example that you can adapt to run commands that will suit your own needs.
SSH onto the server where the Telegraf agent is running.
From the bash command, I can see how many ssh sessions are runnning by using the command,
ps aux | grep [s]shd:
Create a new file called sshd-count.sh in the /usr/local/bin folder.
nano sshd-count.sh
#!/bin/sh
count=$(ps aux | grep -c [s]shd:)
echo $count
Save the file, and then give it execute permissions.
chmod a+x sshd-count.sh
Test the file works,
/usr/local/bin/sshd-count.sh
It should return a number such as 1 or 2 etc.
Now lets create the input in the telegraf.conf
sudo nano /etc/telegraf/telegraf.conf
And add this input
[[inputs.exec]]
commands = ["/usr/local/bin/sshd-count.sh"]
name_override = "ssh_sessions"
timeout = "5s"
data_format = "value"
data_type = "integer"
interval = "1m"
Now test the new input works by testing the exec input filter.
telegraf --test --input-filter=exec
It should return something like
ssh_sessions,host=127.0.0.1 value=1i 1589646245000000000
Restart Telegraf and check its status is active (running)
sudo service telegraf restart
sudo service telegraf status
Now open the Grafana UI, and visit the Explore tab and copy this query from the screen grab below.
If you can see data, then you can now create a dashboard for it and create an alert for it if you want.
If you also have installed Logs Panel, Loki and Promtail on your server and setup your Loki Data Source, as described in the previous lessons, then you can then create Annotation queries and link the Logs Panel with your SSHD Graph.
Your Loki Datasourec Query Log Label can be {unit="ssh.service"}
And you could have a dashboard like this.
I also created an Annotation query which looked for the term "invalid", but also case insensitive. {unit="ssh.service"} (?i)invalid
Note you may want to set your SyslogFacility log level in your sshd_config
sudo nano /etc/ssh/sshd_config
Uncomment SyslogFacility
...
# Logging
SyslogFacility AUTH
...
sudo service sshd restart