Creating Custom Low Level Discovery Rules - Part 1

Video Lecture

Description

Using everything we've seen so far, we can now move onto something much more advanced, and that is to create a custom Low Level Discovery Rule from the ground up.

The concept of a discovery rule, is to have a script that when run, will automatically discover groups of something on your host, or even network. For example, databases on a database server.

Some inbuilt Zabbix agent discovery rules are

• vfs.fs.discovery : Used by Mounted Filesystem Discovery
• net.if.discovery : Used by Network Interface Discovery
• vfs.dev.discovery : Used by Block Devices Discovery

These rules are created and assigned to your hosts when you add certain templates.

You can inspect the output of these rules on your hosts by running,

zabbix_agentd -t vfs.dev.discovery
zabbix_agentd -t net.if.discovery
zabbix_agentd -t vfs.fs.discovery

Each JSON output is dynamically created at runtime and contains a list of macro keys and values that will be specific to what was found from the perspective of the host where it was run. The macros can be used by the discovery rules Item, Trigger, Graph and Host Prototypes.

E.g., vfs.fs.discovery

[
  {
    "{#DEVNAME}": "loop8",
    "{#DEVTYPE}": "disk"
  },
  {
    "{#DEVNAME}": "vda15",
    "{#DEVTYPE}": "partition"
  },
  {
    "{#DEVNAME}": "vda14",
    "{#DEVTYPE}": "partition"
  },
  {
    "{#DEVNAME}": "vda1",
    "{#DEVTYPE}": "partition"
  },
  {
    "{#DEVNAME}": "vda",
    "{#DEVTYPE}": "disk"
  },
  ... etc, many more lines of json
]

In the response above, the keys starting with the # symbol are macros they are used by the prototypes in the specific discovery rules.

To start with this exercise, we will create a discovery rule to monitor chosen services running on a host (Ubuntu 20.04).

The discovery rule will be part of a template that can later be assigned to your hosts.

So to start, we need to create a User Parameter that will run a script, that will output a JSON formatted response containing macro keys and values. This will be used as the key for the discovery rule and will be the script that the discovery rule uses to discover the components that you want discovered.

I have written a Python script that will discover services on my Ubuntu 20.04 and transform the output into a JSON format compatible with what the discovery rule requires.

The Python script will internally run this command to gain its raw data about the services on the host.

systemctl list-unit-files

The script then transforms the response into JSON containing keys and values that will be used as macros while the discovery process sets up the items, triggers, graphs, etc. for the host.

The default response from any command that you run on your host, will need to be transformed into something that Zabbix can read before it can be used as an LLD rule. You are able to use pre-processing rules to transform your incoming data, but in this example, the script with transform the data into JSON before it reaches Zabbix.

I will use Python to transform the data into JSON, but you can use any scripting technique you like that can output a string that looks like a JSON response. A common approach instead of using Python, may be a .sh/.bat/.ps script that runs commands, and reformats using grep, awk, echo and a number of other commands native to the OS.

Now,

Create a folder on your host (if not already exists)

mkdir /home/zabbix

CD into the folder

cd /home/zabbix

And create a new Python file

sudo nano service_discovery.py

Paste this Python code into it and save (Ctrl X)

# Copyright 2020,2021,2022 Sean Bradley https://sbcode.net/zabbix/
import sys
import os
import json

SERVICES = os.popen('systemctl list-unit-files').read()

ILLEGAL_CHARS = ["\\", "'", "\"", "`", "*", "?", "[", "]", "{", "}", "~", "$", "!", "&", ";", "(", ")", "<", ">", "|", "#", "@", "0x0a"]

DISCOVERY_LIST = []

LINES = SERVICES.splitlines()
for l in LINES:
    service_unit = l.split(".service")
    if len(service_unit) == 2:
        if not [ele for ele in ILLEGAL_CHARS if(ele in service_unit[0])]:
            status = service_unit[1].split()
            if len(status) == 1:
                if (status[0] == "enabled" or status[0] == "generated"):
                    DISCOVERY_LIST.append({"{#SERVICE}": service_unit[0]})
            else:
                if (status[0] == "enabled" or status[0] == "generated") and status[1] == "enabled":
                    DISCOVERY_LIST.append({"{#SERVICE}": service_unit[0]})

JSON = json.dumps(DISCOVERY_LIST)

print(JSON)

Note

Special characters "\, ', ", `, *, ?, [, ], {, }, ~, $, !, &, ;, (, ), <, >, |, #, @, 0x0a" are not allowed as parameters in the discovery JSON, so my script excludes any service names including any of those characters. E.g., 'getty@' or 'autovt@'

You can test it using Python,

python3 /home/zabbix/service_discovery.py

Next, you can the test it works from the Zabbix users perspective by running,

sudo -H -u zabbix bash -c 'python3 /home/zabbix/service_discovery.py'

Now to create a user defined parameter in the Zabbix agent configuration.

sudo nano /etc/zabbix/zabbix_agentd.conf

Scroll down to the User-Defined Parameters section and add these new user parameters,

UserParameter=service.discovery,python3 /home/zabbix/service_discovery.py
UserParameter=service.isactive[*],systemctl is-active --quiet '$1' && echo 1 || echo 0
UserParameter=service.activatedtime[*],systemctl show '$1' --property=ActiveEnterTimestampMonotonic | cut -d= -f2

Save (Ctrl-X) and exit.

Test the service.discovery user parameter using

zabbix_agentd -t service.discovery

You can also test that using the zabbix user works,

sudo -H -u zabbix bash -c 'zabbix_agentd -t service.discovery'

Restart Zabbix agent and check status

sudo service zabbix-agent restart
sudo service zabbix-agent status

The response should resemble this below, but containing the various available services depending on your host.

[
  {
    "{#SERVICE}": "accounts-daemon"
  },
  {
    "{#SERVICE}": "apparmor"
  },
  {
    "{#SERVICE}": "cron"
  },
  {
    "{#SERVICE}": "iptables"
  },
  {
    "{#SERVICE}": "nginx"
  },
  {
    "{#SERVICE}": "snmpd"
  },
  {
    "{#SERVICE}": "ssh"
  },
  {
    "{#SERVICE}": "sshd"
  },
  {
    "{#SERVICE}": "syslog"
  },
  {
    "{#SERVICE}": "zabbix-agent"
  }
  ... etc, many more lines of json
]

Ensure all of this works before continuing to part 2.

Useful Links

Zabbix Low Level Discovery

JSON Formatter

Comments