Skip to content


 Zabbix
 Grafana
 Prometheus
 Threejs and TypeScript
 SocketIO and TypeScript
 Blender Topological Earth
 Sweet Home 3D
 Design Patterns Python
 Design Patterns TypeScript
   
 Course Coupon Codes
Three.js and TypeScript
Kindle Edition
$9.99 $14.99 Paperback 
$29.99 $34.99




Design Patterns in TypeScript
Kindle Edition
$9.99 $14.99 Paperback
$19.99 $24.99




Design Patterns in Python
Kindle Edition
$9.99 $14.99 Paperback
$19.99 $24.99




Creating Custom Low Level Discovery Rules - Part 1

Video Lecture

Creating Custom Low Level Discovery Rules - Part 1 Creating Custom Low Level Discovery Rules - Part 1 Creating Custom Low Level Discovery Rules - Part 1

Description

Using everything we've seen so far, we can now move onto something much more advanced, and that is to create a custom Low Level Discovery Rule from the ground up.

The concept of a discovery rule, is to have a script that when run, will automatically discover groups of something on your host, or even network. For example, databases on a database server.

Some Inbuilt Native Zabbix Discovery rules are

  • vfs.fs.discovery : Used by Mounted Filesystem Discovery
  • net.if.discovery : Used by Network Interface Discovery
  • vfs.dev.discovery : Used by Block Devices Discovery

These rules are created and assigned to your hosts when you add certain templates.

You can inspect the output of these rules on your hosts by running,

zabbix_agentd -t vfs.dev.discovery
zabbix_agentd -t net.if.discovery
zabbix_agentd -t vfs.fs.discovery

Each JSON output is dynamically created at runtime and contains a list of macro keys and values that will be specific to what was found from the perspective of the host where it was run. The macros can be used by the discovery rules Item, Trigger, Graph and Host Prototypes.

eg, vfs.fs.discovery

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[
  {
    "{#DEVNAME}": "loop8",
    "{#DEVTYPE}": "disk"
  },
  {
    "{#DEVNAME}": "vda15",
    "{#DEVTYPE}": "partition"
  },
  {
    "{#DEVNAME}": "vda14",
    "{#DEVTYPE}": "partition"
  },
  {
    "{#DEVNAME}": "vda1",
    "{#DEVTYPE}": "partition"
  },
  {
    "{#DEVNAME}": "vda",
    "{#DEVTYPE}": "disk"
  },
  ... etc, many more lines of json
]

In the response above, the keys starting with the # symbol are macros the are used by the prototypes in the specific discovery rules.

To start with this exercise, we will create a discovery rule to monitor chosen services running on a host (Ubuntu 20.04).

The discovery rule will be part of a template that can later be assigned to your hosts.

So to start, we need to create a UserParameter that will run a script, that will output a JSON formatted response containing macro keys and values. This will be used as the key for the discovery rule and will be the script that the discovery rule uses to discover the components that you want discovered.

I have written a Python script that will discover services on my Ubuntu 20.04 and transform the output into a JSON format compatible with what the discovery rule requires.

The Python script will internally run this command to gain it's raw data about the services on the host.

systemctl list-unit-files

The script then transforms the response into JSON containing keys and values that will be used as macros while the discovery process sets up the items, triggers, graphs, etc for the host.

The default response from any command that you run on your host, will need to be transformed into something that Zabbix can read before it can be used as a LLD rule. You are able to use pre processing rules to transform your incoming data, but in this example, the script with transform the data into JSON before it reaches Zabbix.

I will use Python to transform the data into JSON, but you can use any scripting technique you like that can output a string that looks like a JSON response. A common approach instead of using Python, may be a .sh/.bat/.ps script that runs commands, and reformats using grep, awk, echo and a number of other commands native to the OS.

Now,

Create a folder on your host (if not already exists)

mkdir /home/zabbix

CD into the folder

cd /home/zabbix

And create a new Python file

sudo nano service_discovery.py

Paste this Python code into it and save (Ctrl X)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
#!/usr/bin/python
# Copyright 2020 Sean Bradley https://sbcode.net/zabbix/
import sys
import os
import json

SERVICES = os.popen('systemctl list-unit-files').read()

ILLEGAL_CHARS = ["\\", "'", "\"", "`", "*", "?", "[", "]", "{", "}", "~", "$", "!", "&", ";", "(", ")", "<", ">", "|", "#", "@", "0x0a"]

DISCOVERY_LIST = {"data": []}

LINES = SERVICES.splitlines()
for l in LINES:
    service_unit = l.split(".service")
    if len(service_unit) == 2:
        if not [ele for ele in ILLEGAL_CHARS if(ele in service_unit[0])]:
            status = service_unit[1].split()
            if len(status) == 1:
                if (status[0] == "enabled" or status[0] == "generated"):
                    DISCOVERY_LIST["data"].append({"{#SERVICE}": service_unit[0]})
            else:
                if (status[0] == "enabled" or status[0] == "generated") and status[1] == "enabled":
                    DISCOVERY_LIST["data"].append({"{#SERVICE}": service_unit[0]})

JSON = json.dumps(DISCOVERY_LIST)

print(JSON)

Note

Special characters "\, ', ", `, *, ?, [, ], {, }, ~, $, !, &, ;, (, ), <, >, |, #, @, 0x0a" are not allowed as parameters in the discovery JSON so my script excludes any service names including any of those characters. eg, 'getty@' or 'autovt@'

You also need to give it execute permissions,

sudo chmod a+x service_discovery.py

You can the test it works from the Zabbix users perspective by running,

sudo -H -u zabbix bash -c '/home/zabbix/service_discovery.py'

Now to create a UserParameter in the Zabbix agent conf.

sudo nano /etc/zabbix/zabbix_agentd.conf

Scroll down to the UserParameters section and add these new UserParameters,

UserParameter=service.discovery,/home/zabbix/service_discovery.py
UserParameter=service.isactive[*],systemctl is-active --quiet '$1' && echo 1 || echo 0
UserParameter=service.activatedtime[*],systemctl show '$1' --property=ActiveEnterTimestampMonotonic | cut -d= -f2

Save (Ctrl-X) and exit.

Restart Zabbix Agent and check status

sudo service zabbix-agent restart
sudo service zabbix-agent status

Test the service.discovery UserParameter using

zabbix_agentd -t service.discovery

The response should resemble this below, but containing the various available services depending on your host.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
{
  "data": [
    {
      "{#SERVICE}": "accounts-daemon"
    },
    {
      "{#SERVICE}": "apparmor"
    },
    {
      "{#SERVICE}": "cron"
    },
    {
      "{#SERVICE}": "iptables"
    },
    {
      "{#SERVICE}": "nginx"
    },
    {
      "{#SERVICE}": "snmpd"
    },
    {
      "{#SERVICE}": "ssh"
    },
    {
      "{#SERVICE}": "sshd"
    },
    {
      "{#SERVICE}": "syslog"
    },
    {
      "{#SERVICE}": "zabbix-agent"
    }
    ... etc, many more lines of json
  ]
}

Ensure all of this works before continuing to part 2.

Zabbix Low Level Discovery

Online JSON Viewer