JMX Monitoring
Video Lecture
Description
JMX stands for Java Management Extensions. It is a framework for monitoring Java processes, and we can configure Zabbix to connect to a running JMX daemon thread.
To demonstrate, I will install a basic setup of Tomcat server on an Ubuntu 22.04 and enable it to be monitored using JMX.
On the server where you want to run Tomcat, create a separate unprivileged user to run it. I named my user tomcat.
sudo useradd -m -d /opt/tomcat -U -s /bin/false tomcat
Update the package manager cache.
sudo apt update
Install the JDK
sudo apt install default-jdk
Check that Java is working
java -version
The response will be similar to
openjdk version "11.0.16" 2022-07-19
OpenJDK Runtime Environment (build 11.0.16+8-post-Ubuntu-0ubuntu122.04)
OpenJDK 64-Bit Server VM (build 11.0.16+8-post-Ubuntu-0ubuntu122.04, mixed mode, sharing)
Now download the Tomcat core. I downloaded 10.0.23.
wget https://dlcdn.apache.org/tomcat/tomcat-10/v10.0.23/bin/apache-tomcat-10.0.23.tar.gz
Extract the downloaded file
sudo tar xzvf apache-tomcat-10.0.23.tar.gz -C /opt/tomcat --strip-components=1
Grant your tomcat user ownership of the folder where the tar was extracted
sudo chown -R tomcat:tomcat /opt/tomcat/
And make the files executable
sudo chmod -R u+x /opt/tomcat/bin
CD into it and check
cd /opt/tomcat/bin
ls -lh
Start the Tomcat web server.
./catalina.sh start
Visit it in your browser
http://[IP address of server running Tomcat]:8080
You should see the Tomcat homepage.
Now we need to modify the configuration to enable the inbuilt JMX daemon to return its monitoring data.
In the same folder at /opt/tomcat/bin, we create a new file named setenv.sh
sudo nano setenv.sh
Copy and paste the text,
CATALINA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9000 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=[IP address of server running Tomcat]"
Note the option -Djava.rmi.server.hostname=. This is needed for remote connections. If you are connecting to the JMX daemon locally only, then you can omit this parameter.
Also replace the text [IP address of server running Tomcat] with the IP address of your server where you just installed Tomcat.
I have put my Tomcat servers external IP address, since I will be monitoring it remotely from my Zabbix server.
Save by pressing CTRL-X then Y
Check setenv.sh now exists in the folder
ls -lh
Also make the setenv.sh file executable
chmod 750 setenv.sh
Check it again
ls -lh
Change the setenv.sh file owner to tomcat:tomcat
sudo chown -R tomcat:tomcat setenv.sh
And check again
ls -lh
Now we can restart Tomcat
./catalina.sh stop
./catalina.sh start
We should now see the CATALINA_OPTS in the response showing our JMX configuration.
We can check if port 9000 is open to connections.
ss -ntlp | grep 9000
Now to set up Zabbix.
SSH onto your Zabbix server and check if we can telnet to port 9000 on our tomcat servers external IP address.
telnet [IP address of server running Tomcat] 9000
Next to create a JMX host in Zabbix.
I am going to assign the Apache Tomcat by JMX template.
Create the host and name it what you want.
You will find the Apache Tomcat by JMX template in the Templates/Applications host group.
Add the host to any group.
Add the JMX interface.
Set the IP address that Zabbix server (or proxy if host is monitored by proxy) will use to find it.
Set the port to 9000.
We can do various tests to see if it's working. It will not be. There are several more things to do.
We need to install the Java gateway on our Zabbix server, or Zabbix proxy if the host will be monitored by proxy.
sudo apt install zabbix-java-gateway
The Zabbix Java Gateway has its own configuration file. We can leave the defaults. You can view it using,
sudo nano /etc/zabbix/zabbix_java_gateway.conf
Next to update Zabbix servers (or Zabbix proxy) configuration to point to the Java gateway.
sudo nano /etc/zabbix/zabbix_server.conf
Uncomment and change parameters,
JavaGateway=127.0.0.1
JavaGatewayPort=10052
StartJavaPollers=5
Restart the Zabbix server (or proxy if using the Zabbix proxy)
sudo service zabbix-server restart
Do some tests, and run execute now on the items and discovery rules. The JMX icon should light up green, and there will be data in the Monitoring → Latest Data page.
Troubleshooting
Your problems are likely to be firewall, user permissions, JMX daemon related or Zabbix server/proxy configuration. You need to recheck everything.
Java Gateway Configuration Parameter not Set or Empty
You need to install the Zabbix Java Gateway process and set the Zabbix server configuration to point to it. See instructions above.
jmxterm
You can also test independently of Zabbix if the JMX daemon is actually responding, via a shell program called jmxterm.
Furthermore, you can install this on the server running the JMX daemon to test it locally, or on a remote server, such as your Zabbix server or Zabbix proxy where you can also test it from.
Download jmxterm from its official repository at https://github.com/jiaqi/jmxterm
wget https://github.com/jiaqi/jmxterm/releases/download/v1.0.1/jmxterm-1.0.1-uber.jar
Start jmxterm using Java:
java -jar jmxterm-1.0.1-uber.jar
Open a local JMX connection
$>open localhost:9000
or open a remote JMX connection
$>open <IP Address of Server exposing JMX daemon port>:9000
It should respond that the connection is opened, otherwise it will display some kind of error.
Now run some tests
$>beans
$>get -b NIO:name=accepted,type=total *
$>get -b java.lang:type=Memory *
$>get -b NIO:name=accepted,type=total *
Useful Links
Official JMX Monitoring Documentation
Debian/Ubuntu Java Gateway Install Instructions
Configure JMX for Apache Tomcat