Paperback 
Install Second Prometheus Node Exporter
Video Lecture
Description
I will install a Prometheus Node Exporter on a different server and connect to it using the main Prometheus service.
On the other server install it,
sudo apt install prometheus-node-exporter
Check its status.
sudo service prometheus-node-exporter status
It has created a specific user called prometheus. We can inspect it.
id prometheus
ps -u prometheus
ss -ntlp | grep prometheus
It is now exposing the metrics endpoint on http://[your domain or ip]:9100
Now to create a scrape config on the Prometheus server that retrieves metrics from this new URL.
Go back onto your main Prometheus server and edit the existing scrape config for node and add the new metrics endpoint for the other server.
sudo nano /etc/prometheus/prometheus.yml
...
scrape_configs:
...
- job_name: node
# If prometheus-node-exporter is installed, grab stats about the local
# machine by default.
static_configs:
- targets: ['localhost:9100']
- targets: ['IP_ADDRESS_OR_DOMAIN_NAME_OF OTHER_SERVER:9100']
Firewall
If your server is on the public internet, and you haven't configured a dedicated firewall, it may be accessible via the address http://[your domain or IP]:9100/
You can restrict access to port 9100 using iptables.
iptables -A INPUT -p tcp -s <domain name or ip of prometheus server> --dport 9100 -j ACCEPT
iptables -A INPUT -p tcp --dport 9100 -j DROP
iptables -L
Nginx Reverse Proxy
If you have Nginx running on your other server already, with a domain name and an SSL certificate setup, then you can add a location to an existing website config.
For the following instructions, presume that I have already installed Nginx, with a custom domain and installed an SSL certificate as outlined in the sections below, but customized for this other server where I've just installed the Prometheus Node Exporter.
Add the location to your websites existing Nginx configuration
sudo nano /etc/nginx/sites-enabled/YOUR-EXISTING-CONFIGURATION.conf
...
location /metrics {
allow ###.###.###.###;
deny all;
proxy_pass http://localhost:9100/metrics;
}
...
Prometheus Course
If you want to try a more detailed course on Prometheus, then you can visit my Prometheus tutorials.
Useful Links
Grafana 9 and Ubuntu 22.04 Notes
There are no considerable differences to be aware of as shown and discussed in the video.