Elasticsearch Filebeat
Video Lecture
Description
I demonstrate how to setup a Filebeat service to read systemd logs.
Filebeat download instructions can be found at https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation-configuration.html#installation
I downloaded the debian package manager version.
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.16.1-amd64.deb
sudo dpkg -i filebeat-7.16.1-amd64.deb
cd /etc/filebeat
ls -lh
Enable a module for Filebeat to run.
Get a list.
filebeat modules list
I enable the system module
filebeat modules enable system
Now edit the filebeat configuration.
sudo nano /etc/filebeat/filebeat.yml
Update the address of your Elasticsearch server.
output.elasticsearch:
hosts: ["<IP Address of your Elasticsearch Server>:9200"]
Restart and check status.
sudo service filebeat start
sudo service filebeat status
You can also disable a module
filebeat modules disable system
If you enable/disable a module, then restart Filebeat.
sudo service filebeat restart
sudo service filebeat status
Firewall
The Elasticsearch service may or may not have a firewall blocking this new filebeat from sending to it. If you used IPtables from the last lesson, then you can add another IPtables rule to allow the IP address of this new filebeat service to send.
So on my Elasticsearch server, I get the iptables rules line numbers.
iptables -L --line-numbers
I insert the new rule for my new Filebeat services IP before the DROP rule.
iptables -I INPUT 2 -p tcp -s x.x.x.x --dport 9200 -j ACCEPT
Persist changes.
iptables-save > /etc/iptables/rules.v4
Now we can set up a new data source in Grafana, or modify the existing and test it using the explore tab.
If you didn't use IPtables, but your cloud providers firewall options to mange your firewall, then you need to allow this servers IP address, that you just installed Filebeat onto, to send to your Elasticsearch servers IP address on port 9200. I demonstrate setting my firewall rules in the video.
I can verify that my Filebeat can send to my Elasticsearch server by making curl requests from the server running Filebeat.
curl "http://<IP Address of your Elasticsearch Server>:9200"
and to get the name of the new index created by this new Filebeat service,
curl "http://<IP Address of your Elasticsearch Server>:9200/_cat/indices"