Elasticsearch Cluster Tutorial
Video Lecture
| Part | Video Link |
|---|---|
| 1 |  |
| 2 | |
Description
This is a small tutorial about creating a Cluster of Elasticsearch Servers with Metricbeat instances.
I will create 3 identical Ubuntu 20.04 servers in different regions of the world.
I will install Elasticsearch and Metricbeat on them and configure them with identical settings. Note that I am using Metricbeat as an example collector. You can install other beats, such as Filebeat and other collectors instead or in addition to Metricbeat. There are many possibilities.
My servers will from Digital Ocean.
I will select the basic droplets being $10 a month - Ubuntu 20.04, 2GB Ram, 1 CPU, 50GB SSD servers and start them in New York, Amsterdam and Singapore.
I will give them hostnames of ES1, ES2 and ES3.
They all have unique IP addresses which I will need to use in the Elasticsearch and Metricbeat configurations.
I will also name the nodes in the cluster, as node-1, node-2 and node-3.
| Hostname | Node Name | IP Address |
|---|---|---|
| ES1 | node-1 | 203.0.113.1 |
| ES2 | node-2 | 203.0.113.2 |
| ES3 | node-3 | 203.0.113.3 |
Note
The ip addresses used in the above example table are for demonstration only. Replace with the IPs or domain names for each of your Elasticsearch server addresses.
Install Elasticsearch
SSH onto all 3 servers and enter the following commands.
Download and install the Elasticsearch public signing key.
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
Install dependencies
sudo apt-get install apt-transport-https
Save the repository definition
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-7.x.list
Update and install the Elasticsearch package
sudo apt-get update && sudo apt-get install elasticsearch
Edit the Elasticsearch configuration.
sudo nano /etc/elasticsearch/elasticsearch.yml
Modify properties in each elasticsearch.yml by adding your node names and IP addresses.
ES1
cluster.name: mycluster
node.name: node-1
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["203.0.113.1", "203.0.113.2", "203.0.113.3"]
cluster.initial_master_nodes: ["203.0.113.1"]
ES2
cluster.name: mycluster
node.name: node-2
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["203.0.113.1", "203.0.113.2", "203.0.113.3"]
cluster.initial_master_nodes: ["203.0.113.1"]
ES3
cluster.name: mycluster
node.name: node-3
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["203.0.113.1", "203.0.113.2", "203.0.113.3"]
cluster.initial_master_nodes: ["203.0.113.1"]
Note that I named the cluster mycluster. You can name it anything you want containing the letters a-z,- or.
Also, in the above settings, I have chosen my node-1 to be the initial master node. This is only important for when starting the servers for the first time. I will start node-1 first and confirm it has started before starting node-2 and node-3. This is to ensure that all nodes register using the same cluster UUID. After the cluster has started, and all nodes are connected, any of the nodes can be chosen as master node if the current master node in use goes offline for any period of time. In poor network conditions, your master node may change regularly, and all the other nodes will re synchronise with the new agreed master.
Start Elasticsearch Master Node
Start Elasticsearch on ES1 first, wait and confirm its status as active
sudo service elasticsearch start
sudo service elasticsearch status
Check its default response and cluster health.
curl -XGET 'http://localhost:9200/?pretty'
curl -XGET 'http://localhost:9200/_cluster/health?pretty'
There should be no errors.
Take note of the cluster_uuid of the master mode.
curl -XGET 'http://localhost:9200/_cluster/state/master_node?pretty'
Start Elasticsearch Data Nodes
Start the other nodes and confirm statuses are active
sudo service elasticsearch start
sudo service elasticsearch status
Check Health
Now on any of the nodes (master or data), check the cluster health.
curl -XGET 'http://localhost:9200/_cluster/health?pretty'
It should show that number_of_nodes is > 1 and if you have 3 nodes in total, it should say number_of_nodes : 3
If not, then the nodes have probably not detected the master and created there own cluster UUID.
On each other node,
curl -XGET 'http://localhost:9200/_cluster/state/master_node?pretty'
and check if the cluster_uuid matches the cluster_uuid on the master node that you started first. node-1 in my case.
If the cluster_uuid doesn't match, then delete the nodes folder in the data node server,
rm -rf /var/lib/elasticsearch/nodes
and restart
sudo service elasticsearch restart
Check again the cluster health for the correct value
curl -XGET 'http://localhost:9200/_cluster/health?pretty'
In the end, when all nodes are running, they should all agree on the same cluster_uuid when running and all have chosen the same master node.
curl -XGET 'http://localhost:9200/_cluster/state/master_node?pretty'
To see a list of node UUIDs that are active in the cluster,
curl -XGET 'http://localhost:9200/_cluster/state/nodes?pretty'
IP Rules
If your Elasticsearch servers are all public on the internet, then you should create some ip rules to block access.
In my example the IP address of the ES nodes are 203.0.113.1, 203.0.113.2 and 203.0.113.3 so I will create IP rule that allow only them to communicate between each other.
Elasticsearch will use ports 9200 and 9300 by default.
On all 3 ES nodes execute,
#allow 9200 for certain ips and drop everything else
iptables -A INPUT -p tcp -s localhost --dport 9200 -j ACCEPT
iptables -A INPUT -p tcp -s 203.0.113.1 --dport 9200 -j ACCEPT
iptables -A INPUT -p tcp -s 203.0.113.2 --dport 9200 -j ACCEPT
iptables -A INPUT -p tcp -s 203.0.113.3 --dport 9200 -j ACCEPT
iptables -A INPUT -p tcp --dport 9200 -j DROP
#allow 9300 for certain ips and drop everything else
iptables -A INPUT -p tcp -s localhost --dport 9300 -j ACCEPT
iptables -A INPUT -p tcp -s 203.0.113.1 --dport 9300 -j ACCEPT
iptables -A INPUT -p tcp -s 203.0.113.2 --dport 9300 -j ACCEPT
iptables -A INPUT -p tcp -s 203.0.113.3 --dport 9300 -j ACCEPT
iptables -A INPUT -p tcp --dport 9300 -j DROP
#view rules
iptables -L
Note
Replace my example IP address above with your real IP addresses or domain names.
Install Metricbeat
Now that the cluster is confirmed running, its time to start ingesting data into it. I will use Metricbeat since it is a very popular solution and quick to setup.
On each of the master and data nodes, install the Metricbeat service.
curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.10.0-amd64.deb
sudo dpkg -i metricbeat-7.10.0-amd64.deb
Edit the configurations to point to all of the elastic search nodes.
sudo nano /etc/metricbeat/metricbeat.yml
Confirm that the system module is enabled
cd /etc/metricbeat
metricbeat modules list
Start and test status
sudo service metricbeat start
sudo service metricbeat status
Check for indices
curl http://localhost:9200/_cat/indices
Check for cluster health
curl -XGET 'http://localhost:9200/_cluster/health?pretty'
Check who is the master node
curl -XGET 'http://localhost:9200/_cluster/state/master_node?pretty'
Check the ids of each node
curl -XGET 'http://localhost:9200/_cluster/state/nodes?pretty'
Add An Elasticsearch Datasource in Grafana
| Key | Value | Notes |
|---|---|---|
| Name | Elasticsearch |
Or whatever name you want to use |
| URL | http://203.0.113.1:9200 |
IP address or Domain name of your ES Server |
| Index name | metricbeat-7.10.* |
check the correct index name using the curl http://localhost:9200/_cat/indices from the ES server |
| Version | 7.0 | Elasticsearch version 7.10 was used in this tutorial |
Save and Test
Do you have problems connecting?
It is probably ip/firewall rules or the particular ES server is not running.
Add a new rule to each ES server to allow your Grafana server to access port 9200.
Get IP rule line numbers
iptables -L --line-numbers
Insert a rule for your Grafana server at 5 for example. Your Grafana IP address or domain name will be different than mine.
iptables -I INPUT 5 -p tcp -s 203.0.113.123 --dport 9200 -j ACCEPT