Elasticsearch Cluster Tutorial
This is a small tutorial about creating a Cluster of Elasticsearch Servers with Metricbeat instances.
I will create 3 identical Ubuntu 20.04 servers in different regions of the world.
I will install Elasticsearch and Metricbeat on them and configure them with identical settings. Note that I am using Metricbeat as an example collector. You can install other beats, such as Filebeat and other collectors instead or in addition to Metricbeat. There are many possibilities.
My servers will from Digital Ocean.
I will select the basic droplets being $10 a month - Ubuntu 20.04, 2GB Ram, 1 CPU, 50GB SSD servers and start them in New York, Amsterdam and Singapore.
I will give them hostnames of ES1, ES2 and ES3.
They all have unique IP addresses which I will need to use in the Elasticsearch and Metricbeat configurations.
I will also name the nodes in the cluster, as
|Hostname||Node Name||IP Address|
The ip addresses used in the above example table are for demonstration only. Replace with the IPs or domain names for each of your Elasticsearch server addresses.
SSH onto all 3 servers and enter the following commands.
Download and install the Elasticsearch public signing key.
Save the repository definition
Update and install the Elasticsearch package
Edit the Elasticsearch configuration.
Modify properties in each
elasticsearch.yml by adding your node names and IP addresses.
1 2 3 4 5 6
1 2 3 4 5 6
1 2 3 4 5 6
Note that I named the cluster
mycluster. You can name it anything you want containing the letters
Also, in the above settings, I have chosen my
node-1 to be the initial master node. This is only important for when starting the servers for the first time. I will start node-1 first and confirm it has started before starting
node-3. This is to ensure that all nodes register using the same cluster UUID. After the cluster has started, and all nodes are connected, any of the nodes can be chosen as master node if the current master node in use goes offline for any period of time. In poor network conditions, your master node may change regularly, and all the other nodes will re synchronise with the new agreed master.
Start Elasticsearch Master Node
Start Elasticsearch on ES1 first, wait and confirm its status as active
Check its default response and cluster health.
There should be no errors.
Take note of the
cluster_uuid of the master mode.
Start Elasticsearch Data Nodes
Start the other nodes and confirm statuses are active
Now on any of the nodes (master or data), check the cluster health.
It should show that
number_of_nodes is > 1 and if you have 3 nodes in total, it should say
number_of_nodes : 3
If not, then the nodes have probably not detected the master and created there own cluster UUID.
On each other node,
and check if the
cluster_uuid matches the
cluster_uuid on the master node that you started first.
node-1 in my case.
cluster_uuid doesn't match, then delete the nodes folder in the data node server,
Check again the cluster health for the correct value
In the end, when all nodes are running, they should all agree on the same
cluster_uuid when running and all have chosen the same master node.
To see a list of node UUIDs that are active in the cluster,
If your Elasticsearch servers are all public on the internet, then you should create some ip rules to block access.
In my example the IP address of the ES nodes are
203.0.113.3 so I will create IP rule that allow only them to communicate between each other.
Elasticsearch will use ports
9300 by default.
On all 3 ES nodes execute,
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Replace my example IP address above with your real IP addresses or domain names.
Now that the cluster is confirmed running, its time to start ingesting data into it. I will use Metricbeat since it is a very popular solution and quick to setup.
On each of the master and data nodes, install the Metricbeat service.
Edit the configurations to point to all of the elastic search nodes.
Confirm that the system module is enabled
Start and test status
Check for indices
Check for cluster health
Check who is the master node
Check the ids of each node
Add An Elasticsearch Datasource in Grafana
||Or whatever name you want to use|
||IP address or Domain name of your ES Server|
||check the correct index name using the
|Version||7.0||Elasticsearch version 7.10 was used in this tutorial|
Save and Test
Do you have problems connecting?
It is probably ip/firewall rules or the particular ES server is not running.
Add a new rule to each ES server to allow your Grafana server to access port 9200.
Get IP rule line numbers
Insert a rule for your Grafana server at 5 for example. Your Grafana IP address or domain name will be different than mine.