7533743542__opensearch-multinode-cluster-setup-guide

OpenSearch Multi‐Node Cluster Setup Guide

This guide outlines the step-by-step process for safely adding and configuring additional nodes to an existing OpenSearch cluster.

1. Take latest Opensearch Snapshot

• Log in to the OpenSearch Dashboard.

• Verify if snapshot repositories are configured.

• Check the timestamp of the latest snapshot to confirm recent backup.

• Manually take a snapshot before initiating the Multi-Node Setup activity to ensure we have latest data available.

---

2. Backup OpenSearch Templates

• Go to Dev Tools and run:

GET _cat/templates

• Check for any customer-created templates.

• Then run:

GET _template

• Copy the entire output and save it locally (Ctrl + A → Ctrl + C).

You can use this backup with a PUT request if templates need to be restored.

---

3. Launch New Nodes

• Create a new EC2 instance same size as of existing Diagnostic Node:

Type: t3a.large

Note: Depending upon customer's index size and retention period you can update the instance type later in another maintenance window.

AZ: Same as the existing diagnostic node.

• Repeat to launch a second instance.

Note: Provide proper naming conventions to new nodes such as [-oc-diagnostics-1] and [-oc-diagnostics-2].

4. Pre-Configuration Check

• Ensure both new nodes show connected status before proceeding.

---

5. Prepare Directory Structure

Run on both new nodes:

mkdir -p /data/es
chmod 777 /data/es

Also Set Kernel Parameters on all nodes :

echo 'vm.max_map_count=262144' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

Note : The vm.max_map_count setting controls how many memory-mapped areas a process can use, which OpenSearch needs for managing large indices. Setting it to 262144 ensures stability, and applying it via sysctl makes the change immediate and persistent.

6. Validate Hosts and Directories

• Check /etc/hosts :

cat /etc/hosts

• Verify presence of loopback IP (127.0.0.1 or ::1 ).

• Confirm /data/es exists.

---

7. Verify Cluster Name

• Run:

GET /

• Confirm that cluster.name is correct.

---

8. Update Elasticsearch Service Configuration

Modify the Elasticsearch service environment variables:

{
"
plugins.security
.disabled": "true",
"compatibility.override_main_response_version": "true",
"OPENSEARCH_JAVA_OPTS": "-Xmx4g -Xms4g",
"http.max_content_length": "200mb",
"
cluster.name
": "docker-cluster",
"discovery.seed_hosts": "<IP_1>,<IP_2>,<IP_3>",
"cluster.initial_cluster_manager_nodes": "<IP_1>,<IP_2>,<IP_3>"
}

Replace <IP_1>, <IP_2>, <IP_3> with actual node IPs.

discovery.seed_hosts can include all worker nodes; initial_cluster_manager_nodes should include only the 3 master-eligible nodes. Also, update the replica count of elasticsearch to 3.

Note: Validate the ElasticHost URLs configured in Filebeat and Kibana to ensure connectivity is not broken.

9. Confirm Node Connectivity

In Dev Tools , run:

GET /
GET _cat/nodes?v

• Ensure new node IPs are listed.

• If not, restart Elasticsearch pods.

---

10. Clear Data on New Nodes

Run only on the two new nodes :

rm -rvf /data/es/*

---

11. Restart Elasticsearch Pod (if needed)

If new IPs are still missing, restart the pod:

docker restart <elasticsearch# ## #
container
name>

---

12. Final Health Check

Verify cluster health:

GET _cat/health?v

---

✅ Done! Your OpenSearch multi-node cluster is now configured. Monitor node status and cluster logs to ensure everything remains healthy and stable.

Steps for ES-Exporter Multi-Node Setup

---

1. ES Exporter pods are deployed to push the metrics related to Elasticsearch

2. Since we have implemented the Multi-Node Opensearch, setting up the Mulit-Node ES-Exporter will help us gain the metrics from all the instances of Opensearch pods.

3. To enable Multi-Node ES-Exporter setup, update the simply update replica count to 3 in the ES-Exporter service.

4. This will spin-up 3 instances of ES-Exporter all running on 3 different instances along with Opensearch pods.