Removing Nodes from an ES Cluster

Refer to the following steps to remove a node from an Elasticsearch cluster:

1. Ensure sufficient space is available in the remaining ES cluster before removing any ES node(s). If unclear, open a support ticket including a support bundle.

2. If the IPs of the ES nodes you plan to remove are listed in your Swarm Search feed definition, remove them.

3.  If the IPs of the ES nodes you plan to remove are listed in your /etc/caringo/cloudgateway/gateway.cfg file (in the line starting with indexerHosts) on any of your Content Gateways, remove them. Restart the cloudgateway service after making the change.

4. If you are using Telemetry, remove these IPs (or hostnames) from your prometheus.yml file. Restart the Prometheus service.

5. Take note of the ES node as shown in your Swarm Search feed definition for the curl command below. Use the private side addresses if multiple IPs are assigned to the ES node (same subnet as your Swarm nodes).
   Run the following command to remove the desired ES node from the cluster (replace <IP> as appropriate):

   curl -XPUT "<IP address of any ES node>:9200/_cluster/settings" -H 'Content-Type: application/json' -d '{ "transient" :{ "cluster.routing.allocation.exclude._ip" : "<IP of ES node that you want to remove>" } }'

6. Monitor the cluster until the node is no longer participating and the shards have been rebalanced.

   curl -s -XGET "<IP address of any ES node>:9200/_cat/allocation?v"

   That removed ES node should eventually show 0 shards and no longer appear in the shard listing.

   curl -s -XGET "<IP address of any ES node>:9200/_cat/shards?v" | grep "^index"

7. Stop the Elasticsearch service on the removed node and decommission it.

8. Repeat the process for each node you want to remove (one at a time), ensuring sufficient cluster space available in the remaining ES cluster before proceeding.

9. Remove references to the removed node’s IP from the other ES nodes' /etc/elasticsearch/elasticsearch.yml files so that they are not referenced again later. Note cluster.initial_master_nodes is no longer needed after the cluster has formed.

   discovery.seed_hosts: cluster.initial_master_nodes:

   Adjust the following settings in /etc/elasticsearch/elasticsearch.yml file to reflect the changes.

   gateway.recover_after_data_nodes: gateway.expected_data_nodes:

   If unsure what values to set, contact DataCore Support.

10. After making the final configuration changes, perform a rolling restart of your ES cluster.​​​​​​​