Rolling Restart of Elasticsearch

Elasticsearch immediately begins regenerating metadata (reallocating those shards) when an ES node becomes unresponsive. In controlled situations such as a rolling restart, steps can be taken to minimize the impact.

A rolling restart of the ES cluster may be needed to upgrade the Elasticsearch version or to perform maintenance on the server itself such as an OS update or hardware change. This process keeps the cluster operational while taking nodes offline one at a time. Because Elasticsearch prefers to keep data fully replicated and evenly balanced, it must be made to pause rebalancing until the rolling restart completes.

This pausing is performed through ES settings changes, as recommended by Elasticsearch in the bolded links below. The essential process is this:

Start maintenance mode by changing the ES settings.
Complete the maintenance work (such as upgrading ES) and rolling restart.
Stop maintenance mode by restoring the ES settings.

Follow the steps in the Elasticsearch documentation, with this important addition for versions before Elasticsearch 7:

Set the discovery.zen.minimum_master_nodes value in the /etc/elasticsearch/elasticsearch.yml config file to be: (number of master-eligible-nodes/2, rounded down) + 1 when the elasticsearch cluster is first configured.
Verify the number is smaller than the number of currently available nodes. Run the following command now to set that number to be <current_min_master_nodes>: (current number of master-eligible-nodes/2, rounded down) + 1 if not.
```
curl -s -XPUT 'http://ES_NODE:9200/_cluster/settings' \
	--data-binary '{"transient": {"discovery.zen.minimum_master_nodes" : "<current_min_master_nodes>"}}'
```
Follow the Elasticsearch rolling restart procedure for the version:
Reset discovery.zen.minimum_master_nodes to its original value, or adjust it based on the current number of expected available nodes after completing the rolling restart:
```
curl -s -XPUT 'http://ES_NODE:9200/_cluster/settings' \
	--data-binary '{"transient": {"discovery.zen.minimum_master_nodes" : "<original_min_master_nodes>"}}'
```