Rolling Restart of Elasticsearch

Whenever any ES nodes go down, Elasticsearch immediately begins regenerating metadata (reallocating those shards). In controlled situations such as a rolling restart, you can take steps to minimize the impact.

A rolling restart of your ES cluster (keeping it operational while taking nodes offline one at a time) may be needed to upgrade the Elasticsearch version or to perform maintenance on the server itself (such as an OS update or hardware change). Because Elasticsearch prefers to keep data fully replicated and evenly balanced, it must be made to pause rebalancing until the rolling restart completes.

This pausing is performed through ES settings changes, as recommended by Elasticsearch in the bolded links below. The essential process is this:

Start maintenance mode by changing the ES settings.
Complete the maintenance work (such as upgrading ES) and rolling restart.
Stop maintenance mode by restoring the ES settings.

You should follow the steps in the Elasticsearch documentation, with this important addition for versions before Elasticsearch 7:

When you first configured your Elasticsearch cluster, you set the discovery.zen.minimum_master_nodes value in the /etc/elasticsearch/elasticsearch.yml config file to be: (number of master-eligible-nodes/2, rounded down) + 1
Verify that that number is smaller than the number of currently available nodes. If not, run the following command now to set that number to be <current_min_master_nodes>: (current number of master-eligible-nodes/2, rounded down) + 1.
```
curl -s -XPUT 'http://ES_NODE:9200/_cluster/settings' \
	--data-binary '{"transient": {"discovery.zen.minimum_master_nodes" : "<current_min_master_nodes>"}}'
```
Follow the Elasticsearch rolling restart procedure for your version:
After completing the rolling restart, reset discovery.zen.minimum_master_nodes to its original value, or adjust it based on the current number of expected available nodes:
```
curl -s -XPUT 'http://ES_NODE:9200/_cluster/settings' \
	--data-binary '{"transient": {"discovery.zen.minimum_master_nodes" : "<original_min_master_nodes>"}}'
```