Whenever any ES nodes go down, Elasticsearch immediately begins regenerating metadata (reallocating those shards). In controlled situations such as a rolling restart, you can take steps to minimize the impact.
A rolling restart of your the ES cluster (keeping it operational while taking nodes offline one at a time) may be needed to upgrade the Elasticsearch version or to perform maintenance on the server itself (such as an OS update or hardware change). Because Elasticsearch prefers to keep data fully replicated and evenly balanced, it must be made to pause rebalancing until the rolling restart completes.
...
You should follow the steps in the Elasticsearch documentation, with this important addition for versions before Elasticsearch 7:
When you first configured your configure the Elasticsearch cluster, you set the
discovery.zen.minimum_master_nodes
value in the/etc/elasticsearch/elasticsearch.yml
config file to be: (number of master-eligible-nodes/2, rounded down) + 1
Verify that that the number is smaller than the number of currently available nodes. If not, run the following command now to set that number to be <current_min_master_nodes>: (current number of master-eligible-nodes/2, rounded down) + 1.Code Block language bash curl -s -XPUT 'http://ES_NODE:9200/_cluster/settings' \ --data-binary '{"transient": {"discovery.zen.minimum_master_nodes" : "<current_min_master_nodes>"}}'
Follow the Elasticsearch rolling restart procedure for your the version:
After completing the rolling restart, reset
discovery.zen.minimum_master_nodes
to its original value, or adjust it based on the current number of expected available nodes:Code Block language bash curl -s -XPUT 'http://ES_NODE:9200/_cluster/settings' \ --data-binary '{"transient": {"discovery.zen.minimum_master_nodes" : "<original_min_master_nodes>"}}'