Whenever any ES nodes go down, Elasticsearch immediately begins regenerating metadata (reallocating those shards) when an ES node becomes unresponsive. In controlled situations such as a rolling restart, you steps can take steps be taken to minimize the impact.
A rolling restart of your the ES cluster (keeping it operational while taking nodes offline one at a time) may be needed to upgrade the Elasticsearch version or to perform maintenance on the server itself ( such as an OS update or hardware change). This process keeps the cluster operational while taking nodes offline one at a time. Because Elasticsearch wants prefers to keep data fully replicated and evenly balanced, it must be made to pause rebalancing until the rolling restart is donecompletes.
This pausing is done performed through ES settings changes, as recommended by Elasticsearch in the bolded links below. The essential process is this:
Start maintenance mode by changing the ES settings.
Complete the maintenance work (such as upgrading ES) and rolling restart.
Stop maintenance mode by restoring the ES settings.
You should follow Follow the steps in the Elasticsearch documentation, with this important addition for versions before Elasticsearch 7:
...
Set the
discovery.zen.minimum_master_nodes
value in the/etc/elasticsearch/elasticsearch.yml
config file to be: (number of master-eligible-nodes/2, rounded down) + 1 when the elasticsearch cluster is first configured.
Verify
...
the number is smaller than the number of currently available nodes.
...
Run the following command now to set that number to be <current_min_master_nodes>: (current number of master-eligible-nodes/2, rounded down) + 1 if not.
Code Block language bash curl -s -XPUT 'http://ES_NODE:9200/_cluster/settings' \ --data-binary '{"transient": {"discovery.zen.minimum_master_nodes" : "<current_min_master_nodes>"}}'
Follow the Elasticsearch rolling restart procedure for
...
the version:
...
...
This step is only needed if using an elasticsearch version before Elasticsearch 7. Reset
discovery.zen.minimum_master_nodes
to its original value, or adjust it based on the current number of expected available nodes after completing the rolling restart:Code Block language bash curl -s -XPUT 'http://ES_NODE:9200/_cluster/settings' \ --data-binary '{"transient": {"discovery.zen.minimum_master_nodes" : "<original_min_master_nodes>"}}'