Rolling Restart of Elasticsearch

Elasticsearch immediately begins regenerating metadata (reallocating those shards) when an ES node becomes unresponsive. In controlled situations such as a rolling restart, steps can be taken to minimize the impact.

A rolling restart of the ES cluster may be needed to upgrade the Elasticsearch version or to perform maintenance on the server itself, such as an OS update or hardware change. This process keeps the cluster operational while taking nodes offline one at a time. Because Elasticsearch prefers to keep data fully replicated and evenly balanced, it must be made to pause rebalancing until the rolling restart completes.

This pausing is performed through ES settings changes, as recommended by Elasticsearch in the bolded links below. The essential process is:

Change the ES settings to start maintenance mode.
Complete the maintenance work (such as upgrading ES) and rolling restart.
Restore the ES settings to stop maintenance mode.

Follow the steps in the Elasticsearch documentation, specified below:

ONLY for Elasticsearch Clusters Prior to Version 7

Set the discovery.zen.minimum_master_nodes value in the /etc/elasticsearch/elasticsearch.yml config file to be: (number of master-eligible-nodes/2, rounded down) + 1 when the elasticsearch cluster is first configured.
Verify the number is smaller than the number of currently available nodes. Run the following command now, to set that number to be <current_min_master_nodes>: (current number of master-eligible-nodes/2, rounded down) + 1 if not.

curl -s -XPUT 'http://ES_NODE:9200/_cluster/settings' \
	--data-binary '{"transient": {"discovery.zen.minimum_master_nodes" : "<current_min_master_nodes>"}}'

After maintenance is complete, restore discovery.zen.minimum_master_nodes to its original value, or adjust it based on the current number of expected available nodes after completing the rolling restart.

curl -s -XPUT 'http://ES_NODE:9200/_cluster/settings' \
	--data-binary '{"transient": {"discovery.zen.minimum_master_nodes" : "<original_min_master_nodes>"}}'

Follow the Elasticsearch rolling restart procedure for the version:
- ES Rolling Upgrades (7.17): Follow steps 1 and 2 (disable shard allocation and synced flush).
  Disable shard allocation with a curl like this:
  curl -X PUT "ES_NODE:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d' { "persistent": { "cluster.routing.allocation.enable": "primaries" } } '
  Perform a Synced flush with a curl like this (optional):
  curl -X POST "ES_NODE:9200/_flush/synced?pretty"
Stop the Elasticsearch node if you are shutting down the server for maintenance.
systemctl stop elasticsearch
Or, restart the service if you made configuration changes.
systemctl restart elasticsearch
After the server and service have started, you can re-enable shard allocation.
curl -X PUT "ES_NODE:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d' { "persistent": { "cluster.routing.allocation.enable": null } } '
Monitor the cluster health and wait for the status column to switch to green. Once the node is green, all primary and replica shards have been allocated.
curl -X GET "ES_NODE:9200/_cat/health?v=true&pretty"
Monitor the recovery status if it is in yellow or red status.
curl -X GET "ES_NODE:9200/_cat/recovery?pretty"

Swarm Documentation

Rolling Restart of Elasticsearch

ONLY for Elasticsearch Clusters Prior to Version 7

Related content