Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

This is the process for in-place upgrades of Elasticsearch (ES), meaning ones that can use the existing Search feed and index data.

Required
Info
title
Warning

We expect Internet access by default for the upgrade. In case it is not available, ensure that the

necessary

Elasticsearch RPM is in the current directory.


Info
titleRequired
  • This upgrade is for Elasticsearch 6.8.6 and higher only, with a Search feed created on Swarm 11. 
  • For migrating from Elasticsearch 2.3.3 or 5.6.12, see Migrating from Older Elasticsearch.

Upgrading Elasticsearch by script

On each node in your Elasticsearch cluster, follow this process and run the files from your Swarm download bundle:

  1. Query the running Elasticsearch cluster, before upgrading, for the list of nodes.

    Code Block
    languagebash
    curl -i http://ELASTICSEARCH:9200/_cat/nodes

    Note which node is starred. That is the Elasticsearch master node which you should upgrade last to avoid problems electing a new one.

  2. Backup your existing elasticsearch.yml, so that you have a record of any customizations you may have made.
  3. If you have customized path.data, create a symbolic (soft) link: symlink /var/lib/elasticsearch. If you cannot, perform the upgrade manually, as described below. 
  4. When upgrading Elasticsearch 6, start by installing the latest Swarm Search, which is the caringo-elasticsearch-search RPM.

    Code Block
    languagebash
    yum install caringo-elasticsearch-search-VERSION.noarch.rpm


    Info
    titleTip

    If you inadvertently install the Elasticsearch 7 RPM, it will fail with the error: "ES_PATH_CONF must be set to the configuration path chown: cannot access '/etc/elasticsearch/elasticsearch.keystore': No such file or directory". Simply install the caringo-elastisearch-search-7.0.0 RPM now and proceed.


  5. Run the script that installs and configures the upgrade.
    The script detects that Elasticsearch 6 is installed and that discovery.zen.unicast.hosts is configured, so it runs as with --upgrade instead of configuring a new cluster.

    Code Block
    languagebash
    /usr/share/caringo-elasticsearch-search/bin/configure_elasticsearch_with_swarm_search.py


  6. Compare your backup file to the newly created elasticsearch.yml and add back any customizations you need, such as network.host (which defaults to _site_, meaning all interfaces).
  7. Verify all nodes are accounted for, that all shards are assigned, and that the status is green.

    Code Block
    languagebash
    curl -i 'http://ELASTICSEARCH:9200/_cat/health?v'


If Elasticsearch 7 is already installed, the script updates the configuration files and restarts the service for you.

Info
titleTroubleshooting

If the Elasticsearch service fails and journalctl -u elasticsearch shows that access is denied (BootstrapException/AccessDeniedException), change its permissions: 

chown elasticsearch /etc/elasticsearch


Info
titleImportant

If the cluster loses master during the upgrade process and does not recover, that’s fine, just hit Ctrl-C once, whenever the upgrade script is stuck in retrying status checks and proceed to the next node after the script finishes. Review /etc/elasticsearch/elasticsearch.yml and tail/var/log/elasticsearch/<cluster-name>.log for configuration errors.

Once all nodes have started on Elasticsearch 7.5.2, they should be able to elect the master and recover. Health status should go yellow then eventually green. You will have to re-enable shard allocation, otherwise /_cat/health?v will stop at 50% with health status yellow.

Upgrading Elasticsearch manually

If you should need to upgrade manually, these are the steps that the script automates:

  1. It fixes /etc/sysconfig/elasticsearch to be the correct ES6 version (it's also the same as ES7).
  2. It increases the systemd timeout in /etc/systemd/system/elasticsearch.service.d/override.conf (see github.com/elastic/elasticsearch/issues/60140)
  3. After refreshing the config files for Elasticsearch 6, it prompts to continue with the yum upgrade to 7.5.2.
  4. It disables shard allocation and does a POST synced-flush for safer rolling upgrades. 
    Important: Disabling shard allocation or sync-flush can fail to contact the node, but do not proceed to upgrade the next node until the cluster health is green again.
  5. It uninstalls the Prometheus Exporter plugin if it exists.
  6. It shells out to yum to install the Elasticsearch 7 RPM in the current directory or from artifacts.elastic.co, if unavailable.
  7. It updates elasticsearch.yml for version 7 compatibility, including discovery.initial_master_nodes instead of discovery.zen.unicast.hosts, and jvm.options.
  8. It starts the upgraded Elasticsearch 7 and waits for it to be ready.
  9. If the cluster health is green or yellow, it re-enables shard allocation and prompts you to repeat these two steps on the next node.
Info
titleImportant
If the cluster loses master during the upgrade process and does not recover, that’s fine, just hit Ctrl-C once, whenever
  1. the
upgrade script is stuck in retrying status checks and proceed to the
  1. next node
after the script finishes. Review /etc/elasticsearch/elasticsearch.yml and tail/var/log/elasticsearch/<cluster-name>.log for configuration errors.Once all nodes have started on Elasticsearch 7.5.2, they should be able to elect the master and recover. Health status should go yellow then eventually green. You will have to re-enable shard allocation, otherwise /_cat/health?v will stop at 50% with health status yellow
  1. .