Versions Compared

Version Old Version 7 New Version Current
Changes made by

Kyle Miller (Deactivated)

Anjali Tyagi

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Table of Contents
minLevel1
maxLevel2
outlinefalse
typelist
printablefalse

This is the process for in-place upgrades of Elasticsearch (ES) , meaning ones that can use the upgrades, using an existing Search feed Feed and index data.

title
Info

Required

Upgrading Elasticsearch by

...

Script

On each node in

...

an Elasticsearch cluster, follow this process and run the files from

...

the Swarm download bundle:

...

  1. Before upgrading, query the

...

  1. Elasticsearch cluster

...

  1. for the list of nodes.

    Code Block
    languagebash
    curl -i http://ELASTICSEARCH:9200/_cat/nodes

...

  1. Example output:
     Code Block
    [root@elasticsearch Elasticsearch]# curl -i http://<ip-address>:9200/_cat/nodes
HTTP/1.1 200 OK
content-type: text/plain; charset=UTF-8
content-length: 300
<ip-node1> 14 99 1 0.02 0.14 0.15 dilm - <hostname-node1>
<ip-node2>  1 71 1 0.01 0.09 0.08 dilm - <hostname-node2>
<ip-node3> 72 99 0 0.08 0.06 0.08 dilm - <hostname-node3>
<ip-node4> 50 99 0 0.03 0.04 0.05 dilm * <hostname-node4>
<ip-node5> 47 99 1 0.08 0.14 0.22 dilm - <hostname-node5>
    The node with * is the Elasticsearch master node. It is recommended to upgrade the node last to avoid problems electing a new 
...
  1. master node.
  2. Backup 
...
  1. the existing configurations so a record exists of any customizations
...
  1. . Run ./techsupport-bundle-grab.sh from your Support tools directory to backup to upgrade your support tools before running the script. Instructions are here.
  2. Install the latest Swarm Search i.e. caringo-elasticsearch-search RPM.
     Code Block
    languagebash
    yum install caringo-elasticsearch-search-VERSION.noarch.rpm
 Info
...
Tip
...
The error: "ES_PATH_CONF must be set to the configuration path chown: cannot access '/etc/elasticsearch/elasticsearch.keystore': No such file or directory" 
...
is displayed if Elasticsearch 7 RPM was inadvertently installed. Install the caringo-elasticsearch-search
...
 RPM to proceed.
  1. Run the script 
...
  1. to install and 
...
  1. configure the upgrade.
...
  1. If the script detects that Elasticsearch 
...
  1.  is installed and 
...
  1.  configured, 
...
  1.  it runs 
...
  1.  with --upgrade instead of configuring a new cluster, automating the Elasticsearch rolling upgrade steps (https://www.elastic.co/guide/en/elasticsearch/reference/7.17/rolling-upgrades.html).
     Code Block
    languagebash
    /usr/share/caringo-elasticsearch-search/bin/configure_elasticsearch_with_swarm_search.py
 Info
Important
The upgrade requires Internet access to download the Elasticsearch rpm. Place the elasticsearch-*.rpm in the current directory if Internet access is unavailable.
 Note
Warning
...
The configure script requires node.name to be set in /etc/elasticsearch/elasticsearch.yml. This is not the case with a Swarm Cluster Installer (SCI) deployed VM; uncomment the setting and set it to the default value of ${HOSTNAME}.
  1. Compare the backup file to the newly created elasticsearch.yml
...
  1. , add 
...
  1.  any customizations 
...
  1. needed, and restart systemctl restart elasticsearch if the configuration was modified.
Note
thread_pool.write.queue_size: 1000 and prometheus.indices are no longer required.
  1. Verify that all nodes are accounted for, 
...
  1.  all shards are assigned, and 
...
  1.  the status is green.
     Code Block
    languagebash
    curl -i 'http://ELASTICSEARCH:9200/_cat/health?v'
...
The script updates the configuration files and restarts the service 
...
if the upgraded Elasticsearch is already installed.
 Info
...
Troubleshooting
...
Change permissions if the Elasticsearch service fails 
...
and journalctl -u elasticsearch shows 
...
 access is denied (BootstrapException/AccessDeniedException
...
 or ERROR: unable to create temporary keystore):
chown elasticsearch /etc/elasticsearch
 Info
...
Important
...
Type Ctrl-C once 
...
when the upgrade script is stuck in retrying status checks and proceed to the next node after the script finishes if the cluster loses master during the upgrade process and does not recover. Review /etc/elasticsearch/elasticsearch.yml and tail -F /var/log/elasticsearch/<cluster-name>.log for configuration errors.
...
The nodes elect the master and recover once all nodes have started on 
...
the upgraded Elasticsearch. Health status 
...
goes yellow, then eventually green. 
...
Re-enable shard allocation, otherwise /_cat/health?v
...
 stops at 50% with health status yellow.
Upgrading Elasticsearch 
...
Manually
The script automates the following steps for upgrading Elasticsearch manually:
  1. It fixes /etc/sysconfig/elasticsearch to 
...
  1.  the correct ES6 version (
...
  1. the same as ES7+).
  2. It increases the systemd timeout in /etc/systemd/system/elasticsearch.service.d/override.conf (see github.com/elastic/elasticsearch/issues/60140)
...
  1. A prompt to continue with the yum upgrade to the latest ES version appears after refreshing the config files for Elasticsearch 6
...
  1. .
...
  2. It disables shard allocation and 
...
  1. performs a POST synced-flush for safer rolling upgrades. 
 Info
...
Important
Disabling shard allocation or sync-flush can fail to contact the node 
...
 but 
...
does not proceed to upgrade the next node until the cluster health is green again.
  1. It uninstalls the existing Prometheus Exporter plugin
...
  1. .
  2. It shells out to yum to install the Elasticsearch 7 RPM in the current directory or from artifacts.elastic.co, if unavailable
...
  1. .
  2. It updates elasticsearch.yml for 
...
  1. Elasticsearch 7+ compatibility, including 
...
  1.  cluster.initial_master_nodes instead of discovery.zen.unicast.hosts
...
  1.  and jvm.options.
  2. It starts the upgraded Elasticsearch 
...
  1.  and waits for it to be ready.
...
  1. The cluster re-enables shard allocation and prompts 
...
  1.  to repeat these two steps on the next node if the cluster health is green or yellow.