Upgrading from Elasticsearch 1.7.1

Elasticsearch 2.3.3 includes advances in speed, security, scalability, and hardware efficiency and supports compatibility with Linux standards and newer tool releases. Elasticsearch 2.3.3 is a fundamental leap from 1.7.1, so much so that upgrading is actually a migration.

Given the complexities of converting legacy data, the easiest path is to start fresh: provision a new ES cluster (machines or VMs), install ES 2.3.3 on those servers, and create a new feed to this cluster. Swarm 9 will continue to support your old primary feed with the ES 1.7.1 cluster while it builds the index data for the new feed, which means that searching remains available for your users. Once the new feed is built, you make it the primary and the migration is complete. Delete the legacy search cluster entirely unless you have CloudScaler Content Metering data to preserve.

This is an overview of the migration process:

Exceptions

This upgrade strategy does not fit certain situations:

You cannot provision a new search cluster.
You have a business need to convert your CloudScaler Content Metering data.
You use parallel upload in production and need support for uploads to complete during the migration phase (uploads of parts are unaffected).

For such cases, contact Support for assistance.

Pre-Upgrade Checklist

Migration Check	Elasticsearch 2.x has many breaking changes. In particular, if a custom metadata field name includes a period/dot (`x-sample-meta-bad.field.name`), your existing data cannot be upgraded. Elasticsearch recommends that you run its migration plugin to check for problems: Install the migration checker: https://github.com/elastic/elasticsearch-migration/tree/1.x Browse to `http://{Elasticsearch-domain-endpoint}/_plugin/migration/` Select Run checks now. If problems are found, work with DataCore Support to fix your data.
Swarm Requirements	If your cluster is not running Elasticsearch 1.7.1 on Swarm 7.5 or later, upgrade to Swarm 8.2.2 before proceeding with this upgrade. Complete the upgrade of Swarm to 9.0. Swarm 9.0 can work with Elasticsearch 1.7.1 and your existing indices/feeds, but new features in Swarm 9.0 will unavailable until you complete the Elasticsearch upgrade. Upgrading to Swarm 9.0 can be done as a rolling upgrade, without cluster downtime. Note Because of requirements for Elasticsearch 2.3.3, the legacy metering feature in Gateway (CSMeter) is incompatible with Swarm 9.0 and higher. Case-sensitivity - If you enable case-insensitive searching in SCSP (`search.caseInsensitive` `= 1`) , Content Gateway still lets S3 perform the case-sensitive operations it needs. However, if you are upgrading your ES 1.7.1 data, be sure to keep the data consistent by disabling `search.caseInsensitive`. Do not upgrade CloudScaler until this migration is complete. The new Content Gateway requires Elasticsearch 2.3.3.
New ES Cluster	Provision a new set of ES servers (machines or VMs) on which to install Elasticsearch 2.3.3. Ensure that every Elasticsearch node meets the hardware, network, and software requirements, including Java 8. See Hardware Requirements for Elasticsearch. See Preparing the Search Cluster.

Install ES 2.3.3

Important

Never run different versions of Elasticsearch in the same cluster. Ensure that your new Elasticsearch configuration has a different name for the 2.3.3 cluster; otherwise, the new ES servers will join the old ES cluster.

Error rendering macro 'excerpt-include' : No link could be created for 'GOLD:Installing Elasticsearch'.

Note

Swarm Storage lets you create more than one Search feed so that you can transition from using one Elasticsearch cluster to another. During the transition, continue using the primary feed for queries; the second feed is incomplete until it fully clears its backlog. When the second feed is caught up, transition to it (marking it as primary) as soon as reasonable for your operations. When you verify that it is working as the new primary feed target, delete the original feed. Having two feeds is for temporary use only because every feed incurs cluster activity, even when paused.

Switch to ES 2.3.3

Wait until the new feed has completed indexing the cluster, when the feed shows 0 "pending evaluation".
Tip

Set calendar reminders to check on the progress. How many days this takes depends on the cluster's size and load, and you could forget to finish this upgrade.
When the new feed is ready, switch the primary search feed to the new feed ID. In the legacy Admin Console (http://<storage·cluster>:90), go to Settings and change the Default Search feed from <old-feed-id> to <new-feed-id>.
Operate with both feeds for several days. If there is a problem, you can restore the old feed to be primary during troubleshooting.
Upgrade CloudScaler to Content Gateway.
After this confirmation period, delete the old feed. In the legacy Admin Console (http://<storage·cluster>:90), go to Settings and delete the old feed.
Decommission your ES 1.7.1 cluster to reclaim those resources.