Implementing Storage 9.6

Watch Items and Known Issues

The following operational limitations and watch items exist in this release.

• Remove any metrics.* settings from the CSN cluster.cfg, which overrides the setting in Swarm and can cause "Failed to push schema metrics_*" errors. (SWAR-7975)

• EC manifests written in versions 9.5.0 and earlier with an explicit Content-MD5 header fails to replicate in versions 9.6.0 and 9.6.1, allowing for data decay. (SWAR-8111)

• If you delete a bucket, any incomplete multipart upload into that bucket leaves the parts (unnamed streams) in the domain. To find and delete them, use the s3cmd utility (search the Support site for "s3cmd"). (SWAR-7690)

• Newer Dell r740x hardware may need to have the Swarm hardware monitor disabled for proper operation. Contact DataCore support for instructions. (SWAR-7645)

• Dell DX hardware have less chassis-level monitoring information available via SNMP. Contact DataCore Support if this is a concern. (SWAR-7606)

• An issue was resolved where logs reported "FEEDS WARNING: calcFeedInfo() couldn't find realm for xxx". The root cause is fixed; contact DataCore Support so the issue can be resolved if you received such warnings. (SWAR-7556)

• With multipath-enabled hardware, the Swarm console Disk Volume Menu may erroneously show too many disks, having multiplied the actual disks in use by the number of possible paths to them. (SWAR-7248)

See Application and Configuration Guidance.

Upgrading from Previous Versions

Swarm supports rolling upgrades (a single cluster running mixed versions during the upgrade process) and requires no data conversion unless noted for a release. Upgrades can be performed without scheduling an outage or bringing down the cluster. Restart the nodes one at a time with the new version and the cluster continues serving applications during the upgrade process.

Rolling upgrade: Reboot one node at a time and wait for its status to show as "OK" in the UI before rebooting the next node.

Alternative: Reboot the entire cluster at once after the software on all USB flash drives or the centralized configuration location has been updated.

Upgrading from 9.x

Note these installation issues:

• The elasticsearch-curator package may show an error during an upgrade, which is a known curator issue. Workaround: Reinstall the curator: yum reinstall elasticsearch-curator (SWAR-7439)

• Installing the Swarm Search RPM before installing Java can cause the plugin to fail silently. If Gateway startup fails with "Caringo script plugin is missing from indexer nodes", uninstall and reinstall the Swarm Search RPM. (SWAR-7688)

• During a rolling upgrade from 9.0.x–9.2.x, you may see intermittent "WriterMissingRemoteMD5 error token" errors from a client write operation through the Gateway or on writes with gencontentmd5 (or the equivalent). To prevent this, set autoRepOnWrite=0 during the upgrade and restore autoRepOnWrite=1 after it completes. (SWAR-7756)

Upgrading from versions prior to 9.x

• When performing a rolling upgrade from a pre-9.0 version of Swarm, the older nodes receive blocked feed errors due to a change in an internal data structure that defines a feed. This error message is: "Plugin validation error: Unknown attribute indexAlias". Normal feed functionality returns after all nodes are upgraded. If a roll-back to a pre-9.0 version is necessary, you must delete and redefine the feeds, or contact DataCore Support for help with the in-place updating of the feed definitions. (SWAR-7274)

• If the cluster is not running Elasticsearch 1.7.1 on Swarm 7.5 or later, upgrade to Swarm 8.2.2+ before proceeding with this upgrade.

• Complete the upgrade of Swarm to 9.1. Swarm 9.1 can work with Elasticsearch 1.7.1 and the existing indices/feeds, but new features in Swarm 9.1 are unavailable until the Elasticsearch upgrade completes. 

• Provision and install an ES 2.3.3 cluster, and switch to the new feed once it is complete. See Upgrading from Elasticsearch 1.7.1.

• Elasticsearch 2.3.3 has many significant changes from prior releases. Elasticsearch recommends running the migration plugin to check for problems:

  1. Install the migration checker: https://github.com/elastic/elasticsearch-migration/tree/1.x

  2. Browse to http://{Elasticsearch-domain-endpoint}/_plugin/migration/

  3. Select Run checks now.

  4. If problems are found, work with Support to fix the data.

• If you are currently using Content Router, it must be decommissioned prior to upgrading.

• Do not upgrade Content Gateway (formerly CloudScaler) until the ES migration is complete. The new Content Gateway requires Elasticsearch 2.3.3.