New Features

Performance: This release of Swarm Storage enhances both memory management and cluster performance:

• Swarm cluster startup has been optimized to guarantee the fastest sequencing. Now volume mounting must complete and the persistent settings must be processed before any needed recovery activities can commence. (SWAR-8911)

• Swarm nodes shut down faster, allowing for quicker rebooting of Swarm clusters. (SWAR-8891)

• Swarm nodes with limited physical memory can now respond better under high client loads. (SWAR-8870)

• Swarm's memory management has been improved, which enables higher loads for client writes. (SWAR-8816)

Stability: This release also includes changes that improve Swarm stability and administration:

• Better handling of newly added hotplug volumes results in clients receiving fewer 503 Service Unavailable responses. (SWAR-8887)

• HP cycles now cleanse all traces of removed volumes from the cluster, greatly reducing the chance that recovery can be started erroneously for a volume already recovered. (SWAR-8836)

• Reworking of cluster operations has reduced spurious "Cannot contact node" announcements during maintenance rebooting of multiple nodes. (SWAR-8848)

• When secure logging (security.secureLogging) is enabled, Swarm removes more sensitive information from AUDIT-level messages. (SWAR-8790)

Additional Changes

These items are other changes, including those that come from testing and user feedback.

OSS Versions 

See https://perifery.atlassian.net/wiki/spaces/public/pages/2443805627 for the complete listing of packages and versions.

Fixed in 11.3.0

• Drive light plug-in control is restored for hardware in mpt3sas enclosures, including Western Digital Ultrastar Serv60. (SWAR-8934)

• For some feed statistics, feed accounting resets and requires a reboot to correct the statistic. (SWAR-8854)

Upgrade Impacts

Use the supported versions of Swarm components if running an older version of Elasticsearch:

These items are changes to the product function that may require operational or development changes for integrated applications. Address the upgrade impacts for each of the versions since the version currently running:

Watch Items and Known Issues

The following watch items are known:

• Volumes newly formatted in Swarm 11.1, 11.2, or 11.3 to use encryption-at-rest cannot be downgraded to Swarm 11.0 or earlier without a special procedure to prevent data loss. Contact DataCore Support before any such downgrade with encrypted volumes. (SWAR-8941)

• Infrequent WARNING messages, "Node/Volume entry not published due to lock contention (...); action will be retried," may appear in logs. Unless they are frequent, they may be ignored. (SWAR-8802)

• If a node mounts an encrypted volume that is missing the encryption key in the configuration, the node fails to mount all disks in the node. (SWAR-8762)

• S3 Backup feeds do not backup logical objects greater than 5 GB. (SWAR-8554)

• When restarting a cluster of virtual machines that are UEFI-booted (versus legacy BIOS), the chassis shut down but does not come back up. (SWAR-8054)

These are standing operational limitations:

• If downgrading from Swarm 11.0, CRITICAL errors may appear on the feeds. To stop the errors, edit the existing feed definition names via the Swarm UI or legacy Admin Console. (SWAR-8543)

• If the Elasticsearch cluster is wiped, the Storage UI shows no NFS config. Contact DataCore Support for help repopulating the SwarmFS config information. (SWAR-8007)

• If a bucket is deleted, any incomplete multipart upload into that bucket leaves the parts (unnamed streams) in the domain. To find and delete them, use the s3cmd utility (search the Support site for "s3cmd" for guidance). (SWAR-7690)

• Removing subcluster assignments in the CSN UI creates invalid config parameters preventing the unassigned nodes from booting. (SWAR-7675)

• Logs showed the error "FEEDS WARNING: calcFeedInfo(etag=xxx) cannot find domain xxx, which is needed for a domains-specific replication feed". The root cause is fixed; if such warnings are received, contact DataCore Support so the issue can be resolved. (SWAR-7556)

• If a feed is subject to a prolonged outage, a node reboot may be required for it to resume progress after the outage is cleared. If progress is not resolved after the reboot, contact DataCore Support. This has been resolved in 12.1.0 (SWAR-9062)

• If Elasticsearch 6.8.6 blocks an index due to low disk space, this needs to be issued against each index (index_*, csmeter*, metrics*) in the read_only_allow_delete state. This is no longer an issue after upgrading to Swarm 12 / Elasticsearch 7 as it automatically unblocks when disk space frees up. (SWAR-8944)

  curl -i -XPUT "<ESSERVERIP>:9200/<INDEXNAME>/_settings" -d '{"index.blocks.read_only_allow_delete" : null}' -H "Content-Type: application/json"

Upgrading Swarm

Note these installation issues when upgrading Swarm:

• The elasticsearch-curator package may show an error during an upgrade, which is a known curator issue. Workaround: Reinstall the curator. (SWAR-7439)

  yum reinstall elasticsearch-curator

• Do not install the Swarm Search RPM before installing Java. If Gateway startup fails with "Caringo script plugin is missing from indexer nodes", uninstall and reinstall the Swarm Search RPM. (SWAR-7688)

Proceed to https://perifery.atlassian.net/wiki/spaces/public/pages/2443804878 to upgrade Swarm 9 or higher.