Swarm Storage 14.0.1 Release

New Features

Erasure Coding Improvements — Swarm 14 includes Erasure Coding related improvements:

• With Swarm 14.0 erasure coded (EC) objects that are indexed will include the field "ec_encoding" that gives the current EC coding of the object. Non-EC objects will not have this field. (SWAR-6653)
• Swarm now computes the data footprint of Erasure Coding EC segments and whole replicas objects separately during each HP cycle so that the relative space usage of whole replicas vs EC can inform space usage policy decisions. (SWAR-9160)

Additional Changes

These items are other changes, including those that come from testing and user feedback.

• OSS Versions — See Third-Party Components for Storage 14.0.1 for the complete listing of packages and versions for this release.
• Fixed in 14.0
  • Reboot loop due to a bad drive: When a volume fails at mount time, the Swarm node will recognize the volume as failed and alert the cluster to the failure. The node will operate with the remaining volumes, so physically removing the volume may be necessary. (SWAR-9189)

  • Remove legacy nonce handling: Remove scsp.forceLegacyNonce settings from the node.cfg files prior to upgrading to 14.0. (SWAR-9108)

  • Bucket listings: During a node reboot, such as a rolling reboot of the cluster, a newly booted node would temporarily return an empty result set for a listing query. (SWAR-9083)
  • S3 backup feed: A 5G object size limitation has been removed. (SWAR-8554)

Upgrade Impacts

These items are changes to the product function that may require operational or development changes for integrated applications. Address the upgrade impacts for each of the versions since the one you are currently running:

Impacts for 14.0

• Change in the node.cfg file — The previously deprecated sysctl section of the node.cfg file has been removed. If it is necessary to set kernel runtime parameters, use kernel.sysctlFileUrl (introduced in Swarm 12.0) instead. (SWAR-8968)

  Settings changes
  • Updated:

    • All sysctl.* settings have been removed. (SWAR-8968)

    • support.reportPeriod default has been changed to 21600 (6 hours). (SWAR-8424)
    • The following settings are now persisted cluster settings that can be updated via SNMP and the UI (SWAR-9115)
      
      • cluster.enforceTenancy
      • cluster.proxyIPList
      • ec.maxManifests
      • ec.minParity
      • ec.segmentSize
      • feeds.retry
      • health.parallelWriteTimeout
      • health.underreplicationAlertPercent
      • health.underreplicationTolerance
      • health.persistentUnderreplicationAlertPercent
      • log.obscureUUIDs
      • scsp.clientPoolTimeout
      • scsp.defaultContextReplicas
      • scsp.defaultROWAction
      • scsp.maxWriteTime
      • scsp.validateOnRead
      • search.numberOfShards
• Swarm storage node metrics are deprecated and will be replaced in the next major release by the graphs and reporting from Grafana and Prometheus Node Exporter. The storage administration UI has been updated to allow for metrics to be turned off. Clear metrics.target from your configuration, uninstall caringo-elasticsearch-metrics, and curl -XDELETE 'http://ELASTICSEARCH:9200/metrics-*' to clear the space in your Elasticsearch cluster. (SWAR-8982)
• Differences in scsp.forceLegacyNonce configuration depending on the version you're upgrading from (SWAR-9020):

• If you are currently running a Swarm Storage version prior to 11.1, and upgrading to 11.1, 11.2, 11.3, 12.0 or 12.1:

  Before upgrading, set scsp.forceLegacyNonce=true in your node.cfg file. After the upgrade, when the cluster is fully up, update scsp.forceLegacyNonce=false using swarmctl and change scsp.forceLegacyNonce=false in your node.cfg file.

  If you are currently running a Swarm Storage version 11.1, 11.2, 11.3, 12.0 or 12.1 and upgrading to another version from that list:

  Before upgrading, verify that scsp.forceLegacyNonce=false is in your node.cfg file and verify using swarmctl that scsp.forceLegacyNonce=false in your cluster.

• Subcluster assignments can no longer be blank, and CSN installations with mixed subcluster assignments will have the unassigned nodes unable to boot, showing an error in contacting their time source. Be sure to supply a subcluster for each node if any named subcluster is specified in your cluster. (SWAR-7675)

  Use swarmctl to check or change settings

  Use 'swarmctl -C scsp.forceLegacyNonce' to check the value of scsp.forceLegacyNonce.

  Use 'swarmctl -C scsp.forceLegacyNonce -V False' to set the value to false.

  For more details, see https://support.cloud.caringo.com/tools/Tech-Support-Scripts-Bundle-swarmctl.pdf.

Watch Items and Known Issues

The following watch items are known:

• If a node mounts an encrypted volume that is missing the encryption key in the configuration, the node will fail to mount all of the disks in the node. (SWAR-8762)
• S3 Backup feeds do not yet backup logical objects larger than 5 GB; those writes will fail with a CRITICAL log message. (SWAR-8554)
• When restarting a cluster of virtual machines that are UEFI-booted (versus legacy BIOS), the chassis shut down but do not come back up. (SWAR-8054)
• With multipath-enabled hardware, the Swarm console Disk Volume Menu may erroneously show too many disks, having multiplied the actual disks in use by the number of possible paths to them. (SWAR-7248)

These are standing operational limitations:

• If you wipe your Elasticsearch cluster, the Storage UI will show no NFS config. Contact Support for help repopulating your SwarmFS config information. (SWAR-8007)
• If you delete a bucket, any incomplete multipart upload into that bucket will leave its parts (unnamed streams) in the domain. To find and delete them, use the s3cmd utility (search the Support site for "s3cmd" for guidance). (SWAR-7690)
• If you remove subcluster assignments in the CSN UI, doing so creates invalid config parameters that will prevent the unassigned nodes from booting. (SWAR-7675)
• You may see false 404 Not Found and other SCSP errors during rolling reboot in versions 11.1 through 12.0.1. To mitigate this problem, set scsp.forceLegacyNonce=False in the cluster configuration. You will need to remove this setting before upgrading to 12.1.0 or later. (SWAR-9020)
• When you use certificates with HAProxy, S3 Backup restoration to the cluster may be blocked if the certificate is not located where Swarm expects it. From 12.1, a clearer error message draws attention to the issue for S3 backup and replication feeds that are blocked due to invalid X.509 ("SSL") certificates. (SWAR-8996)

To upgrade Swarm 9 or higher, proceed now to How to Upgrade Swarm. If you need to migrate from Swarm 8.x or earlier, contact Support for guidance.