Additional Changes

Changes include versions and fixes based on testing and user feedback.

OSS Versions

See Third-Party Components for Storage 16.0 for the complete listing of packages and versions for this release.

Fixed in 16.0

• S3 Object-Locking Requests: Fixed a rare problem that could have caused data loss when rapid S3 object-locking requests were initiated through the Gateway on the same object version. (SWAR-9626)

• Synchronous Indexing: Swarm 16.0 will issue a critical log message if synchronous indexing is configured but failing at a high rate. The setting “scsp.synchronousIndexingFailuresPerHourLimit” controls this behavior. (SWAR-9692)

• Versioned Buckets: Reduced the chance of incorrect (and seemingly duplicated) objects in listing queries related to versioned buckets. (SWAR-9790)

• Rapid Deletes: Rapid deletes of versions of a single object can lead to 503 request with "currentVersionInFlux" errors. The object might return a 404 when accessed by name. Reduced the chance of 503 request persisting on subsequent “Delete” attempts. (SWAR-9806)

• EC Versioned Object: When new EC versioned object writes are deleted, no segments are left behind, which speeds up space reclamation. (SWAR-9850)

• NTP Misconfiguration: Improved protection against negative time jumps caused by NTP misconfiguration. (SWAR-9907)

• Replication Feeds and S3 Backup Feeds Performance: The number of replication threads actually used was potentially smaller than the amount specified in the feed definition for replication feeds and S3 backup feeds. After upgrade, tuning of the specified threads may be required. (SWAR-9911)

• Shards: The default number of shards in any new Elasticsearch index created by a Search Feed has changed from 5 to 10. (SWAR-9926)

• Memory Use: Improved the request failure handling that led to abnormal memory use in some cases. (SWAR-9946)

• Physical Console: The setting “security.securePhysicalConsole=True” can be used to limit the actions that can be performed via console to improve cluster security. A reboot is required for the setting to take effect. (SWAR-9949)

• Drive Identification: Improved handling of drive identification in the (blue) system menu. (SWAR-9952)

• Versioned Object Writes: Improved failure handling of versioned object writes. (SWAR-9962)

• UDP Traffic Handling: Improved UDP traffic handling in clusters with UDP multicast disabled. (SWAR-9969)

• Versioned Objects: Fixed an issue where frequently updated versioned objects might be inaccessible after rebooting. (SWAR-9998)

• Abandoned Elasticsearch Records: Removed a source of abandoned Elasticsearch records following S3 multi-delete operations. (SWAR-10005)

• Minor Elasticsearch upgrade to 7.17.14.

Watch Items and Known Issues

The following watch items are known:

• Configuring elasticsearch.yml's network.host (https://www.elastic.co/guide/en/elasticsearch/reference/7.16/important-settings.html#network.host) to "__site__" might not choose the right IP to allow master election if the server is multi-homed. Modify elasticsearch.yml to enter a specific IP for the node, and the configuration script will preserve it. (SWAR-9350)
  If you run into this issue, the fix is to:

  • systemctl stop elasticsearch on all ES nodes

  • remove all the contents of the path.data directory

  • change network.host: <IP of ES NIC in the Storage VLAN>

  • systemctl start elasticsearch

• Elasticsearch can fail to start and return a warning "unable to load JNA native support library", which is due to SELinux setting “noexec” on /tmp.
  For Elasticsearch 7.5.2, edit “/etc/elasticsearch/jvm.options” replacing the line "-Djava.io.tmpdir=${ES_TMPDIR}" with "-Djava.io.tmpdir=/var/log/elasticsearch". With Elasticsearch 7.17, uncomment the "Environment=ES_TMPDIR=/usr/share/elasticsearch/tmp" line in /etc/systemd/system/elasticsearch.service.d/override.conf and create that directory. (SWAR-9347)

• Swarm versions 10.0 onward are vulnerable to kernel issues manifested on some Intel CPUs. Symptoms include lowered performance, long mount times, and cluster instability. Swarm versions 14.1 and later provide a workaround for this issue, see https://caringo.atlassian.net/wiki/spaces/KB/pages/2973204604. (SWAR-9055)

• The search configuration utility was upgraded to caringo-elasticsearch-search-7.2.0-1.noarch.rpm and Elasticsearch 7.17.14 is now the default version installed. It now preserves rack awareness settings in elasticsearch.yml and can be used to upgrade an SCI Elasticsearch node which does not have node.name. If you are running Elasticsearch 7.5.2 or other earlier version, do not install the 7.17.14 rpm directly. Instead, run the latest configuration script configure_elasticsearch_with_swarm_search.py. The configuration script upgrades /etc/sysconfig/elasticsearch, so that the default Elastic-bundled JDK is used. (SWAR-9896)

• The 15 series releases can exhibit a known issue when attempting to hot plug drives into a Swarm node that uses the Broadcom/LSI HBA driver based on the kernel version in the release. Customers may experience situations where the hot plug of a known good drive fails, requiring a node restart to allow the drive to be recognized. (SWAR-9873)

• A chunked transfer APPEND of an EC object will fail with a 501. Updates to an object being manipulated by an ongoing multipart upload will cause that upload to fail with a 409, RequiresBasisGeneration.

• Customizations to an elasticsearch 6.8.6 /etc/elasticsearch/elasticsearch.yml path.data and network.host fields will be lost when running the configuration script to upgrade to elasticsearch 7 if the upgrade does not complete. This can happen if the new elasticsearch 7 rpm is not in the current directory and cannot be downloaded. Reapply your customizations as this will not affect upgrades starting with elasticsearch 7.5.2. (SWAR-9977)

• Customers who have set ec.convertVersionedObjects=true should revert this setting until a later release due to a potential data loss issue with this setting. (SWAR-10001)

• Customers with high data turnover and continuous disk defragmentation are subject to abnormal process growth during mount, which causes process crashes and instability. This issue will be addressed in version 16.1. (SWAR-10056)

• Swarm Storage 16.0 has a restriction where multipart upload requests partially completed on nodes running 15.x or earlier releases, do not complete successfully if these requests are continued or have completed on Swarm 16.0. This known issue is expected to be resolved so that an upgrade from 15.x to 16.1 or later, with proper settings changes, can allow multipart uploads in mixed version clusters (such as during a rolling upgrade) to complete successfully. If the cluster operation does not allow for stopping multipart upload operations during the upgrade, contact Support for an upgrade procedure to mitigate the issue. (SWAR-10054)

• Customers need to wait 1 min or more to get the changes done in feed definition to be effective throughout the cluster. (SWAR-10007)

• Swarm will attempt to determine if the volumes being used on a chassis are indeed separate devices. The serial number associated with each volume will be used for this determination. With default settings, Swarm will consider separate volumes on the same device as the same for protection purposes. EC segment placement may be impacted by this consideration. This behavior change will not impact most hardware-based clusters, but VM installations with virtual disks may need to lower the k value of their EC encoding in order to allow EC writes. (SWAR-10042)

These are standing operational limitations:

• The Storage UI shows no NFS config if the Elasticsearch cluster is wiped. Contact DataCore Support for help in repopulating the SwarmFS config information. (SWAR-8007)

• Any incomplete multipart upload into a bucket leaves the parts (unnamed streams) in the domain if the bucket is deleted. To find and delete those parts, use the s3cmd utility (search the Support site for "s3cmd" guidance). (SWAR-7690)

• The chassis shuts down but does not come back up when restarting a cluster of virtual machines that are UEFI-booted (versus legacy BIOS). (SWAR-8054)

• Invalid config parameters that prevent the unassigned nodes from booting are created if subcluster assignments are removed in the CSN UI. (SWAR-7675)

• Upgrading an elasticsearch 6.8.6 node using the configure script in caringo-elasticsearch-search-7.2.0 will fail with "ERROR: error parsing version". Please use caringo-elasticsearch-search-7.2.1 instead. (SWAR-10100)

• On node reboots, some feed statistics that are not persisted across boots may show up incorrectly. It will fix itself eventually. Currently, there is no workaround for this. (SWAR-9720)

To upgrade Swarm 9 or higher, proceed to How to Upgrade Swarm. For migration from Swarm 8.x or earlier, contact DataCore Support for guidance.

Deprecation

• The search.caseInsensitive is now deprecated and will be removed in a future release. (SWAR-10085)