Additional Changes

Changes include versions and fixes based on testing and user feedback.

OSS Versions

See [DRAFT] Third-Party Components for Storage 16.1.2 for the complete listing of packages and versions for this release.

Fixed in 16.1.2

• The request to deleted named streams that showed up with the incorrect context in the domain listing will now respond with "<domain_name>/context_not_found". (SWAR-10006)

• For listing requests, if Elasticsearch servers are offline/unreachable, the response error code will be 503 instead of 500 along with the following header. (SWAR-10018)

  • Castor-System-Error-Token: ReaderUnavailableIndexer

  • Castor-System-Error-Text: Indexer server unavailable

• Improved recoveries of dense clusters when multiple volumes retire. (SWAR-10047)

• DELETE performance has been optimized. (SWAR-10089)

• Upgrading an Elasticsearch 6.8.6 node using the configure script in caringo-elasticsearch-search-7.2.x has been fixed. (SWAR-10100)

• Swarm prioritizes Elasticsearch ingest nodes for indexing if node.roles are configured for ingest nodes. Re-discovery of Elasticsearch nodes will happen every hour. (SWAR-10135)

• Introducing "protection level = device" config setting; partitioned volumes within a device are treated as one for erasure-coded objects, preventing data loss when the device is retired. (SWAR-10042)

• Configure a cluster with a minimum of four nodes to guarantee high availability and failover in the event of a node failure. (SWAR-10139)

Watch Items and Known Issues

The following watch items are known:

• Customizations to an Elasticsearch 6.8.6 /etc/elasticsearch/elasticsearch.yml path.data and network.host fields will be lost when running the configuration script to upgrade to Elasticsearch 7 if the upgrade does not complete. This can happen if the new Elasticsearch 7 rpm is not in the current directory and cannot be downloaded. Reapply your customizations as this will not affect upgrades starting with Elasticsearch 7.5.2. (SWAR-9977)

• False 404s might be seen in the Swarm responses during heavy load situations. To mitigate this, please verify “cip.processes = 2”, and keep increasing the cip.queryMinimumTimeout value gradually until the 404 occurrences are reduced. (SWAR-10120)

• When using search.perDomainIndex=True (under Support guidance), the number of supported domains is limited based on the number of data nodes in the Elasticsearch cluster and search.numberOfShards. For example, Five data nodes support 5x600 shards at search.numberOfShards=6; each domain requires 6x2 (primary and replica) shards. If more than 250 domains are created, new indices cannot be created resulting in the domain listing that returns a 503 ReaderUnavailableIndex. The castor.log will show errors EFD19, EIP15, and EIP02.
  For example, "Validation Failed: this action would add [6] shards, but the cluster currently has maximum [999]/[1000] normal open shards". Error reporting will be improved in a future release.

These are standing operational limitations:

• The Storage UI shows no NFS config if the Elasticsearch cluster is wiped. Contact DataCore Support for help in repopulating the SwarmFS config information. (SWAR-8007)

• Any incomplete multipart upload into a bucket leaves the parts (unnamed streams) in the domain if the bucket is deleted. To find and delete those parts, use the s3cmd utility (search the Support site for "s3cmd" guidance). (SWAR-7690)

• The chassis shuts down but does not come back up when restarting a cluster of virtual machines that are UEFI-booted (versus legacy BIOS). (SWAR-8054)

• Invalid config parameters that prevent the unassigned nodes from booting are created if subcluster assignments are removed in the CSN UI. (SWAR-7675)

• On node reboots, some feed statistics that are not persisted across boots may show up incorrectly. It will fix itself eventually. Currently, there is no workaround for this. (SWAR-9720)

• Customers need to wait 1 minute or more to get the changes done in feed definition to be effective throughout the cluster. (SWAR-10007)

To upgrade Swarm 9 or higher, proceed to How to Upgrade Swarm. For migration from Swarm 8.x or earlier, contact DataCore Support for guidance.

Deprecation

• The search.caseInsensitive is now deprecated and will be removed in a future release. (SWAR-10085)