Changes for Storage 9.2

New Features

This release focuses enhancements in the following areas to improve the operation and ease of maintenance of your storage cluster:

Metadata Typing Expanded

In addition to indexing content metadata as string types, Swarm now supports alternative representations — such as numeric, time, and geo-location — for the same metadata. This provides applications with powerful query capabilities when performing searching operations. A metadata field value of "30.3675, -97.7878" can simultaneously be searched for as a string value or as a latitude/longitude geographic coordinate. Swarm automatically adds supported type representations for metadata as it is indexed. See Custom Metadata Typing.

Create a feed on the new schema (see Upgrading an Existing Index) to use this typing; a feed does not need to be recreated if custom metadata is not used or this typing feature is not needed. 

S3 Multipart Listing Compatibility Improvements

Some S3 applications naively codify formatting validations for multipart operations contrary to AWS S3 API guidance. Swarm mimics AWS S3 behavior with regard to the formatting of the opaque ETag string on multipart upload and listing operations to support these applications:

• The Composite-Content-MD5 header is now indexed in Elasticsearch for newly completed multipart writes. The header is persisted in a COPY operation and dropped on PUT or APPEND.

• A new header, Castor-System-CompositeMD5, is provided as a trailing header if the Content-MD5 is available on all parts when completing a multipart write operations. This is used by the Content Gateway as the Etag element of the equivalent S3 response to a multipart complete.

• In support of the HTTP/1.1 specification revisions (see RFC 7231), Gateway now uses a new "gencontentmd5" query argument in lieu of the "Expect: Content-MD5" request header. As clarified by RFC 7231, values other than "100-continue" are deprecated for an Expect header. Swarm continues to accept requests that contain "Expect: Content-MD5".

Volume Upgrades and Management Simplification

Swarm volumes that hold mostly small objects and are formatted by Swarm 7.5 or earlier can be automatically upgraded in-place to better handle objects <100KB. This upgrade expands the journal space and eliminates a "full" volume due to journal exhaustion. Prior to this improvement, an administrator had to manually retire and reformat the old volume to expand the journal space. Contact DataCore Support if you have such volumes to upgrade.

Additional management simplifications in this release improve handling of virtual machine volumes, encrypted volumes, and event reporting:

• Log messages include additional physical details about volumes, such as SCSI locator (bus ID) and volume serial number, for events such as volume retires and failures. This makes it easier for an administrator to locate and verify the physical device in the hardware chassis.

• When a volume is mounted or formatted, the logs affirmatively report when the volume is encrypted.

• Swarm now supports volumes that provide no serial numbers or erroneously provide the same serial number for all volumes on a controller. Although the best practice for VMs is to configure them for accurate tracking, this is no longer a strict requirement.

Performance and Stability Improvements

• Load balancing of small objects (<100KB) onto newly added volumes is improved.

• Hot-plug add/remove of volumes is improved for rapidly occurring events on multiple volumes in a chassis.

• When using a replication feed, the target cluster's replicate-on-write policy is honored, which both reduces the load impact to the cluster and eliminates any time before full protection for the replicated objects.

Upgrade Impacts

These items are changes to the product function that may require operational or development changes for integrated applications.

Additional Changes

These items are other changes and improvements including those that come from testing and user feedback.

• OSS Updates

  • Intel Network Driver updates include ixgbe 5.0.4, ixgbevf 4.0.3, hpsa 3.4.14-0, megaraid_sas 9.102.00.00, and mpt2sas/mpt3sas 09.102.00.00. (SWAR-7446)

  • For better multipath support, the Linux kernel is now 4.4.47. (SWAR-7446)

  • NTP is updated to version 4.2.8p9 to address recently discovered security issues. (SWAR-7366)

• Improvements

  • The new 'preserve' query argument works with COPY requests to guarantee any custom metadata existing on the object is carried over to the copy. To overwrite an existing value, include the header name with the new value on the request. (SWAR-6661)

  • Swarm now accepts a proxy username and password to allow health reports to be sent via a password-enabled proxy. (SWAR-6831)

  • After the cluster is booted, the Swarm admin user and password may now be removed from the security.administrators setting of the configuration file value, provided that 'smnp' is the only remaining user defined. (SWAR-7329)

  • Swarm now allows a fields=all query argument on bucket and domain listing requests to return all available fields on each record in the response. (SWAR-7034)

  • Bucket or domain listing queries with start and size values exceeding the limit of 10000 return a 500 response. This erroneous query type now returns a 400 response with detail about the error. (SWAR-7442)

  • To comply with RFC 7232, Swarm no longer attempts to limit DELETE, PUT, or COPY requests using the "If-Modified-Since" header. This header continues to be appropriate on GET and HEAD requests. (SWAR-7254)

  • SCSP now supports use of the version query argument in contexts where versioning is disabled; operations that reference the current version proceed normally, but any other etag results in a 404 - Not Found. (SWAR-7325)

  • The setting metrics.targets now allows either spaces or commas as separators between items (host names or IP addresses). The metrics curator configuration settings "clusters" and "hosts" allows either spaces or commas as item separators. (SWAR-7435)

• Resolved Issues

  • Feed progress metrics showed erroneous bursts of activity in an otherwise quiet cluster. (SWAR-7383)

  • Hotplug removing a volume from a multipath-enabled chassis caused CPU utilization to be dominated by errant kernel processes, which required powering off the chassis. (SWAR-7389)

  • In a CSN environment, Swarm uses the disk.minGB to decide which volumes to include in the calculation of the ideal number of Swarm processes, guaranteeing it skips disks too small to be mounted. Non-CSN users set chassis.processes manually. (SWAR-6805)

  • In an encryption-enabled cluster, hotplugging a new volume caused extraneous console warnings about volumes already present. (SWAR-7452)

  • Metadata header names provided on a multipart write completion request became lowercased on the completed object. The case of the headers is now preserved. (SWAR-7240)

  • Not specifying log.host in a cluster's node.cfg file resulted in periodic node reboots. (SWAR-7440)

  • Recreating a deleted bucket or domain no longer results in CRITICAL log messages about abandoned content. (SWAR-7388)

  • Search feeds in clusters to which versioned objects are replicated may have duplicated or missing versions in listing queries that use the versions query argument. (SWAR-7424)

  • Swarm returned a 501 if a bad cluster name was supplied on the Castor-System-Cluster header. It now correctly returns 502. (SWAR-7184)

  • Volumes classified as stale (offline for more than 2 weeks) are now shown as offline on the corresponding Swarm process. The volume identify light is activated for 1 hour at boot for offline volumes so they can be identified and removed. (SWAR-7425)

  • When an EC range read is satisfiable from a single segment and that segment is unavailable and not the first segment in the set, data is erroneously returned from the first segment, resulting in the appearance of corrupt data. (SWAR-7460)

  • With the igb driver, insufficient channels caused inhibit efficient interrupt handling, resulting in lower overall cluster performance. (SWAR-7163)

  • When upgrading from 9.0.x, the caringo-elasticsearch-search RPM did not upgrade correctly from the prior version, requiring an uninstall first. (SWAR-7216)

  • Enabling scsp.autoContentMD5Computation caused COPYs to return invalid Content-MDs, GETs to alter/remove Content-MD5, and prevented successful creation of EC objects in some circumstances. (9.2.1: SWAR-7517)