Changes

• Elasticsearch 6: This release of Gateway adds support for Elasticsearch 6.8.6 while maintaining compatibility with Elasticsearch 5.6.12. See Migrating from Older Elasticsearch. If also running SwarmFS, upgrade it to 2.4 after migrating to Elasticsearch 6. (CLOUD-3131)

• New Grafana Dashboard: The Grafana dashboard for Gateway 6.3 covers details of CPU load, operations, connections, and HTTP status codes. See Prometheus Node Exporter and Grafana.
  

• Faster Clipping: Reworked connection handling has improved the performance of video clip processing. (CLOUD-3147)

• S3 Compatibility

  • Multipart writes are long-running operations with initial and final responses. For S3 compatibility, the initial response now returns x-amz-version-id with the value of the expected ETag. If there is an error completing the write, there is no new object, and the expected ETag given is not valid. (CLOUD-3141)

  • The S3 protocol in this release of Gateway has over 30 internal changes, in areas such as ACL, CORS, V4 signatures, and multipart writes to maintain parity with the evolution of AWS S3. (CLOUD-3142)

Fixed

• Resolved an issue that can trigger a Field Data Circuit Breaker error in Elasticsearch.  (6.3.1: CLOUD-3172)

• A valid COPY request on an alias (unnamed mutable) object returned 404 Not Found. (6.3.1: CLOUD-3170)

• S3 signature errors occurred related to Date/x-amz-date with clients such as rclone. (6.3.1: CLOUD-3157)

Upgrade Impacts

To upgrade from a version of Gateway 6, see Upgrading Gateway. If migrating from Elasticsearch 2.3.3 and are ready to upgrade from Gateway 5, see Upgrading from Gateway 5.x.

Address the upgrade impacts for each of the versions since currently running version:

Watch Items and Known Issues

These are known operational limitations and watch items that exist for Gateway.

• When using the default RHEL/CentOS configuration of IPTABLES, traffic to the Gateway is blocked unless action is taken to disable IPTABLES or to enable inbound traffic to the front-end protocol port(s).

• Gateway is not compatible with the fingerprint scanner module for Linux PAM. If it is installed, remove it by running: yum remove fprintd-pam

• SCSP reading operations that request a Content-MD5 hash validation and for which there is a hash mismatch causes a storage node to be temporarily removed for the Gateway's connection pool due to the way Swarm reports a hash validation failure.

• Swarm Integrity Seal upgrades cannot be performed through Gateway. They may be done directly to the back-end Swarm cluster.

• If the HTTP cache control headers If-Modified-Since and If-Unmodified-Since are used, review the discussion of these in the Storage SCSP Development.

The following are known issues in this release:

• A valid APPEND request on an alias (unnamed) object returns 404 Not Found. (CLOUD-3181)

• The AWS S3 SDK for C# does not properly sign S3-compatible requests with spaces in the name unless the domain contains ".s3." or ".s3-". See https://github.com/aws/aws-sdk-net/issues/933. (CLOUD-3068)

• When buckets are created, the x-amz-storage-class header is not preserved. (CLOUD-3062)

• The Gateway error "Failed reading from client" on a PUT due to "EofException: Early EOF" may occur when clients do not send the full body. This may point to a bug in the client's retry logic, such as not resetting the position marker to the beginning of the file or part. (CLOUD-3010)

• During new object creation as part of renaming with ?newname, Gateway does not verify the user has permission to create the new object name (although it is highly likely, because it is a write within the same context). (CLOUD-2966)

• An s3cmd or rclone server-side copy request may time out on a multipart copy for >5GB objects (s4cmd performs it correctly). Workaround: After verifying it is not the HTTPS proxy timing out, increase the client timeout: set s3mcd socket_timeout = 600 in ~/.s3cfg or use rclone copy --timeout=10m --contimeout=2m caringo:mybucket/5gb caringo:mybucket/subfolder/. (CLOUD-2949)

• Listings with max-keys may be shorter than expected because CommonPrefixes are included in the count of keys returned. (CLOUD-2917)

• Uploading files / photos using Panic's Transmit app on iOS fails due to a 403 Invalid Signature error. (CLOUD-2886)

• Usernames are case-insensitive, but listings exclude a token if the username (myadmin) does not match the case used when the token was created (myAdmin). (CLOUD-2837)

• Multipart PUT requests via recent Cyberduck versions fail with 403 SignatureDoesNotMatch when using AWS Signature Version 4. Install the Caringo .cyberduckprofiles from Using the Cyberduck application with Content Gateway S3 which force V2 signatures. (CLOUD-2799)

• If a policy document includes a Principal that has plural "users" or "groups" instead of "user" or "group", the policy fails to take effect without warning. (CLOUD-2783)

• Versioning-enabled buckets with large numbers of objects may generate Gateway server.log warnings that can be safely ignored: "S3BucketRequestHandler: WARNING: problem with versioned bucket listing. Number of CommonPrefix (2000) exceeds max-size limit (1000)." (CLOUD-2643)

• 403 S3 V4 Signature mismatch errors may result when using Cyberduck with the "pound" proxy in front of Gateway S3. Workaround: Disable the Expect header in the Cyberduck preferences, or (recommended) use a different proxy such as HAProxy. (CLOUD-2628)

• When Gateway cannot connect to Elasticsearch nodes, the errors may erroneously report this as being related to Storage nodes. (CLOUD-2595)

• Because of issues with Range and ETag header handling, video playback of .mp4 streams may not work correctly when served via the Gateway S3 port. It does work when served via the Gateway SCSP port. (CLOUD-1964)

• Gateway caches the Swarm version from the "Server:" response header, so after upgrading Swarm Gateway must be restarted to consistently see the new version. (CLOUD-1271)

• Gateway responds with a 500 (Internal Server Error) instead of 400 (Bad Request) if the size of the metadata headers sent to Swarm is too large. (CLOUD-800)

• The S3 bucket listing StorageClass response element always reports STANDARD. (CLOUD-766)

• If an S3 client escapes URI path characters such as "/", the Gateway audit log escapes the "%" characters used by the client as escape characters. URI audit log processing for S3 clients requires double-unescaping when this occurs. (CLOUD-703)