Implementing Content Gateway 5.3

Watch Items and Known Issues

These are known operational limitations and watch items that exist in this release of Gateway.

When using the default RHEL/CentOS configuration of IPTABLES, traffic to the Gateway is blocked unless action is taken to disable IPTABLES or to enable inbound traffic to the front-end protocol port(s).
Gateway is not compatible with the fingerprint scanner module for Linux PAM. If it is installed, remove it by running:
yum remove fprintd-pam
SCSP reading operations requesting a Content-MD5 hash validation and for which there is a hash mismatch cause a storage node to be temporarily removed for the Gateway's connection pool due to the way Swarm reports a hash validation failure.
Swarm Integrity Seal upgrades cannot be performed through Gateway. They may be done directly to the back-end Swarm cluster.
If the HTTP cache control headers If-Modified-Since and If-Unmodified-Since are used, review the discussion of these in the Storage SCSP Development.

The following are known issues in this release.

During new object creation as part of renaming with ?newname, Gateway does not verify the user has permission to create the new object name (although it is highly likely, because it is a write within the same context). (CLOUD-2966)
Verbose log4j messages are printed on the console during service start. These have no operational impact and may be ignored. (CLOUD-2964)
Listings with max-keys may be shorter than expected because CommonPrefixes are included in the count of keys returned. (CLOUD-2917)
S3 listings that specify a delimiter can return multiple CommonPrefixes entries. That confuses some S3 clients like Caringo Drive, leading to missing directories. (CLOUD-2888)
Uploading files/photos using Panic's Transmit app on iOS fails due to a 403 Invalid Signature error. (CLOUD-2886)
Gateway 5.2.2 and earlier do not output the NextMarker field in S3 listings, which can cause some S3 clients such as Caringo Drive, rclone, and Transmit to show only 1000 files in a directory or to miss some subdirectories. (CLOUD-2871)
Listings exclude a token if the case of the username (myadmin) does not match that used when the token was created (myAdmin). (CLOUD-2837)
Multipart PUT requests via recent Cyberduck versions fail with 403 SignatureDoesNotMatch when using AWS Signature Version 4. Install the Caringo .cyberduckprofiles from Using the Cyberduck application with Content Gateway S3 which force V2 signatures. (CLOUD-2799)
If a policy document includes a Principal that has plural "users" or "groups" instead of "user" or "group", the policy fails to take effect without warning. (CLOUD-2783)
On macOS, you must use Transmit 4.x or >=5.0.5 and Gateway 5.2.2 to avoid 403 Invalid Signature errors. (CLOUD-2777)
Versioning-enabled buckets with large numbers of objects may generate Gateway server.log warnings that can be safely ignored: "S3BucketRequestHandler: WARNING: problem with versioned bucket listing. Number of CommonPrefix (2000) exceeds max-size limit (1000)." (CLOUD-2643)
403 S3 V4 Signature mismatch errors may result when using Cyberduck with the "pound" proxy in front of Gateway S3. Workaround: Disable the Expect header in the Cyberduck preferences, or (recommended) use a different proxy, such as "haproxy". (CLOUD-2628)
When Gateway cannot connect to Elasticsearch nodes, the errors may erroneously report this as being related to Storage nodes. (CLOUD-2595)
Because of issues with Range and ETag header handling, video playback of .mp4 streams may not work correctly when served via the Gateway S3 port. It does work when served via the Gateway SCSP port. (CLOUD-1964)
Gateway caches the Swarm version from the "Server:" response header; therefore, after upgrading Swarm, you must restart Gateway to consistently see the new version. (CLOUD-1271)
Gateway install fails if Internet access in not available. (CLOUD-1216)
Gateway responds with a 500 (Internal Server Error) instead of 400 (Bad Request) if the size of the metadata headers sent to Swarm is too large. (CLOUD-800)
The S3 bucket listing StorageClass response element always reports STANDARD. (CLOUD-766)
If an S3 client escapes URI path characters such as "/", the Gateway audit log escapes the "%" characters used by the client as escape characters. URI audit log processing for S3 clients requires double-unescaping when this occurs. (CLOUD-703)

Upgrading from Release 5.x

The Gateway software components packaged as RPMs in the distribution and this section describes the method to upgrade existing 5.x installations. Although these steps include instructions for routing traffic away from Gateways during the upgrade process, you may safely skip these if the deployment does not include a load balancer.

Verify Swarm storage cluster is upgraded to version 9 or later. This includes completing the migration to Elasticsearch 2.3 metadata search servers in conjunction with the Swarm 9 upgrade.
In the load balancer, disable traffic for one Gateway at a time and perform the following upgrade steps for that Gateway. Allow traffic to continue flowing to the other Gateways.
Upgrade the traffic-disabled Gateway.
1. Stop the Gateway service.
  RHEL/CentOS 6
  service cloudgateway stop
  RHEL/CentOS 7
  systemctl stop cloudgateway
2. Upgrade the Gateway service:
3. If using RHEL/CentOS 7, reload the systemd control scripts:
4. Review and merge any configuration changes to the gateway.cfg and logging.cfg files in the /etc/caringo/cloudgateway directory. The new distribution versions are the *.rpmnew files.
5. Start the Gateway service:
  RHEL/CentOS 6
  RHEL/CentOS 7
6. Verify the Gateway service is running. See the log and curl examples in "Verifying the upgrade," below.
7. Upgrade the Content Portal:
In the load balancer, re-enable client traffic to the newly upgraded Gateway and repeat the process for the remaining Gateways.

Upgrading from Release 4.x

The Gateway software components packaged as RPMs in the distribution and this section describes the method to upgrade existing 4.x installations. Although these steps include instructions for routing traffic away from Gateways during the upgrade process, skip these if the deployment does not include a load balancer. Since Gateway 4.x is only certified on RHEL/CentOS 6, the commands shown here are only applicable for RHEL/CentOS 6.

Verify Swarm storage cluster is upgraded to version 9 or later. This includes completing the migration to Elasticsearch 2.3 metadata search servers in conjunction with the Swarm 9 upgrade.
In the load balancer, disable traffic for one Gateway at a time and perform the following upgrade steps for that Gateway. Allow traffic to continue flowing to the other Gateways.
Upgrade the traffic-disabled Gateway.
1. Stop the Gateway service:
2. Upgrade the Gateway service:
3. Review and merge any configuration changes to the gateway.cfg and logging.cfg files in the /etc/caringo/cloudgateway directory. The new distribution versions are the *.rpmnew files.
4. Start the Gateway service:
5. Verify the Gateway service is running. See the log and curl examples in "Verifying the upgrade," below.
6. Upgrade the Content Portal:
In the load balancer, re-enable client traffic to the newly upgraded Gateway and repeat the process for the remaining Gateways.

Verifying the upgrade

After upgrading, you can check the Gateway's server log (default: /var/log/caringo/cloudgateway_server.log) to verify there are no startup errors. You are looking to verify the SCSP and/or S3 protocol services are running. This is an example from a server with both protocols enabled:

Verify the Gateway is responding to Management API requests by requesting the API version. This command is run from the Gateway and assumes the SCSP protocol is running on port 8080. Adjust the port and/or interface IP address if the configuration is different.

You should get an HTTP 200 response and a JSON body returned with that request.