S3 Protocol Architecture

The S3 protocol personality is a front-end storage protocol for client applications. It runs within the Gateway itself. All Gateway deployment and scaling features apply when using the S3 front-end protocol.

An administrator can configure the Gateway to run several ways:

  • the SCSP protocol

  • the S3 protocol

  • both protocols at the same time

Gateways can be scaled horizontally with any combination of front-end protocols as needed for a particular deployment. This allows for a heterogeneous mix of client types utilizing the same Swarm storage cluster and sharing content with each other. Performing this allows an administrator to provide a unified object storage platform.

Sharing Storage across S3 and SCSP

Content Gateway provides the mechanism to unify the back-end Swarm object storage so S3 applications and SCSP applications can share content. In a unified object storage deployment, a device like a load balancer is used to route incoming client traffic through the appropriate port number or pool of Gateway servers.

Routing Methods

These are some routing methods usable for a unified object storage front-end:

IP Address

Listen to multiple virtual IP addresses and distribute traffic based upon the IP address used by the client.



X-Forwarded-Host

Use Layer 7 inspection of the requests to distribute traffic based upon headers contained in the client requests.

The X-Forwarded-Host header can be used by a proxy or load balancer to allow client applications to use a different host name for each supported HTTP storage protocol while sharing the same storage domain for the content. Consider the storage domain castor.example.com with two protocol-specific host names scsp.castor.example.com and s3.castor.example.com. Each host resolves to a different IP address so a load balancer can direct the traffic to the appropriate Gateway server pool for the chosen storage protocol.

The load balancer needs to add the header to direct storage requests to the shared storage domain: 

X-Forwarded-Host: castor.example.com

DNS

Cause the DNS name resolution to be different for the clients using one storage protocol than for the clients using another protocol.

Clients using S3 may resolve the storage domain castor.example.com to 10.100.100.81 while clients using SCSP resolve the same storage domain to 10.100.100.82. This method requires the administrator has control of the hosts where the client applications run so as to allow alteration of the DNS/host resolution but does not require in-line modification of the HTTP requests.

OPTIONS

Route request method OPTIONS with "Origin" to the S3 port rather than default to the SCSP port (which happens when an Authorization header does not exist or have "AWS").

S3 must handle "bucket in Host" style requests because SCSP reports the domain was not found.

curl -i -H 'Origin: http://www.example.com' -H 'Access-Control-Request-Method: PUT' -X OPTIONS https://mycorsbucket.elsewhere.com/

Pattern-Matching

Use Layer 7 inspection of the requests to switch incoming traffic based on the object storage protocol. See below:

This is performed by looking for the distinctive S3 Authorization header pattern or one of the query string arguments for authenticated S3 requests. The pattern-matching rules for these authenticated S3 requests are as follows.

  • Headers

Use pattern-matching on the request header.

Authorization ~= ^AWS.*
  • Arguments

Use pattern-matching on the query string arguments.

  • Absence

Use pattern-matching to test for the absence of all AWS request patterns.

The request is either an anonymous S3 request or an SCSP request. Since anonymous S3 requests do not create, update, or delete content, they are most likely GET requests, and it is safe to allow SCSP to handle these. Using SCSP for video (e.g., mp4) objects also avoids an issue with If-Range handling on the Gateway S3 port.

 

© DataCore Software Corporation. · https://www.datacore.com · All rights reserved.