{"type":"doc","content":[{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"toc","parameters":{"macroParams":{"minLevel":{"value":"1"},"maxLevel":{"value":"2"},"outline":{"value":"false"},"type":{"value":"list"},"printable":{"value":"false"}},"macroMetadata":{"macroId":{"value":"1cf3e495-c0b8-4196-a199-2d91893672a3"},"schemaVersion":{"value":"1"},"title":"Table of Contents"}},"localId":"9ce774c6-d7da-4927-935d-abb6247b4edf"}},{"type":"paragraph","content":[{"text":"Amazon S3 is two distinct things: \"S3 The Service\" and \"S3 The Protocol\". Since the S3 protocol reflects characteristics of the Amazon service that may not be applicable outside of Amazon, the Gateway adapts these characteristics so they make sense when hosting storage within the environment. These adaptations are transparent to most applications and enhance the protocol features by making use of the unique strengths of Swarm storage.","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"AWS Regions versus Storage Domains","type":"text"}]},{"type":"paragraph","content":[{"text":"Amazon AWS currently has fewer than ","type":"text"},{"text":"two dozen geographic regions","type":"text","marks":[{"type":"link","attrs":{"href":"https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/"}}]},{"text":" worldwide, each of which is shared by thousands of end-users. Choose a permanent region for each of the S3 buckets based upon latency, cost, and any applicable regulatory requirements. ","type":"text"}]},{"type":"paragraph","content":[{"text":"An ","type":"text"},{"text":"Amazon AWS region is roughly analogous to a Swarm storage domain. A Swarm ","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"1778e987-6614-487f-a95d-a40f0469debd"}}]},{"text":"bucket is tied to a domain as an S3 bucket is tied to a region","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"1778e987-6614-487f-a95d-a40f0469debd"}},{"type":"textColor","attrs":{"color":"#091e42"}}]},{"text":". They are fundamentally different in the following ways:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"AWS regions are finite, but any number of Swarm domains can be created; this allows achieving the optimal granularity for assigning content user permissions.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#091e42"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"AWS buckets must have a name unique across ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#091e42"}}]},{"text":"all","type":"text","marks":[{"type":"textColor","attrs":{"color":"#091e42"}},{"type":"em"}]},{"text":" regions, but Swarm buckets need only be unique within the domain.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#091e42"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"AWS buckets are fixed geographically, but Swarm content dynamically distributes across the entire Swarm cluster, according to the content protection settings.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#091e42"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"AWS buckets can be replicated to a differently named bucket, but Swarm replication preserves the domain, bucket, and object names identically across every target cluster, which supports content sharing and direct DR fail-over.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#091e42"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"AWS buckets can replicate in one direction, but Swarm clusters can use mirrored replication. A domain","type":"text","marks":[{"type":"textColor","attrs":{"color":"#091e42"}}]},{"text":"'s content can be created, updated, and accessed across distributed storage clusters while remaining synchronized.","type":"text"}]}]}]},{"type":"panel","attrs":{"panelType":"success"},"content":[{"type":"heading","attrs":{"level":3},"content":[{"text":"Best Practice","type":"text"}]},{"type":"paragraph","content":[{"text":"Domain creation and allocation in Swarm is lightweight, so be generous: provide each client (or business unit) a separate storage domain. Performing this benefits both sides:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"For storage admins, usage tracking and storage management are easier.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"For storage end-users, access control policies are uncomplicated, and bucket naming is more flexible.","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Bucket Location","type":"text"}]},{"type":"paragraph","content":[{"text":"The Amazon S3 ","type":"text"},{"text":"GET Bucket Location","type":"text","marks":[{"type":"em"}]},{"text":" request returns the AWS region in which the bucket is located. By default, Gateway returns an empty value for the location, which S3 clients interpret as us-east-1. If another region is desired, there are two options:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Supply the location in the bucket creation operation using LocationConstraint.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Set the [s3]region option in the Gateway configuration file to the preferred region. This applies to all buckets unless the location is specified during creation.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"If you require the prior behavior of returning the cluster name, set [s3]region to that cluster name.","type":"text"}]},{"type":"panel","attrs":{"panelType":"success"},"content":[{"type":"heading","attrs":{"level":3},"content":[{"text":"Best Practice","type":"text"}]},{"type":"paragraph","content":[{"text":"Unless you have an application that requires otherwise, either use the default, or choose another valid AWS region, as there are clients that validate the location against a list of known AWS regions.","type":"text"}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Storage Class","type":"text"}]},{"type":"paragraph","content":[{"text":"Amazon S3 allows clients to set a storage class preference in the ","type":"text"},{"text":"x-amz-storage-class","type":"text","marks":[{"type":"code"}]},{"text":" header, which defines the data durability and access frequency of content. See the ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://aws.amazon.com/s3/storage-classes/"}},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"The S3 protocol tags ","type":"text"},{"text":"all","type":"text","marks":[{"type":"em"}]},{"text":" objects with the ","type":"text"},{"text":"x-amz-storage-class-meta","type":"text","marks":[{"type":"code"}]},{"text":" header and includes the client application's requested class or STANDARD, if none is specified. Bi-directional translation between the AWS S3 ","type":"text"},{"text":"x-amz-storage-class","type":"text","marks":[{"type":"code"}]},{"text":" header and the Swarm ","type":"text"},{"text":"x-amz-storage-class-meta","type":"text","marks":[{"type":"code"}]},{"text":" header is performed for S3 protocol operations.","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Virtual Hosting of Buckets","type":"text"}]},{"type":"paragraph","content":[{"text":"Amazon S3 allows virtual hostnames to bucket mappings within the storage service. This is accomplished by creating a DNS alias (CNAME record) for a virtual host name pointing to one of the Amazon S3 region endpoints. An example is to allow the web request to be mapped to the real Amazon S3 URL:","type":"text"}]},{"type":"codeBlock","attrs":{"language":"xml"},"content":[{"text":"http://www.fred.com/hello.html\nhttp://s3.amazonaws.com/www.fred.com/hello.html","type":"text"}]},{"type":"paragraph","content":[{"text":"The HTTP ","type":"text"},{"text":"Host","type":"text","marks":[{"type":"code"}]},{"text":" header can be used to specify the bucket name if the virtual host is mapped to the ","type":"text"},{"text":"bucket","type":"text","marks":[{"type":"em"}]},{"text":" named ","type":"text"},{"text":"www.fred.com","type":"text","marks":[{"type":"code"}]},{"text":" in the US Standard Region.","type":"text"}]},{"type":"paragraph","content":[{"text":"The S3 protocol also supports this mapping of virtual hosts to buckets. To accomplish this, the storage administrator configures the DNS server to perform wildcard resolution of host names to the front-end IP address of the Gateway. The Gateway then searches for a storage domain within Swarm starting with the value of the ","type":"text"},{"text":"Host","type":"text","marks":[{"type":"code"}]},{"text":" header and then recursively popping off the leftmost word using period (\".\") as a delimiter until it finds a match or runs out of words. The previously popped words are concatenated back together using periods and the result becomes the bucket name for the request when a storage domain is found.","type":"text"}]},{"type":"paragraph","content":[{"text":"As an example, consider the storage domain in Swarm called ","type":"text"},{"text":"fred.com","type":"text","marks":[{"type":"code"}]},{"text":" containing a bucket called ","type":"text"},{"text":"www","type":"text","marks":[{"type":"code"}]},{"text":" and an object within called ","type":"text"},{"text":"hello.html","type":"text","marks":[{"type":"code"}]},{"text":". The normal method to access this object is with the URL ","type":"text"},{"text":"http://fred.com/www/hello.html","type":"text","marks":[{"type":"code"}]},{"text":", which has the following HTTP request headers:","type":"text"}]},{"type":"codeBlock","attrs":{"language":"xml"},"content":[{"text":"GET /www/hello.html HTTP/1.1\nHost: fred.com","type":"text"}]},{"type":"paragraph","content":[{"text":"Create DNS entries for ","type":"text"},{"text":"www.fred.com","type":"text","marks":[{"type":"code"}]},{"text":" and ","type":"text"},{"text":"fred.com","type":"text","marks":[{"type":"code"}]},{"text":" to setup the virtual host mapping from ","type":"text"},{"text":"www.fred.com","type":"text","marks":[{"type":"code"}]},{"text":" to the Swarm storage domain and bucket. This is an example where the Content Gateway's front-end IP address is ","type":"text"},{"text":"10.100.100.81","type":"text","marks":[{"type":"code"}]},{"text":" and a wildcard match is used.","type":"text"}]},{"type":"codeBlock","attrs":{"language":"xml"},"content":[{"text":"fred.com A 10.100.100.81\n*.fred.com CNAME fred.com","type":"text"}]},{"type":"paragraph","content":[{"text":"After the DNS entries are in place, the ","type":"text"},{"text":"hello.html","type":"text","marks":[{"type":"code"}]},{"text":" object is now accessible with the additional URL ","type":"text"},{"text":"http://www.fred.com/hello.html","type":"text","marks":[{"type":"code"}]},{"text":". The HTTP request headers arriving at the Gateway look like this:","type":"text"}]},{"type":"codeBlock","attrs":{"language":"xml"},"content":[{"text":"GET /hello.html HTTP/1.1\nHost: www.fred.com","type":"text"}]},{"type":"paragraph","content":[{"text":"The Gateway first checks for the non-existent storage domain ","type":"text"},{"text":"www.fred.com","type":"text","marks":[{"type":"code"}]},{"text":" and then removes ","type":"text"},{"text":"www","type":"text","marks":[{"type":"code"}]},{"text":", the leftmost word, and finds the storage domain ","type":"text"},{"text":"fred.com","type":"text","marks":[{"type":"code"}]},{"text":" in the storage cluster. The Gateway then transparently modifies the HTTP request headers by prefixing the URI path with the removed word ","type":"text"},{"text":"www","type":"text","marks":[{"type":"code"}]},{"text":" and shortening the ","type":"text"},{"text":"Host","type":"text","marks":[{"type":"code"}]},{"text":" header as follows:","type":"text"}]},{"type":"codeBlock","attrs":{"language":"xml"},"content":[{"text":"GET /www/hello.html HTTP/1.1\nHost: fred.com","type":"text"}]},{"type":"paragraph","content":[{"text":"The Gateway searches for a storage domain by removing the leftmost words until a domain is found or until it reaches a null hostname if the bucket name contains periods, ","type":"text"},{"text":"hires.images","type":"text","marks":[{"type":"code"}]},{"text":" with a virtual host name of ","type":"text"},{"text":"hires.images.fred.com","type":"text","marks":[{"type":"code"}]},{"text":". For example, it tests for the existence of the domains ","type":"text"},{"text":"hires.images.fred.com","type":"text","marks":[{"type":"code"}]},{"text":" and ","type":"text"},{"text":"images.fred.com","type":"text","marks":[{"type":"code"}]},{"text":" before finding ","type":"text"},{"text":"fred.com","type":"text","marks":[{"type":"code"}]},{"text":". The request results in an error if the search reaches a null hostname. The ","type":"text"},{"text":"[caching] domainExistenceRefresh","type":"text","marks":[{"type":"code"}]},{"text":" configuration parameter in ","type":"text"},{"text":"gateway.cfg","type":"text","marks":[{"type":"code"}]},{"text":" is used to optimize domain existence testing.","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Identity Management and S3 Authentication Tokens","type":"text"}]},{"type":"paragraph","content":[{"text":"The Gateway's S3 protocol uses an external identity management system (IDM) for users and groups, similar to a federation with AWS IAM, and uses an internal system for managing authentication tokens, similar to AWS temporary security credentials. These authentication tokens are created and managed within the Swarm cluster.","type":"text"}]},{"type":"paragraph","content":[{"text":"Authenticated requests to the S3 protocol follow the AWS S3 request signing rules for v2 and v4 signatures whereby each request includes a header in one of the following forms:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Authorization: AWS AccessKey:Signature","type":"text","marks":[{"type":"code"}]},{"text":" (v2 signature)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Authorization: AWS4-HMAC-SHA256 Credential,SignedHeaders,Signature","type":"text","marks":[{"type":"code"}]},{"text":" (v4 signature)","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"The construction of the Authorization header is automatically handled by S3 SDKs and S3 applications. For the elements of the header's value string and the S3 HMAC authentication mechanism, see the ","type":"text"},{"text":"AWS S3 documentation","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs.aws.amazon.com/s3"}}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"Each S3 client needs at least one authentication token to authenticate S3 protocol requests to the Gateway. The Access Key value is the ID of an authentication token.","type":"text"}]},{"type":"paragraph","content":[{"text":"See ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://perifery.atlassian.net/wiki/spaces/public/pages/2443822615"}},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"The ","type":"text"},{"text":"X-User-Secret-Key-Meta","type":"text","marks":[{"type":"code"}]},{"text":" header is required when creating the token object when creating authentication tokens for use with S3 (also referred to as \"S3 authentication tokens\"). The value of this header becomes the Secret Access Key (or \"Secret Key\") used to sign S3 requests. As previously mentioned, the Access Key becomes the token cookie's value returned by the create request.","type":"text"}]},{"type":"paragraph","content":[{"text":"This example shows that an authentication token is created by the user ","type":"text"},{"text":"gcarlin","type":"text","marks":[{"type":"code"}]},{"text":" with an S3 secret key of ","type":"text"},{"text":"abcdefg","type":"text","marks":[{"type":"code"}]},{"text":" and an expiration time of 365 days from now. Note: this request uses HTTP basic authentication to create the token.","type":"text"}]},{"type":"codeBlock","attrs":{"language":"xml"},"content":[{"text":"POST /.TOKEN/ HTTP/1.1 \nAuthorization: Basic Z2NhcmxpbjpmdW5ueQ==\nHost: abc.cloud.example.com\nX-User-Secret-Key-Meta: abcdefg \nX-User-Token-Expires-Meta: +365\nContent-Length: 0","type":"text"}]},{"type":"paragraph","content":[{"text":"This is an excerpt from the Gateway's response.","type":"text"}]},{"type":"codeBlock","attrs":{"language":"xml"},"content":[{"text":"HTTP/1.1 201 Created \nSet-Cookie: token=722bfb49aa8365897a3e774d539038ce; expires=Fri, 06-Jun-2017 18:44:52 GMT; path=/","type":"text"}]},{"type":"paragraph","content":[{"text":"Construct the S3 Authorization header using the following to sign S3 requests with the created token:","type":"text"}]},{"type":"codeBlock","content":[{"text":"AccessKey=722bfb49aa8365897a3e774d539038ce\nSecretAccessKey=abcdefg","type":"text"}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"heading","attrs":{"level":3},"content":[{"text":"Note","type":"text"}]},{"type":"paragraph","content":[{"text":"Tokens that contain an S3 secret key are used to sign S3 storage requests and may not be used to authenticate SCSP storage operations.","type":"text"}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Error Response for Missing Resource","type":"text"}]},{"type":"paragraph","content":[{"text":"The S3 protocol includes an extra XML ","type":"text"},{"text":"Resource","type":"text","marks":[{"type":"code"}]},{"text":" tag in the error response to provide additional details for troubleshooting errors regarding non-existent content. This is an example showing the additional field.","type":"text"}]},{"type":"codeBlock","attrs":{"language":"xml"},"content":[{"text":"\n\tNoSuchKey\n\tThe specified key does not exist.\n\t/mybucket/missingFile.txt \n\t03B8CD915CD6C3A5 \n\tmissingFile.txt\n","type":"text"}]},{"type":"paragraph","content":[{"text":"The format of the ","type":"text"},{"text":"Resource","type":"text","marks":[{"type":"code"}]},{"text":" tag is: \"","type":"text"},{"text":"/{bucketName}/{objectName}","type":"text","marks":[{"type":"code"}]},{"text":"\".","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Multipart Uploads","type":"text"}]},{"type":"paragraph","content":[{"text":"AWS S3 requires every part (except the last) of a multipart upload must be at least 5 MB in size. The Gateway's S3 protocol implementation does not impose this limitation and allows each part of a multipart upload to be of any size.","type":"text"}]},{"type":"paragraph","content":[{"text":"Multipart writes are long-running operations with initial and final responses. For S3 compatibility, the initial response returns ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#091e42"}}]},{"text":"x-amz-version-id","type":"text","marks":[{"type":"code"}]},{"text":" with the value of the expected ETag. There is no new object if there is an error completing the write, and the expected ETag given is not valid. (v6.3)","type":"text","marks":[{"type":"textColor","attrs":{"color":"#091e42"}}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"List after Update Timing","type":"text"}]},{"type":"paragraph","content":[{"text":"After an object is created, the delay before that object appears in a list operation can vary depending upon the Swarm metadata feed batch timeout setting. New objects are available within two seconds following a create when the batch timeout is set to 1 second. The ","type":"text"},{"text":"Amazon S3 documentation","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs.aws.amazon.com/s3"}}]},{"text":" has specific developer guidance about this eventual consistency behavior.","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"PUT Object Copy Metadata","type":"text"}]},{"type":"paragraph","content":[{"text":"The AWS S3 ","type":"text"},{"text":"PUT Object Copy","type":"text","marks":[{"type":"em"}]},{"text":" request creates a duplicate of an existing object and, when the ","type":"text"},{"text":"x-amz-copy-source","type":"text","marks":[{"type":"code"}]},{"text":" header is included with the request, copies the ","type":"text"},{"text":"x-amz-meta-*","type":"text","marks":[{"type":"code"}]},{"text":" custom metadata from the source object.","type":"text"}]},{"type":"paragraph","content":[{"text":"Although Swarm allows for more custom metadata patterns than this, the metadata matching ","type":"text"},{"text":"x-amz-meta-*","type":"text","marks":[{"type":"code"}]},{"text":" pattern is copied during a ","type":"text"},{"text":"PUT Object Copy","type":"text","marks":[{"type":"em"}]},{"text":" operation.","type":"text"}]},{"type":"paragraph","content":[{"type":"inlineExtension","attrs":{"extensionType":"com.atlassian.confluence.macro.core","extensionKey":"excerpt-include","parameters":{"macroParams":{"":{"value":"Object Versioning"},"nopanel":{"value":"true"}},"macroMetadata":{"macroId":{"value":"c17e433a-61fb-4ad9-9157-33248d18c2c1"},"schemaVersion":{"value":"1"},"title":"Insert excerpt"}},"localId":"292a89fe-ab05-40bf-937e-a53900b6a0ea"}}]}],"version":1}

Browser not supported