The individual parts of a larger object are ready to upload when the initiate request is complete. Swarm allows any number of parts in a multipart upload. (v9.1)
Create a POST request with the object name used in the initiate request, the upload ID returned from the initiate request, and a unique part number for each part to upload a part.
The part uploads for an upload ID must include the same domain query argument if the initiate request included a domain query argument for a specific domain. The part uploads must not include an encoding query argument as they inherit whatever was specified in the initiate request unlike the initiate request. A failed part upload can be retried without affecting the outcome of the multipart upload.
A part is stored as an immutable object whose Content-UUID is returned in the request response when it is successfully uploaded. Client applications must keep track of the part number used for the upload and the Content-UUID Swarm assigned when stored to eventually complete the multipart upload, as described below.
Parts are unnamed
Each part is an immutable object that returns a Content-UUID even if the initiated object is named, even though a POST on a named object does not ordinarily return that header. The parts are tenanted in the same domain as the destination object, but parts are unnamed, so they cannot reside in buckets.
Using object part numbers
Swarm uses part numbers to identify the position of each part in an object. Include the upload ID and a unique part number for each part so Swarm can assemble the parts in the correct order when uploading parts. Non-sequential part numbers for each part can be selected (2, 4, 6, 8), but Swarm assembles the parts in sequential order.
Record each part number and corresponding Content-UUID. This information is required to complete the multipart write procedure.
Uploading a part
The following must be included to upload each part:
The object name or UUID (or Content-UUID, for immutable objects) returned by the initiate request
The upload ID returned from the initiate request
A unique part number for each part uploaded
POST /ObjectNameorUUID?partNumber=PartNumber&uploadId=UploadId HTTP/1.1 Host: cluster.example.com User-Agent: Swarm Client/0.1 Content-Length: 43402 Expect: 100-continue [ content ]
The content to be uploaded for the part is placed in the body of the request, like a normal POST operation.
Uploading a part by copying from an existing object
Create a POST request that uses the content from existing objects if the required parts currently exist in the storage cluster. Swarm creates an EC copy of the object for the multipart write when the part copy request is completed. The request fails if the source object does not exist or cannot be read from the specified range. This process leaves the source and destination versions unrelated to each other.
Content Gateway
The user making the "PUT with copy" request must have read access to the source object when going through Gateway.
The additional headers used in this request specify the source object and the range in bytes. The source object must be specified in the x-castor-copy-source header by UUID or object name in bucket/object-name format:
curl -i "$HOST/fd9cf39f056fb0dd858d8fb288c22885?PartNumber=3 &UploadID=ddd080eb400bd5531f580191e3c5a916dd66c7c1e3244dc6cad46183097677e6dd66c7c1e3244dc6cad46183097677e60P" -XPOST -H "x-castor-copy-source: a08212d59b5bd306a52008dfef335be2" -H "x-castor-copy-source-range: 5-8" HTTP/1.1 201 Created Location: http://192.168.1.171:80/09938e338c3590b93855d7cca2179aec Location: http://192.168.1.109:80/09938e338c3590b93855d7cca2179aec Volume: 3f5ef63dab992ebcf28e092bb56103c3 Volume: 12e08e29145f277501a6490b602ea287 Manifest: ec Castor-System-UploadID: ddd080eb400bd5531f580191e3c5a916dd66c7c1e3244dc6cad46183097677e6dd66c7c1e3244dc6cad46183097677e60P Content-UUID: 09938e338c3590b93855d7cca2179aec Last-Modified: Tue, 27 Sep 2016 20:49:46 GMT Entity-MD5: 9g0GoVLSYSXc/PMI4FWKbQ== Stored-Digest: f60d06a152d26125dcfcf308e0558a6d Castor-System-Encoding: zfec 1.4(1, 1, 524288, 200000000) Castor-System-Version: 1475009386.549 Etag: "4c760a34ee534bcdba91680919378e2e" Content-Range: bytes 5-8/10 Replica-Count: 2 Date: Tue, 27 Sep 2016 20:49:46 GMT Server: CAStor Cluster/8.2.a Content-Length: 46 Content-Type: text/html Keep-Alive: timeout=14400 <html><body>New stream created</body></html>
The Content-MD5 is applied to the range read if a gencontentmd5 query argument (or the deprecated Expect: Content-MD5 header) is applied to a part copy with a range read.
These are the arguments and headers that are required for a part upload request that copies data from an existing object:
POST /ObjectNameorUUID?partNumber=PartNumber&uploadId=UploadId HTTP/1.1 x-castor-copy-source: uuid/name x-castor-copy-source-domain: domain_name
Response Headers
The result code for the operation (which always responds with chunked encoding) is in the trailing header Castor-System-Result when the x-castor-copy-source header is used.
See "Response Headers for Multipart Writes" in Completing the Multipart Write.
-copy-source- headers
The following headers in the POST request for part uploads are optional, except for the first. They perform the same filtering as the regular headers of those names (range, if-match, etc.) performed against the source object being copied if included.
Note
Error responses on conditional headers come back immediately, in place of a 202 (Accepted for processing) response. Condition failures (such as the ETags not matching) are reported in the initial HTTP response, not the castor-system-result header.
Type | Header | Notes |
|---|---|---|
Source | x-castor-copy-source | Required. Must be a valid name or UUID. |
Domain | x-castor-copy-source-domain | Required (unless untenanted) |
Range | x-castor-copy-source-range: bytes=first-last | If the range values are out of bounds for the data, the request returns 416 (Range Not Satisfiable). Tip: To copy from a start range to the end of the object, omit the end range. (v11.1) |
Conditional | x-castor-copy-source-if-match: "<ETag>" | The ETag must be enclosed in quotes. |
x-castor-copy-source-if-unmodified-since: <timestamp> | Uses the format of the standard HTTP last-modified header. |
Validating the uploaded parts
Include a gencontentmd5 query argument or Content-MD5 header in the POST argument to return a Content-MD5 header to validate the content of the uploaded part. See Content-MD5 Checksums. (v9.2)
Verify each part has an MD5 stored with it if intending to validate the full transfer on the complete using the Composite-Content-MD5 header.