| Table of Contents |
|---|
To work with very large objects or objects of unknown length, you need to use the advanced options that are incorporated in Swarm Elastic Content Protection:
Erasure coding (EC), which segments and stores large objects efficiently and securely
Multipart Write, which divides an object into multiple parts and uploads them simultaneously
These are key terms used in Swarm elastic content protection:
Chunked transfer encoding | Used in WRITE, UPDATE, and APPEND SCSP methods to send objects of an undetermined content length to a storage cluster. The exact request header is:
See RFC 7230 3.3.1.
|
|---|
| |
Erasure coding | Describes one of the ways an object can be protected in a storage cluster. A large object written to the cluster using erasure coding is automatically stored on disk as a set of data and parity segments. This process ensures both content protection and optimal storage usage for large objects. Swarm has configuration parameters that enable an object to be automatically erasure-coded on the drive. |
|---|---|
Manifest | Swarm object containing a list of the segments that comprise a large object. |
Dividing Objects with Erasure Coding
...
Attempting to store an object larger than 4TB will result in a 400 Bad Request response immediately after the write is submitted.
| Info |
|---|
Increasing allowed object sizeTo store objects larger than 4TB, increase the limit that is set by ec.maxSupported (defaults to 4398046511104) and also set ec.segmentSize (defaults to 200000000) to a value proportionately larger. On a full read, Swarm must load the entire manifest; increasing the segment size minimizes the size of the manifest and so the number of socket connections required to read an entire EC object. (SWAR-7823) |
Storing Streaming Media
Streaming media is supported using industry-standard chunked transfer encoding. Your application can now stream digital media or other types of data to the cluster without knowing the object size in advance. The size of the object is limited only by the available space in the cluster (up to 4TB). Attempting to store a chunked encoded object larger than 4TB will result in a 400 Bad Request response (see note above).
...