To work with very large objects or objects of unknown length, you need to use the advanced options that are incorporated in Swarm Elastic Content Protection:

• Erasure Coding EC (EC), which segments and stores large objects efficiently and securely

• Multipart Write, which divides an object into multiple parts and uploads them simultaneously

These are key terms used in Swarm elastic content protection:

Dividing Objects with Erasure Coding

Swarm allows writing large objects of known length using the erasure coding option incorporated in the Swarm Elastic Content Protection. With this option, you can divide the object into smaller segments and encode it with additional parity segments that provide data protection.

Additionally, you can write (POST, PUT, COPY, APPEND) objects of unknown length to a cluster using standard HTTP chunked transfer encoding. Objects sent to the cluster using chunked transfer encoding are erasure-coded when stored on disk, using the encoding type specified by either cluster configuration or request query arguments. This feature allows you to store large objects and streaming media in the cluster.

Storing Large Objects

You can store an object as large as 4TB in the cluster. Erasure coding is seamless and transparent to the application, automatically partitioning the object into segments, encoding them, and distributing the segments throughout the cluster. When you configure the cluster, you set the threshold for when objects become erasure coded; in addition, applications can control which objects get erasure-coded on an individual object basis. See Erasure Coding EC.

Attempting to store an object larger than 4TB results in an HTTP 400 Bad Request response immediately after the write is submitted.

Storing Streaming Media

Streaming media is supported using industry-standard chunked transfer encoding. Your application can now stream digital media or other types of data to the cluster without knowing the object size in advance. The size of the object is limited only by the available space in the cluster (up to 4TB). Attempting to store a chunked encoded object larger than 4TB results in an HTTP 400 Bad Request response (see note above).

Any object written with HTTP chunked transfer encoding must be erasure-coded and cannot be replicated. If you write an object and specify both erasure coding and replication in the header (for example, combining an encoding=5:2 query argument with a lifepoint header with a reps= parameter), the write operation results in an HTTP 400 Bad Request response.