Replication is a proven and valuable mechanism to ensure verify data integrity, but the cost per GB of storage can get become high as object sizes and cluster sizes growexpand. A complementary data protection strategy, erasure coding (EC), provides high data durability with a smaller footprint. Swarm manages EC and replication together to optimize cost-effectiveness, converting objects between them seamlessly and dynamically, based on the policies that you set.
| Table of Contents | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
How EC
...
Works
Erasure coding breaks the original object into multiple data segments (k) and computes additional parity segments (p) based on the content of the data segments. This results in m total segments (k + p = m) typically being being distributed to m different nodes (or subclusters) in the storage cluster (see the ec.protectionLevel setting).
The erasure coding encoding level is expressed as a tuple in this format:
| Code Block | ||
|---|---|---|
| ||
{data segments}:{parity segments} |
For very large objects, Swarm creates multiple sets of erasure segments for very large objects. The object breakdown into one or more erasure sets is transparent to external applications. A GET or HEAD of an erasure-coded object uses the same syntax as a replicated object.
The following illustration represents how erasure coding works:
How
...
Much EC
...
Protects
Swarm still reads the object as long as there are still k total segments (any combination of original data or parity) remaining in the cluster . In other words, the protection against drive if a hard disk or a node containing an erasure segment fails. Protection against disk failure for the object is equal to the number of specified parity p segments.
For example, because the The segments from a 5:2 (5 data segments with 2 parity segments for a total of 7 segments) or 8:2 (8 data and 2 parity segments for 10 total segments) erasure code are distributed to different nodes, they are protected against the loss of any two nodes because they are distributed to different nodes. An erasure-coded object is immediately retrievable when accessed even if some segments are missing. However, regenerating Regenerating the missing erasure set segments is still performed in a self-healing, cluster-initiated manner (similar to the recovery process for replicated objects) to protect against further drive disk loss. This process kicks off automatically when a missing volume is detected and automatically regenerates any missing segments.
Regardless of settings, Swarm always seeks the widest possible distribution of content (whether replicas or EC segments) regardless of settings, to maximize data protection. Swarm Swarm applies the following 2p+2 protection levels as possible:
One segment per subcluster … up to p segments per subcluster
One segment per node … up to p segments per node
One segment per volume
How
...
Much Disk Space EC Saves
The amount of drive disk space (or footprint) used for erasure-coded objects depends on the ratio of data to parity segments in the specified encoding.
Use the following formula to roughly calculate the drive space that you can expect to see disk space expected to be used by an EC object with one set of erasure segments:
(total segments ÷ data segments ) × object size | = object footprint |
|---|---|
((k+p) ÷ k) × GB | = total GB |
How footprint changes with different EC encoding (versus 3 reps)
1 GB object with 5:2 encoding: ((5 + 2) ÷ 5) × 1 GB = 1.4 GB (vs. 3 GB for replication)
3 GB object with 5:2 encoding: ((5 + 2) ÷ 5) × 3 GB = 4.2 GB (vs. 9 GB for replication)
3 GB object with 7:3 encoding: ((7 + 3) ÷ 7) × 3 GB = 4.3 GB (vs. 9 GB for replication)
...
...
Note
Additional system metadata is written with each EC segment
...
which adds about 16 bytes per segment.
See See Hardware Requirements for Storage for how to size and optimize your hardware for erasure coding.
See Cluster Protection Planning for how to determine the required number of nodes and optimize EC performance in your a cluster.children