Implementing Replication Policy

Owned by Rumena Zlatkova (Deactivated)
Last updated: Dec 19, 2023 by Bala Harish A(AC)
5 min read

Based on the context (location in the cluster) policies can be created to control how many copies of a replicated object Swarm maintains. Swarm allows maintaining separate replication policies for the cluster and any domain and bucket. 

Understanding how Swarm chooses among policies is crucial to design policies because Swarm allows specifying different policies at multiple levels simultaneously. Replication specified at the object level have first priority, but the context policies constrain the minimum and maximum, as well as provide the default:

See https://perifery.atlassian.net/wiki/spaces/public/pages/2443821972 for setting lifepoints with replication.

See https://perifery.atlassian.net/wiki/spaces/public/pages/2443821292 for using replication query arguments.

Specifying Replication

Replication is one of several content protection features in Swarm. All are controlled by way of policies. Swarm evaluates policies for each object based on the cluster and context values:

  • Policy-related settings for the cluster (required) 

  • Policy-related headers on domain and bucket objects (optional)

For an overview of Swarm policies and how to set them at the cluster level, see Configuring Cluster Policies.

The replication policy has three parameters. All are required at the cluster level because they establish the baseline for how the cluster replicates. To lock out lower-level policies apply the optional anchored parameter to cluster and domains.

Deprecated

The cluster setting policy.replicas replaces the following three, which are all deprecated: scsp.minReplicas, scsp.maxReplicas, scsp.defaultReplicas

Note

Replication policy does not apply to context (bucket and domain) objects themselves, which follow the cluster configuration settings scsp.defaultContextReplicas and scsp.maxContextReplicas.

Using this 2-part setting define a different replication policy at the cluster and each context (domain and bucket) level:

Parts

Scope

Value

Notes

Parts

Scope

Value

Notes

replicas

required

Affects the current context level.



Swarm returns 400 (Bad request) if value is missing or out of range.



Replaces cluster-level settings:

  • scsp.minReplicas

  • scsp.maxReplicas

  • scsp.defaultReplicas

min:n

1 ≤ n ≤ 16

For the objects in this context, sets the lower limit for the number of replicas required to keep.

Overrides any policy or lifepoint header reps= constraint of a lower value.

Recommendations

  • Use min:1 and write those objects with reps=1, but leave the default reps at 2 or 3 to maintain single replicas of unimportant objects.

  • For reps=3, use the default scsp.defaultROWAction = immediate, unless the write load is heavy and sustained, in which case protect performance by using scsp.defaultROWAction = full.



max:n

min ≤ n ≤ 20

For the objects in this context, sets the upper limit for the number of replicas and for k+p for each EC/ECP write request.

Overrides any policy or lifepoint header reps= constraint of a higher value.

Small clusters: Set this value to the total number of nodes in the cluster, which avoids needless errors
when lifepoint headers request more replicas than there are nodes.

default:n

min ≤ n ≤ max

For the objects in this context, sets the default number of replicas if none is specified by the current lifepoint or request. 

For most cases, min and default values match.

anchored

optional

Affects contexts below the current.

none

Used with cluster and domain policies. Applies the policy to all levels below, overriding any lower-level policies.

Summary of value constraints:

1 ≤ min ≤ default ≤ max ≤ 20

Example: 

Starting with the cluster set the replication policy for each level with the appropriate naming and method:

Scope

Type

Name, SNMP MIB

Configuration Method

Scope

Type

Name, SNMP MIB

Configuration Method

Cluster

Configuration
setting

policy.replicas

policyReplicas

Cluster policies are captured in a single setting including all three required parameters, for a baseline: 

Use the snmpset command on the cluster's settings object to change and persist the cluster-wide policies. See Using SNMP with Swarm.

Domain and
Bucket

Persisted header

Policy-Replicas

policyReplicas

Context (domain and bucket) policies are captured in persisted headers overriding some or all cluster policies. 

Use a write method (POST for new, COPY for update) on the domain object to set the policy values.

Use a read method (GET, HEAD) on the domain object to check the policy values.

Replication is subject to policy resolution on each write request. For contexts, Swarm uses the policy at the given level unless a level above it is anchored. Swarm uses the anchored policy.

A new domain has a separate required policy for named objects (which bucket are unable to override). Tenanted unnamed objects follow this policy regardless of "anchored":

Writing a domain policy

This new bucket increases the maximum, but note this policy is canceled by any policies "anchored" above it at the domain or cluster level:

Writing a bucket policy

How Replication Policies Resolve

The goal of policy resolution is to determine the correct number of replicas to keep for a content object if subject to a set of conflicting policies. This evaluation must be bounded by a minimum and maximum, and, as with all types of Swarm policies, must honor the anchored option, which inhibits evaluation at lower levels. 

Swarm evaluates replication policy according to scope, which is tied to object type (because types can be stored in a certain type of context):

Scope

Object Type

Min = Highest Among

Max = Lowest Among

Default = First Found,
Constrained by Min/Max

Scope

Object Type

Min = Highest Among

Max = Lowest Among

Default = First Found,
Constrained by Min/Max

Cluster

  • Untenanted unnamed

  • Cluster setting

  • Cluster setting

  1. Lifepoint

  2. Cluster setting 

Domain

  • Tenanted unnamed

  • Cluster setting

  • Domain object header

  • Cluster setting

  • Domain object header

  1. Lifepoint

  2. Domain header

  3. Cluster setting 

Bucket

  • Named

  • Cluster setting

  • Domain object header

  • Bucket object header

  • Cluster setting

  • Domain object header

  • Bucket object header

  1. Lifepoint

  2. Bucket header

  3. Domain header

  4. Cluster setting 

Every cluster update must supply all three values (min, max, default), but any subset is allowed for domain, bucket, and unnamed. They can be in any order, and any case.

Affected Objects

Example

Effect

Affected Objects

Example

Effect

cluster

Untenanted
unnamed

policy.replicas: min:2 max:10 default:4

Constrains the lifepoint value for reps: 2 ≤ reps ≤ 10.
Defaults to 4 if reps is unspecified.

domain

Tenanted
unnamed

Policy-Replicas: min:1 max:8

Constrains the lifepoint value for reps: 1 ≤ reps ≤ 8.
Defaults to 4 (from the cluster setting) if reps is unspecified.

bucket

Named

Policy-Replicas: default:2 max:6

Constrains the lifepoint value for reps: 1 (per domain setting) ≤ reps ≤ 6.
Defaults to 2 if reps is unspecified.

Conflict Example

Policies at different levels can conflict, such as in this example:

First Evaluation

= Invalid

First Evaluation

= Invalid

Cluster: min:1 max:10

  • Domain:  min:2 max:5

    • Bucket:  min:6 max:8

Most constrained values:

  • min:6 max:5

The most restrictive min is 6, and the most restrictive max is 5, which is not a valid pair, because the max is lower than the min. Given such a conflict, Swarm logs a warning message and reevaluates the policy, omitting the lowest level in the hierarchy (bucket, in this example), until the conflict is resolved. The lowest level is dropped because Swarm privileges the values of the cluster owner over those of the domain owner, over those of the bucket owner.

First Evaluation

= Invalid

Second Evaluation

= Valid

First Evaluation

= Invalid

Second Evaluation

= Valid

Cluster: min:1 max:10

  • Domain:  min:2 max:5

    • Bucket:  min:6 max:8

Most constrained values:

  • min:6 max:5

Cluster: min:1 max:10

  • Domain:  min:2 max:5

    • Bucket: min:6 max:8

Most constrained values:

  • min:2 max:5





© DataCore Software Corporation. · https://www.datacore.com · All rights reserved.