Configuring ROW Replicate On Write

Owned by Rumena Zlatkova (Deactivated)
Last updated: Dec 12, 2023 by Bala Harish A(AC)
4 min read

Swarm creates all replicas requested at once, in parallel when executing a Replicate On Write (ROW) command for data protection. Objects are safe (replicated on other nodes) using ROW even if a disk failure occurs immediately after a write or update completes.

ROW versus HP for Replication

Without ROW, the Health Processor (HP) manages all replication in the background, the HP checks its replication constraint when writing an object. The HP creates a duplicate of the updated version and deletes the older replicas on the cluster nodes when updating an existing object.

With ROW, Swarm creates another instance of the object on another node immediately, as part of the write. ROW verifies two or more object replicas (instances) exist in the cluster before the client write request is completed. If the object includes a constraint for creating more copies, those additional copies are made during the normal health checking process. While ROW may temporarily restrict client responsiveness as objects are created and replicated at the same time, your objects are protected from a single volume failure right away.

ROW Benefits

ROW Effects

ROW Benefits

ROW Effects

  • Immediately protects an object from a single-volume failure rather than waiting
    for the Health Processor to duplicate the object during normal operations.

  • Quickly deletes older replicas to verify all versions are current.

  • Quickly invalidates cached domain versions in the cluster so the latest version is implemented.

  • Takes a bit longer to write than a single replica.

  • Is more likely to fail because of the additional preconditions.

ROW Commands

ROW is enabled by default. ROW-related commands can be issued in these ways:

Configuration

Disable ROW with this setting in node.cfg. (See https://perifery.atlassian.net/wiki/spaces/public/pages/2443811510)

scsp.replicateOnWrite = false

SNMP

Set autoRepOnWrite=0 to disable the setting without restarting the cluster. (See https://perifery.atlassian.net/wiki/spaces/public/pages/2443811583)

Important

Once autoRepOnWrite is been changed using SNMP, it is stored in the persistent settings. Any future changes must be using SNMP or the Swarm UI to override that setting.

Query Argument

Use a replicate query argument when needing to override the cluster-wide ROW configuration. (See https://perifery.atlassian.net/wiki/spaces/public/pages/2443821675) 

ROW enabled: To override the cluster defaults, use the query argument with an integer. The most common usage, replicate=1, allows the write to succeed with only one instance of the object, which effectively disables ROW for the write:

POST /?replicate=1

ROW Disabled: If you have ROW disabled in the cluster, you can achieve ROW by adding the query argument with immediate or full to each write:

POST /?replicate=immediate POST /?replicate=full

Note

The replicate=immediate option quickly invalidates cached bucket versions in the cluster so the latest version is implemented in the cluster when creating or updating a bucket. It also prevents subsequent permission errors because out-of-date permissions are used from the prior version.

Configuration Settings

Following are the configuration settings that control replication for the storage cluster.

scsp.replicateOnWrite

ROW is enabled by default. You can disable ROW in the storage cluster by changing the setting to scsp.replicateOnWrite = false.

The settings scsp.replicateOnWrite and scsp.defaultROWAction are evaluated together.  When scsp.replicateOnWrite is true (the default), Swarm withholds sending the write verification to the client until the number of replicas specified by the scsp.defaultROWAction setting (default 2) are created successfully. 

Important

Any query arguments in the request override the configuration settings.

scsp.defaultROWAction

scsp.defaultROWAction is a positive integer or full (a special keyword). Full indicates that the number of replicas is the number determined by the greater of the object's reps lifepoint or the policy.replicas default setting for that object. The limits defined by the cluster policy.replicas min and policy.replicas max settings are enforced on the number of replicas created in a ROW request as well.

The replicate query argument serves the same purpose with the similar acceptable values.

See .

policy.replication

Your replication policies (policy.replicas: min:# max:# default:# [anchored]) and the policy evaluation process affect how Swarm applies ROW. These settings define the baseline range of allowable replicas:

Replication Configured

Effect

Value

Description

Replication Configured

Effect

Value

Description

scsp. replicateOnWrite

lower limit

true | false

Swarm starts evaluation with the value of replicateOnWrite: 1 replica if false and 2 replicas if true (the default).

policy.replicas max

upper limit

min ≤ n ≤ 20

Swarm determines how many replicas are made for an object and limits the SCSP request to that number. The replication policy default is the key value: if it is 2 and the request asks for more, it receives 2 on the write.

When the replicate query argument is used, the remaining policy.replication parameters have these effects:

Replication Requested

Meaning

Effect on Evaluation

Evaluates to

Description

Replication Requested

Meaning

Effect on Evaluation

Evaluates to

Description

replicate=immediate

For use if ROW is disabled in the cluster. Only two replicas must be created before Swarm sends the response; if more replicas are required, those additional replicas are created after the response.

lower limit

2

Only 2 replicas get created synchronously.

replicate=full

For use if ROW is disabled in the cluster. All required replicas must be created before Swarm sends the response.

default value

policy.replicas default

The policy.replicas default evaluated for the object determines the number of replicas Swarm creates synchronously.

replicate={integer}

How many replicas must be created synchronously for this write, overriding cluster settings (within constraints). If policy.replicas max or the lifepoint reps is less than this integer, Swarm uses the smaller of those values.

default limit

{integer} ≤ policy.replicas default

The policy.replicas default evaluated for the object serves as an upper limit on {integer} to determine the number of replicas Swarm creates synchronously.

See  and .

© DataCore Software Corporation. · https://www.datacore.com · All rights reserved.