{"type":"doc","content":[{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"toc","parameters":{"macroParams":{"minLevel":{"value":"1"},"maxLevel":{"value":"2"},"outline":{"value":"false"},"type":{"value":"list"},"printable":{"value":"false"}},"macroMetadata":{"macroId":{"value":"f9852bcc-5fb5-4b3b-9986-e60915ccb403"},"schemaVersion":{"value":"1"},"title":"Table of Contents"}},"localId":"af9d6d56-aa09-4b56-870a-fa4295a6c92e"}},{"type":"bodiedExtension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"excerpt","parameters":{"macroParams":{"hidden":{"value":"true"},"atlassian-macro-output-type":{"value":"INLINE"}},"macroMetadata":{"macroId":{"value":"fc4a9b84-d3be-4dce-b76f-da4544a59a26"},"schemaVersion":{"value":"1"},"title":"Excerpt"}},"localId":"8901f865-d2db-4eea-addc-18854ed66ea5"},"content":[{"type":"paragraph","content":[{"text":"Internal use:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"PRD","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs.google.com/document/d/1zPl2fykWnm8ZudcY8zf9N49R5OlJfivK6RemojR4730/edit"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"SCSP SEND","type":"text","marks":[{"type":"link","attrs":{"href":"https://perifery.atlassian.net/wiki/spaces/public/pages/2443821576"}}]}]}]}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"What is Synchronous Write to Remote Sites?","type":"text"}]},{"type":"paragraph","content":[{"text":"Swarm's ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://perifery.atlassian.net/wiki/spaces/public/pages/2443814668"}},{"text":" are mature and resilient, with each feed syncing content iteratively and asynchronously, linking one source cluster to one target destination. Replication feeds keep target clusters continuously updated with object changes in the source cluster, and having multiple replication feeds allows the distribution of content across a set of remote Swarm clusters. Remote synchronous write is limited by inter-cluster bandwidth and the constraints of overall request timeouts. Hence, it should not be thought of as a synchronization guarantee. Remote synchronous write are built on the Swarm asynchronous feed, which acts as a safety net to ensure the eventual replication of new writes that may not have been synchronously replicated.","type":"text"}]},{"type":"paragraph","content":[{"text":"Starting with Swarm 12, configure individual domains and buckets to perform an object write across all remote sites at the same time. Write completion to a given bucket or domain is delayed until ","type":"text"},{"text":"every","type":"text","marks":[{"type":"em"}]},{"text":" remote replication is completed when remote synchronous writes are enabled. There are two situations that require synchronous write to remote sites:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"An application requires assurance that backups (replicas) are committed to every site.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The content publication needs newly added content to be readable from ","type":"text"},{"text":"any","type":"text","marks":[{"type":"em"}]},{"text":" remote site immediately upon the write completion.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Remote Site Synchronous Write","type":"text","marks":[{"type":"em"}]},{"text":" or ","type":"text"},{"text":"Remote Synchronous Write","type":"text","marks":[{"type":"em"}]},{"text":" is the option to have a client create or update a Swarm object and have that change committed in ","type":"text"},{"text":"all active remote","type":"text","marks":[{"type":"em"}]},{"text":" clusters immediately before the client receives a success code. An object written this way has the same UUID and metadata across all Swarm clusters as if it were replicated normally.","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"heading","attrs":{"level":3},"content":[{"text":"RSW","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Synchronous Write to Remote Sites is the same as ","type":"text"},{"text":"RSW","type":"text","marks":[{"type":"strong"}]},{"text":". All commands and log messages use the acronym ","type":"text"},{"text":"RSW","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"},{"type":"hardBreak"},{"text":"This is separate from ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://perifery.atlassian.net/wiki/spaces/public/pages/2443811678"}},{"text":" , which guarantees immediate replication within the ","type":"text"},{"text":"same","type":"text","marks":[{"type":"em"}]},{"text":" cluster.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"RSW is implemented in Gateway using the SCSP SEND method. RSW requires writing to Gateway, and Gateway uses the ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://caringo.atlassian.net/wiki/spaces/public/pages/2443821576"}},{"text":" method for the Swarm nodes. This allows replicating objects normally.","type":"text"}]}]}]}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"heading","attrs":{"level":3},"content":[{"text":"Warning","type":"text"}]},{"type":"paragraph","content":[{"text":"Write completion to a given bucket or domain is delayed until ","type":"text"},{"text":"every","type":"text","marks":[{"type":"em"}]},{"text":" remote ","type":"text"},{"text":"replica","type":"text","marks":[{"type":"strong"}]},{"text":" is completed when remote synchronous writes are enabled. RSW might require increasing ","type":"text"},{"text":"threads","type":"text","marks":[{"type":"code"}]},{"text":" and ","type":"text"},{"text":"maxConnections","type":"text","marks":[{"type":"code"}]},{"text":" / ","type":"text"},{"text":"maxConnectionsPerRoute","type":"text","marks":[{"type":"code"}]},{"text":" configuration items (","type":"text"},{"type":"inlineCard","attrs":{"url":"https://perifery.atlassian.net/wiki/spaces/public/pages/2443810201/Gateway+Configuration#%5Bgateway%5D"}},{"text":"). Monitor replication status (","type":"text"},{"text":"rswfeeds","type":"text","marks":[{"type":"code"}]},{"text":") and timings (","type":"text"},{"text":"rswtime","type":"text","marks":[{"type":"code"}]},{"text":") in cloudgateway_audit.log (","type":"text"},{"type":"inlineCard","attrs":{"url":"https://perifery.atlassian.net/wiki/spaces/public/pages/2443810251/Gateway+Logging#Audit-Logging"}},{"text":" and","type":"text"},{"type":"inlineCard","attrs":{"url":"https://perifery.atlassian.net/wiki/spaces/public/pages/2443822661"}},{"text":"). Note Gateway 8.0.4 and later fixes issues with connection handling that could lead to ","type":"text"},{"text":"Too many open files","type":"text","marks":[{"type":"code"}]},{"text":" errors when replication fails.","type":"text"}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Implementing Remote Targets","type":"text"}]},{"type":"paragraph","content":[{"text":"A remote synchronous write engages as many replication targets as are active ","type":"text"},{"text":"at the time of the write","type":"text","marks":[{"type":"em"}]},{"text":". Start using remote synchronous write with one remote cluster and add additional remote clusters later. Remove or change remote sites in the future while continuing to use RSW.","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Complete the upgrade to Swarm 12 or higher, including the latest versions of Content Gateway, Content UI, and Swarm Storage. Current versions are required for this feature, but no additional components need to be installed.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Complete provisioning and initialization for the additional remote Swarm clusters participating in remote synchronous writing.","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Content protection policies","type":"text","marks":[{"type":"link","attrs":{"href":"https://perifery.atlassian.net/wiki/spaces/public/pages/2443812020"}}]},{"text":" do ","type":"text"},{"text":"not","type":"text","marks":[{"type":"em"}]},{"text":" need to match across all clusters. Versioning may not be enabled (which enlarges the footprint) on any but the primary cluster.","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Create separate replication feeds for each of the remote Swarm clusters in the primary (source) cluster. See ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://perifery.atlassian.net/wiki/spaces/public/pages/2443814668"}},{"text":" for more details.","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Navigate to ","type":"text"},{"text":"Cluster > Feeds","type":"text","marks":[{"type":"strong"}]},{"text":", select ","type":"text"},{"text":"+ Add","type":"text","marks":[{"type":"strong"}]},{"text":", and select ","type":"text"},{"text":"Replication","type":"text","marks":[{"type":"strong"}]},{"text":" in the Swarm UI.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Leave the legacy setting ","type":"text"},{"text":"Propagate Deletes","type":"text","marks":[{"type":"strong"}]},{"text":" enabled.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Choose ","type":"text"},{"text":"Replicate via direct POST","type":"text","marks":[{"type":"strong"}]},{"text":" rather than the legacy GET mode for the remote cluster's ","type":"text"},{"text":"Replication Mode","type":"text","marks":[{"type":"strong"}]},{"text":". ","type":"text"}]}]}]}]}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"heading","attrs":{"level":3},"content":[{"text":"Note","type":"text"}]},{"type":"paragraph","content":[{"text":"Replication-type feeds with a Status of ","type":"text"},{"text":"Active","type":"text","marks":[{"type":"strong"}]},{"text":" participate in RSW if enabled.","type":"text"}]}]},{"type":"mediaSingle","attrs":{"layout":"center"},"content":[{"type":"media","attrs":{"width":1347,"id":"5a09ab7e-0349-440b-b647-84205b3c8af8","collection":"contentId-2443818831","type":"file","height":337}}]},{"type":"orderedList","attrs":{"order":4},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Set up replication feeds in the remote clusters if the remote clusters are accepting content to be mirrored ","type":"text"},{"text":"back","type":"text","marks":[{"type":"em"}]},{"text":" to the primary cluster.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Verify that the feeds are running successfully in each of the target clusters, populating objects from the source cluster.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Enabling Remote Synchronous Writes","type":"text"}]},{"type":"paragraph","content":[{"text":"Click the ","type":"text"},{"text":"Settings","type":"text","marks":[{"type":"strong"}]},{"text":" (gear icon) and set the ","type":"text"},{"text":"Properties","type":"text","marks":[{"type":"strong"}]},{"text":" to be in force for that domain or bucket in the Content UI. The policy change takes effect on ","type":"text"},{"text":"Save","type":"text","marks":[{"type":"strong"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"Constraints:","type":"text","marks":[{"type":"strong"}]},{"text":" As with the Storage Policies (replicas, encoding, versioning) for domains and buckets, this setting is constrained by the cluster's configuration. The constraints are whether the cluster has any replication feeds that are configured and whether they are in ","type":"text"},{"text":"Active","type":"text","marks":[{"type":"code"}]},{"text":" status for Remote Sites; no synchronous writes occur if there are not any remote clusters being replicated.","type":"text"}]},{"type":"table","attrs":{"layout":"full-width","width":1800.0,"localId":"9d2b3608-33b5-4c7d-b23a-9d4971b58219"},"content":[{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[48.0]},"content":[{"type":"mediaSingle","attrs":{"layout":"align-start","width":45.0},"content":[{"type":"media","attrs":{"width":627,"id":"75fcd7dd-865d-4ff9-948b-819c1f7b8680","collection":"contentId-2443818831","type":"file","height":790}}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[1423.0]},"content":[{"type":"heading","attrs":{"level":3},"content":[{"text":"Domains","type":"text"}]},{"type":"paragraph","content":[{"text":"The option ","type":"text"},{"text":"must","type":"text","marks":[{"type":"em"}]},{"text":" be enabled at this level if synchronous writes to remote sites to include ","type":"text"},{"text":"unnamed objects","type":"text","marks":[{"type":"strong"},{"type":"em"}]},{"text":" (which do not reside in buckets) is desired.","type":"text"}]},{"type":"paragraph","content":[{"text":"The best practice is to enable the option at the level of those buckets if synchronous writes for certain buckets are desired.","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Synchronous Write Enabled:","type":"text","marks":[{"type":"strong"}]},{"text":" Toggle to explicitly enable or discontinue the feature.","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"For a domain, enabling it means that ","type":"text"},{"text":"each of its buckets","type":"text","marks":[{"type":"em"}]},{"text":" also default to having it enabled, unless inheritance is overridden.","type":"text"}]}]}]}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[48.0]},"content":[{"type":"mediaSingle","attrs":{"layout":"align-start","width":45.0},"content":[{"type":"media","attrs":{"width":612,"id":"7861b057-d865-4394-8e00-cbb82cda7a30","collection":"contentId-2443818831","type":"file","height":763}}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[1423.0]},"content":[{"type":"heading","attrs":{"level":3},"content":[{"text":"Buckets","type":"text"}]},{"type":"paragraph","content":[{"text":"Disable ","type":"text"},{"text":"Inherit Policy","type":"text","marks":[{"type":"strong"}]},{"text":" and explicitly enable it for ","type":"text"},{"text":"each ","type":"text","marks":[{"type":"em"}]},{"text":"affected bucket unless enabling synchronous write at the domain level:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Inherit Policy:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"(default)","type":"text","marks":[{"type":"em"}]},{"text":" Select to accept the domain's policy, which is disabled by default until the domain ","type":"text"},{"text":"explicitly","type":"text","marks":[{"type":"em"}]},{"text":" enables it.","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"With inheritance off, the bucket does not respond to domain-level changes. ","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Synchronous Write - Enabled:","type":"text","marks":[{"type":"strong"}]},{"text":" Toggle to explicitly enable or disable the feature.","type":"text"}]}]}]}]}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Managing RSW Manually","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"heading","attrs":{"level":3},"content":[{"text":"Required","type":"text"}]},{"type":"paragraph","content":[{"text":"PutPolicy","type":"text","marks":[{"type":"code"}]},{"text":" and ","type":"text"},{"text":"GetPolicy","type":"text","marks":[{"type":"code"}]},{"text":" permissions are required to set or change RSW settings.","type":"text"}]}]},{"type":"paragraph","content":[{"text":"Manage and verify remote synchronous write settings from the ","type":"text"},{"text":"command line","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"78be33e8-c876-4593-a6a0-d03a218b791d"}}]},{"text":":","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"To ","type":"text"},{"text":"enable","type":"text","marks":[{"type":"strong"}]},{"text":" remote synchronous write, run a curl command that specifies ","type":"text"},{"text":"one domain or bucket","type":"text","marks":[{"type":"em"}]},{"text":" that is included in remote synchronous writes:","type":"text"}]},{"type":"codeBlock","content":[{"text":"curl -X PUT http:///_admin/manage/tenants/_system/domains//rsw?state=enabled --anyauth -u admin:password \ncurl -X PUT http:///_admin/manage/tenants/_system/domains//buckets//rsw?state=enabled --anyauth -u admin:password ","type":"text"}]},{"type":"paragraph","content":[{"text":"Repeat the ","type":"text"},{"text":"RS","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"9ff1220d-75d4-4938-8246-29a580a22a12"}}]},{"text":"W enabling command on each remaining domain and/or bucket that is part of remote synchronous write.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"To ","type":"text"},{"text":"disable","type":"text","marks":[{"type":"strong"}]},{"text":" remote synchronous write from the command line, run a curl command that specifies ","type":"text"},{"text":"one domain or bucket","type":"text","marks":[{"type":"em"}]},{"text":" that is excluded in remote synchronous writes:","type":"text"}]},{"type":"codeBlock","content":[{"text":"curl -X PUT http:///_admin/manage/tenants/_system/domains//rsw?state=disabled --anyauth -u admin:password \ncurl -X PUT http:///_admin/manage/tenants/_system/domains//buckets//rsw?state=disabled --anyauth -u admin:password ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"To ","type":"text"},{"text":"inherit ","type":"text","marks":[{"type":"strong"}]},{"text":"the domain setting again after having had it disabled on the bucket, change the state back to ","type":"text"},{"text":"unset","type":"text","marks":[{"type":"code"}]},{"text":":","type":"text"}]},{"type":"codeBlock","content":[{"text":"curl -X PUT http:///_admin/manage/tenants/_system/domains//buckets//rsw?state=unset --anyauth -u admin:password ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"To ","type":"text"},{"text":"verify","type":"text","marks":[{"type":"strong"}]},{"text":" the RSW status of a given domain or bucket, run the RSW command:","type":"text"}]},{"type":"codeBlock","content":[{"text":"curl -I http:///_admin/manage/tenants/_system/domains//rsw \ncurl -I http:///_admin/manage/tenants/_system/domains//buckets//rsw ","type":"text"}]}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Logging for Remote Writes","type":"text"}]},{"type":"paragraph","content":[{"text":"A file is completely written to the original cluster and then sent to the remote clusters before the client is notified of completion, so this incurs some latency for the request when a large file is written with remote synchronization. The new object is accessible from the source cluster even if errors prevent immediate replication in any remote clusters.","type":"text"}]},{"type":"paragraph","content":[{"text":"In Gateway log messages, find Remote Synchronous Write statuses by looking for the prefix \"","type":"text"},{"text":"RSW: ","type":"text","marks":[{"type":"code"}]},{"text":"\". ","type":"text"}]},{"type":"paragraph","content":[{"text":"At the ","type":"text"},{"text":"Info ","type":"text","marks":[{"type":"strong"}]},{"text":"level of logging, a remote replication report displays:","type":"text"}]},{"type":"codeBlock","content":[{"text":"2020-06-17 23:59:03,396 INFO [qtp412788346-59|9403B57E570AFA2C] RSW: domain1/bucket1/object1 replicated 2 of 2 feeds in 51.63s ","type":"text"}]},{"type":"paragraph","content":[{"text":"When enabling ","type":"text"},{"text":"Debug ","type":"text","marks":[{"type":"strong"}]},{"text":"levels (see ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://perifery.atlassian.net/wiki/spaces/public/pages/2443810251"}},{"text":"), view all detail statuses, which show that the remote writes are resolved before the RSW completes:","type":"text"}]},{"type":"codeBlock","content":[{"text":"2020-06-17 23:59:03,396 DEBUG [qtp412788346-59|9403B57E570AFA2C] RSW: Received RSW final response from swarm: 200\n2020-06-17 23:59:03,396 DEBUG [qtp412788346-59|9403B57E570AFA2C] RSW: -- header Feed-1-Status = 0\n2020-06-17 23:59:03,396 DEBUG [qtp412788346-59|9403B57E570AFA2C] RSW: -- header Feed-1-StatusTime = Wed, 17 Jun 2020 23:59:03 GMT\n2020-06-17 23:59:03,396 DEBUG [qtp412788346-59|9403B57E570AFA2C] RSW: -- header Feed-2-Status = 0\n2020-06-17 23:59:03,396 DEBUG [qtp412788346-59|9403B57E570AFA2C] RSW: -- header Feed-2-StatusTime = Wed, 17 Jun 2020 23:58:31 GMT\n2020-06-17 23:59:03,396 INFO [qtp412788346-59|9403B57E570AFA2C] RSW: domain1/bucket1/object1 replicated 2 of 2 feeds in 51.63s\n2020-06-17 23:59:03,396 DEBUG [qtp412788346-59|9403B57E570AFA2C] ScspProxyRequestHandler: RSW Result = rsw 2/2 ok ","type":"text"}]},{"type":"paragraph","content":[{"text":"The feed numbers refer to the ","type":"text"},{"text":"ID","type":"text","marks":[{"type":"strong"}]},{"text":" field of the feed definition. ","type":"text"},{"text":"Feed-1-Status = ","type":"text","marks":[{"type":"code"}]},{"text":" is a count; how many objects are ","type":"text"},{"text":"remaining","type":"text","marks":[{"type":"em"}]},{"text":" for the feed to handle.","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"0 means the feed is empty (completed).","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"1 means a timeout occurred.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"2 or higher indicates an error.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"The ","type":"text"},{"text":"RSW result","type":"text","marks":[{"type":"strong"}]},{"text":" is 1/2 or 0/2 instead of 2/2 if anything remains in the feed that cannot be remotely written.","type":"text"}]},{"type":"paragraph"}],"version":1}

Browser not supported