Replication Feeds
Using Replication Feeds
Replication feeds enable the replication of objects from one or more domains or the entire cluster to another cluster. Replication provides extra protection and can be a crucial component of a high-availability design. Replication occurs as a background process that is considered "best effort" and subject to inter-cluster bandwidth. For those who require more deterministic replication, a remote synchronous write feature can be employed in gateways serving the source cluster of a replication feed.
Types of Replication
What type of replication method is chosen and how to configure it depends on whether a legacy Swarm implementation exists and on the needs for securing replication traffic over untrusted networks. (v10.0)
Secure Replication: Swarm Storage supports remote replication over a WAN, so replication feeds can operate through the Content Gateway. When a replication feed is defined, specify which replication mode to use: either the legacy bidirectional GET method of replication (which may be needed for specific application compatibility or network requirements) or the recommended direct POST method, which offers better performance and flow management. With Swarm Storage 10.0 and later, TLS/SSL security can be implemented as it fits the implementation:
Upload a trusted certificate to Swarm
Replicate to an SSL offloader that services the target cluster
Replicate from a forward proxy on the source cluster.
See https://perifery.atlassian.net/wiki/spaces/public/pages/2443814996 and https://perifery.atlassian.net/wiki/spaces/public/pages/2443815713.
Replication Methods: Below are two replication methods available, along with the configuration variants of each that are supported. For best performance, choose direct POST replication, which can go through Gateway. GET replication is the legacy method, which may be needed for application compatibility or networking requirements.
Note
Using the legacy Bidirectional GET for remote replication requires populating the Storage configuration setting cluster.proxyIpList
for any cluster using a reverse proxy. The setting is a comma-separated list of reverse proxy IP addresses or names, including ports in name:port
format. If using Direct POST replication, this setting can be populated or left blank, as it has no effect.
Adding a Replication Feed
To add a feed to the cluster, click the +Add button at the top right of the page and then select the Replication button.
When a replication feed is defined, set the scope and select which type (Replication Mode) is in force and with what speed (number of concurrent Threads), if using direct POST:
The following table describes the data entry fields in the dialog box.
ID (Existing feeds) | Read-only; system-assigned identifier |
---|---|
Status (Existing feeds) | Read-only; the current feed processing state. |
Name | The name attached to a feed. |
Scope | The feed filters that were selected for the replication feed. The object is replicated to the domain(s) indicated in this field. Gateway adminDomainDo not create the same domain in two clusters; always create it in the source cluster and then replicate it to the target. A Gateway must use an independent adminDomain, at least temporarily, if the Gateway is in front of the target cluster. (CLOUD-2785)
The field value allows pattern matching with the Python regular expression (RE) syntax, so multiple domain names can be matched. The exception to the RE matching is that the "{m,n}" repetition qualifier may not be used. An example domain list value using RE is:
|
Target Remote Cluster Name | The configuration setting for the target cluster (the configure the Gateway setting |
Proxy or Host(s) | The IP address(es) or host name(s) of either:
Enter each address with a comma or space between them to enter two or more node IP addresses. |
Port | Defaults to 80. Allows specifying a custom port for the remote cluster. |
Replication Mode | Defaults to Direct POST. Choose replication via direct POST (recommended) or bidirectional GET. Switching modes does not require a feed restart. (v9.6) For best performance, choose direct POST replication, which can go through Gateway. GET replication is the legacy method, which may be needed for application compatibility or networking requirements. |
Threads | Replication via direct POST. The default replication speed (20 simultaneous threads) is best for same-sized clusters with minimal replication backlog. When processing backlog, the new default push thread count is 20 per node (in releases prior to v15.0, it was 6 per volume) Reduce the threads to avoid overwhelming a smaller target cluster. For faster replication against a backlog, increase the threads temporarily and monitor bandwidth and cluster performance, as boosting the speed stresses both clusters. |
SSL Server | Replication via direct POST. Defaults to none. Enable Require trusted SSL if replicating over an untrusted network; Allow untrusted SSL is available but not intended for production systems. (v10.0) |
Remote Admin Name/Password | Inherit from source cluster: Uncheck the enabled box, if the remote cluster user name is different from the source cluster name in the same realm. Then enter: User/Password credentials
|
Propagate Deletes
The legacy option Propagate Deletes is deprecated; it existed to allow setting a target cluster to preserve all deleted objects. This need is now covered by https://perifery.atlassian.net/wiki/spaces/public/pages/2443812258; historical versions of deleted objects can be accessed to recover mistakenly deleted content. Versioning to the target cluster serving as the archive can be limited, to minimize space usage. (v11.1)
Note these restrictions and behaviors if an existing feed has this option specified:
With Versioning - Always propagate deletes when using https://perifery.atlassian.net/wiki/spaces/public/pages/2443812258 in the cluster.
Without Versioning - The target cluster maintains deleted content carrying no verifiable indication of the deleted status if this option is disabled.
Using Feed Actions
Clicking on an existing feed in the Feeds list opens the Feed Settings page, with the existing settings populated. The Actions (gear) icon menu at the top right supports multiple feed actions, appropriate to the type of feed:
Pause / Resume | Manual Pause: Feed processing may occasionally need to be paused to perform system maintenance. Pause the search feed before stopping the Elasticsearch service in the search cluster when upgrading an Elasticsearch cluster. Return to the action menu and select the Resume action to resume feed processing after completing system maintenance. Automatic Pause: When the setting feeds.pauseDisconnectPerHourLimit is non-zero and the number of disconnects for a node exceeds the value (default 1000), the feed is automatically paused with a CRITICAL error message. |
---|---|
Refresh | Object data is sent to the feed target in near real-time (NRT) as it is written or updated. Any objects unable to be processed immediately are retried each HP cycle until successful, at which point they are marked as complete and are not resent. Select the Refresh option from the feed action menu, which verifies and rehydrates all previously sent content to a remote cluster if a data loss failure occurs on the remote feed target and restoration from backup is not possible. This process takes some time, as it must revisit all objects in the cluster. |
Delete | When a feed is deleted, it frees source cluster resources. This process does not affect the objects previously pushed to the remote target. Select the Delete option from the feed action menu and verify the intention to permanently delete the feed. The deleted feed is removed from the remaining cluster nodes within 60 seconds. |
View Feed Table | Displays the SNMP Repository Dump for the selected node, for feed diagnostics (see below). |
Troubleshooting Feeds
Feed Diagnostics: Double-click a blocked feed to open the settings page, click the gear icon, and select View feed table, which displays the SNMP Repository Dump for the selected node to troubleshoot. (v2.0)
Review the feedPluginState status to identify the blockage.
Example:feedPluginState blocked: Destination cluster onyx1 reports invalid request: Castor-System-Cluster value must refer to a remote cluster on RETRIEVE request
Idle Feeds: A feed can appear to be idle with items still queued for processing. Plan for the fact that feed status reporting is a best-effort snapshot, not a low-latency or guaranteed transaction mechanism.
Feed Prioritization: Domain and bucket context objects are prioritized for all types of feeds; this improves usability when remote sites are initiated.
Retries for Blocked Feeds: Blocked feeds are retried every 20 minutes, but if the definition for a blocked feed is changed, it triggers an immediate attempt with the new definition, which may clear the blockage. (v10.1)
© DataCore Software Corporation. · https://www.datacore.com · All rights reserved.