Understanding Feeds

Feeds is the object-routing mechanism in the Swarm storage cluster that uses intermittent channel connections to distribute data to these targets:

• Elasticsearch cluster, for object metadata search

• Remote Swarm storage clusters, for object replication and mirroring

• S3 cloud service bucket, for object replication dedicated to disaster recovery

Swarm supports three types of feeds that you can implement:

• Metadata Search (recommended, and required for Content Gateway and UI) provides real-time metadata indexing and ad-hoc search capabilities within Swarm by name or metadata. The Elasticsearch service collects the metadata for each object and updates the search database in your Swarm network. When you create a new object, domain, or bucket, the service collects only the metadata and not the actual content.
  See Search Feeds.

• Remote Replication enables object replication directly to an external storage cluster without API intervention. When Swarm recognizes new or updated objects, it copies these objects to an internal queue. At specific intervals based on the retryWait attribute settings, the plug-in moves the queued objects to the targeted cluster.
  See Replication Feeds.

• S3 Backup provides off-premises Swarm disaster recovery from an S3 cloud service. It allows backup and selective restoration of some or all of your Swarm cluster contents.
  See S3 Backup Feeds.

Because feeds involve communication beyond the Swarm cluster, they require a reliable network connection between the source and target cluster, as well as the source cluster and the Elasticsearch cluster.

Search feeds work on the entire cluster, but replication-type feeds have optional domain filtering (inclusive and exclusive). They can copy over to the target all content in the source cluster or the content in specific domains.

Feed Behaviors

• Feeds operate continuously to keep up with source cluster changes.

• A single object can be associated with up to eight feed definitions.

• The source cluster processes all UUIDs and names stored in the source cluster based on your feed configurations. As objects are added to the cluster, Swarm adds the UUIDs and names to the assigned feed queue. Swarm logic processes the queue and notifies the target cluster and the Elasticsearch server that feeds are available.

• All feed changes can take up to 60 seconds to propagate from the source node to the targeted nodes in your cluster.

• After an object is replicated, it is not replicated again unless you update a feed. When this occurs, Swarm reevaluates all objects in the source cluster against the new feed definition. If required, Swarm reinitiates another replication to the targeted cluster.