Viewing and Editing Feeds - Legacy Admin Console

Owned by Rumena Zlatkova (Deactivated)
Last updated: Dec 07, 2023 by Bala Harish A(AC)
11 min read

Deprecated

The Legacy Admin Console (port 90) is still available but has been replaced by the Swarm Storage UI. (v10.0)

There are three types of feeds be created in Swarm:

  1. Replication

  2. Search

  3. S3 Backup - see S3 Backup Feeds. (v11.0)

A Replication Feed performs replication continuously and adaptively:

  • Replicates all objects in the source cluster or domain.

  • Performs continuous replication to keep up with source cluster intake, within available connectivity and bandwidth to the target cluster.

  • Verifies object replication to the targeted cluster as soon as possible.

  • Uses an intermittent connection (such as HTTP) to move content from the source to the target cluster.

A Search Feed is an object-routing mechanism in the storage cluster using intermittent channel connections to distribute data to Elasticsearch for object metadata searching. The source cluster processes all UUIDs and names stored in the source cluster based on the feed configurations in the Swarm Admin Console Settings page. As objects are added to the cluster, Swarm adds the UUIDs and names to the assigned feed queue and notifies the target Elasticsearch server feed data is available.

Note

Multiple search feeds can exist to populate different ES clusters, but one can be designated as the primary, for searching from Swarm.

Viewing Cluster Feeds

The Cluster Feeds page appears when clicking Feeds in the Swarm Admin Console. Feeds are an object routing mechanism in the storage cluster using intermittent channel connections to distribute data to one or more targeted storage clusters.

The following can be sent using Feeds:

  • Metadata content from the source cluster to the search servers (Search Feed)

  • Objects from the source cluster or a particular domain to a targeted cluster (Replication Feed)

Cluster Feeds

Column

Description

Column

Description

Feed Name / Feed ID

The name of the feed and the corresponding number uniquely identifying the feed.

Estimated Completion Time

The estimated time required to process and clear the current backlog of replication feeds based on the last 60 minutes.
This estimate assumes conditions do not change.

Percent Completed

The dynamic report of progress to completion, as a percentage.

Avg Objects / Min

The average number of objects processed per minute based on the processing rate during the last 60 minutes.

Nodes Reporting

The number of cluster nodes that sent feed reports to be aggregated.

Each feed provides color status indicators showing the current state. The following table describes the status colors and corresponding states for search and replication feeds.

Color

State

Description

Color

State

Description

Green

OK

The feed replicated successfully to the targeted search server or cluster node.

Gray

Paused

The feed is suspended because of an ongoing fast volume recovery (FVR) issue on the node.

Orange

Blocked

The destination node or cluster is offline or not accepting feeds.

  • For Search feeds, the search service may be offline.

  • For Replication feeds, the remote cluster may be offline.

Red

Config Error

The feed is misconfigured.

Viewing a Replication Feed

The Replication Feed window provides the statistics for a replication feed sent to a targeted Replication cluster. Depending on the configuration, a typical configuration can have more than one replication feed.

Swarm logs a critical message and updates the status color and state in the window for quick identification a problem when an issue occurs for a particular Replication feed.

The following data appears when expanding the Replication Feed:

Replication Feed

Column

Description and States

Column

Description and States

Node Feed State

The current feed processing state to the Replication cluster. The state can be:

  • OK. Operating normally.

  • Paused. Temporarily paused due to volume recovery.

  • Blocked. Processing blocked due to a transient condition. The Node Plugin State indicates Blocked as well. (See below.)

Node Plugin State

The HTTP transaction state to the Replication cluster. The state can be:

  • Idle. No pending work or replication at this time.

  • Gathering. Gathering replication entries for a future request.

  • Replicating. Processing ongoing requests to the destination cluster.

  • Blocked. Waiting for a transient error to clear. (See below.)

  • Permanent Error. Permanently stuck due to configuration errors.|

Nodes Reporting

The IP addresses of each reporting node contributing to the aggregate report.

Nodes appear together as a group if all target cluster nodes report successfully. Nodes appear in separate rows with corresponding feed state if one or more nodes does not report successfully.

  Each IP address hyperlinks to the SNMP Tools and Monitoring Systems page for the reporting node, which provides node-specific detail for each feed.

Aggregated Current Object Processing

Provides the aggregated statistics for new or updated objects. These statistics include:

  • Pending Evaluation. Objects evaluated before assignment to the replication feed.

  • Unprocessed. Objects queued for processing by the target cluster.

  • Successes. Objects replicated by the target cluster.

  • Retrying. Objects rejected by the target cluster due to a server or network problem. These objects continue to be transmitted until they are accepted by the target cluster.

Aggregated Versioned Object Processing

Provides the aggregated statistics for Object Versioning:

  • Pending Evaluation. Versions evaluated before assignment to the replication feed.

  • Unprocessed. Versions queued for processing by the target cluster.

  • Successes. Versions processed by the target cluster.

  • Retrying. Versions rejected by the target cluster, due to a server or network problem. These objects continue to be transmitted until they are accepted.

 Aggregated Delete Event Processing

 Provides the aggregated statistics for deleted objects. These statistics include:

  • Pending Evaluation. Object deletes received but not evaluated by the target cluster.

  • Unprocessed. Object deletes queued for processing by the target cluster.

  • Successes. Object deletes replicated by the target cluster.

  • Retrying. Object deletes not processed by the target cluster due to a server or network problem. The object deletes continue to be transmitted until they are processed by the target cluster.

Viewing a Search Feed

The Search Feed window provides statistics for a search feed sent to a targeted Elasticsearch cluster. A typical configuration has one search feed.

Swarm logs a critical message and updates the status color and state in the window for quick identification a problem when an issue occurs for a particular search feed.

Note

Elasticsearch proactively monitors the disk space on the nodes. Elasticsearch suspends itself on nodes as soon as space falls below 5% and automatically unblocks itself when space becomes available, without requiring Swarm to restart.

The following status summaries appear when expanding the Search Feed:

Column

Description and States

Column

Description and States

Node Feed State

The current feed processing state to the search cluster nodes. The state can be:

  • OK. Operating normally.

  • Paused. Temporarily paused due to volume recovery.

  • Blocked. Blocked due to a transient condition. The Node Plugin State indicates Blocked as well. (See below.)

Node Plugin State

The HTTP transaction state to the search server. The state can be:

  • Idle. No pending work or replication at this time.

  • Indexing. Processing ongoing requests to the search server.

  • Blocked. Waiting for a transient error to clear. (See below.)

  • Permanent Error. Permanently stuck due to configuration errors.

Nodes Reporting

The number of nodes and corresponding IP address contributing to the aggregate report.

Nodes appear together as a group if the search nodes report successfully. Nodes appear in separate rows with corresponding feed state if one or more nodes does not report successfully.

Each IP address hyperlinks to the SNMP Tools and Monitoring Systems page for the reporting node, which provides node-specific detail for each feed.

Aggregated Current Object Processing

Provides the aggregated statistics for new or updated objects:

  • Pending Evaluation. Objects evaluated before assignment to the search feed.

  • Unprocessed. Objects queued for processing by the search service.

  • Successes. Objects processed by the search service.

  • Retrying. Objects rejected by the search service, due to a server or network problem. These objects continue to be transmitted until they are accepted.

Aggregated Versioned Object Processing

Provides the aggregated statistics for Object Versioning:

  • Pending Evaluation. Versions evaluated before assignment to the search feed.

  • Unprocessed. Versions queued for processing by the search service.

  • Successes. Versions processed by the search service.

  • Retrying. Versions rejected by the search service, due to a server or network problem. These objects continue to be transmitted until they are accepted.

 Aggregated Delete Event Processing

 Provides the aggregated statistics for deleted objects:

  • Pending Evaluation. Object deletes not belonging to the search feed.

  • Unprocessed. Object deletes queued for processing by the search service.

  • Successes. Object deletes processed by the search service.

  • Retrying. Object deletes not processed by the search service, due to a server or network problem. The object deletes continue to be transmitted until they are processed.

Adding a Feed

Implement Feeds through the Cluster Settings page.

Adding a Replication Feed

The Add Replication Feed dialog box allows entering the configuration settings for a replication feed to a target cluster.

Feed Name

The friendly name attached to this feed. This name appears in the Feeds row in the Settings page.

Replicate All Objects

Enable to replicate all objects in the source cluster to the target cluster, regardless of domain.

The filtering options become available and must be populated with valid values if disabling this option.

Domains to Replicate

Required. Specify one or more domains to include, by name (hrfoo.example.com, itfoo.example.com) or by wildcard (.*foo).

Domains to Exclude

Optional. Specify one or more domains to exclude from the set of domains to replicate, by name (abc123.example.com, abc456.example.com) or by wildcard (abc.*).

Replicate Objects in No Domain

Optional. Includes unnamed objects not tenanted in any domain.

Enabling this option creates a filtered feed that only replicates these unnamed objects if leaving the domain lists above unspecified.

Propagate Deletes

Enable to keep object deletes synchronized with the source cluster. Disable to prevent objects from being deleted in the target cluster.

Replication Mode

Specifies replication via direct POST (recommended) or bidirectional GET.  

For best performance, choose direct POST replication, which can go through Gateway. GET replication is the legacy method, which may be needed for application compatibility or networking requirements. Switching modes does not require a feed restart. (v9.6)

Replication Threads

Replication via direct POST only. The default replication speed (6 simultaneous threads) is best for same-sized clusters with minimal replication backlog. (v9.6)

To avoid overwhelming a smaller target cluster, reduce the threads. For faster replication against a backlog, increase the threads temporarily, but monitor bandwidth and cluster performance, as boosting the speed stresses both clusters.

Remote Cluster Proxy or Cluster Host(s)

The IP address of either:

  • One or more nodes in the target cluster.

  • A reverse proxy host that routes to the target cluster. 

To enter two or more node IP addresses, enter each address separated by a comma or spaces.

Remote Cluster Proxy or Cluster Host(s) Port

Defaults to 80. Allows specifying a custom port for the remote cluster. (v9.6)

Remote Cluster
Name

The configuration setting for the target cluster (the cluster.name value in the .cfg file of the target cluster).

Remote Cluster
Administrative User
Name

The administrative user name of the target cluster.
Enter a value in this field only if the remote cluster user name is different from the source cluster name in the same realm.

Remote Custer
Administrative
Password

The administrative password of the target cluster.
Enter a value in these fields only if the remote cluster password is different from the password on the source cluster in the same realm.

To add a replication feed:

  1. Open the Swarm Admin Console and click Settings.

  2. In the Feed Name row, click Add Replication.

  3. Enter the administrator name and password when prompted for authentication.

  4. In the Add Replication Feed window, complete the fields as described above.

  5. Click Add. The new feed appears in the Feed Name row in the Cluster Settings page and propagates to the targeted nodes in the cluster within 60 seconds.

  6. Click Update. A Success dialog box appears.

  7. Click Close.

Adding a Search Feed

The Add Search Feed dialog box allows entering the configuration settings for a search feed to the Elasticsearch server.

 

The following table describes the data entry fields in the dialog box.

Field

Description

Field

Description

Feed name

The name attached to the feed.

Search server(s)

The IP addresses or server names resolvable by DNS. Separate each server name with a space if entering more than one server.
DNS must be configured on both the source and target clusters.

Search server port

The default port for a host.

Search full metadata

Enabled - Swarm indexes all object metadata, including baseline and client metadata fields.
Disabled - Swarm indexes the baseline metadata fields.

See Metadata Field Matching for a list of baseline and custom fields.

Feed batch size

The maximum number of objects sent concurrently to be processed. The default is 100.

Feed batch timeout

The maximum amount of time (in seconds) before a batch is resent to be processed after a timeout. The default is 1.

To Add a Search Feed

  1. Open the Swarm Admin Console and click Settings.

  2. In the Feed Name row, click Add Search.

  3. Enter the administrator name and password when prompted for authentication.

  4. In the Add search feed window, complete the fields as described above.

  5. Click Add. 
    The new feed appears in the Feed Name row in the Cluster Settings page and propagates to the targeted nodes in the cluster within 60 seconds.

  6. Click Update. 
    A Success dialog box appears.

  7. Click Close.

Feed Actions

Deleting a Replication Feed

Source cluster resources are freed when deleting a feed. This process does not affect the objects previously pushed to the target cluster.

To delete a feed:

  1. In the Swarm Admin Console click Settings.

  2. In the Cluster Settings window, locate the Feed Name section.

  3. Identify the feed to be deleted and select the corresponding Delete checkbox.

  4. Click Update. 
    A Success dialog box appears.

  5. Click Close. 
    The deleted feed is removed from the remaining cluster nodes within 60 seconds.

Editing a Search Feed

To edit a feed:

  1. Open the Swarm Admin Console, and click Settings.

  2. In the Cluster Settings window, locate the Feed Name section.

  3. Select the feed to edit and click Edit.

  4. In the Edit search feed window, make any changes as needed to the appropriate fields.

  5. Set the Refresh feed option based on the Elasticsearch server configuration:

    • Select Refresh feed to hydrate a new search server.

    • Deselect Refresh feed to update a running feed on a current search server.

  6. Click Update. 
    Updates appear in the Cluster Settings window and propagate to the remaining cluster nodes within 60 seconds.

Pause Feed

Pause the feed or unpause it by toggling the checkbox option when editing a feed.

Pause the search feed before stopping the Elasticsearch service in the search cluster when upgrading the Elasticsearch cluster.

See Installing Elasticsearch.

Refresh Feed

As objects are written or updated, metadata is sent to the search servers in near real-time (NRT). Any objects that cannot be processed immediately are retried each HP cycle until they succeed, at which point they are marked as complete and are not resent. Refresh the feed if a data loss failure occurs on the Elasticsearch servers and restore from backup is not possible, which verifies and rehydrates all metadata content.

Select either to refresh all search data for the feed or to add the changes to the running feed, without data verification, when editing a feed.

  • Enabled: Resends all object metadata to the Elasticsearch server. An index is created if an Elasticsearch index for the cluster does not exist. To recreate an existing index (such as for case-insensitive searching where case-sensitive was previously used), drop the existing index before refreshing the feed. Select Refresh feed to reverify the feed content on the new or moved server if a new Elasticsearch server is installed in the domain or are moving it to another domain.

  • Disabled: Adds changes to the running feed without resending previously processed objects. Deselect Refresh feed to add the changes to the running feed if updating current search service.

Deleting a Search Feed

Source cluster resources are freed when deleting a feed. This process does not affect the objects previously pushed to the search server.

To Delete a Feed

  1. Open the Swarm Admin Console, and click Settings.

  2. In the Cluster Settings window, locate the Feed Name section.

  3. Identify the feed to be delete and select the corresponding Delete checkbox.

  4. In the Security dialog box, enter the administrator name and password and click OK.

  5. Click Update. A Success dialog box appears.

  6. Click Close. Changes propagate to the targeted nodes in the cluster within 60 seconds.

Delete the search data previously sent by the feed. 

Troubleshooting Blocked Feeds

Troubleshoot the feed state to the Replication cluster by reviewing the Feeds Table data in the SNMP Tools and Monitoring Systems page for each reporting cluster node if the Node Feed State indicates blocked.

For more information about the SNMP Tools and Monitoring Systems tables, see the SNMP MIB Reference file included in the top level of the Swarm product distribution ZIP file.

To troubleshoot feeds:

  1. Log in to the Swarm Admin Console as an Administrator.

  2. In the console, click  Feeds . 

    • In the Cluster Feeds window, the  orange status highlights the blocked node feed and plugin states for the Feed.

    • The  Nodes Reporting box lists the IP addresses for any offline replication cluster nodes or nodes not accepting feeds.

  3. In the  Nodes Reporting  box, click a reporting node IP address. 
    The  Feed Table  in the SNMP Tools and Monitoring Systems page appears for the selected node.

  4. Review the feedPluginState status to identify the blockage. 
    Example: 

    feedPluginState blocked: Destination cluster onyx1 reports invalid request: Castor-System-Cluster value must refer to a remote cluster on RETRIEVE request

  5. Repeat to troubleshoot the remaining blocked IP addresses.

© DataCore Software Corporation. · https://www.datacore.com · All rights reserved.