1 Adding a Search Feed
- 1.1 Adding a Feed to an Existing Index
2 Using Feed Actions
3 Troubleshooting Feeds

Adding a Search Feed

To add a Search feed in the cluster, click the +Add button at the top of the Feeds page and then add a Search feed. Verify the [storage cluster] managementPassword is set properly in the gateway.cfg file if errors are encountered during feed creation. Correct the value and restart the gateway service if a change is needed.

Naming: Swarm applies a naming scheme guaranteeing Elasticsearch index names are always unique, within and across clusters. Swarm creates a new Elasticsearch index and alias for that feed if a second search feed is created through the UI. (v9.0)
Multiples: Swarm allows the creation of more than one Search feed facilitating the transition from using one Elasticsearch cluster to another. During the transition, continue using the primary feed for queries; the second feed is incomplete until it fully clears its backlog. When the second feed is caught up, transition to it (apply Make primary to the second feed) as soon as reasonable for current operations. Delete the original feed once the new primary feed target is verified as working. Having multiple feeds is usually for temporary use only because every feed incurs cluster activity, even when paused.

Important

Restart all Gateway servers to pick up the new feed and update any https://perifery.atlassian.net/wiki/spaces/public/pages/2443810580 if the default search feed is changed (apply Make primary to a new feed). No restarts are needed if only a feed definition is updated.

The following table describes the data entry fields in the dialog box.

ID (Existing feeds)	Read-only; system-assigned identifier
Status (Existing feeds)	Read-only; the current feed processing state. Primary - Flags the Search feed used for all search queries. Only one feed can be Primary. Set from the Feeds command menu.
Name	The name attached to this feed.
Batch Size	Defaults to 100. The maximum number of objects sent concurrently to be processed.
Batch Timeout (Seconds)	Defaults to 1. The maximum amount of time (in seconds) before a batch is resent to be processed after a timeout.
Search Full Metadata	Enabled - (default) Swarm storage indexes all object metadata, including baseline and custom metadata fields. Disabled - Swarm storage indexes only the baseline metadata fields. See Metadata Field Matching for a list of baseline and custom fields.
Server Host(s) or IP(s)	The IP addresses or server names are resolvable by DNS. Separate with a comma or space if entering more than one. DNS must be configured on both the source and target clusters. Important Refresh the feed to prevent it becoming blocked if the list of ES servers on an active feed is changed.
Server Port	Defaults to 9200. The default port for a host.
Alias (Existing feeds)	Read-only; system-assigned name by which Elasticsearch references the Swarm feed.

Adding a Feed to an Existing Index

Use one of the following methods if creating a new search feed pointing to an existing index:

Create Alias

Create a new ES alias pointing to the existing index:
curl -i -X POST <ES-node>:9200/_aliases \ -d '{ "actions": [ { "add": { "index": "EXISTING·INDEX·NAME", "alias": "NEW·ALIAS·NAME" } } ] }'
Create a new index feed and specify the new ES alias created:
curl -i -X POST --anyauth -u admin:ourpwdofchoicehere <swarm-node>:91/api/storage/feeds \ -d '{ "actions": [ { "add": { "index": "EXISTING·INDEX·NAME", "alias": "NEW·ALIAS·NAME" } } ] }'

Remap Alias

Create an index feed through the UI as usual.
Remap the ES alias to the existing index:
curl -i -X POST <ES-node>:9200/_aliases \ -d '{ "actions": [ { "remove": { "index": "NEW·INDEX·NAME", "alias": "NEW·ALIAS·NAME" } }, { "add": { "index": "EXISTING·INDEX·NAME", "alias": "NEW·ALIAS·NAME" } } ] }'
Delete the new index created for the new feed:

Using Feed Actions

Clicking on an existing search feed in the Feeds list opens its Feed Settings page, with the existing settings populated. The gear icon menu at the top right supports multiple feed actions, appropriate to the type of feed:

Pause / Resume	Occasionally it is desirable to pause feed processing to perform system maintenance. Pause the search feed before stopping the Elasticsearch service in the search cluster. Return to the action menu and select the Resume action to resume feed processing after completing system maintenance.
Make Primary	Select the 'Make primary' option from the feed actions menu to change which search feed is the primary feed used for all search queries for Search feeds only.
Refresh	Object data is sent to the feed target in near real-time (NRT) as they are are written or updated. Any objects unable to be processed immediately are retried each HP cycle until successful, at which point they are marked as complete and are not resent. Select the Refresh option from the feed action menu, which verifies and rehydrates all previously sent content to the Elasticsearch cluster, if a data loss failure occurs on the remote feed target and a restore from backup cannot be completed. This process takes some time, as it must revisit all objects in the cluster. For search feeds, if an Elasticsearch index for the cluster does not exist, it is created. To recreate an existing index 'fresh' (such as for case-insensitive searching where case-sensitive was previously used), drop the existing index before refreshing the feed.
Delete	When deleting a feed, it frees source cluster resources. To delete a feed, select the Delete option from the feed action menu and verify intention to permanently delete the feed. The deleted feed is removed from the remaining cluster nodes within 60 seconds. Delete the search data previously sent by the feed if desired.
View Feed Table	Displays the SNMP Repository Dump for the selected node, for feed diagnostics and troubleshooting (see below).

Troubleshooting Feeds

Feed Diagnostics: To troubleshoot the blocked feed, double-click it to open its settings page, click the gear icon, and select View feed table, which displays the SNMP Repository Dump for the selected node. (v2.0)
Review the feedPluginState status to identify the blockage.
Idle Feeds: A feed can appear to be idle with items still queued for processing. Plan for the fact that feed status reporting is a best-effort snapshot, not a low-latency or guaranteed transaction mechanism.
Feed Prioritization: Domain and bucket context objects are prioritized for all types of feeds; this improves usability when initiating remote sites.
Retries for Blocked Feeds: Blocked feeds are retried every 20 minutes, but if the definition for a blocked feed is changed, it triggers an immediate attempt with the new definition, which may clear the blockage. (v10.1)
Blocked Search Feeds: Swarm marks the feed with the status Blocked and messages report it is missing if Swarm cannot find the Elasticsearch index associated with a search feed. Delete the feed and recreate it with the same settings if the search index is gone.