Info

If using search.perDomainIndex=True under Support guidance, then reduce search.numberOfShards before creating the Search Feed since each domain will have its own index. A value of 5 shards should support over 200M objects in a domain while keeping the shard between 20 GB and 50GB to achieve optimal performance as too many small shards will increase CPU usage and very large shards can cause slower searches and will take longer to recover after a failure. It is important to minimize the number of shards needed as Elasticsearch limits each (data) node to 600 shards. This includes primary and replica shards. Content Gateway's daily csmeter indices, with 1 shard and a default 100 day retention (retentionDays), must also be counted.

The hardware platform for the Elasticsearch servers can scale from one virtual machine to multiple physical machines. Scaling Elasticsearch involves both the number of nodes and the number of shards for each index.

Number of Nodes and Shards

Elasticsearch data is organized into indices, which are made up of one or more shards. Each shard is an instance of a Lucene index, which is like a self-contained search engine that indexes and handles queries for one specific portion of Elasticsearch data. Elasticsearch distributes these shards across the cluster of nodes. Refer to the Elasticsearch guidelines.

Scaling is critical: Having too few nodes slows indexing and searching, and too few shards can make them expand excessively large (over 50 GB), which limits the cluster's ability to recover from failures.

Best Practice

Add more Elasticsearch nodes for faster indexing and searching. Nodes require RAM and SSD resources but are easy to configure and join the cluster.
Do not exceed 200M unique objects per ES node. Exceeding this limit negatively impacts performance.
Boost the shard count if expecting multi-billion objects or a lot of custom metadata. Review Elasticsearch shard sizing guidelines and consult DataCore Support for a recommended number.

Max Objects *	Minimum ES Nodes	Shards	Notes

Max Objects *	Minimum ES Nodes	Shards	Notes
200M	1	5	Must set shard replicas to zero. (see below) Do not use it without a robust ES backup strategy.
600M	3	5	This was the default shard count before Swarm 16.
1000M	5	12	Swarm 16 increases the default shard count to 12.
1600M	8	15
2000M	10	20

* Some workloads like Veeam Backup produce a high listing load which requires more elasticsearch nodes, up to one node per shard. Veeam also produces a high object delete and retention period update load, resulting in deleted documents being almost half of the elasticsearch index. Deleted documents are gradually “merged” out of the index, but they have the same impact on listing performance and sizing requirements as much as live documents.

* Very large object sizes may have the effect of many more objects. An application creates an equal number of Swarm objects if an application backs up large VM snapshots to Swarm. A large object count in Swarm is created, needing a larger ES cluster, if the backup application is chunking and storing every 5MB of each snapshot as a separate Swarm object (which has separate metadata).

Increasing the Shards

The Swarm setting search.numberOfShards allows adjusting the number of shards on new search indices as the implementation is scaled. The setting has no effect on existing indices, it must be set before creating a new Search Feed. (v12.0)

To adjust the shard number, update the configuration and start a new feed based on the new shard count.

Set search.numberOfShards to the new value in the cluster.cfg.
Reboot the cluster, or apply it to the running cluster using the command in the Support Tools bundle, as directed by Support:
swarmctl -a -C search.numberOfShards -V 20
Create a new search feed using the new shard count:
- No listing downtime: Create a new feed, make it Primary once it completes, and then delete the old one.
- With listing downtime: Delete the existing ES index and Refresh the feed.

Single-Node ES Clusters

The default shard protection (1 replica) causes a permanent yellow status when an Elasticsearch cluster has one node. This is only suitable for demo or proof of concept deployments. The yellow status occurs because the primary and replica shards are recommended to be hosted on separate nodes. To deploy a single-node ES cluster, explicitly change the shard replicas to zero.

Important

Change the replicas to zero after installing the ES RPMs and before configuring feeds, metrics, or metering, and implement a robust ES backup strategy.

Update the elasticsearch.yml files with the new hosts and change the shard replicas back to 1 if deploying additional nodes later.

Configure the indices to have zero replicas using the script provided if configuring an ES cluster with one ES data node:

/usr/share/caringo-elasticsearch-search/bin/configure_replicas.py -r 0 -e <ES-SERVER>

Use the help command to view the complete options for changing the number of replicas on existing indices.

/usr/share/caringo-elasticsearch-search/bin/configure_replicas.py --help