Introduction

The purpose of this article is to provide sizing guidance regarding how much data a Veeam Backup and Replication as well as Veeam M365 Server can store in a Swarm domain.

With Swarm 16.1.2 and Gateway 8.1.0 we introduced a new feature called “Index per Domain”, this means within our Elasticsearch database we will now store metadata from each domain in separate indices. It is enabled by setting search.perDomainIndex=True before creating a Search Feed.

Data in Elasticsearch is organized into indices. Each index is made up of one or more shards. Each shard is an instance of a Lucene index, which you can think of as a self-contained search engine that indexes and handles queries for a subset of the data in an Elasticsearch cluster.

There are some sizing consideration an administrator needs to review first.

Elastic recommends the following regarding shards in their official documentation:

Example capacity planning

Let’s assume we have an Elasticsearch cluster with 5 data nodes and 5 shards per index.

Veeam Backup and Replication uses long path names for their objects:

This is roughly 275 characters on average ( depends on the custom folder name you chose when creating the backup repository ).

1 shard can therefor hold 50G/1.1kb = 46.5 Million documents.

Total number of shards planned at start will be 10 (5 primary + 5 replicas)

This is set as follows:

So one index(per domain) can hold about = 10 shards * 46.5 Millions docs per shard = 465 Million documents(including the replicas)

This means for optimal use of a domain a maximum of 232 Million objects are recommended, after which performance will start to degrade.

If you are using the default storage optimization aka block size of 1MB , this means that in the best case scenario where Veeam writes data into perfect 1MB sized objects you will be able to write 200 TB to the domain. This includes the storage footprint of immutability ( depends on your chosen backup retention settings ).

How can you monitor the size of your shards ?

Using Curl via a shell:

Example output:

The column “sto” shows the datastore disk space usage for the shard.

Using Grafana:

Our Dashboard “Swarm Search v8.1” has a panel showing this metric

What can I do if my shard exceeds 50GB ?

You have 3 options:

Conclusion

In the example above we had 5 data nodes, with 600 shards per node this means Swarm can support up to 250 domains ( some indices are needed for csmetrics, kibana, etc.. )

If you require more domains that can be supported by your existing Elasticsearch cluster size, then you will need to grow the size of your Elasticsearch cluster, specifically nodes with “data” role.

If you are projecting to exceed 200TB per domain then you need to either increase the storage optimization aka block size to our recommended size of 4MB to 8MB or think about how to distribute this workload over multiple domains for optimal performance.