The best practices for using DataCore Swarm object storage as the capacity tier in a Veeam Backup and Replication Scale-out Backup Repository include sizing and configuration guidelines and operational tips for optimizing the backup and recovery processes. These practices may be updated periodically as new lessons are learned in the field.
We recommend reading Veeam’s best practice documentation before reading this KB article, https://bp.veeam.com/vbr/2_Design_Structures/D_Veeam_Components/D_backup_repositories
| Table of Contents |
|---|
Use multiple SOBR
Currently Veeam supports only one object storage bucket per SOBR configuration. We recommend using multiple SOBRs pointing to different buckets and storage domains per client and workload type ( VB365 Backup, Archive backup jobs etc.. )
This will allow you to take advantage of future support for separate elasticsearch indices per storage domain.
| Note |
|---|
In order to use a SOBR a Veeam Enterprise (up to 2 SOBR) or an Enterprise Plus (unlimited SOBRs) license is required. |
Each SOBR requires its own performance tier (primary storage), whereby each SOBR could be a subdirectory of the same primary storage.
Another advantage of multiple SOBR is that you can set different retention policies per SOBR.
Backup Job Storage Settings
Veeam allows you to configure a block size per Backup Job. The block size affects deduplication and incremental backup size. By default blocks are compressed and compression ratio is about 50%.
The smaller the block size, the more calls are needed to the object storage to upload the data. DataCore recommends to use “Local Target (large blocks)”. This is equal to a 4MB (=2MB after compression) object size. Performance Tier storage requirements depend on this setting, too. Consult with Veeam or your partner before setting this.
| Note |
|---|
Be aware that backup copy jobs inherit the Storage optimization of the backup job its backing up, you will not be able to modify it on “backup copy” type jobs |
For VBR, setting the block size to 8MB for best performance (up to 3.5x faster) requires a PowerShell script. Veeam provided us with an official tool to change storage block size on all backup jobs using a specific object storage backup repository. You need to run this script from the VBR PowerShell console.
| Tip |
|---|
For more details on how to set the storage block size please see Veeam V12 Guidelines |
SOBR Offloading
For the SOBR offloading the aggregated number of concurrent tasks of all performance tier extents is used. If an extent has no limit on concurrent tasks, SOBR offload uses unlimited number of tasks (up to the # of backup chains).
Each SOBR offload task will use up to 64 connections to the capacity tier. This is defined by S3ConcurrentTaskLimit in the Windows registry settings. ( HKLM\SOFTWARE\Veeam\Veeam Backup and Replication\S3ConcurrentTaskLimit , requires reboot to apply ).
We recommend approx. 2,000 connections for on-prem object storage.
The default value of 64 (decimal) is applied to every worker process, meaning that the “SOBR Offloading” job overall may be using hundreds or thousands of concurrent connections, resulting in sub-optimal performance.
The performance will depend on the Veeam setup in the environment. If you have a single repository with 8 concurrent tasks, and the above setting is left standard, you may have the ability to over power the object store. For example 8 x 64 connections , which may result in high CPU usage.
Since Veeam 11 additionally the number of tasks can be throttled on the repository see image.
...
To calculate the number of S3 operations that are processed in parallel (max) use this calculation: Available Repository Task Slots * S3ConcurrentTaskLimit = Max used S3 operations in parallel.
Use the Veeam Object Storage IO sizing calculator https://calculator.veeam.com/vse/ for the best setting.
| Info |
|---|
To verify the current SOBR offload job uses new setting check %programdata%/Veeam/Backup/SOBR_Offload_name-of-sobr/VMname/*.gate.log and search for S3ConcurrentTaskLimit. |
Task throttling
Max Concurrent Tasks per Backup Proxy, by default Veeam recommends this to be set to 2x the number of CPU cores in your VBR server. (Apply the same modifications on the VM backup proxy).
For more info
...
see https://helpcenter.veeam.com/docs/backup/vsphere/limiting_tasks.html?ver=120
...
Handling multiple Veeam Backup Agents
Veeam recommends the use of VBR “managed mode” which provides network traffic rules via their backup proxies. In this mode, all data protection and administration tasks are performed by a backup administrator in Veeam Backup & Replication.
Reference: Veeam Agent Management Infrastructure - Veeam Agent Management Guide
Optimization inside DataCore Swarm
By default, the cloud gateway 7.10.4 uses 50 threads to execute S3 MultiDelete requests, you can improve performance by increasing MaxRecursiveDeleteThreads to 250 in /etc/caringo/cloudgateway/gateway.cfg
MaxRecursiveDeleteThreads must always be lower than threads and maxConnections
so if you set MaxRecursiveDeleteThreads then we recommend also changing:
Example for a cluster of 4 Swarm Storage Nodes
threads = 400
maxConnections = 400
MaxConnectionsPerRoute = (maxConnections / Total Swarm Storage Nodes)
| Note |
|---|
If you add more Gateways, be aware that all the parameters above need to be divided by the number of gateways you have in your environment. |
Final Remarks
For additional technical details refer to the Veeam v12 Guidelines Veeam V12 Guidelines
For additional configuration steps, refer to the deployment guide for using Integrating Swarm Object Storage with Veeam Backup and Replication.