| Table of Contents | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
The best practices for using DataCore Swarm object storage as the capacity tier in a Veeam Backup and Replication Scale-out Backup Repository include sizing and configuration guidelines and operational tips for optimizing the backup and recovery processes. These practices may be updated periodically as new lessons are learned in the field.
We recommend reading Veeam’s best practice documentation before reading this KB article, https://bp.veeam.com/vbr/2_Design_Structures/D_Veeam_Components/D_backup_repositories
...
Use
...
Multiple SOBR
Currently, Veeam supports only one object storage bucket per SOBR configuration. We recommend using multiple SOBRs pointing to different buckets and storage domains per client and workload type (VB365 Backup, Archive backup jobs etc.). )
This will allow you to take advantage of future support for separate elasticsearch indices per storage domain.
| Note |
|---|
CautionIn order to To use a SOBR a Veeam Enterprise (up to 2 SOBR) or an Enterprise Plus (unlimited SOBRs) license is required. |
Each SOBR requires its own performance tier (primary storage), whereby each SOBR could be a subdirectory of the same primary storage.
Another advantage of multiple SOBR is that you can set different retention policies per SOBR.
Backup Job Storage Settings
Veeam allows you to configure a block size per Backup Job. The block size affects deduplication and incremental backup size. By default, blocks are compressed and the compression ratio is about 50%.
The smaller the block size, the more calls are needed to the object storage to upload the data. DataCore recommends to use “Local Target (large blocks)”. This is equal to a 4MB (=2MB after compression) object size. Performance Tier storage requirements depend on this setting, too. Consult with Veeam or your partner before setting this.
| Note |
|---|
CautionBe aware that backup copy jobs inherit the Storage optimization of the backup job its backing up, you . You will not be able to modify it on “backup copy” type jobs. |
Note
For VBR, setting the block size to 8MB for best performance (up to 3.5x faster) requires a PowerShell script. Veeam provided us with an official tool to change storage block size on all backup jobs using a specific object storage backup repository. You need to run this script from the VBR PowerShell console.
...
See Veeam V12 Guidelines for more details on how to set the storage block size
...
.
SOBR Offloading
For the SOBR offloading the aggregated number of concurrent tasks of all performance tier extents is used. If an extent has no limit on concurrent tasks, SOBR offload uses an unlimited number of tasks (up to the # of backup chains).
Each SOBR offload task will use up to 64 connections to the capacity tier. This is defined by S3ConcurrentTaskLimit in the Windows registry settings. (HKLM\SOFTWARE\Veeam\Veeam Backup and Replication\S3ConcurrentTaskLimit , requires a reboot to apply).
We recommend approx. 2,000 connections for on-prem object storage.
...
The performance will depend on the Veeam setup in the environment. If you have a single repository with 8 concurrent tasks, and the above setting is left standard, you may have the ability to over power overpower the object store. For example: 8 x 64 connections, which may result in high CPU usage.
Since Veeam 11 additionally , the number of tasks can be throttled on the repository see image.
...
To calculate the number of S3 operations that are processed in parallel (max), use this calculation: Available Repository Task Slots * S3ConcurrentTaskLimit = Max used S3 operations in parallel.
Use the Veeam Object Storage IO sizing calculator https://calculator.veeam.com/vse/ for the best setting.
| Info |
|---|
InfoTo verify the current SOBR, offload job uses the new setting check %programdata%/Veeam/Backup/SOBR_Offload_name-of-sobr/VMname/*.gate.log and search for S3ConcurrentTaskLimit. |
Task
...
Throttling
Max Concurrent Tasks per Backup Proxy, by default Veeam recommends this to be set to 2x the number of CPU cores in your VBR server. (Apply the same modifications on the VM backup proxy).
For more info see https://helpcenter.veeam.com/docs/backup/vsphere/limiting_tasks.html?ver=120
...
Handling
...
Multiple Veeam Backup Agents
Veeam recommends the use of VBR “managed mode” which provides network traffic rules via their backup proxies. In this mode, all data protection and administration tasks are performed by a backup administrator in Veeam Backup & and Replication.
Reference: Veeam Agent Management Infrastructure - Veeam Agent Management Guide
...
https://helpcenter.veeam.com/docs/backup/agents/agents_infrastructure.html?ver=120
Optimization Inside DataCore Swarm
By default, the cloud gateway 7.10.4 uses 50 threads to execute S3 MultiDelete requests, you can improve performance by increasing MaxRecursiveDeleteThreads to 250 in /etc/caringo/cloudgateway/gateway.cfg
...
MaxConnectionsPerRoute = (maxConnections / Total Swarm Storage Nodes)
| Note |
|---|
WarningIf you add more Gateways, be aware that all the parameters above need to be divided by the number of gateways you have in your environment. |
Final Remarks
For additional technical details refer to the Veeam v12 Guidelines , see Veeam V12 Guidelines.
For additional configuration steps, refer to see the deployment guide for using Integrating Swarm Object Storage with Veeam Backup and Replication.