| Table of Contents | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
The purpose of this KB document is to provide our recommendations for the use of Veeam V12.
This information was gathered from support cases as well as and certification testing of Veeam V12.
...
VB365 = Veeam Backup for Microsoft 365
VBR = Veeam Backup and Replication
VBA = Veeam Backup Agent for Windows
General Configuration
...
Guidelines
To avoid creating buckets with too many objects and to keep bucket listing performance under control, we recommend:
Use large storage object size in VBR and VBA (4MB or higher)
Assign a A new dedicated bucket and storage domain per clientmust be used for every SOBR and workload ( VB365 backup, archive backup jobs etc ) . This will allow you to take advantage of future support for separate elasticsearch indices per storage domain.
Keep VB365 and VBR in different buckets, as VB365 does not provide the ability to control storage object sizessupport versioning or object locking.
Veeam backup agent (VBA) now allows direct to cloud backup, it does however not provide you with a way to limit concurrent connections via the Proxy Threads like VBR does. You will need to define this at the load balancer level. Some examples are provided at the end of this article.
Be mindful of the capacity requirement if your backup schedule is aggressive and rapidly deletes data. Deleting data in Swarm is not immediate; it relies on a background health process to convert deleted (trapped) space back into free space and you will need sufficient free space to ingest new data while the trapped space reclamation occurs. For more information see KB article How do I fix trapped space?
When using multiple gateways in your environment make sure to use a real load balancer, DNS round robin is not a valid load balancing configuration. We recommend using the least connection load balance algorithm.
| Info |
|---|
Least Connections:The system passes a new connection to the gateway that has the least number of current connections in the pool. |
Veeam Backup for Microsoft 365 (VB365)
| Warningnote |
|---|
WarningVB365 does not support versioning or object locking. |
VB365 only has a single way to configure the number of threads (default is 64) it uses to communicate with the backup repository.
Reference: https://helpcenter.veeam.com/docs/vbo365/guide/vbo_threads_and_limit.html?ver=70
In In a mixed environment if your backup repository is overwhelmed with too many concurrent connections we recommend reducing the backup proxy threads.
VB365 stores data items in chunks to the object storage repository. These chunks are before compression 5MB for Exchange data and 8MB for SharePoint and OneDrive data. Veeam documentation says that you can assume 40-50% compression efficiency.
| Noteinfo |
|---|
ImportantVB365 has hardcoded chunk sizes, those are not configurable. |
| Note |
|---|
CautionVB365 stores meta-data in their own objects. These objects are small in size (100KB or less) and can make up for up to 50% of the overall number of objects in an object storage repository. |
Source reference material: https://bp.veeam.com/vb365/guide/design/sizing/objectstorage.html
Veeam Backup Agent for Windows (VBA)
When using direct to cloud backup option, using larger block size can yield up to 3.5x faster performance. We recommend at least 4MB block size.
This is configured in the backup job properties → Bucket → Advanced (button) → Storage Tab.
...
You can configure network throttling in settings section. (but not thread/task concurrency)
...
Veeam Backup and Replication (VBR)
| Infonote |
|---|
ImportantMake sure to install and use V12 Cumulative Patch 2 3 or higher which was released on April 12, where they increased the concurrent delete threads were increased from 1 to 10, affecting the performance of all their backup jobs. Reference: https://www.veeam.com/kb4420 |
| Note |
|---|
WarningData in an object storage bucket or container must be managed solely by Veeam Backup & Replication, including retention and data management. Enabling lifecycle rules is not supported, and may result in backup and restore failures. For more information, see https://helpcenter.veeam.com/docs/backup/vsphere/object_storage_repository_cal.html?ver=120#general-considerations-and-limitations. |
Using larger block size can yield up to 3.5x faster performance. We recommend at least 4MB block size.
This is done in the backup job properties → Storage section → Advanced Settings → Storage Tab.
...
| Note |
|---|
WarningBe aware that backup copy jobs inherit the Storage optimization of the backup job its backing up, you will not be able to modify it on “backup copy” type jobs. |
| Info |
|---|
RequiredSetting the block size to 8MB for best performance requires a PowerShell script. Veeam provided us with an official tool to change storage block size on all backup jobs using a specific object storage backup repository. You need to run this script from the VBR PowerShell console.I have provided the manual steps as well in case you wish to only do it against 1 specific backup job in your environment |
Note
The script is vendor agnostic, it will simply ask you which backup repository it should use when searching for associated backup jobs and then apply the desired block size.
| View file | ||
|---|---|---|
|
...
By default our , the cloud gateway 7.10.4 uses 50 threads to execute S3 MultiDelete requests, you can improve performance by increasing MaxRecursiveDeleteThreads recursiveDeleteMaxThreads to 250 in /etc/caringo/cloudgateway/gateway.cfgNote: MaxRecursiveDeleteThreads
| Note |
|---|
recursiveDeleteMaxThreads can overwhelm your Elasticsearch cluster if its not sized for the workload, in some situations you may have to reduce this configuration to get less CPU load on Elasticsearch. |
recursiveDeleteMaxThreads must always be lower than threads and maxConnections
so if you set MaxRecursiveDeleteThreads recursiveDeleteMaxThreads then we recommend also changing:
...
| bgColor | #DEEBFF |
|---|
Example for a cluster of 4 Swarm Storage Nodes
threads = 400
maxConnections = 400
MaxConnectionsPerRoute = (maxConnections / Total Swarm Storage Nodes)
Note
...
If you add more Gateways, be aware that all the parameters above need to be divided by the number of gateways you have in your environment.
Your Performance Tier has an option to use per-machine backup files, which is the recommended option, this increases the number of concurrent threads VBR uses, but in some scenario’s this can overwhelm your backup repository.
Example: if If you setup a backup job to backup 2 VM’s and have this option enabled as well as 8 backup proxy threads then Veeam will use 128 concurrent connections to communicate with the backup repository.
| Info |
|---|
ImportantConcurrent Connections = Backup Proxy Threads x AWS S3 SDK (uses 8 threads per operation) x Backup Job Total VM’s |
...
Additional
...
Configurable Parameters on the VBR
...
Side
Backup Proxy Settings
Max Concurrent Tasks per Backup Proxy, by default Veeam recommends this to be set to 2x the number of CPU cores in your VBR server. ( apply Apply the same modifications on the VM backup proxy).Note: refer to
| Info |
|---|
TipRefer to the official documentation |
...
Network Traffic Throttling on the backup proxy
| Infonote |
|---|
TipNew in V12 is the ability to set multiple traffic rules per backup proxy. |
...
Object Storage Repository Settings
Limit the concurrent tasks against the backup repository if it is overwhelmed.
...
Handling
...
Multi-VBA
...
Clients
Veeam recommends the use of VBR “managed mode” which provides network traffic rules via their backup proxies. In this mode, all data protection and administration tasks are performed by a backup administrator in Veeam Backup & Replication.
referenceReference: https://helpcenter.veeam.com/docs/backup/agents/agents_infrastructure.html?ver=120
If you decide to use the standalone mode then the following simple example HAproxy scenario’s can provide you some level of protection against over-subscribing the object storage target.
Scenario-1:
...
Simple Max Connections
You can define maxconn at the backend side,
...
| bgColor | #DEEBFF |
|---|
...
backend servers
server s1 192.168.30.10:80 check maxconn
...
30
After 30 concurrent connections the rest is queued up...
You can also define how long clients are queued with the following directive:
...
If you timeout you get a 503, which Veeam will retry. ( up to a maximum of 40 mins )
Scenario-2: Sliding
...
| bgColor | #DEEBFF |
|---|
frontend website
bind :80
...
Window Option (For Burst Traffic Handling)
frontend website
bind :80
stick-table type ipv4 size 100k expire 30s store http_req_rate(10s)
...
http-request track-sc0
...
src
...
http-request deny deny_status 503 if { sc_http_req_rate(0) gt 20 }
...
default_backend servers
Return 503 if the rate > 20 concurrent requests in the last 10s.
...