Field Notice: Veeam v11 products failed offload jobs due to false missing objects warnings (objects still exist erroneously)

Products Affected:

Veeam Backup and Replication v11 and higher
Veeam Backup for Office 365 v5 and higher

Description:

During heavy recursive S3 delete operations Swarm storage nodes can fail to delete all metadata entries from Elasticsearch.

This leads to listing requests returning ghost entries – objects or versions that were already deleted.

Any attempt to access these ghost entries returns an HTTP 404 response which leads to failed Veeam jobs.

Neither refreshing the Search Feed nor attempting to delete the object again fixes this.

Problem Symptom:

Failed Veeam offload jobs show the following error message: (truncated for readability)

REST API error: S3 error: The specified key does not exist. Failed to load object.

Solution:

This requires a manual cleanup procedure if you are already experiencing this issue. Please open a ticket with DataCore Swarm support ticket including the Veeam logs containing the 404 errors.

Whether you have experienced this issue or not please apply the settings changes below to avoid new occurrences of this bug.

Run the following 4 commands on your Swarm Cluster Services (SCS) server. This enables synchronous indexing in Swarm and increases the wait time so multi-delete operations are more likely to keep Elasticsearch in sync.