Improving Performance of Delimiter-Based S3 Listings Using CommonPrefixes

Overview

When performing S3 listings that rely on the delimiter feature (for example, folder-style navigation), some environments may observe slower response times. This is typically related to how Elasticsearch processes prefix-based aggregations used by the DataCore Cloud Gateway to generate CommonPrefixes.

This KB article explains the symptoms, expected behavior, and recommended configuration adjustments.

Symptoms

You may notice one or more of the following:

• S3 delimiter-based listings take longer than expected.

• Listing latency increases as the number of objects or prefixes grows.

• Deep or complex folder structures result in slower response times.

• Performance may degrade further during heavy listing or indexing activity.

Once indexing pressure reduces, the listing performance usually improves.

Cause

The Cloud Gateway uses Elasticsearch aggregations to compute folder listings. These aggregations may take longer to complete when:

• Many objects exist under the same prefix.

• The prefix hierarchy is deep.

• Elasticsearch cache entries are invalidated during ingest operations.

• The environment is under index-heavy load.

This behavior is expected under certain workloads and object counts.

Recommended Configuration

To improve performance of delimited S3 folder listings, review the following recommended configuration values in:

Backup Application Optimization

After making changes, restart the Cloud Gateway service:

Additional Information

• Listing performance is expected to vary during periods of heavy listing and indexing.

• No functional impact occurs. Only listing latency is affected.

• Improvements and tuning strategies are being evaluated for future releases.