| Table of Contents | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...
The listing cache solves a scalability problem with the gateway's delimited folder listing functionality. To determine if a folder has subfolders, an Elasticsearch query has to enumerate all objects with the folder name as a prefix to their object names. This can run into the millions of objects for large buckets. When such queries are issued repeatedly and at high frequencies, the resulting CPU use brings an entire Elasticsearch cluster to a halt. This has a crippling effect on any sizeable Veeam deployment
Limitations
Client-Specific Binding: Bound to a dedicated client, with no cross-gateway sharing allowed. The gateway must be able to intercept every write and delete that happens in Swarm.
Non-Persistent Cache: The disk/memory cache is discarded by default on restart.
Limited Lifecycle and RecursiveDeletion Support: No support for bucket lifecycle policies, delete lifepoints, or recursive deletes. All writes and deletes must originate from the gateway.
Memory Constraints: Caching large volumes of data can quickly consume system memory. Misconfiguring cache sizes can lead to memory exhaustion or excessive eviction, reducing cache effectiveness.
Delimiters Support: Custom delimiters are not yet supported, only forward slash "/".
Replication Support: <pending engineering feedback>, right now do not setup replication when LC is enabled.
Prerequisites
The listing cache prototype can be installed as on a regular gateway. No extra config is required. Increasing Java heap size is mandatory, and the disk storing /var/spool/caringo/cloudgateway should be SSD with preferably >=100Gb free capacity.
Memory, CPU and Disk Requirements
...
| Panel | ||
|---|---|---|
| ||
vim /etc/sysconfig/cloudgateway HEAP_MIN="12228m" |
Create disk cache partition
| Panel | ||
|---|---|---|
| ||
vgcreate swarmspool /dev/sdb Persist it by adding at the end of /etc/fstab /dev/mapper/swarmspool-diskcache /var/spool/caringo xfs defaults 0 0 |
Limitations
...
Client-Specific Binding: Bound to a dedicated client, with no cross-gateway sharing allowed. The gateway must be able to intercept every write and delete that happens in Swarm.
...
Non-Persistent Cache: The disk/memory cache is discarded by default on restart.
...
...
Delimiters Support: Custom delimiters are not yet supported, only forward slash "/".
...
Replication Support: <pending engineering feedback>, right now do not setup replication when LC is enabled.
How Does Listing Cache Work
...
How to Enable Listing Cache
The steps for enabling Listing Cache in a production environment depend on the platform or technology stack. The procedure to enable Listing Cache in Swarm is outlined below:
| Info |
|---|
Validate your system support Listing Cache. Many frameworks or databases like Redis, Memcached, or certain web frameworks support caching. |
Set Add in the /etc/caringo/cloudgateway/gateway.cfg.
| Code Block |
|---|
[storage_cluster] disableListingCache=false |
...
After testing in a staging environment, roll out the Listing Cache to production by deploying the necessary configurations and code changes.
Monitor performance impact closely during the rollout phase.
Optional. Pre-warm the cache with commonly accessed listings before enabling it in production, so the initial requests are served from the cache.
...