/
Listing Operations

Listing Operations

Owned by Rumena Zlatkova (Deactivated)
Last updated: Jan 03, 2024 by Bala Harish A(AC)
4 min read

Listing operations are a specialized class of searching that usually have a context constraint of a domain or a bucket, except in the case of listing untenanted unnamed objects or finding annotation objects. The user is typically interested in the hierarchy or membership within a context when performing listing operations. Since listing operations are in fact searches, other searching options and metadata constraints can be used in combination with them.

Encoding

List query response bodies are UTF-8 encoded when non-ASCII characters are included. Applications must decode the response body from UTF-8 prior to interpreting the list body when reading and writing non-ASCII characters.

Listing Domain Contents

Domains are a context that contains bucket objects and unnamed objects. Buckets are identified by name. Unnamed objects are identified by UUID and are either mutable (alias) or immutable.

Basic form of listing a domain
GET /?format=json&domain=myDomain
Filter buckets that start with a string
GET /?format=json&domain=myDomain&prefix=Southwest_
Listing unnamed objects within a domain
GET /?format=json&domain=myDomain&stype=unnamed
Listing everything within a domain
GET /?format=json&domain=myDomain&stype=all

Note

Swarm assumes stype=bucket when listing a domain, so it returns a list of bucket names unless a specific stype is requested.

Listing a Bucket Context

Buckets are a specific context type that belongs to a domain and can contain only named objects. This operation is similar to listing a domain context with additional URI element of the bucket name added.

Listing bucket contents
GET /myBucket?format=json&domain=myDomain

Note

Swarm assumes stype=named when listing a bucket because no other stype is valid.

Listing Untenanted Unnamed Objects

Objects may still be listed by specifying an empty domain context if the storage cluster includes unnamed objects not contained in a domain (untenanted).

Listing untenanted unnamed objects
GET /?format=json&domain=&stype=unnamed

Storage in Use

The storage in use for a domain or bucket context can be dynamically queried using the du query argument. This is valid for domains and buckets.

  • du=withreps requests the total storage impact of objects.

  • size=0 prevents the return of the request body (the calculated value is returned in the header).

Argument

Header Result

Description

Argument

Header Result

Description

du=withoutreps

Castor−Bytes−Used:
2189243

The HTTP/1.1 response header returns a Castor-System-Bytes-Used field that indicates a summary of the storage used by the objects within that context for du=withoutreps.

Swarm calculates the space from the sum of the body bytes of all relevant and distinct objects. This calculation does not consider the number of replicas for each distinct object.

du=withreps 

Castor−Bytes−Used−With−Reps: 109416625

The HTTP/1.1 response header returns a Castor-System-Bytes-Used-With-Reps field that indicates a summary of the storage used by the objects within that context for du=withreps.

Swarm calculates the space from the sum of the body bytes of all relevant objects using each object's maximum reps value from the object lifepoint headers and the assigned value from the cluster multiplied by the object's size. A 100 MB object with reps=2 consumes 200 MB of space. The same object with reps=3 consumes 300 MB of space. Stored with an erasure coding value of reps=4:2, the object consumes 150 MB of storage.

The maximum of reps=3 is chosen for the calculation and the storage impact is recorded as 300 MB if the object has three lifepoints that include all previous example values. Every object has the metadata field sizewithreps that records its space impact.

Important

The Castor-System-Bytes-Used and Castor-System-Bytes-Used-With-Reps fields are computed on-demand. Depending upon the object count within the context, they can consume computational resources on the search servers. Applications should only request du operations when the space calculations are required.

Domain Storage in Use

The option of selecting the types of objects to consider using the stype argument is available when querying the storage in use for a domain.

Argument summary:

  • du=withreps requests the full storage impact of objects.

  • size=0 prevents the return of the request body. The calculated value is returned in the header.

  • stype=named|unnamed|all selects the types of objects included in the calculation.

Space used by all content in domain
GET /?format=json&domain=myDomain&du=withreps&size=0&stype=all   HTTP/1.1 200 OK  Gateway-Request-Id: 26F809F67D883E6D  Content-Type: application/json; charset=utf-8  Castor-System-Object-Count: 17  Castor-System-Bytes-Used-With-Reps: 121590  [snip]
Space used by named objects
GET /?format=json&domain=myDomain&du=withreps&size=0&stype=named    HTTP/1.1 200 OK  Gateway-Request-Id: C6A8D293950C1FD5  Content-Type: application/json; charset=utf-8  Castor-System-Object-Count: 12  Castor-System-Bytes-Used-With-Reps: 121422  [snip]
Space used by unnamed objects
GET /?format=json&domain=myDomain&du=withreps&size=0&stype=unnamed    HTTP/1.1 200 OK  Gateway-Request-Id: 3D79D93B73A07E35  Content-Type: application/json; charset=utf-8  Castor-System-Object-Count: 2  Castor-System-Bytes-Used-With-Reps: 168  [snip]

Bucket Storage in Use

There are only named objects within the bucket context when querying the storage in use for a bucket. The stype argument is not required. 

Argument summary:

  • du=withreps requests the total storage impact of objects.

  • size=0 prevents the return of the request body (the calculated value is returned in the header).

Calculating space used by named objects in a bucket
GET /mybucket?format=json&domain=myDomain&du=withreps&size=0    HTTP/1.1 200 OK  Gateway-Request-Id: 100601F9E31D5ECC  Content-Type: application/json; charset=utf-8  Castor-System-Object-Count: 1000  Castor-System-Bytes-Used-With-Reps: 22016  [snip]

Annotations in Existence

Submit a listing query that sets the argument "decorates" equal to the ETag of the target object in question to retrieve any annotation objects that may exist in the cluster for a given object:

Listing annotation objects for given ETag
GET /?format=json&domain=&decorates=8c2c582c216a1f088c3652bced5a5f91

See Metadata Annotation.

Related content

© DataCore Software Corporation. · https://www.datacore.com · All rights reserved.