Setting Quotas

Owned by Rumena Zlatkova (Deactivated)
Last updated: Mar 18, 2024 by Anjali Tyagi
9 min read

Quota Essentials

The Gateway manages all quota actions; as with Metering, there is no additional service for installation and monitoring. Use the Content UI to set quotas on a context, as content protection and versioning are set . A quota policy combines these elements:

  1. Scope - where the quota applies: a specific bucket, a domain, or an entire tenant

  2. Limits - how much storage and/or network usage is allowed

  3. Action - what happens in response overage, from notification to complete lockout

  4. Notification - who to email with information about the quota status change

  5. Override - (optional) alternate action in response of overage with an expiration time

Quota processing relies on periodic queries of Elasticsearch where historical bandwidth and storage metering is maintained. Gateway continually evaluates the metered data against the configured limits on tenants, domains, and buckets to determine when limits are exceeded. The action and optional override are used to determine the restrictions imposed while a quota remains above the limit.

During quota evaluation, if multiple limits are exceeded, including limits from different scopes within the hierarchy, then the most restrictive action or override are applied. See Example Quota Scenarios below for examples of multiple limits interacting within the scope hierarchy.

An expiration date must be provided when an administrator uses an override to force an action different from the computed one. Overrides are designed as a temporary measure to alter the effects of exceeding a quota limit without requiring the action to be changed. The override is automatically invalidated by the Gateway so an administrator does not need to remove any overrides put in place upon reaching the expiration date. The quota's defined action again applies if the limit is exceeded once an override expires.

Purpose for Quotas

No quotas exist by default. All users are entitled to as much network bandwidth and storage space as the storage cluster allows. Design quota policies using metrics to enforce service level agreements to create a cloud service managing and billing tenants. Scope owners may want to define self-imposed quota limits, such as to put a safety cap on the overall usage to control the bill. 

Quota Metrics

Quota policies monitor and allow limiting two classes of metrics: bandwidth usage and/or storage usage:

  • Bandwidth: total of network bytes IN plus bytes OUT. Bytes IN refers to data sent from the client application to the front-side interface of the Gateway. Bytes OUT refers to data sent from the Gateway to the client application. These measures include the content body of the HTTP requests. Bandwidth from replication feeds is included when a feed is routed through the Gateway. Bandwidth from Management API requests are not included.

  • Storage in use is one of two types: Raw and Logical. Raw storage refers to the actual amount of disk space used within the cluster based on the replication/EC factor for the objects. Logical storage refers to the summation of the objects’ Content-Length headers. One type is allowed per quota policy. 

Cascading Limits

Assign quotas to one or more contexts: tenants, domains, and buckets. Swarm applies quota states in this order: tenants, then domains, then buckets. A bucket’s quota is capped by the limits for the domain and tenant, and a domain’s quota is capped by the limits for the tenant. Define quota limits that, when combined across a scope, exceed the limits allowed for the scope (such as 12 domains with 1-TB limits being housed in a tenant with a 10-TB limit). This flexibility simplifies the task of quota creation and makes it easier to manage quotas at lower levels.

Important

Exceeding a metric limit at a higher scope level cascades the overage across all lower scopes. All storage domains and all buckets are over quota, regardless of the individual metrics if a tenant’s limits are exceeded. An empty bucket can be over quota because the domain is over quota.

Over-Provisioning

They show the limits as specified for the scope level when metric limits are displayed in the Content Portal. The limits display as if they are unlimited if no quota exists at the scope level. The metric limit for a bucket with a quota displays the bucket’s metric limits even if they exceed the domain’s metric limits or the tenant’s metric limits. This allows administrators to over-provision storage for lower scopes. The Content Portal displays those limits in the lower scopes as they are possible before the limits at the higher scopes are exceeded.

Enabling Quotas

By default, the Quotas feature is disabled globally, and it must first be enabled through Gateway. The Gateway configuration has a [quota] section for enabling, controlling, and customizing notifications for quotas.

See Gateway Configuration.

Defining a Quota

Navigate to the Properties (gear icon) of the tenant, domain, or bucket to define a quota. Click Enable to view the policy settings:

Enable

The feature must be deliberately enabled for this scope, after which all required values must be entered to Save.

Storage Type

Choose to monitor storage usage by

  • Raw Storage (the total footprint on disk of all objects, replicas, segments, manifests)

  • Logical Storage (the sum of the uploaded content and any versions).

Storage/Bandwidth Limit

Limits are set in units of MB, GB, TB, or PB. These units are multiplies of 1024, not 1000.

Note

Bandwidth resets automatically based upon a calendar month: the first day of the month at 00:00:00 UTC. The status automatically changes to OK when a metric falls back within the allowed limit or resets for the new month. 

Actions

Consequences for exceeding a quota, from least restrictive to most:

  1. Notify Only: Sends notification but does not restrict any operations

  2. Read/Delete Only: Blocks operations that may increase storage (no writes, updates)

  3. Read Only: Blocks operations that change or add content (no writes, updates, deletes)

  4. Lockout: Blocks all operations

Override

Sets an alternate Action with an Expiration date as a temporary policy override, such as to grant a grace period for the client to resolve the overage. Overrides may impose a more restrictive Action than the policy, such as may be used if a tenant is delinquent in payments.

Email Notifications

Lists recipients of email notifications, who are notified when a limit is exceeded or is no longer exceeded. 

Every time an overage begins or ends, the Gateway sends an email to every address defined within the quota policy's notification email list. These notifications contain the scope, the metric’s name and limit, the time of detection, and the resulting quota state that now applies to the scope. The scope is identified as the tenant name, the domain name, or the domain + bucket name. Notifications are specific to the scope where the overage occurs. For example, if a tenant quota is exceeded, notifications do not cascade down to the people in the domain and bucket quotas within the tenant. Limits cascade to lower levels, but notifications do not.

Quota Effects

The Content Portal displays an alert message about the overage on its Contents page and provides the scope in which it has occurred (the current level or higher) when a metric limit has been exceeded by a tenant, domain, or bucket. Note: the overage may not be the fault of the current context. A user of a domain or bucket within the tenant sees an alert message that quota levels are exceeded, even if the domain or bucket is using none of the allowed usage if a tenant has exceeded the storage quota:

Note

Quota metrics are measured periodically, so there can be a lag between when a metric limit is actually exceeded and when the overage is detected. A similar lag can exist going the other direction, as an overage ends. Lag is not longer than 60 minutes while the typical lag is 5 to 15 minutes in a typical deployment.

HTTP Response for Overage - The HTTP response indicates where the overage occurred, the metric exceeded, and the current quota state when a storage action fails due to an exceeded quota. For example, if a domain’s storage limit is exceeded while a bucket’s storage limit is not exceeded and the quota state is “read only,” a write operation to the bucket returns an error stating the domain’s storage quota is exceeded.

Quota Headers

Define quota limits on context objects: tenants, domains, and buckets. Quota management makes use of metadata headers stored with these context objects.

In the header names, {M} is one of these usage metrics:

  • bandwidth (network bandwidth inbound and outbound)

  • rawstorage (storage in use as the total footprint on disk)

  • storage (storage in use as the total logical size only)

Header

 Value

Description

Header

 Value

Description

x-caringo-meta-quota-{M}-limit

= {state} ; {limit}

  • limit = target for this particular metric

  • state = “notify”, “nowrite”, “read”, “lock”

The configured quota limit and the Action (consequence) when the limit is exceeded.

x-caringo-meta-quota-{M}-current

= {state} ; {value} ; {timestamp}

  • value = current for this particular metric

  • state = “ok”, “notify”, “nowrite”, “read”, “lock”

  • timestamp = ISO-8601 timestamp of last update

Last measured value of storage and bandwidth metrics as queried from Elasticsearch. This header is computed by Gateway and is recommended not be set externally.

x-caringo-meta-quota-{M}-override

= {state} ; {user} ; [ {deadline} ]

  • deadline = ISO-8601 timestamp of expiration

  • state = “ok”, “notify”, “nowrite”, “read”, “lock”

  • user = who set the override

Optional. The state and expiration time for the temporary override.

x-caringo-meta-quota-email

= {addresses}

  • addresses = comma-separated list of email addresses

One or more email addresses to notify about quota state changes

Notifications are sent whe the current state of a metric changes at the context level (tenant, domain, or bucket). The message text is similar to the error messages returned to blocked storage operations.

The Gateway Configuration has a [quota] section where the email service, sender, and content of the email notifications for quotas are customized.

API for Quotas

Set and clear quotas and check on quota states using the Content Management API.

See Methods for Quotas.

Example Quota Scenarios

Here are two detailed scenarios for quota enforcement actions and how they apply within the scope hierarchy of tenants, domains, and buckets.

Tenant alpha: storage quota 1.0PB, limit action "read/delete only"     Domain alpha-one: no quota     Bucket mike: bandwidth quota 100TB, limit action "locked"
    Domain alpha-two: no quota         Bucket november: no quota

During use, the sum of the storage in the two domains for tenant alpha expand past 1.0 PB. Tenant alpha’s storage quota has now been exceeded and enters a “read/delete only” state. This restriction also propagates down and applies to domains alpha-one and alpha-two and all buckets. No one is allowed to add more content -- only read and delete operations.

Later, as reading operations continue on bucket mike within domain alpha-one, the bandwidth total passes 100 TB and bucket mike’s quota is exceeded and enters a “locked” state.

At this point, no activity is allowed for bucket mike since it is “locked” while the other domains and buckets within tenant alpha remain in a “read/delete only” state.

After the end of the month is reached and bucket mike’s bandwidth is reset to zero, bucket mike is no longer exceeding the bandwidth quota. Bucket mike returns to the inherited state of “read/delete only” because the tenant is still in a “read/delete only” state.

Tenant bravo: bandwidth quota 500GB, limit action "locked"     Domain bravo-three: storage quota 2.0PB, limit action "read only"         Bucket oscar: no quota

During use, domain bravo-three exceeds the storage quota of 2.0 PB and enters a “read only” state. Because of inheritance, this also means bucket oscar is now read-only. Later, bucket papa within domain bravo-four exceeds the bandwidth limit of 250 GB and enters a “notify only” state. Because all actions are still allowed, bucket papa can continue to use additional bandwidth. As this continues, tenant bravo eventually exceeds the bandwidth total of 500 GB and enters and “locked” state. Due to inheritance, this locked state now applies to domains bravo-three and bravo-four and to the buckets. Thus, bucket oscar and papa are now locked because of the tenant quota limit.

A few frantic phone calls after this lockout happens, the administrator agrees to override tenant bravo’s bandwidth quota with “notify only” for the remainder of the month. Thus, tenant bravo’s effective state is now “notify only” and all storage operations are again allowed for the lower scopes except for bucket oscar. Bucket oscar remains in a “read only” state because it is still over the storage limit, and no override exists for the quota. Bucket papa remains in “notify only” state, which is the same coming from tenant bravo, thus, storage operations are restored even though bucket papa remains over the limit.

© DataCore Software Corporation. · https://www.datacore.com · All rights reserved.