Overview

Object Locking prevents object versions from being deleted or overwritten for a fixed amount of time or indefinitely. It is applied to an object version to meet regulatory requirements that require WORM storage or to add another protection layer against object changes and deletion. 

There is a strong connection between Object Locking and Versioning. Object Locking does not lock objects, but individual object versions. Therefore, a user can create new object versions even though the object is locked, but it is impossible to delete or change any locked version of the object. 

There are two types of Object Locking:

• Retention - Specifies a fixed period (“retention period”) during which the object version remains locked. During this retention period, the object is WORM-protected and cannot be overwritten or deleted. After the period expires, the lock goes away automatically from the object.

• Legal hold - Keeps the object locked until the legal hold is explicitly removed. 

These two types of object locking are orthogonal, independent of each other, and can be used simultaneously.

Retention Periods

A retention period locks an object version for a fixed amount of time. Until that fixed amount of time has expired, one cannot delete or change the object version. 

There are three different ways to set a retention period on an object version: 

• Newly created objects can inherit a default retention period configured on the bucket.

• Explicitly set a retention period when creating a new object. This overrides the default retention period configured on the bucket if present.

• Explicitly set a retention period on an existing object version. 

A bucket's default retention period specifies the duration in days or years, for which, every object version placed in the bucket is locked. While placing an object in the bucket, the Gateway calculates a retention period for the object version by adding the specified duration to the object version's creation timestamp.

An already set retention period can always be extended. Submit a new lock request for the object version with a retention period longer than the current period. The Gateway replaces the existing retention period with the new, longer period.  Any user with permission to place an object retention period can extend a retention period. 

Open

Retention Modes

There are two retention modes that impact what can be done with objects under retention: 

• In governance mode, some users can be granted permission to shorten or remove a retention period if necessary.

• In compliance mode, any user including the admin user, cannot overwrite or delete a protected object version. When an object is locked in the compliance mode, the retention mode cannot be changed, and the retention period cannot be shortened. 

The default retention mode and retention period can be set independently at the bucket level. The retention mode always applies to the individual objects carrying it, not to the bucket or cluster as a whole.

Legal Hold

A legal hold prevents an object version from being overwritten or deleted like a retention period. A legal hold does not have an associated retention period and remains in effect until removed.  

Legal holds are independent of retention periods and retention modes. As long as the bucket contains an object that has Object Locking enabled, the user can place and remove legal holds regardless of whether the specified object version has a retention period set or not. Placing a legal hold on an object version does not affect the retention mode or retention period for that object version.

Open

Important

A legal hold cannot be applied as a default at the bucket level.

Design

The Object Locking feature is implemented fully in the Gateway using lifepoints, a Swarm feature, which is used to prevent deletion of locked objects until a certain date has passed. It is supported by both S3 and SCSP protocols.

Despite Gateway relying on lifepoints, it remains possible for applications to impose user-defined lifepoints on objects along with object locks. The Gateway verifies correct semantics in all cases without any additional behavior needed from the application side. In the case of any conflicts between user-defined lifepoints and object locks, the object lock always wins. 

Object Locking is compatible with the S3 protocol and always pertains to specific object versions. A prerequisite for applying it on a bucket is that versioning must be enabled on the bucket.

Metadata Headers Related to Object Locking

Object Locking uses the following headers.

• On Buckets

  x-object-lock-meta-status: ENABLED (absent means DISABLED) x-object-lock-meta-default: <GOVERNANCE|COMPLIANCE>[:<duration>]

   The duration of a bucket's default retention period is expressed as “<integer>y” or “<integer>d” for the number of years/days. 

• On Objects

  x-object-lock-meta-mode: <GOVERNANCE|COMPLIANCE>  x-object-lock-meta-retain-until-date:<date> x-object-lock-meta-legal-hold: ON (absent means OFF) x-object-lock-meta-original-lifepoints: <original lifepoints> lifepoint: [<date>] deletable=no (for retention period) lifepoint: [] deletable=no (for legal hold)

  The above headers are listed using the SCSP names and the corresponding S3 names start with x-amz-*.  The SCSP headers are effectively stored with objects. The S3 names are mapped onto the SCSP counterparts and back on the fly.

Internal to Gateway, all header values are treated case insensitive. Dates are in rfc1123 format, e.g., “Wed, 12 Dec 2016 15:59:02 GMT”. For S3, these are translated in to ISO8601 format. 

• The x-object-lock-meta-retain-until-date header applies to retention periods and specifies the end date of the retention period.

• The x-object-lock-meta-legal-hold header applies to the legal hold.  

The x-object-lock-meta-original-lifepoints header stores the complete set of user-defined delete/deletable lifepoint headers found on the object when the retention period/legal hold is applied. The original delete/deletable lifepoint headers are removed.

The lock lifepoint protects the object against deletion in Swarm, so be it through user requests or built-in functionalities like HP or bucket policies. The lock lifepoint is computed as follows:

• If the object is locked with a retention period, then the lock lifepoint end date matches the end date of the retention period. For legal hold, the lock lifepoint has no end date. 

• Go over the list of original lifepoints and append those whose end date is later than the one from the lock lifepoint. In the case of the legal hold, there is no such end date so none of the original lifepoints get appended.

The purpose of storing the set of original lifepoints is to allow later modifications/removal of the object lock to recompute/reinstate the original lifepoints as they are before the object locking. 

The purpose of appending the “later lifepoints” to the lock lifepoint is to allow Swarm to act on them as it normally does once the lock lifepoint has expired naturally, without any gateway intervention. For legal hold, there must always be a gateway intervention to remove the lock, so the original lifepoints are reinstated at that time.

Enabling Object Locking on a Bucket

Before locking any objects, verify Object Locking is enabled on the bucket. S3 allows enabling Object Locking on new buckets carrying no objects, but Gateway does not impose this restriction. The user must have PutBucketObjectLocking permission to enable/disable Object Locking on a bucket. The user must have GetBucketObjectLocking permission to query the current Object Locking status. 

The request to enable Object Locking can fail with the following errors:

Enabling Object Locking on a bucket comes down to storing the x-object-lock-meta-status and optionally x-object-lock-meta-default headers on the bucket context object. Using S3, one enables/inspects Object Locking config on a bucket using the following calls: 

• https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectLockConfiguration.html  

• https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectLockConfiguration.html  

Using SCSP, one enables Object Locking on a bucket as follows:

PUT /<bucket>?objectlock=<defaultmode> [ :<defaultperiod> ] where;

• The default mode is either governance or compliance. 

• The optional default period is a number of years (y) or days (d), e.g., 1y or 20d. 

In this call, the user can omit either default mode, default duration, or both. The defaults can be modified or removed at any time via additional PUT command.

Use GET /<bucket>?objectlock to query object locking status of a bucket. This returns the following response headers:

And the response body says:

Object locking is enabled on bucket <bucket> with default mode <mode> [ and default duration <duration>] 

No response headers are present if the bucket does not have object locking enabled and the response body says:

Object locking disabled

Locking an Object at Creation Time

The client adds the following headers in the S3 PutObject/SCSP POST request to create a new object with an immediate retention period in effect; 

This takes precedence over the default bucket retention mode and duration if present. 

• The Gateway looks for corresponding defaults at the bucket level if any one of these two headers are omitted from the request.

  • It takes the corresponding values from there if found.

  • The request fails with an HTTP 400 error as both are needed for a successful retention lock if either mode or retain-until-date is still missing. 

• The object is written as a normal unlocked object, despite being written to a bucket that has Object Locking enabled if both headers are omitted from the request and there is no default set at the bucket level. The client can request an immediate legal hold by specifying the header: 

Use of these headers requires that the user has PutObjectRetention/PutObjectLegalHold permission; otherwise the request fails with an HTTP 403 error. The request fails with an HTTP 412 error if the bucket does not have Object Locking enabled.  

The Gateway forwards these headers when creating a new object on Swarm, and also creates a lock lifepoint instructing Swarm to not delete the object before the retention period expires. 

Finally, the original lifepoint headers are preserved in x-object-lock-meta-original-lifepoints: <original lifepoints>

Configuring Retention Limits

It is now possible to configure retention limits at root, domain, and bucket levels. The root config lives in gateway.cfg as item [object_locking]retentionLimit = 100y (default)]. When specifying the value, supported units are y(ear), d(ay), h(our), m(minutes), and s(seconds). It allows specifying only one value/unit combination, hence, syntax such as 1y2d is NOT supported. Typical config uses days or years. The smaller units are mainly intended for testing.

The older config [object_locking]retentionMaxYears = 100 is still supported but deprecated in favor of the more flexible config.

The domain/bucket retention limits live as metadata header X-Object-Lock-Meta-Retention-Limit on the context objects.

When evaluating the retention duration for an object, it is always the smaller root/domain/bucket limit that applies.

Domain and bucket retention limits can be set on existing domain/bucket via the following SCSP requests, using the same value/unit syntax;

To delete a retention limit, use:

A GET or HEAD with the following query args returns a response carrying the objectlocking metadata including the new retention limit metadata header X-Object-Lock-Meta-Retention-Limit.

To set or delete a retention limit, you need new permission PutDomainObjectLockingLimit (for domain) and PutBucketObjectLockingLimit (for bucket). A typical cloud service provider would reserve these permissions for itself.

Any changes to retention limits are written to the audit log, tagged with LIMIT:xxx where xxx is the limit being set, or NONE if the limit is being removed.

You need the GetBucketObjectLocking permission to query the current bucket retention limit (together with other metadata like bucket defaults). To query the current domain retention limit, you need the GetDomainObjectLocking permission, which is new. A domain carries no retention defaults, only an optional retention limit.

You can make regular GET/HEAD requests (ie. without ?objectlock query arg) on buckets and domains with original logic. If you have the required Get{Domain|Bucket}ObjectLocking permission, the response will also contain the objectlocking metadata headers, including retention limit if present, together with all other metadata.

Max Retention Config 

S3 allows defining a “max-retention-duration" limit in the policy. The Gateway has the new configuration option to offer a similar capability to S3. Using a single new config flag to approximate this functionality:

Supported units are y(ear), d(ay), h(our), m(minutes), and s(seconds). The smaller units are intended for testing.

The default limit value is 100 years if unspecified. It is assumed a year is 365 days when performing conversions between numbers of days and years. 

In the SCSP/S3 APIs, any user-specified value exceeding the limit is silently capped to the limit. 

Managing Retention on an Existing Object

Enabling/disabling retention on an object requires that the user has PutObjectRetention permission. The user must have GetObjectRetention permission to query the current retention status. The client must explicitly specify the versionId of the object version to lock. 

The Gateway then creates a new variant of that version on which it stores the extra Object Locking headers. This variant is protected by the retention period. For the client, there is no distinction between the variant and the original object version; the following headers are added or changed:

Introducing or extending a retention period is always possible, but there are restrictions to shortening/removing a retention period on an object that is already under the retention:

• In compliance mode, this is not permitted

• In governance mode, it requires that the user has a special permission BypassGovernanceRetention. 

An S3 request must explicitly include x-amz-bypass-governance-retention:true as a request header with any request that requires overriding governance mode. Using S3, one enables/inspects a retention period on an object using the following calls:

• https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectRetention.html  

• https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectRetention.html  

 Using SCSP, one enables/inspects a retention period on an object using the following calls: 

 Use GET /<bucket>/<object>?version=<uuid>&objectlock to query the current object lock status, and the response carries the following headers:  

And the response body says: 

Object is locked in <mode> mode until <date> 

When called on an object that is not under retention, none of the headers are present and the response body says:  

Object is not locked 

Both S3 and SCSP allow retrieving Object Lock information using the regular object HEAD and GET requests. Assuming that the user has GetObjectRetention permission, the information is returned in the form of the above response headers. The response body is not affected. 

Managing Legal Hold on an Existing Object

Enabling/disabling legal hold requires PutObjectLegalHold permission for the user. To query the current legal hold status, the user needs GetObjectLegalHold permission.

Using S3, the user can enable/inspect legal hold using the following calls:

• https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectLegalHold.html  

• GetObjectLegalHold - Amazon Simple Storage Service

 Using SCSP, the user can enable/inspect legal hold on an object using the following calls:

The response carries the following headers:  

x-object-lock-meta-legal-hold: on. 

And the response body says:  

Object is locked in legal hold 

When it is called on an object that is neither under the legal hold nor retention, the header is not present, and the response body says:  

Object is not locked 

Both S3 and SCSP allow retrieving legal hold information using regular object HEAD and GET requests. Assuming the user has GetObjectLegalHold permission, the information is returned in the form of the above response headers. The response body is not affected. 

Combined Retention and Legal Hold

An object can fall under both retention mode and legal hold at the same time. In the SCSP protocol, querying and deleting such combined locks are handled via a uniform GET and DELETE API (as opposed to S3 which has separate APIs for querying/deleting retention and legal hold). 

For query, use GET /<bucket>/<object>?version=<uuid>&objectlock[=<locktype>]

When querying the object lock status without specifying the lock type, response headers for both retention and the legal hold display, and the response body contains both status texts, separated by a newline.

One can also query lock status for one specific lock type, being either legal hold or retention. The corresponding permission is required. 

To delete, use DELETE /<bucket>/<object>?version=<uuid>&objectlock=<locktype>. Using SCSP one can remove either the retention or legal hold using DELETE and can specify the appropriate query argument objectlock=<locktype>, where locktype is legal hold or retention.

Enforcing the Object Lock

It is worth repeating that Amazon S3’s definition of Object Locking locks object versions, and not objects. It is perfectly possible to overwrite an object that is locked, and now, the overwritten version continues to exist and is protected from changes or deletion.

Moreover, object versions are immutable in Swarm; any object version is protected from modification by design. The exception to this is deletion, which the lock lifepoints now protect against:

• The users' attempts to delete

• Any automated delete attempts by Swarm. 

Swarm rejects any delete requests for indelible objects with a 403 error. An S3 client can delete the delete marker, making the previous version visible again in the Portal. An upcoming Portal release allows visibility of delete markers and version history in a bucket listing.

For SCSP, this is an update to gateway.cfg to pick the desired behavior which is either ‘fail deletes the locked objects with a 403 error’ or ‘mimic the S3 behavior’. 

New Policy Actions

The following new policy actions related to Object Locking are introduced:

Interactions with Other Functionalities

Content UI

The Content Portal also supports Object Locking.

• There are different icons based on each Object Locking state and default of the bucket in the bucket listing view.

• The object versions are locked at the bucket level.

• Both retention and legal hold can be applied on a single object version if necessary.

Recursive Deletes

To avoid the conflict of recursive deletes attempting to remove a locked object, Gateway first checks if there are any objects under retention or legal hold and refuses the recursive delete if so. The request fails with an HTTP 412 error if the recursive delete cannot be performed due to the Object Locking.

While the recursive delete of a domain or bucket does not immediately result in deletion of locked object versions, instead this makes it less practical to find and access. Significantly, it defeats the built-in safety checks that prevent versioning from getting disabled, which results in the deletion of the locked object versions. For that reason, the Gateway requires an additional config setting in the gateway.cfg;

A warning is logged if this setting is present. The header into the recursive delete request is X-Object-Lock-Meta-Bypass-Recursive-Delete-Check: true. When encountering this header, the Gateway skips the aforementioned checks and allows the recursive delete to proceed. 

Lifepoints

The original set of user-defined lifepoints is preserved in a separate header and can be reinstated when the object lock is removed. This applies to deletable/delete lifepoints, all other types of lifepoints are left as is. 

The Gateway adds a deletable=no lock lifepoint to protect locked objects from inadvertent deletion. In the case of the retention period, the lock lifepoint has the same end date as the retention period. The lock lifepoint includes the subset of user-defined lifepoints with a later end date than the retention period. This allows those lifepoints to automatically resume taking effect as soon as the retention period expires. 

In the case of the legal hold, the lock lifepoint has no end date, and no user-defined lifepoints are included in it.

Append

SCSP APPEND creates a new version when versioning is enabled. Any lock either retention or legal hold, which is applied to the object version, is applied to the new version created by the APPEND operation.

Audit Logging

Object Locking operations are audit logged. Since object locks can also be requested as part of the object PUT/POST/COPY requests, the Gateway tags the request’s audit log line with additional object lock information, rather than inserting new log lines.  

The tags are appended to the audit log line, enclosed in ‘[]’ brackets. Multiple tags (both legal hold and retention) are separated by a comma. Object locking tags are always prefixed with string OBJLCK. This keeps a door open for other kinds of tags in the future.