Custom Metadata Headers

Owned by Rumena Zlatkova (Deactivated)
Last updated: Jan 02, 2024 by Bala Harish A(AC)
3 min read

Custom metadata headers are created as a means to pass data required by applications. Including custom metadata on stored objects increases the usefulness of content: it provides information that can be indexed by Elasticsearch and used to find, filter, and analyze the content later.

Note

Swarm stores these headers and supplied values without parsing, validation, or modification.

Work with custom metadata through the WRITE, UPDATE, and COPY methods. The COPY method allows updating and adding to the metadata on objects after the initial WRITE.

Tip

With COPY requests, add the preserve query argument to guarantee any custom metadata existing on the object is carried over to the copy. To overwrite an existing value, include the header name with the new value on the request. (v9.2)

See https://perifery.atlassian.net/wiki/spaces/public/pages/2443820996.

Requirements for Custom Names

Characters: For best compatibility going forward, Swarm restricts to these characters in custom metadata header names (v9.1):

  • letters (both cases, although case-insensitive is consistent with HTTP/1.1 RFC)

  • numbers

  • dash (hyphen)

  • underscore

Elasticsearch and Dots

Some versions of Elasticsearch (such as 2.3.3) do not allow dots in normal field names. When indexing objects, Swarm converts any dots in custom metadata field names (x_foo_meta_2016.12) to underscores (x_foo_meta_2016_12). (v9.1)

Formats: Follow one of these two naming formats when custom headers are defined, or they are silently ignored and not persisted to the storage cluster: 

  • x-*-meta

  • x-*-meta-*

x-ExampleCorp-meta-color: blue

Requirements for Custom Values

To specify more than one value for the same header, list the values on the same line, separated by commas.

x-color-meta: blue, green

For metadata values, use 7-bit US-ASCII characters, or else follow RFC 2047 guidelines for alternate character sets.

x-xml-meta-data: <size>large</size><color>blue</color><specialorder/>

Verify the total length of all persisted metadata, keys and values, does not exceed 32 KB. Metadata over 32 KB results in a 400 Bad Request error response from Swarm.

Sample Scenario for Custom Metadata

Assume a domain of "example.com" with a bucket called "surveillance", created for storing the company's surveillance videos. 

To add a video, POST to the bucket, specifying the Content-Type of the video and including custom metadata to document the video's duration, camera location, and camera model:

Use a HEAD command to verify the video is successfully stored:

The custom metadata is what makes it possible and practical to find videos. Suppose that an incident occurred in the Annex 2 building; to find surveillance video that may be relevant to the investigation, search the surveillance bucket for video taken at Annex 2 during that time span:

The search correctly found a video of interest in the surveillance bucket and returned the object: 2017/03/22/20170311-972-9928817883.mp4

© DataCore Software Corporation. · https://www.datacore.com · All rights reserved.