Custom metadata headers are created as a means to pass data required by applications. Including custom metadata on stored objects increases the usefulness of content: it provides information that can be indexed by Elasticsearch and used to find, filter, and analyze the content later.

Note

Swarm stores these headers and supplied values without parsing, validation, or modification.

Work with custom metadata through the WRITE, UPDATE, and COPY methods. The COPY method allows updating and adding to the metadata on objects after the initial WRITE.

Tip

With COPY requests, add the preserve query argument to guarantee any custom metadata existing on the object is carried over to the copy. To overwrite an existing value, include the header name with the new value on the request. (v9.2)

See SCSP Headers.

Requirements for Custom Names

Characters: For best compatibility going forward, Swarm restricts to these characters in custom metadata header names (v9.1):

letters (both cases, although case-insensitive is consistent with HTTP/1.1 RFC)
numbers
dash (hyphen)
underscore

Elasticsearch and Dots

Some versions of Elasticsearch (such as 2.3.3) do not allow dots in normal field names. When indexing objects, Swarm converts any dots in custom metadata field names (x_foo_meta_2016.12) to underscores (x_foo_meta_2016_12). (v9.1)

Formats: Follow one of these two naming formats when custom headers are defined, or they are silently ignored and not persisted to the storage cluster:

x-*-meta
x-*-meta-*

x-ExampleCorp-meta-color: blue

Requirements for Custom Values

To specify more than one value for the same header, list the values on the same line, separated by commas.

x-color-meta: blue, green

Important

Do not reuse the same header with different values.

For metadata values, use 7-bit US-ASCII characters, or else follow RFC 2047 guidelines for alternate character sets.

x-xml-meta-data: <size>large</size><color>blue</color><specialorder/>

Verify the total length of all persisted metadata, keys and values, does not exceed 32 KB. Metadata over 32 KB results in a 400 Bad Request error response from Swarm.

Sample Scenario for Custom Metadata

Assume a domain of "example.com" with a bucket called "surveillance", created for storing the company's surveillance videos.

To add a video, POST to the bucket, specifying the Content-Type of the video and including custom metadata to document the video's duration, camera location, and camera model:

curl -i --location-trusted -X POST --post301	\
	--data-binary @20170311-972-9928817883.mp4 	\
	-H "Expect: 100-continue" 					\
	-H "x-example-meta-Start-Time: 2017-03-11T12:00:01.678Z" \
	-H "x-example-meta-End-Time: 2017-03-11T13:00:00.421Z" 	\
	-H "x-example-meta-Building: Annex 2" \
	-H "x-example-meta-Location: 972" \
	-H "x-example-meta-CameraModel: SWDSK-850004A-US" \
	-H "Content-Type: video/mp4" \
	-H "Content-Disposition: inline" \
	"http://example.com/surveillance/2017/03/22/20170311-972-9928817883.mp4"

HTTP/1.1 100 Continue
Date: Mon, 27 Mar 2017 17:15:26 GMT
Server: CAStor Cluster/9.2.0
Content-Length: 0

HTTP/1.1 201 Created
Location: http://192.168.1.12:80/surveillance/2017/03/22/20170311-972-9928817883.mp4 \
	?domain=example.com
Location: http://192.168.1.13:80/surveillance/2017/03/22/20170311-972-9928817883.mp4 \
	?domain=example.com
Volume: 8aff01dbe86d6ff1f27b5872bfc8e840
Volume: cef223aa1bfc13e356203fdede8489e4
Manifest: ec
Last-Modified: Mon, 27 Mar 2017 17:15:25 GMT
Castor-System-Encoding: zfec 1.4(2, 1, 524288, 200000000)
Castor-System-Version: 1490634925.750
Etag: "c04b7eac90a3f22292581080c32fdd07"
Replica-Count: 2
Date: Mon, 27 Mar 2017 17:17:16 GMT
Server: CAStor Cluster/9.2.0
Content-Length: 46
Content-Type: text/html
Keep-Alive: timeout=14400
<html><body>New stream created</body></html>

Use a HEAD command to verify the video is successfully stored:

curl --head \
	--location-trusted "http://example.com/surveillance/2017/03/22/20170311-972-9928817883.mp4"

HTTP/1.1 301 Moved Permanently
Date: Mon, 27 Mar 2017 17:22:50 GMT
Server: CAStor Cluster/9.2.0
Location: http://192.168.1.12:80/surveillance/2017/03/22/20170311-972-9928817883.mp4
	?domain=example.com&auth=2db96e4590e029966aecfd0dd96da7e9
Content-Length: 0
Keep-Alive: timeout=14400

HTTP/1.1 200 OK
Castor-System-CID: fd20ce977b35d0509205b27977d697d3
Castor-System-Cluster: example.com
Castor-System-Created: Mon, 27 Mar 2017 17:15:25 GMT
Castor-System-Name: 2017/03/22/20170311-972-9928817883.mp4
Castor-System-Version: 1490634925.750
Content-Disposition: inline
Content-Type: video/mp4
Last-Modified: Mon, 27 Mar 2017 17:15:25 GMT
x-example-meta-Building: Annex 2
x-example-meta-CameraModel: SWDSK-850004A-US
x-example-meta-End-Time: 2017-03-11T13:00:00.421Z
x-example-meta-Location: 972
x-example-meta-Start-Time: 2017-03-11T12:00:01.678Z
Manifest: ec
Content-Length: 1500964975
Etag: "c04b7eac90a3f22292581080c32fdd07"
Castor-System-Path: /example.com/surveillance/2017/03/22/20170311-972-9928817883.mp4
Castor-System-Domain: example.com
Volume: 8aff01dbe86d6ff1f27b5872bfc8e840
Date: Mon, 27 Mar 2017 17:22:50 GMT
Server: CAStor Cluster/9.2.0
Keep-Alive: timeout=14400

The custom metadata is what makes it possible and practical to find videos. Suppose that an incident occurred in the Annex 2 building; to find surveillance video that may be relevant to the investigation, search the surveillance bucket for video taken at Annex 2 during that time span:

curl -i --location-trusted "http://192.168.1.11/surveillance\
	?domain=example.com\
	&format=json&fields=all\
	&content-type=video/mp4\
	&x-example-meta-Building=Annex%202\
	&x-example-meta-Start-Time:date=<2017-03-11T12:17:23Z\
	&x-example-meta-End-Time:date=>2017-03-11T12:17:23Z"

HTTP/1.1 200 OK
Castor-System-Alias: fd20ce977b35d0509205b27977d697d3
Castor-System-CID: 72203a85b0f9d7a64a7625c114f8a886
Castor-System-Cluster: example.com
Castor-System-Created: Mon, 27 Mar 2017 16:37:38 GMT
Castor-System-Name: surveillance
Castor-System-Version: 1490632658.361
X-Timestamp: Mon, 27 Mar 2017 16:37:38 GMT
Last-Modified: Mon, 27 Mar 2017 17:26:00 GMT
Transfer-Encoding: chunked
Content-Type: application/json; charset=utf-8
Castor-Object-Count: 1
Castor-System-Object-Count: 1
Date: Mon, 27 Mar 2017 17:26:00 GMT
Server: CAStor Cluster/9.2.0
Keep-Alive: timeout=14400
[{
	"sizewithreps": 2251447463,
	"contextid": "fd20ce977b35d0509205b27977d697d3",
	"content_type": "video/mp4",
	"name": "2017/03/22/20170311-972-9928817883.mp4",
	"x_example_meta_end_time:date": "2017-03-11T13:00:00.421Z",
	"@timestamp": 1490635036512,
	"x_example_meta_building": "Annex 2",
	"x_example_meta_location:date": 972000,
	"x_example_meta_location": "972",
	"x_example_meta_cameramodel": "SWDSK-850004A-US",
	"domainid": "72203a85b0f9d7a64a7625c114f8a886",
	"x_example_ meta_start_time:date": "2017-03-11T12:00:01.678Z",
	"hash": "c04b7eac90a3f22292581080c32fdd07",
	"timestamp": 1490635036512,
	"x_example_meta_location:double": 972,
	"last_modified": "2017-03-27T17:15:25.748400Z",
	"bytes": 1500964975,
	"content_disposition": "inline",
	"x_example_meta_location:long": 972,
	"x_example_me ta_end_time": "2017-03-11T13:00:00.421Z",
	"x_example_meta_start_time": "2017-03-11T12:00:01.678Z"
}]

The search correctly found a video of interest in the surveillance bucket and returned the object: 2017/03/22/20170311-972-9928817883.mp4

Swarm Documentation

Custom Metadata Headers