/
Custom Metadata Headers

Custom Metadata Headers

Custom metadata headers are created as a means to pass data required by applications. Including custom metadata on stored objects increases the usefulness of content: it provides information that can be indexed by Elasticsearch and used to find, filter, and analyze the content later.

Note

Swarm stores these headers and supplied values without parsing, validation, or modification.

Work with custom metadata through the WRITE, UPDATE, and COPY methods. The COPY method allows updating and adding to the metadata on objects after the initial WRITE.

Tip

With COPY requests, add the preserve query argument to guarantee any custom metadata existing on the object is carried over to the copy. To overwrite an existing value, include the header name with the new value on the request. (v9.2)

See SCSP Headers.

Requirements for Custom Names

Characters: For best compatibility going forward, Swarm restricts to these characters in custom metadata header names (v9.1):

  • letters (both cases, although case-insensitive is consistent with HTTP/1.1 RFC)

  • numbers

  • dash (hyphen)

  • underscore

Elasticsearch and Dots

Some versions of Elasticsearch (such as 2.3.3) do not allow dots in normal field names. When indexing objects, Swarm converts any dots in custom metadata field names (x_foo_meta_2016.12) to underscores (x_foo_meta_2016_12). (v9.1)

Formats: Follow one of these two naming formats when custom headers are defined, or they are silently ignored and not persisted to the storage cluster: 

  • x-*-meta

  • x-*-meta-*

x-ExampleCorp-meta-color: blue

Requirements for Custom Values

To specify more than one value for the same header, list the values on the same line, separated by commas.

x-color-meta: blue, green

Important

Do not reuse the same header with different values.

For metadata values, use 7-bit US-ASCII characters, or else follow RFC 2047 guidelines for alternate character sets.

x-xml-meta-data: <size>large</size><color>blue</color><specialorder/>

Verify the total length of all persisted metadata, keys and values, does not exceed 32 KB. Metadata over 32 KB results in a 400 Bad Request error response from Swarm.

Sample Scenario for Custom Metadata

Assume a domain of "example.com" with a bucket called "surveillance", created for storing the company's surveillance videos. 

To add a video, POST to the bucket, specifying the Content-Type of the video and including custom metadata to document the video's duration, camera location, and camera model:

curl -i --location-trusted -X POST --post301 \ --data-binary @20170311-972-9928817883.mp4 \ -H "Expect: 100-continue" \ -H "x-example-meta-Start-Time: 2017-03-11T12:00:01.678Z" \ -H "x-example-meta-End-Time: 2017-03-11T13:00:00.421Z" \ -H "x-example-meta-Building: Annex 2" \ -H "x-example-meta-Location: 972" \ -H "x-example-meta-CameraModel: SWDSK-850004A-US" \ -H "Content-Type: video/mp4" \ -H "Content-Disposition: inline" \ "http://example.com/surveillance/2017/03/22/20170311-972-9928817883.mp4" HTTP/1.1 100 Continue Date: Mon, 27 Mar 2017 17:15:26 GMT Server: CAStor Cluster/9.2.0 Content-Length: 0 HTTP/1.1 201 Created Location: http://192.168.1.12:80/surveillance/2017/03/22/20170311-972-9928817883.mp4 \ ?domain=example.com Location: http://192.168.1.13:80/surveillance/2017/03/22/20170311-972-9928817883.mp4 \ ?domain=example.com Volume: 8aff01dbe86d6ff1f27b5872bfc8e840 Volume: cef223aa1bfc13e356203fdede8489e4 Manifest: ec Last-Modified: Mon, 27 Mar 2017 17:15:25 GMT Castor-System-Encoding: zfec 1.4(2, 1, 524288, 200000000) Castor-System-Version: 1490634925.750 Etag: "c04b7eac90a3f22292581080c32fdd07" Replica-Count: 2 Date: Mon, 27 Mar 2017 17:17:16 GMT Server: CAStor Cluster/9.2.0 Content-Length: 46 Content-Type: text/html Keep-Alive: timeout=14400 <html><body>New stream created</body></html>

Use a HEAD command to verify the video is successfully stored:

curl --head \ --location-trusted "http://example.com/surveillance/2017/03/22/20170311-972-9928817883.mp4" HTTP/1.1 301 Moved Permanently Date: Mon, 27 Mar 2017 17:22:50 GMT Server: CAStor Cluster/9.2.0 Location: http://192.168.1.12:80/surveillance/2017/03/22/20170311-972-9928817883.mp4 ?domain=example.com&auth=2db96e4590e029966aecfd0dd96da7e9 Content-Length: 0 Keep-Alive: timeout=14400 HTTP/1.1 200 OK Castor-System-CID: fd20ce977b35d0509205b27977d697d3 Castor-System-Cluster: example.com Castor-System-Created: Mon, 27 Mar 2017 17:15:25 GMT Castor-System-Name: 2017/03/22/20170311-972-9928817883.mp4 Castor-System-Version: 1490634925.750 Content-Disposition: inline Content-Type: video/mp4 Last-Modified: Mon, 27 Mar 2017 17:15:25 GMT x-example-meta-Building: Annex 2 x-example-meta-CameraModel: SWDSK-850004A-US x-example-meta-End-Time: 2017-03-11T13:00:00.421Z x-example-meta-Location: 972 x-example-meta-Start-Time: 2017-03-11T12:00:01.678Z Manifest: ec Content-Length: 1500964975 Etag: "c04b7eac90a3f22292581080c32fdd07" Castor-System-Path: /example.com/surveillance/2017/03/22/20170311-972-9928817883.mp4 Castor-System-Domain: example.com Volume: 8aff01dbe86d6ff1f27b5872bfc8e840 Date: Mon, 27 Mar 2017 17:22:50 GMT Server: CAStor Cluster/9.2.0 Keep-Alive: timeout=14400

The custom metadata is what makes it possible and practical to find videos. Suppose that an incident occurred in the Annex 2 building; to find surveillance video that may be relevant to the investigation, search the surveillance bucket for video taken at Annex 2 during that time span:

curl -i --location-trusted "http://192.168.1.11/surveillance\ ?domain=example.com\ &format=json&fields=all\ &content-type=video/mp4\ &x-example-meta-Building=Annex%202\ &x-example-meta-Start-Time:date=<2017-03-11T12:17:23Z\ &x-example-meta-End-Time:date=>2017-03-11T12:17:23Z" HTTP/1.1 200 OK Castor-System-Alias: fd20ce977b35d0509205b27977d697d3 Castor-System-CID: 72203a85b0f9d7a64a7625c114f8a886 Castor-System-Cluster: example.com Castor-System-Created: Mon, 27 Mar 2017 16:37:38 GMT Castor-System-Name: surveillance Castor-System-Version: 1490632658.361 X-Timestamp: Mon, 27 Mar 2017 16:37:38 GMT Last-Modified: Mon, 27 Mar 2017 17:26:00 GMT Transfer-Encoding: chunked Content-Type: application/json; charset=utf-8 Castor-Object-Count: 1 Castor-System-Object-Count: 1 Date: Mon, 27 Mar 2017 17:26:00 GMT Server: CAStor Cluster/9.2.0 Keep-Alive: timeout=14400 [{ "sizewithreps": 2251447463, "contextid": "fd20ce977b35d0509205b27977d697d3", "content_type": "video/mp4", "name": "2017/03/22/20170311-972-9928817883.mp4", "x_example_meta_end_time:date": "2017-03-11T13:00:00.421Z", "@timestamp": 1490635036512, "x_example_meta_building": "Annex 2", "x_example_meta_location:date": 972000, "x_example_meta_location": "972", "x_example_meta_cameramodel": "SWDSK-850004A-US", "domainid": "72203a85b0f9d7a64a7625c114f8a886", "x_example_ meta_start_time:date": "2017-03-11T12:00:01.678Z", "hash": "c04b7eac90a3f22292581080c32fdd07", "timestamp": 1490635036512, "x_example_meta_location:double": 972, "last_modified": "2017-03-27T17:15:25.748400Z", "bytes": 1500964975, "content_disposition": "inline", "x_example_meta_location:long": 972, "x_example_me ta_end_time": "2017-03-11T13:00:00.421Z", "x_example_meta_start_time": "2017-03-11T12:00:01.678Z" }]

The search correctly found a video of interest in the surveillance bucket and returned the object: 2017/03/22/20170311-972-9928817883.mp4

Related content

© DataCore Software Corporation. · https://www.datacore.com · All rights reserved.