Custom Metadata Headers
Custom metadata headers are created as a means to pass data required by applications. Including custom metadata on stored objects increases the usefulness of content: it provides information that can be indexed by Elasticsearch and used to find, filter, and analyze the content later.
Note
Swarm stores these headers and supplied values without parsing, validation, or modification.
Work with custom metadata through the WRITE, UPDATE, and COPY methods. The COPY method allows updating and adding to the metadata on objects after the initial WRITE.
Tip
With COPY requests, add the preserve query argument to guarantee any custom metadata existing on the object is carried over to the copy. To overwrite an existing value, include the header name with the new value on the request. (v9.2)
See SCSP Headers.
Requirements for Custom Names
Characters: For best compatibility going forward, Swarm restricts to these characters in custom metadata header names (v9.1):
letters (both cases, although case-insensitive is consistent with HTTP/1.1 RFC)
numbers
dash (hyphen)
underscore
Elasticsearch and Dots
Some versions of Elasticsearch (such as 2.3.3) do not allow dots in normal field names. When indexing objects, Swarm converts any dots in custom metadata field names (x_foo_meta_2016.12
) to underscores (x_foo_meta_2016_12
). (v9.1)
Formats: Follow one of these two naming formats when custom headers are defined, or they are silently ignored and not persisted to the storage cluster:
x-*-meta
x-*-meta-*
x-ExampleCorp-meta-color: blue
Requirements for Custom Values
To specify more than one value for the same header, list the values on the same line, separated by commas.
x-color-meta: blue, green
Important
Do not reuse the same header with different values.
For metadata values, use 7-bit US-ASCII characters, or else follow RFC 2047 guidelines for alternate character sets.
x-xml-meta-data: <size>large</size><color>blue</color><specialorder/>
Verify the total length of all persisted metadata, keys and values, does not exceed 32 KB. Metadata over 32 KB results in a 400 Bad Request error response from Swarm.
Sample Scenario for Custom Metadata
Assume a domain of "example.com
" with a bucket called "surveillance
", created for storing the company's surveillance videos.
To add a video, POST to the bucket, specifying the Content-Type of the video and including custom metadata to document the video's duration, camera location, and camera model:
curl -i --location-trusted -X POST --post301 \
--data-binary @20170311-972-9928817883.mp4 \
-H "Expect: 100-continue" \
-H "x-example-meta-Start-Time: 2017-03-11T12:00:01.678Z" \
-H "x-example-meta-End-Time: 2017-03-11T13:00:00.421Z" \
-H "x-example-meta-Building: Annex 2" \
-H "x-example-meta-Location: 972" \
-H "x-example-meta-CameraModel: SWDSK-850004A-US" \
-H "Content-Type: video/mp4" \
-H "Content-Disposition: inline" \
"http://example.com/surveillance/2017/03/22/20170311-972-9928817883.mp4"
HTTP/1.1 100 Continue
Date: Mon, 27 Mar 2017 17:15:26 GMT
Server: CAStor Cluster/9.2.0
Content-Length: 0
HTTP/1.1 201 Created
Location: http://192.168.1.12:80/surveillance/2017/03/22/20170311-972-9928817883.mp4 \
?domain=example.com
Location: http://192.168.1.13:80/surveillance/2017/03/22/20170311-972-9928817883.mp4 \
?domain=example.com
Volume: 8aff01dbe86d6ff1f27b5872bfc8e840
Volume: cef223aa1bfc13e356203fdede8489e4
Manifest: ec
Last-Modified: Mon, 27 Mar 2017 17:15:25 GMT
Castor-System-Encoding: zfec 1.4(2, 1, 524288, 200000000)
Castor-System-Version: 1490634925.750
Etag: "c04b7eac90a3f22292581080c32fdd07"
Replica-Count: 2
Date: Mon, 27 Mar 2017 17:17:16 GMT
Server: CAStor Cluster/9.2.0
Content-Length: 46
Content-Type: text/html
Keep-Alive: timeout=14400
<html><body>New stream created</body></html>
Use a HEAD command to verify the video is successfully stored:
curl --head \
--location-trusted "http://example.com/surveillance/2017/03/22/20170311-972-9928817883.mp4"
HTTP/1.1 301 Moved Permanently
Date: Mon, 27 Mar 2017 17:22:50 GMT
Server: CAStor Cluster/9.2.0
Location: http://192.168.1.12:80/surveillance/2017/03/22/20170311-972-9928817883.mp4
?domain=example.com&auth=2db96e4590e029966aecfd0dd96da7e9
Content-Length: 0
Keep-Alive: timeout=14400
HTTP/1.1 200 OK
Castor-System-CID: fd20ce977b35d0509205b27977d697d3
Castor-System-Cluster: example.com
Castor-System-Created: Mon, 27 Mar 2017 17:15:25 GMT
Castor-System-Name: 2017/03/22/20170311-972-9928817883.mp4
Castor-System-Version: 1490634925.750
Content-Disposition: inline
Content-Type: video/mp4
Last-Modified: Mon, 27 Mar 2017 17:15:25 GMT
x-example-meta-Building: Annex 2
x-example-meta-CameraModel: SWDSK-850004A-US
x-example-meta-End-Time: 2017-03-11T13:00:00.421Z
x-example-meta-Location: 972
x-example-meta-Start-Time: 2017-03-11T12:00:01.678Z
Manifest: ec
Content-Length: 1500964975
Etag: "c04b7eac90a3f22292581080c32fdd07"
Castor-System-Path: /example.com/surveillance/2017/03/22/20170311-972-9928817883.mp4
Castor-System-Domain: example.com
Volume: 8aff01dbe86d6ff1f27b5872bfc8e840
Date: Mon, 27 Mar 2017 17:22:50 GMT
Server: CAStor Cluster/9.2.0
Keep-Alive: timeout=14400
The custom metadata is what makes it possible and practical to find videos. Suppose that an incident occurred in the Annex 2 building; to find surveillance video that may be relevant to the investigation, search the surveillance bucket for video taken at Annex 2 during that time span:
curl -i --location-trusted "http://192.168.1.11/surveillance\
?domain=example.com\
&format=json&fields=all\
&content-type=video/mp4\
&x-example-meta-Building=Annex%202\
&x-example-meta-Start-Time:date=<2017-03-11T12:17:23Z\
&x-example-meta-End-Time:date=>2017-03-11T12:17:23Z"
HTTP/1.1 200 OK
Castor-System-Alias: fd20ce977b35d0509205b27977d697d3
Castor-System-CID: 72203a85b0f9d7a64a7625c114f8a886
Castor-System-Cluster: example.com
Castor-System-Created: Mon, 27 Mar 2017 16:37:38 GMT
Castor-System-Name: surveillance
Castor-System-Version: 1490632658.361
X-Timestamp: Mon, 27 Mar 2017 16:37:38 GMT
Last-Modified: Mon, 27 Mar 2017 17:26:00 GMT
Transfer-Encoding: chunked
Content-Type: application/json; charset=utf-8
Castor-Object-Count: 1
Castor-System-Object-Count: 1
Date: Mon, 27 Mar 2017 17:26:00 GMT
Server: CAStor Cluster/9.2.0
Keep-Alive: timeout=14400
[{
"sizewithreps": 2251447463,
"contextid": "fd20ce977b35d0509205b27977d697d3",
"content_type": "video/mp4",
"name": "2017/03/22/20170311-972-9928817883.mp4",
"x_example_meta_end_time:date": "2017-03-11T13:00:00.421Z",
"@timestamp": 1490635036512,
"x_example_meta_building": "Annex 2",
"x_example_meta_location:date": 972000,
"x_example_meta_location": "972",
"x_example_meta_cameramodel": "SWDSK-850004A-US",
"domainid": "72203a85b0f9d7a64a7625c114f8a886",
"x_example_ meta_start_time:date": "2017-03-11T12:00:01.678Z",
"hash": "c04b7eac90a3f22292581080c32fdd07",
"timestamp": 1490635036512,
"x_example_meta_location:double": 972,
"last_modified": "2017-03-27T17:15:25.748400Z",
"bytes": 1500964975,
"content_disposition": "inline",
"x_example_meta_location:long": 972,
"x_example_me ta_end_time": "2017-03-11T13:00:00.421Z",
"x_example_meta_start_time": "2017-03-11T12:00:01.678Z"
}]
The search correctly found a video of interest in the surveillance bucket and returned the object: 2017/03/22/20170311-972-9928817883.mp4
Related content
© DataCore Software Corporation. · https://www.datacore.com · All rights reserved.