Custom Metadata Typing

Info

The user can disable metadata typing to save space with the search.enableCustomMetadataTyping setting:
search.enableCustomMetadataTyping = false
The user must review this setting before creating any new search feed. This setting is available since Swarm v14.1, but the default is ‘False’ in v15.0.

Swarm indexes custom metadata as strings by default. Swarm types the custom metadata if the data follows standards for numeric, time, or geospatial formats to perform query operations and to manage data by type. Swarm applies typing to Custom Metadata Headers on all context objects (domains, buckets) and content objects (named, alias, immutable, untenanted). (v9.2)

Tip

Including a non-digit, non-"e" character (such as ~) in the string to prevent additional typing if the string data exists that may match the pattern for a number or date.

Data Types for Custom Headers

On POST and COPY, data types Swarm parses and recognizes in custom headers are:

  • String (default)

  • Numeric (double and long)

  • Date (ISO 8601)

  • Geo-point (latitude, longitude)

Swarm Search indexes on these types to run queries against these types in the custom metadata.

Important

Convert dashes (hyphens) to underscores which is how the headers are indexed for Elasticsearch if querying directly to Elasticsearch rather than through Swarm.

Type

Example Header

Example Search

Notes

Type

Example Header

Example Search

Notes

String

"x-foo-meta: ASCII string."

&x-foo-meta=ASCII%20string

The default datatype. See the Elasticsearch documentation onhttps://www.elastic.co/guide/en/elasticsearch/reference/2.3/string.html.

Double

"x-foo-meta: 34567.123"

&x-foo-meta:double=34567.123

See the Elasticsearch documentation on https://www.elastic.co/guide/en/elasticsearch/reference/2.3/number.html.

Long

"x-foo-meta: 93e27"

&x-foo-meta:long=93e27

To prevent overflow, Swarm indexes numeric values only up to 64 bits of precision.

See the Elasticsearch documentation on https://www.elastic.co/guide/en/elasticsearch/reference/2.3/number.html.

Geo-point

"x-foo-meta: 11, -33"
"x-foo-meta: (11, -33)"
"x-foo-meta: 11.22, -33.44"

&x-foo-meta:geo=11,-33

A 2-tuple of two numeric values. The first value is lat (latitude); the second value is lon (longitude).

For search operations, evaluations are performed element by element.

See the Elasticsearch documentation on https://www.elastic.co/guide/en/elasticsearch/reference/2.3/geo-point.html.

Date

"x-foo-meta: 1420070400001" (milliseconds)
"x-foo-meta: 208704067" (seconds)
"x-foo-meta: 2016-12-11"
"x-foo-meta: 2016-12-11T12:45:30"
"x-foo-meta: 2016-12-11T12:45:30.678Z"
"x-foo-meta: 2016-12-11T12:45:30.678-01"
"x-foo-meta: 2016-12-11T12:45+01:23"

&x-foo-meta:date=1420070400001

Counts of time since the epoch can be numbers up to 13 digits:

  • Digits between 11 and 13 are read as a date in milliseconds-since-the-epoch

  • Digits fewer than 11 are read as a date in seconds-since-the-epoch

For dates, the time and zone are optional. An offset from UTC (Zulu) can be appended to the time as ±[hh]:[mm], ±[hh][mm], or ±[hh].

See the ISO 8601 and the Elasticsearch documentation on https://www.elastic.co/guide/en/elasticsearch/reference/2.3/date.html.

 

Querying Custom Metadata by Type

Query for custom metadata typed by using the additional fields when making SCSP queries to Swarm:

  • &x-foo-meta performs string-based queries

  • &x-foo-meta:long performs integer/long typed queries

  • &x-foo-meta:double performs double typed queries

  • &x-foo-meta:date performs date-typed queries

  • &x-foo-meta:geo performs geo-point-typed queries

Here is an example of a query that filters between a range of timestamps, stored in custom headers:

curl -i --location-trusted "http://192.168.1.11/surveillance?domain=example.com&format=json&fields=all\ &content-type=video/mp4\ &x_example_meta_Building=Annex%202\ &x_example_meta_Start_Time:date=>2017-03-11T12:00Z\ &x_example_meta_End_Time:date=<2017-03-11T12:30Z"

Upgrading an Existing Index

The index needs to be rebuilt if existing custom metadata is currently stored as strings and it is desired to take advantage of typing across all existing custom metadata.

© DataCore Software Corporation. · https://www.datacore.com · All rights reserved.