{"type":"doc","content":[{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"toc","parameters":{"macroParams":{"minLevel":{"value":"1"},"maxLevel":{"value":"2"},"outline":{"value":"false"},"type":{"value":"list"},"printable":{"value":"false"}},"macroMetadata":{"macroId":{"value":"f8acde36-d7fb-45ed-bfc2-363bcc747238"},"schemaVersion":{"value":"1"},"title":"Table of Contents"}},"localId":"efb9dfd0-c477-4ff8-8499-5bae74b5f40b"}},{"type":"paragraph","content":[{"text":"In addition to updating object metadata directly (via ","type":"text"},{"text":"COPY","type":"text","marks":[{"type":"link","attrs":{"href":"/wiki/spaces/public/pages/2443821408"}}]},{"text":"), append additional metadata to existing objects without altering the original. This provides a method to extend the metadata of immutable objects, including historical versions, because each object's create date, original metadata, and version sequence remain undisturbed. Annotations provide an additional method for finding and managing objects, such as storing S3 object-level ACLs for the Gateway to enforce.","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"heading","attrs":{"level":3},"content":[{"text":"Important","type":"text"}]},{"type":"paragraph","content":[{"text":"Swarm cannot be downgraded to an earlier version once this feature is used.","type":"text"}]}]},{"type":"paragraph","content":[{"text":"Benefits","type":"text","marks":[{"type":"strong"}]},{"text":" - Keeping metadata annotation separate from the object itself provides several advantages:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Add helpful metadata without changing the object’s create date, original metadata, and version sequence.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Retrieve objects as originally written, so applications can distinguish between what was original and what was added later.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Annotate ","type":"text"},{"text":"immutable","type":"text","marks":[{"type":"em"}]},{"text":" objects.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Annotate ","type":"text"},{"text":"historical versions","type":"text","marks":[{"type":"em"}]},{"text":" of objects, independent of the current version of the object. This is keenly important when the metadata is derived from analysis performed on the data, which changes from version to version, or when capturing information about specific versions.","type":"text"}]}]}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"heading","attrs":{"level":3},"content":[{"text":"Note","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"This lightweight implementation of annotation does not rely on the annotator and the target object interacting, and the objects do not operate as a pair. For example, there is no single request that returns both objects' headers, and there is no method to merge and resolve conflicts between them.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"It is recommended to use ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#172b4d"}}]},{"type":"inlineCard","attrs":{"url":"https://perifery.atlassian.net/wiki/spaces/public/pages/2443821408"}},{"text":" ","type":"text"},{"text":"operation to update the metadata on an object so the metadata is directly available in listings. This COPY operation is efficient even for large objects, assuming they are erasure-coded, because the manifest stream is copied. For S3, use the PUT with ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#172b4d"}}]},{"text":"copy","type":"text","marks":[{"type":"textColor","attrs":{"color":"#172b4d"}},{"type":"link","attrs":{"href":"https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html"}}]},{"text":" operation, specifying the same source and destination.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#172b4d"}}]}]}]}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Annotation Cleanup","type":"text"}]},{"type":"paragraph","content":[{"text":"There are two key features of this annotation method: (1) ","type":"text"},{"text":"validation","type":"text","marks":[{"type":"em"}]},{"text":" that target objects exist before annotations are written, and (2) the Health Processor's ","type":"text"},{"text":"automated tracking and cleanup","type":"text","marks":[{"type":"em"}]},{"text":" of annotation objects after the target object is removed. A target object annotated may be removed from Swarm in one of several ways:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"SCSP Delete","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"SCSP Write (invalidating the old version)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Lifepoint Delete","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Recursive delete of a parent context (domain or bucket)","type":"text"}]}]}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"heading","attrs":{"level":3},"content":[{"text":"Note","type":"text"}]},{"type":"paragraph","content":[{"text":"The Health Processor it logs a “DECORATION DELETE” AUDIT-level message when purging an annotation during garbage collection. Annotation objects \"decorate\" a targeted content object.","type":"text"}]}]},{"type":"paragraph","content":[{"text":"Regardless of the type of Swarm object annotated (named, alias, immutable, historical version) and the protection type (replicated or erasure-coded), metadata annotation operate largely the same way:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Swarm deletes the orphaned annotation during garbage collection if an annotation is created and the target object is later deleted.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The target object is completely unaffected if an annotation is created and later deleted. ","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"For named objects only, Swarm replaces the annotation object with a delete marker.","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Swarm deletes both recursively if deleting a domain or a bucket containing both the original object and the annotation.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Create separate annotations for any historical versions if updating a versioned object; Swarm deletes the orphaned annotation during garbage collection when deleting a version.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Two outcomes occur based on the position in the version chain if creating and later deleting an annotation on a versioned object: ","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Historical versions","type":"text","marks":[{"type":"em"}]},{"text":": Swarm removes the annotation. ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Current versions","type":"text","marks":[{"type":"em"}]},{"text":": Swarm replaces the annotation object with a delete marker.","type":"text"}]}]}]}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Creating Annotations","type":"text"}]},{"type":"paragraph","content":[{"text":"Metadata annotation makes use of a persisted header, ","type":"text"},{"text":"Castor-System-Decorates","type":"text","marks":[{"type":"code"}]},{"text":", which is the ETag of the target object the annotation object is extending (decorating). This is an annotation object, subject to special Health Processor management, if this header is present. The header is valid for all Swarm object types (immutable, alias, and named), but not for context objects (domains and buckets). Both the annotator (decorator) and annotated target object may be versioned. ","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"heading","attrs":{"level":3},"content":[{"text":"Tip","type":"text"}]},{"type":"paragraph","content":[{"text":"Although it is common to create annotations as metadata-only (","type":"text"},{"text":"Content-Length: 0","type":"text","marks":[{"type":"code"}]},{"text":") objects, annotations are complete objects. Data may be included as part of the annotation.","type":"text"}]}]},{"type":"paragraph","content":[{"text":"Create a new annotation object create an object pointing to the ETag of the target and includes the custom metadata to be added, such as GPS coordinates extracted from an existing, uploaded photo:","type":"text"}]},{"type":"heading","attrs":{"level":5},"content":[{"text":"Extending Metadata with Post-Processed Data","type":"text","marks":[{"type":"strong"}]}]},{"type":"codeBlock","attrs":{"language":"xml"},"content":[{"text":"Content-Length: 0\nCastor-System-Decorates: 9282727ffcca3a09e0843281aafc13af\nX-GPS-Meta-Longitude: 36; 16; 48.36000000000589\nX-GPS-Meta-Latitude: 115; 10; 20.79299999981990","type":"text"}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"heading","attrs":{"level":3},"content":[{"text":"Note","type":"text"}]},{"type":"paragraph","content":[{"text":"The ETag has no quotes, even though returned ETags are quoted. The ETag must be of a content-containing object, not a delete marker.","type":"text"}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"heading","attrs":{"level":3},"content":[{"text":"Alias Objects","type":"text"}]},{"type":"paragraph","content":[{"text":"The current ETag of an alias object, not the permanent UUID, must be used for creating annotations. Swarm returns a 400 - Bad Request error if using the UUID.","type":"text"}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Searching for Annotations","type":"text"}]},{"type":"paragraph","content":[{"text":"In the annotation (decorator) object’s Elasticsearch record, the ","type":"text"},{"text":"Castor-System-Decorates","type":"text","marks":[{"type":"code"}]},{"text":" header value is indexed under the key ","type":"text"},{"text":"decorates","type":"text","marks":[{"type":"strong"}]},{"text":", and the Elasticsearch configuration templates include the ","type":"text"},{"text":"decorates ","type":"text","marks":[{"type":"strong"}]},{"text":"field. Most Swarm queries return this value, if present, as part of the results.","type":"text"}]},{"type":"paragraph","content":[{"text":"Query argument","type":"text","marks":[{"type":"strong"}]},{"text":" - Use a “","type":"text"},{"text":"decorates=","type":"text","marks":[{"type":"code"}]},{"text":"” query argument in Swarm listing queries to find annotation objects for a given ETag (or earlier query result “hash”).","type":"text"}]},{"type":"paragraph","content":[{"text":"See ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://perifery.atlassian.net/wiki/spaces/public/pages/2443821892"}},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Sample Scenario for Annotations","type":"text"}]},{"type":"paragraph","content":[{"text":"Suppose a company needs to store surveillance videos as immutable objects (as protection from tampering) in the domain \"","type":"text"},{"text":"swarm.example.com","type":"text","marks":[{"type":"code"}]},{"text":"\". To add a video, use the normal POST, adding the Content-Type of the video and custom metadata for the video's duration, camera location, and camera model:","type":"text"}]},{"type":"codeBlock","attrs":{"language":"shell"},"content":[{"text":"curl -i --location-trusted -X POST --post301 \\\n--data-binary @20170311-972-9928817883.mp4 \\\n-H \"Expect: 100-continue\" \\\n-H \"x-example-meta-Start-Time: 2017-03-11T12:00:01.678Z\" \\\n-H \"x-example-meta-End-Time: 2017-03-11T13:00:00.421Z\" \\\n-H \"x-example-meta-Building: Annex 2\" \\\n-H \"x-example-meta-Location: 972\" \\\n-H \"x-example-meta-CameraModel: SWDSK-850004A-US\" \\\n-H \"Content-Type: video/mp4\" \\\n-H \"Content-Disposition: inline\" \\\n\"http://swarm.example.com/\"\n\nHTTP/1.1 201 Created\nLocation: http://192.168.1.11:80/e970b3280d5501571c8c6fe9d6838557?domain=swarm.example.com\nLocation: http://192.168.1.12:80/e970b3280d5501571c8c6fe9d6838557?domain=swarm.example.com\nVolume: b3381183a1cfc620d960db3eae1d086d\nVolume: 604a44d1a351045553b5481391af0810\nManifest: ec\nContent-UUID: e970b3280d5501571c8c6fe9d6838557\nLast-Modified: Tue, 28 Mar 2017 19:19:48 GMT\nCastor-System-Encoding: zfec 1.4(2, 1, 524288, 200000000)\nCastor-System-Version: 1490728788.934\nEtag: \"681b2470307b9260fb83542903e51828\"\nReplica-Count: 2\nDate: Tue, 28 Mar 2017 19:22:19 GMT\nServer: CAStor Cluster/9.2.0\nContent-Length: 46\nContent-Type: text/html\nKeep-Alive: timeout=14400\nNew stream created","type":"text"}]},{"type":"paragraph","content":[{"text":"To verify the video is successfully stored, use a HEAD command:","type":"text"}]},{"type":"codeBlock","attrs":{"language":"shell"},"content":[{"text":"curl --head --location-trusted \"http://swarm.example.com/e970b3280d5501571c8c6fe9d6838557\"\n\nHTTP/1.1 200 OK\nCastor-System-CID: 7e7fd5d747d244726af93c726672408b\nCastor-System-Cluster: swarm.example.com\nCastor-System-Created: Tue, 28 Mar 2017 19:19:48 GMT\nContent-Disposition: inline\nContent-Type: video/mp4\nLast-Modified: Tue, 28 Mar 2017 19:19:48 GMT\nx-example-meta-Building: Annex 2\nx-example-meta-CameraModel: SWDSK-850004A-US\nx-example-meta-End-Time: 2017-03-11T13:00:00.421Z\nx-example-meta-Location: 972\nx-example-meta-Start-Time: 2017-03-11T12:00:01.678Z\nManifest: ec\nContent-Length: 1500964975\nEtag: \"681b2470307b9260fb83542903e51828\"\nCastor-System-Domain: swarm.example.com\nVolume: b3381183a1cfc620d960db3eae1d086d\nDate: Tue, 28 Mar 2017 19:24:25 GMT\nServer: CAStor Cluster/9.2.0\nKeep-Alive: timeout=14400","type":"text"}]},{"type":"paragraph","content":[{"text":"The custom metadata is what makes it possible and practical to identify video of interest. Suppose an incident occurs in the Annex 2 building. Search for immutable video taken at Annex 2 during the time span to find surveillance video relevant to the investigation:","type":"text"}]},{"type":"codeBlock","attrs":{"language":"shell"},"content":[{"text":"curl -i --location-trusted \"http://swarm.example.com/?domain=swarm.example.com&format=json&fields=all\\\n&stype=immutable\\\n&content-type=video/mp4\\\n&x-example-meta-Building=Annex%202\\\n&x-example-meta-Start-Time:date=<2017-03-11T12:17:23Z\\\n&x-example-meta-End-Time:date=>2017-03-11T12:17:23Z\"\n\nHTTP/1.1 200 OK\nCastor-System-Alias: 7e7fd5d747d244726af93c726672408b\nCastor-System-CID: ffffffffffffffffffffffffffffffff\nCastor-System-Cluster: swarm.example.com\nCastor-System-Created: Tue, 28 Mar 2017 19:19:29 GMT\nCastor-System-Name: swarm.example.com\nCastor-System-Owner: @CAStor administrator\nCastor-System-Version: 1490728769.536\nX-Timestamp: Tue, 28 Mar 2017 19:19:29 GMT\nLast-Modified: Tue, 28 Mar 2017 19:26:30 GMT\nTransfer-Encoding: chunked\nContent-Type: application/json; charset=utf-8\nCastor-Object-Count: 1\nCastor-System-Object-Count: 1\nDate: Tue, 28 Mar 2017 19:26:30 GMT\nServer: CAStor Cluster/9.2.0\nKeep-Alive: timeout=14400\n[{\n \"contextid\":\"7e7fd5d747d244726af93c726672408b\",\n \"x_example_meta_start_time\":\"2017-03-11T12:00:01.678Z\",\n \"x_example_meta_end_time:date\":\"2017-03-11T13:00:00.421Z\",\n \"@timestamp\":1490728939869,\n \"domainid\":\"7e7fd5d747d244726af93c726672408b\",\n \"last_modified\":\"2017-03-28T19:19:48.932400Z\",\n \"bytes\":1500964975,\n \"hash\":\"681b2470307b9260fb83542903e51828\",\n \"x_example_meta_location:double\":972.0,\n \"content_disposition\":\"inline\",\n \"sizewithreps\":2251447463,\n \"content_type\":\"video/mp4\",\n \"timestamp\":1490728939869,\n \"x_example_meta_location:long\":972,\n \"x_example_meta_location:date\":972000,\n \"x_example_meta_end_time\":\"2017-03-11T13:00:00.421Z\",\n \"name\":\"e970b3280d5501571c8c6fe9d6838557\",\n \"castor_stream_type\":\"immutable\",\n \"x_example_meta_building\":\"Annex 2\",\n \"x_example_meta_location\":\"972\",\n \"x_example_meta_cameramodel\":\"SWDSK-850004A-US\",\n \"x_example_meta_start_time:date\":\"2017-03-11T12:00:01.678Z\"\n}] ","type":"text"}]},{"type":"paragraph","content":[{"text":"The search correctly finds a video of interest: ","type":"text"},{"text":"e970b3280d5501571c8c6fe9d6838557","type":"text","marks":[{"type":"code"}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Adding Metadata Annotation","type":"text"}]},{"type":"paragraph","content":[{"text":"With the video stored securely, suppose the organization also needs to run an application to perform facial recognition on the video. An application generates data when it is run, including both information on the algorithm/settings and the detailed results. The original video object must remain read-only to serve as evidence, so the derived data and metadata must be stored with a method associating it with the original object without altering it.","type":"text"}]},{"type":"paragraph","content":[{"text":"The solution is to annotate the video with a decoration object (which can be named or unnamed) to associate the results with the original video.","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"heading","attrs":{"level":3},"content":[{"text":"Important","type":"text"}]},{"type":"paragraph","content":[{"text":"The ","type":"text"},{"text":"Castor-System-Decorates","type":"text","marks":[{"type":"code"}]},{"text":" header always refers to the ETag of the original video, not the GUID; this is a precaution against the annotation becoming orphaned if the target object is mutable. ","type":"text"}]}]},{"type":"codeBlock","attrs":{"language":"shell"},"content":[{"text":"curl -i -X POST --post301 --location-trusted -d @results \\\n-H \"Castor-System-Decorates: 681b2470307b9260fb83542903e51828\" \\\n-H \"x-VideoAnalysis-meta-Algorithm: facial-recognition\" \\\n-H \"x-VideoAnalysis-meta-Version: 8.7\" \\\n-H \"Content-Type: application/vnd.analysis.facerec\" \\\n\"http://swarm.example.com\"\n\nHTTP/1.1 201 Created\nLocation: http://192.168.1.13:80/0cb2d9e90a3341b10bc9dba27f27259c?domain=swarm.example.com\nLocation: http://192.168.1.12:80/0cb2d9e90a3341b10bc9dba27f27259c?domain=swarm.example.com\nVolume: 6bd38289c2a8fb314caf902d9811fb87\nVolume: 604a44d1a351045553b5481391af0810\nManifest: ec\nContent-UUID: 0cb2d9e90a3341b10bc9dba27f27259c\nLast-Modified: Tue, 28 Mar 2017 20:26:08 GMT\nCastor-System-Encoding: zfec 1.4(2, 1, 524288, 200000000)\nCastor-System-Version: 1490732768.888\nEtag: \"867c10c9e6649313a3a5eed2cc76f307\"\nReplica-Count: 2\nDate: Tue, 28 Mar 2017 20:26:12 GMT\nServer: CAStor Cluster/9.2.0\nContent-Length: 46\nContent-Type: text/html\nKeep-Alive: timeout=14400\nNew stream created","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"heading","attrs":{"level":3},"content":[{"text":"Tip","type":"text"}]},{"type":"paragraph","content":[{"text":"Create annotations as part of the intake process that stores the original objects in Swarm.","type":"text"}]}]},{"type":"paragraph","content":[{"text":"To find any annotations producing facial recognition on the original object, search for objects that decorate the video and also qualify the search to look for facial recognition results:","type":"text"}]},{"type":"codeBlock","attrs":{"language":"shell"},"content":[{"text":"curl -i --location-trusted \"http://swarm.example.com/?domain=swarm.example.com&format=json&stype=immutable&fields=all\\\n&decorates=681b2470307b9260fb83542903e51828\\\n&x_videoanalysis_meta_algorithm=facial%20recognition\"\n\nHTTP/1.1 200 OK\nCastor-System-Alias: 7e7fd5d747d244726af93c726672408b\nCastor-System-CID: ffffffffffffffffffffffffffffffff\nCastor-System-Cluster: swarm.example.com\nCastor-System-Created: Tue, 28 Mar 2017 19:19:29 GMT\nCastor-System-Name: swarm.example.com\nCastor-System-Owner: @CAStor administrator\nCastor-System-Version: 1490728769.536\nX-Timestamp: Tue, 28 Mar 2017 19:19:29 GMT\nLast-Modified: Tue, 28 Mar 2017 20:36:40 GMT\nTransfer-Encoding: chunked\nContent-Type: application/json; charset=utf-8\nCastor-Object-Count: 1\nCastor-System-Object-Count: 1\nDate: Tue, 28 Mar 2017 20:36:40 GMT\nServer: CAStor Cluster/9.2.0\nKeep-Alive: timeout=14400\n[{\n \"sizewithreps\":11684987,\n \"contextid\":\"7e7fd5d747d244726af93c726672408b\",\n \"content_type\":\"application/vnd.analysis.facerec\",\n \"name\":\"0cb2d9e90a3341b10bc9dba27f27259c\",\n \"castor_stream_type\":\"immutable\",\n \"timestamp\":1490732772033,\n \"@timestamp\":1490732772033,\n \"domainid\":\"7e7fd5d747d244726af93c726672408b\",\n \"decorates\":\"681b2470307b9260fb83542903e51828\",\n \"x_videoanalysis_meta_algorithm\":\"facial-recognition\",\n \"x_videoanalysis_meta_version:long\":9,\n \"x_videoanalysis_meta_version:date\":8700,\n \"x_videoanalysis_meta_version\":\"8.7\",\n \"hash\":\"867c10c9e6649313a3a5eed2cc76f307\",\n \"last_modified\":\"2017-03-28T20:26:08.888400Z\",\n \"x_videoanalysis_meta_version:double\":8.7,\n \"bytes\":7789991\n}]","type":"text"}]},{"type":"paragraph","content":[{"text":"The search correctly finds an annotation: ","type":"text"},{"text":"0cb2d9e90a3341b10bc9dba2 ","type":"text","marks":[{"type":"code"}]}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"heading","attrs":{"level":3},"content":[{"text":"Note","type":"text"}]},{"type":"paragraph","content":[{"text":"Elasticsearch is a NoSQL (non-relational) database that does not support joins directly, so queries for a primary object and an annotation object cannot be combined.","type":"text"}]}]},{"type":"paragraph"}],"version":1}

Browser not supported