Swarm 11.3 - Updated August 2020

• Swarm Performance: This release of Swarm Storage enhances both cluster performance and memory management. Cluster shutdowns and startups are faster, and better memory management and support for nodes with limited memory improves Swarm performance under high client loads. This release also includes changes that improve Swarm stability and administration, through better handling of volume removal and hotplugging, smoother rebooting, and stronger logging security.

• Folder Listing Service: With Gateway 7.0, folder listing support across Swarm clients (such as SwarmFS and S3) has been both completely rearchitected and also newly centralized within Content Gateway. Folder listing is what renders the virtual folders (prefixes) on named Swarm objects (such as FY2019/Q3/object.jpg) in to familiar folders on users' file systems. The new service makes full use of Elasticsearch 6 features and is no longer bound by ES listing limits. Centralization means that future listing improvements are easier and faster to roll out.

• UI Changes for NFS Exports: Dependency on Elasticsearch is removed for NFS export definitions. The Swarm UI is updated to reflect the less complex NFS definitions.

• SwarmFS Redesign: With 3.0, SwarmFS removes dependency on Elasticsearch versioning and makes full use of the new folder listing service in Content Gateway 7.0. The new architecture brings many benefits to SwarmFS implementations, such as centralized authentication through Gateway, improved query security, and freedom to move the Elasticsearch cluster to a more secure network location.

Swarm 11.2 - Updated June 2020

• Next-Generation SEND Method: This release of Swarm Storage and Content Gateway focuses on adding support for the next generation of SCSP SEND, which are foundational to future capabilities. SCSP SEND allows forcing an object to be written immediately another cluster for which a replication feed exists.

Swarm 11.1 - Updated April 2020

• Grafana Dashboards for Swarm Monitoring: To offer sophisticated visualization of the Prometheus Node Exporter and related Swarm data, DataCore has published public Grafana dashboards for monitoring Swarm implementations. Search the dashboards for Caringo to see all dashboards for Swarm products and features: https://grafana.com/grafana/dashboards?search=caringo. See https://perifery.atlassian.net/wiki/spaces/public/pages/2443812753.
  

  Customized dashboards are available for the following products: 

  • Swarm System Monitoring (Separate dashboards for Storage 11.0 and Storage 10.2.): - Covers cluster health, capacity, indexing, licensing, temperature, and network and CPU loads, as well as cluster-wide operations:
    
    
    

  • Gateway Monitoring (for Gateway 6.3) - covers details of CPU load, operations, connections, and HTTP status codes:
    
    
    

  • Video Clipping (optional, for Gateway 6.2+) - covers numbers, rates, and error counts for video clipping requests (see https://perifery.atlassian.net/wiki/spaces/public/pages/2990833665); errors are counted by stage (preprocessing, processing, postprocessing), to help with troubleshooting:
    
    
    

• Gateway Support for Untenanted Objects: Untenanted objects are unnamed objects that are written to Swarm without specifying a domain. Gateway adds support for untenanted objects, and accepts the Swarm setting enforceTenancy=false and provides Content Metering metrics for these objects. Upgrading to Content Gateway is possible if still using SCSP Proxy due to untenanted unnamed objects. Gateway 6.2.0 accepts untenanted objects, so it is a drop-in replacement for SCSP Proxy, which is deprecated. With enforceTenancy=false, untenanted objects can continue to be created with existing client applications. Note: untenanted objects are incompatible with the Content UI.

• Elasticsearch 6: Swarm supports and ships with Elasticsearch 6, which is a version allowing upgrades-in-place (without reindexing) going forward several releases. Both ES2 and ES5 are deprecated in the next release. 

• Python 3 throughout Swarm: All Swarm Storage usage of Python 2 is uniformly upgraded to Python 3, which brings with it a small performance boost, up to 20% improvement for high loads.

• Modernization: Extensive work has modernized the Linux kernel to Debian 10 and the drivers and components, which allowed for comprehensive updates across Swarm's third-party tools and dependencies.

• Large Cluster Support: This release includes performance improvements for very large clusters, which benefits clusters of all sizes.

• Faster Uploads from Content UI: To speed the performance of large uploads, the part size for multipart uploads has been increased to 25 MB, which is a common S3 client default part size.

Swarm 11.0 - Updated September 2019

• S3 Backup for DR: Swarm allows tiering to public cloud services for convenient and affordable off-premises storage for disaster recovery (DR). Content Gateway supports Amazon S3, which has the widest support in the industry, so S3-compatible endpoints are the first cloud destination from Swarm. By implementing an S3 backup feed from Swarm, backups become continuous, have minimal latency, and need minimal intervention and monitoring. The S3 Backup leverages Swarm's mature feed mechanism, which offers long-term iteration over objects in the cluster, proven method for tracking work as it is performed, and mechanisms for TLS connections and forward proxies. Having the parallelism of the entire cluster makes best use of network bandwidth, while sending the backups through a forward proxy enables bandwidth throttling.
  

  S3 Backup occurs as an integral part of an operating Swarm cluster. After the feed is started, the progress can be monitor with warnings of blockages and particular object failures, as with any other feed. The S3 Backup feed honors the versioning settings in the cluster, as enabled, disabled, or suspended throughout the domains and buckets. The feed keeps the backup current and trimmed: when disabling Swarm versioning on buckets or domains, delete buckets or domains, or have object lifepoints expire, the Swarm feeds mechanism processes the expired content as deleted, allowing the S3 Backup feed to clear them from the S3 bucket.

• S3 Backup Restore: The Restore tool runs outside of Swarm, using a command-line interface for executing the data and restoration tasks. Restore what is needed: either the entire cluster, or portions. Swarm supports bulk restores at the granularity of cluster, domain, or bucket, as well as more surgical restores of a few objects. Run multiple copies to achieve a faster, parallel recovery. 

• Video Clipping / Partial File Restore: As soon as a video is uploaded into a bucket in Swarm, it is viewable and sharable from the Content UI. With the new Video Clipping controls optionally installed, portions can be excerpted out and stored as new, standalone videos within Swarm. The tool saves the clip into the same bucket as the source video, creating a default name that includes the original name and the start and end times of the clip. Each clip created is a standalone video, not a stub pointing to a range in the original; therefore, there is no dependency on the original, which speeds and simplifies distribution.
  

• Content Sharing: A new Share button appears next to the name when selecting an object in a Content UI listing to view it. The button opens a menu of commands for content sharing, including copying the URL to the local clipboard, downloading the file locally, and opening the default email program to email the link to someone else. 
  

• Large Uploads through Content UI: The Content UI file uploader has been redesigned to write directly to Swarm storage and bypass spooling altogether, which removes the prior 4 GB limit. The Content UI accepts more and larger files and is able to recover and resume uploads that encounter errors, and the uploader is compatible with Swarm containerization.

• Containerization Architecture: The architecture work of Swarm 10 continues with build-out of support for containerization, so Swarm storage nodes can be managed in containers.

• Prometheus Node Exporter: The Prometheus Node Exporter preview has new, global-friendly naming for the node exporter metrics files, and the statistics have richer state information, including node status (idle, mounting, initializing, retiring).

• Platform IPMI Credential Storage: Platform Server can store the IPMI username and password for an externally managed Swarm chassis. Credentials do not need to entered when Platform runs power on/off commands using IPMI over LAN by storing the credentials.

• SwarmFS Tuning: The SwarmFS 2.3 release adds several new Advanced settings for tuning SwarmFS behavior and performance in different implementations.

• S3 Compatibility: Content Gateway continues to keep pace with the evolving S3 protocol changes to maintain best-in-class compatibility with applications written for the AWS S3 protocol.

• FileFly 3.1: The 3.1 release of FileFly features new, generic support for S3-compatible endpoints as well as numerous performance and UI improvements.