How to purge FileFly content from Swarm

When a file on a Source disk is migrated via FileFly, it is replaced with a stub file in the Source disk. FileFly creates an object in Swarm that stores the file contents and its metadata. Given the nature of FileFly's data protection, deleting the stub does nothing to purge unwanted content from Swarm storage, which is a 2-step process: (1) demigrate files from Swarm + (2) scrub objects from Swarm.

A "Scrub Destination Policy" requires demigrating the unwanted files. The purpose of a Scrub policy is to remove the copy of data residing in Swarm after the data has been demigrated from primary storage.

Manual Demigration

Caution

Do not run out of disk space. Perform demigrations in batches if in doubt.

Follow this process to delete files and verify they are no longer consuming space in Swarm:

Demigrate the files.
- Create a Rule in the FileFly Admin Portal to demigrate based on a filename pattern or attributes, such as size, date, or owner.
- Access the files or change the owner.
Delete the files (not stubs) no longer needed to be archived in Swarm.
Important: If a stub file is accidentally deleted, use DrTool to recreate it (search for Recreating Stubs in the FileFly Guide).
FileFly keeps track of which files are demigrated by way of metadata objects stored in Swarm. These metadata references stay available in case a stub file needs to be recreated to restore access to the data before the grace period expires once a file is demigrated.
Wait the (configurable) 24-hour grace period, then run a Scrub (Aggressive) task.
The task deletes the objects related to those files from Swarm. Monitor what objects FileFly stores in Swarm by using a listing request:
curl -iL "http://SWARM/filefly-destination-bucket?format=json&fields=x-alt-meta-name,name,tmborn,x-alt-meta-path"
Allow several HP cycles for the space to become fully available in Swarm. Use the Swarm UI to monitor the trapped space being freed (see Viewing and Managing the Cluster).

Scripted Demigration

To delete files in a batch and a rule cannot be created to demigrate all files or there is not enough space on disk to demigrate the files for deletion, use a script to read the last few bytes of a file (to trigger a demigration) and then perform a delete.

The PowerShell script below reads a file listing in C:\testdata\ then reads each filepath in a loop and outputs the last few lines of the file. Removing the item triggers a full demigration job.

(Get-ChildItem C:\testdata\ -Recurse).fullname > filename.txt
ForEach ($files in Get-Content .\filename.txt){
echo $files >> filesdone.txt
Get-Content -tail 5 $files | Out-File and2bits.txt
Remove-Item $files
}

Because each file's migration state has been changed to demigrated when a scrub runs, the object is deleted after the global grace period is past.