...
| Anchor | ||||
|---|---|---|---|---|
|
1.1 provides an overview of a FileFly system. All communication between FileFly components is secured with Transport Layer Security (TLS). The individual components are described below.
| Anchor | ||||
|---|---|---|---|---|
|
FileFly System Overview
...
After completing the installation process, FileFly Tools must be configured via the Admin Portal web interface. The FileFly Admin Portal will be opened automatically and can be found later via the Start Menu.
The interface will lead you through the process for installing your license.
For production licensed installations, a 'Backup & Scrub Grace Period' setup page will be displayed. Please read the text carefully and set the minimum grace period as appropriate and after consulting with your backup plan – see also 7.2. This value may be revised later via the 'Settings' page.
...
Note: The servers that form the High-Availability Gateway cluster must NOT be members of a Windows failover cluster.
For further DNS recommendations, refer to 2.1.
| Anchor | ||||
|---|---|---|---|---|
|
...
Run the Caringo FileFly Agent.exe
Select install location
Select migration or Gateway role as appropriate, refer to 2.3.1
If installing a FileFly Gateway, select the desired plugins
Follow the instructions to activate the agent via FileFly Admin Portal
...
For further information see 5.3.1.
Important: If any type of clustering is used, ensure that FileFly Agent for Windows is installed on ALL cluster nodes.
...
Installation of the FileFly FPolicy Server software requires careful preparation of the NetApp Filer and the FileFly FPolicy Server machines. Instructions are provided in 4.2.
Note: Legacy 7-Mode Filers require a different procedure at FileFly FPolicy Server installation time – see 4.3.
| Anchor | ||||
|---|---|---|---|---|
|
...
Generally, antivirus software will not cause demigrations during normal file access. However, some antivirus software will demigrate files when performing scheduled file system scans.
Prior to production deployment, always check that installed antivirus software does not cause unwanted demigrations. Some software must be configured to skip offline files in order to avoid these inappropriate demigrations. Consult the antivirus software documentation for further details.
If the antivirus software does not provide an option to skip offline files during a scan, Caringo FileFly Agent may be configured to deny demigration rights to the antivirus software. Refer to 13.5 for more information.
It may be necessary for some antivirus products to exempt the Caringo FileFly Agent process from real-time protection (scan-on-access). For example, using Microsoft Security Essentials (MSE), it is necessary to add e.g. C:\Program Files\Caringo FileFly\FileFly Agent\<version>\mwi_clmb.exe to the 'Excluded Processes' list. Be sure to update the exclusion whenever FileFly is upgraded.
...
Check for other applications that open all the files on the whole volume. Audit scheduled processes on the file server – if such processes cause unwanted demigration, it may be possible to block them (see 13.5).
| Anchor | ||||
|---|---|---|---|---|
|
...
Configure email notifications to monitor system health and Task activity – see 5.10
Enable syslog on agents – see 13.1
| Anchor | ||||
|---|---|---|---|---|
|
...
Run 'Gather Statistics' Policies
Examine reports
Examine Server statistics – see 5.3
For more detail, examine demigrates in file server agent.log files
...
Requires: Source(s) Included in Community Edition: yes
Generate statistics report(s) for file sets at the selected Source(s). Optionally include statistics by file owner. By default, owner statistics are omitted which generally results in a faster policy run. Additionally, rules may be used to specify a subset of files on which to report rather than the whole source.
Statistics reports can be retrieved from FileFly Admin Portal – see 5.8.6.
| Anchor | ||||
|---|---|---|---|---|
|
...
Requires: Source(s) Included in Community Edition: yes
Scan all stubs present on a given Source, revalidating the relationship between the stubs and the corresponding files on secondary storage. This operation is required following a restore from backup and should be performed on the root of the restored source volume.
If only Write Once Read Many (WORM) destinations are in use, this policy is not required.
Important: This revalidation operation MUST be integrated into backup/restore procedures, see 7.2.
| Anchor | ||||
|---|---|---|---|---|
|
...
Requires: Source(s), Rule(s) Included in Community Edition: no
Generate a disaster recovery file for Caringo FileFly DrTool by analyzing files at the selected Source(s). FileFly DrTool can use the generated file(s) to recover or update source files.
Note: DrTool files generated from Source will account for renames.
DrTool files can be retrieved from FileFly Admin Portal – see 5.8.7.
| Anchor | ||||
|---|---|---|---|---|
|
...
When creating a production deployment plan, please refer to 2.6.
| Anchor | ||||
|---|---|---|---|---|
|
...
See Installing FileFly Agent for Windows 2.3.3
| Anchor | ||||
|---|---|---|---|---|
|
...
This section describes Windows-specific considerations only and should be read in conjunction with 2.6.
| Anchor | ||||
|---|---|---|---|---|
|
...
This section describes support for 'Cluster-mode' NetApp Filers. For '7-mode' Filers (that is, 7.x Filers and 8.x Filers operating in '7-mode'), see 4.3.
| Anchor | ||||
|---|---|---|---|---|
|
...
Caringo FileFly FPolicy Servers require EXCLUSIVE use of SMB connections to their associated NetApp Vservers. This means Explorer windows must not be opened, drives must not be mapped, nor should any UNC paths to the filer be accessed from the FileFly FPolicy Server machine. Failure to observe this restriction will result in unpredictable FPolicy disconnections and interrupted service.
When creating a production deployment plan, please refer to 2.6.
| Anchor | ||||
|---|---|---|---|---|
|
...
Except when following the procedure in 4.2.5, Robocopy must not be used with the /b (backup mode) switch when copying FileFly NetApp stubs.
When in backup mode, robocopy attempts to copy stub files as-is rather than demigrating them as they are read. This behavior is not supported.
Note: The /b switch requires Administrator privilege – it is not available to normal users.
...
This section describes support for NetApp Filers 7.3 and above including 8.x Filers operating in '7-mode'. For version 9.x Filers and 8.x Filers running in 'Cluster-mode', see 4.2.
| Anchor | ||||
|---|---|---|---|---|
|
...
Caringo FileFly FPolicy Servers require EXCLUSIVE use of SMB connections to their associated NetApp filers/vFilers. This means Explorer windows must not be opened, drives must not be mapped, nor should any UNC paths to the filer be accessed from the FileFly FPolicy Server machine.
Demigrations cannot be triggered by applications running locally on the FileFly FPolicy Servers since the Filer ignores these requests. This is an FPolicy restriction.
When creating a production deployment plan, please refer to 2.6.
| Anchor | ||||
|---|---|---|---|---|
|
...
When using a High-Availability configuration, be sure to use the same netapp.cfg across all nodes and remember to restart each node's service.
...
The swarm scheme should only be used when accessing Swarm storage nodes directly.
If accessing Swarm storage via a CloudScaler Gateway, the cloudscaler scheme must be used instead, see 4.5.
Note: FileFly software does not support access to storage nodes via an SCSP Proxy.
...
Run the Caringo FileFly Agent.exe, select the FileFly Gateway role (see 2.3.3) and select FileFly Swarm Plugin on the 'Components' page
Follow the prompts to complete the installation
...
Tick 'Include metadata HTTP headers' to store per-file metadata with the destination objects, such as original filename and location, content-type, owner and timestamps – see 4.4.7 for details. File extension to content-type mappings may be customized by editing the swarm-mimetypes file, found in C:\Program Files\Caringo FileFly\data\swarm.data\.
Also tick 'Include Content-Disposition' to include original filename for use when downloading the target objects directly using a web browser.
...
Caringo CloudScaler provides a multi-tenanted object storage platform built upon Swarm storage nodes. The FileFly cloudscaler scheme must only be used when accessing the storage via CloudScaler. To store data on Swarm nodes directly, the swarm scheme must be used instead, see 4.4.
| Anchor | ||||
|---|---|---|---|---|
|
...
Run the Caringo FileFly Agent.exe, select the FileFly Gateway role (see 2.3.3) and select FileFly CloudScaler Plugin on the 'Components' page
Follow the prompts to complete the installation
...
Tick 'Include metadata HTTP headers' to store per-file metadata with the destination objects, such as original filename and location, content-type, owner and timestamps – see 4.5.7 for details. File extension to content-type mappings may be customized by editing the cloudscaler-mimetypes file, found in C:\Program Files\Caringo FileFly\data\cloudscaler.data\.
Also tick 'Include Content-Disposition' to include original filename for use when downloading the target objects directly using a web browser.
...
Run the Caringo FileFly Agent.exe, select the FileFly Gateway role (see 2.3.3) and select FileFly S3 Plugin on the 'Components' page
Follow the prompts to complete the installation
...
Select 'Amazon AWS S3'
If required, fill in the 'HTTPS Proxy' section (not recommended for performance reasons)
Enter your Amazon Web Services (AWS) account details
Select authentication 'Signature Type'
AWS4-HMAC-256 is required for newer Amazon data centers
AWS2 may be faster – it is safe to try this first
Click Manage Buckets…
Click New to create a new bucket
Click Options to set storage options for the selected bucket (see 4.6.3)
To copy a URI to the clipboard for use in the Admin Portal Destination object:
click Get Migration URI to select a partition
Optionally, check 'Allow Reduced Redundancy (via s3rr:// URIs)'
...
Reduced Redundancy Storage (RRS) is a slightly lower cost Amazon S3 storage option (when compared to the S3 Standard storage class) where data is replicated fewer times. Care should be taken when assessing whether the lower durability of RRS is appropriate.
Reduced Redundancy must be enabled via Caringo FileFly S3 Config, see 4.6.5.
| Anchor | ||||
|---|---|---|---|---|
|
...
Run the Caringo FileFly Agent.exe, select the FileFly Gateway role (see 2.3.3) and select FileFly S3 Plugin on the 'Components' page
Follow the prompts to complete the installation
...
Run the Caringo FileFly Agent.exe, select the FileFly Gateway role (see 2.3.3) and select FileFly Azure Plugin on the 'Components' page
Follow the prompts to complete the installation
...
Run the Caringo FileFly Agent.exe, select the FileFly Gateway role (see 2.3.3) and select FileFly Google Plugin on the 'Components' page
Follow the prompts to complete the installation
...
Servers are machines with activated agents – see 5.3. Status and health information for each Server is shown on the 'Servers' tab.
...
Sources are volumes or folders upon which Policies may be applied (i.e., locations on the network from which files may be Migrated) – see 5.4.
| Anchor | ||||
|---|---|---|---|---|
|
Destinations are locations to which Policies write files (i.e., locations on the network to which files are Migrated) – see 5.5.
| Anchor | ||||
|---|---|---|---|---|
|
Rules are used to filter the files at a Source location so that only the required subset of files is acted upon – see 5.6.
| Anchor | ||||
|---|---|---|---|---|
|
Policies specify which operations to perform on which files. Policies bind Sources, Rules and Destinations – see 5.7.
| Anchor | ||||
|---|---|---|---|---|
|
Tasks define schedules for Policy execution – see 5.8.
Note: The Caringo FileFly Webapps service needs to run continuously in order to launch scheduled tasks.
...
View the Global Task Log
Stop All Tasks
Suspend/Start Scheduler to disable/enable scheduled Task execution
Click the name of a Task to reveal the details of the particular Task run
Click Details to expand all running/recent Task details – see 5.9.1
Clear the 'Recent Task History'
Show/Hide Successful Tasks in the 'Recent Task History' section
...
Note: To add a new node to an existing cluster, refer to 5.3.4.
| Anchor | ||||
|---|---|---|---|---|
|
...
A server may be reactivated by following the same procedure as for adding a new server – see 5.3.1.
| Anchor | ||||
|---|---|---|---|---|
|
...
From the 'Sources' tab, click Create Source
Name the Source and optionally enter a comment
Optionally, tag the Source by either entering a new tag name, or selecting an existing tag from the drop-down box
Create a URI using the browser panel (see 5.4.5)
Optionally, select inclusions and exclusions – see 5.4.4
Note: To exclude a directory from being actioned use a Rule. See 11.
Tip: On the 'Overview' tab, click on the Create Source 'Quick Link' to go directly to the 'Create Source' page.
...
| Anchor | ||||
|---|---|---|---|---|
|
Directory Inclusions & Exclusions
Within a given Source, individual directory subtrees may be included or excluded to provide greater control over which files are eligible for policy operations. Excluded directories will not be traversed.
In the Source editor, once a URI has been entered/created, the directory tree may be expanded and explored in the 'Directory Inclusions & Exclusions' panel ( 5.1). By default, all directories will be ticked, marking them for inclusion.
Branches of the tree are collapsed automatically as new branches are expanded. However, directories representing the top of an inclusion/exclusion remain visible even if the parent is collapsed.
Ticking/unticking a directory will include/exclude that directory and its subdirectories recursively. Note that the root directory (the Source URI) may also be unticked.
The 'other dirs' entry represents both subdirectories that may be created in the future, as well as subdirectories not currently shown because their parent directories are collapsed.
When a Source's inclusions and exclusions are edited at a later date, the Validate and edit button must be clicked prior to modifying the contents of the panel. Validation verifies that directories specified for inclusion/exclusion still exist, and assists with maintaining the consistency of the configuration if they do not.
...
From the 'Rules' tab, click Create Rule
Name the Rule and optionally enter a comment
Optionally, to omit the files that match this Rule, check Negate
Complete the following as required:
Note: Creating a compound rule is detailed later, see 5.6.11.
Tip: On the 'Overview' tab, click on the Create Rule 'Quick Link' to go directly to the 'Create Rule' page.
...
wildcard patterns, e.g. *.doc (see 5.6.5)
regular expressions, e.g. /2004-06-[0-9][0-9]\.log/ (see 5.6.6)
Notes:
files match if any one of the patterns in the list match
all whitespace before and after each file pattern is ignored
patterns starting with '/' match the entire path from the Source URI
patterns NOT starting with '/' match files in any subtree
patterns are case-insensitive
...
The 'Patterns' field uses the same format as the 'File Matching Patterns' field see 5.6.4
Windows users are of the form domain\username
...
From the 'Rules' tab, click Create Compound Rule
Name the Rule and optionally enter a comment
Optionally, to omit the files that match this Compound Rule, check Negate
Click on the 'Combine logic' drop-down box and choose the logic type (see Combine Logic 5.6.12)
From the 'Available' box in the 'Rules' section, select the names of the Rules to be combined into the Compound Rule, and click Add
To remove a Rule from the 'Selected' box, select the Rule name and click Remove
...
From the 'Tasks' tab, click Create Task
Name the Task and optionally enter a comment
In the 'Policies' section, select Policies from the 'Available' list using the Add/Remove buttons
Select the times to execute the Policies from the 'Schedule' section
Optionally, enable completion notification – see 5.9.3
Tip: On the 'Overview' tab, click on the Create Task 'Quick Link' to go directly to the 'Create Task' page.
...
Click Download DrTool Files on the 'Task Details' page to access the results of Policies that produce DrTool Files (i.e. the 'Create DrTool File From Destination' and 'Create DrTool File From Source' operations).
The 'Download DrTool Files' page also allows files from previous runs of the Task to be downloaded. To configure retention options for these files see 5.10.
| Anchor | ||||
|---|---|---|---|---|
|
...
The operation counts are updated in real time as the task runs. Operations will automatically be executed in parallel, see 13.4 for more details.
Note: The locked, skipped and failed counts are not shown if they are zero.
If multiple Tasks are scheduled to run simultaneously, the common elements are grouped in the 'Running Tasks' section and the Tasks are run together using a single traversal of the file system.
When a Task has finished running, summary information for the Task is displayed in the 'Recent Task History' section on the 'Overview' tab, and details of the Task are listed in the log file.
Tip: click the Task name next to the log links in the expanded view of a running or finished task to jump straight to the 'Task Details' page to access statistics, DrTool files etc.
FileFly Admin Portal can also be configured to send a summary of recent Task activity by email, see 5.10.
| Anchor | ||||
|---|---|---|---|---|
|
...
When a Task finishes running, regardless of whether it succeeds or fails, a completion notification email may be sent as a convenience to the administrator. This notification email contains summary information similar to that available in the 'Recent Task History' section of the 'Overview' tab.
To use this feature, email settings must be configured beforehand – see 5.10. Notifications for a given task may then be enabled either by:
...
The Force Backup Now button allows a backup of the current configuration to be taken without waiting for the next scheduled backup time.
Please refer to 6.2 for further information.
| Anchor | ||||
|---|---|---|---|---|
|
Specify work hours and work days which may be used by migration policies to pause migration activity during the busy work period.
Individual policies may then be configured to pause during work hours – see 5.7 for supported operations.
...
Configuration backup can be scheduled on the Admin Portal's 'Settings' page – see 5.10. A default schedule is created at installation time to backup configuration once a week.
Configuration backup files include:
...
Install the same version of Caringo FileFly Agent or FileFly FPolicy Server as normal (see 2.3.3)
Stop the 'Caringo FileFly Agent' service
Restore the contents of the following directories from backup:
C:\Program Files\Caringo FileFly\data\FileFly Agent\
C:\Program Files\Caringo FileFly\logs\FileFly Agent\
Restart the 'Caringo FileFly Agent' service
...
Ensure that the restoration of stubs is included as part of your backup & restore test regimen.
When using Scrub policies, ensure the Scrub grace period is sufficient to cover the time from when a backup is taken to when the restore and Post-Restore Revalidate steps are completed (see below).
It is strongly recommended to set the global minimum grace period accordingly to guard against the accidental creation of scrub policies with insufficient grace. To update this setting, see 5.10.
Important: It will NOT be possible to safely restore stubs from a backup set taken more than one grace period ago.
...
| Anchor | ||||
|---|---|---|---|---|
|
Please consult 4.2.5 for information regarding snapshot restore on Cluster-mode NetApp Filers.
...
On the Admin Portal 'Overview' tab, click Suspend Scheduler
Run the Caringo FileFly Tools.exe installer
Upgrade all FileFly Agents and FileFly FPolicy Servers (see 8.2)
Resolve any warnings displayed on the 'Overview' tab
On the 'Overview' tab, click Start Scheduler
...
The 'File Pattern' field will match either filenames only (and search within all directories), or filenames qualified with directory paths in the same manner as filename patterns in FileFly Admin Portal Rules – see 5.6.4.
For the purposes of file pattern matching, the top-level directory is considered to be the top level of the entire URI path. This may be different to the top-level of the original Source URI.
...
All files may be recovered either as a batch process using the command line (see 9.7) or interactively as follows:
...
When recovering to a new location, always use an up-to-date DrTool file generated by a 'Create DrTool File From Source' Policy.
To rewrite source file URIs to the new location, use the -csu command line option to update the prefix of each URI. Once these URI substitutions have been applied (and checked in the GUI) files may be recovered as previously outlined. The -csu option is further detailed in 9.7.
Important: DO NOT create stubs in a new location and then continue to use the old location. To avoid incorrect reference counts, only one set of stubs should exist at any given time.
...
In FileFly DrTool, source files may be updated to reflect a destination URI change through use of the -cmu command line option – detailed in 9.7.
To apply the destination URI substitution to existing files on the source, select 'Update All Source Files…' from the Edit menu. When given the option, elect to update substituted entries only.
Note: This operation must always be performed using an up-to-date DrTool file generated by a 'Create DrTool File From Source' Policy.
...
Run a 'Create DrTool File From Destination' Policy, see 3.9.
| Anchor | ||||
|---|---|---|---|---|
|
...
Excluding directories at fixed locations is most easily achieved using the 'Directory Inclusions & Exclusions' panel in the Source editor – see 5.4.4.
The example of excluding Wilma's 'Personal' directory can be accomplished by unticking that directory, as shown in 11.1.
| Anchor | ||||
|---|---|---|---|---|
|
...
Create a Rule to match PDF files within a 'DOC' directory
Add this Rule to the Policy
Edit the policy (see 5.7.3)
Add the Rule created in step 1; the selected Rules for the policy will now be 'all files modified more than 6 months ago' AND the newly created exclusion Rule
Save the policy
...
Create a Rule to match PDF files within a 'DOC' directory that is one directory deep in the Source.
Add this Rule to the 'Migrate Home Directories' policy
Edit the policy (see 5.7.3)
Add the Rule created in step 1; the selected Rules for the policy will now be 'all files modified more than 6 months ago' AND the newly created exclusion Rule
Save the policy
...
Normally, the administration password is changed on the 'Settings' page as needed – see 5.10.
However, should the system administrator forget the username or password entirely, the credentials may be reset as follows:
...
| Anchor | ||||
|---|---|---|---|---|
|
As described in 4.1.7, on Windows file systems, FileFly can monitor stub deletion events in order to make corresponding secondary storage files eligible for removal using Scrub Policies.
This feature is not enabled by default. It must be enabled on a per-volume basis either by specifying volume GUIDs (preferred) or drive letters. Volume GUIDs may be determined by running the Windows mountvol command or powershell Get-WmiObject -Class win32_volume. For Windows clustered volumes, the cluster volume must be specified using a volume GUID.
Note: This feature should not be configured to monitor events on backup destination volumes. In particular, some basic backup tools such as Windows Server Backup copy individual files to VHDX backup volumes in a manner which is not supported and so such volumes must not be configured for Stub Deletion Monitoring. Of course, deletions may still be monitored on source data volumes.
...
For many errors, the most salient details are to be found in the last line of the error tree (or the last line of the first branch of the error tree). Consider the following last line:
[11] ERR_DMSOCKETUTIL_GETROUNDROBINCONNECTEDSOCKET_FAILED [ERR_ADD_COUL DCOULD_NOT_RESOLVE_HOSTNAME] [host was [svr1279.example.com]]
It is fairly clear that this error represents a failure to resolve the server hostname svr1279.example.com. As with any other software, the administrator's next steps will include checking the spelling of the DNS name, the server's DNS configuration and whether the hostname is indeed present in DNS.
...
A description of the issue – be sure to include:
how long the issue has been present
how regularly the issue occurs
any changes made to the environment or configuration
any specific circumstances which trigger the issue
does the issue occur for a particular file and/or server?
Caringo FileFly version
Swarm and CloudScaler versions
Operating System(s)
fltmc instances command output from migration source servers
Saved system info (.NFO) from msinfo32.exe
Plugins in use
Source and Destination URIs
Applicable Log Files
see 14.1 for log locations
include Admin Portal logs
include source agent logs
include destination/gateway agent logs
remember to include all nodes in each agent cluster
zip the entire log folders wherever possible
Generate a system configuration file (support.zip) by clicking the link on the Admin Portal 'About' page
Any other error messages – include screenshots if necessary
...