4 Sources and Destinations

The following pages describe the characteristics of the Sources and Destinations supported by Caringo FileFly^TM. Planning, setup, usage and maintenance considerations are outlined for each storage platform.

IMPORTANT: Read any relevant sections of this chapter prior to deploying FileFly^TM in a production environment.

4.1 Supported Schemes

• win for Sources that are Windows NTFS or ReFS volumes
• netapp for Sources on a NetApp Filer
• swarm for Destinations using Caringo Swarm
• cloudscaler for Destinations using Caringo CloudScaler

4.1.1 Supported Features

4.2 Microsoft Windows

4.2.1 Migration Support

Windows NTFS volumes may be used as migration sources. On Windows Server 2012 R2 and above, ReFS volumes may also be used as migration sources.

Windows stub files can be identified by the ‘O’ (Offline) attribute in Explorer. Depending on the version of Windows, files with this flag may be displayed with an overlay icon.

Stub Deletion Monitoring

On Windows, the FileFly Agent monitors stub deletions to identify secondary storage files that are no longer referenced in order to maximize the usefulness of Scrub Policies. This feature extends not only to stubs that are directly deleted by the user, but also to other cases of stub file destruction such as overwriting a stub or renaming a different file over the top of a stub.

4.2.2 Planning

When creating a production deployment plan, please refer to §2.6 .

Cluster Support

Clustered volumes managed by Windows failover clusters are supported. However, the Cluster Shared Volume (CSVFS) feature is NOT supported. As a result, on Windows Server 2012 and above, when configuring a ‘File Server’ role in the Failover Cluster Manager, ‘File Server for general use’ is the only supported File Server Type. The ‘Scale-Out File Server for application data’ File Server Type is NOT supported.

When using clustered volumes in FileFly^TM URIs, ensure that the resource FQDN appropriate to the volume is specified rather than the FQDN of any individual node.

4.2.3 Setup

Installation

See Installing FileFly Agent for Windows §2.3.3

4.2.4 Usage

URI Format

win://{servername}/{drive letter}/[{path}]

Where:

• servername – Server FQDN or Windows Failover File Server Resource FQDN
• drive letter – Windows volume drive letter

Examples:

win://fs1.example.com/d/projects

win://fs2.example.com/e/

Note: Share names and mapped drives are not supported.

4.2.5 Interoperability

Microsoft DFS Namespaces (DFSN)

DFSN is supported. FileFly^TM Sources must be configured to access volumes on individual servers directly rather than through a DFS namespace. Users and applications may continue to access files and stubs via DFS namespaces as normal.

Microsoft DFS Replication (DFSR)

DFSR is supported for:

• Windows Server 2016
• Windows Server 2012 R2
• Windows Server 2008 R2

FileFly Agents must be installed (selecting the migration role during installation) on EACH member server of a DFS Replication Group prior to running migration tasks on any of the group’s Replication Folders.

If adding a new member server to an existing Replication Group where FileFly^TM is already in use, FileFly Agent must be installed on the new server first.

When running policies on a Replicated Folder, sources should be defined such that each policy acts upon only one replica. DFSR will replicate the changes to the other members as usual.

Read-only (one-way) replicated folders are NOT supported. However, read-only CIFS shares can be used to prevent users from writing to a particular replica as an alternative.

Due to the way DFSR is implemented, care should be taken to avoid writing to stub files that are being concurrently accessed from another replica.

In the rare event that DFSR-replicated data is restored to a member from backup, ensure that DFSR services on all members are running and that replication is fully up-to-date (check for the DFSR ‘finished initial replication’ Windows Event Log message), then run a Post-Restore Revalidate Policy using the same source used for migration.

Retiring a DFSR Replica

Retiring a replica effectively creates two independent copies of each stub, without updating secondary storage. To avoid any potential loss of data:

1. Delete the contents of the retired replica (preferably by formatting the disk)
2. Run a Post-Restore Revalidate Policy on the remaining copy of the data

If it is strictly necessary to keep both, now independent, copies of the data and stubs, then run a Post-Restore Revalidate Policy on both copies separately (not concurrently).

Preseeding a DFSR Replicated Folder Using Robocopy

The most common use of Robocopy with FileFly^TM stubs is to preseed or stage initial synchronization. When performing such a preseeding operation:

• both servers must have FileFly Agent installed before preseeding
• preseed using robocopy with the /b flag (to copy stubs as-is)
• for new Replicated Folders, ensure that the ‘Primary member’ is set to be the original server, not the preseeded copy
• once preseeding is complete and replication is fully up-to-date (check for the DFSR ‘finished initial replication’ Windows Event Log message), it is recommended to run a Post-Restore Revalidate Policy on the original FileFly^TM Source

Note: If the process above is aborted, be sure to delete all preseeded files and stubs (preferably by formatting the disk) and then run a Post-Restore Revalidate Policy on the original FileFly^TM Source.

Robocopy (Other Uses)

Robocopy will, by default, demigrate stubs as they are copied. This is the same behavior as Explorer copy-paste, xcopy etc..

Robocopy with the /b flag (backup mode – must be performed as an administrator) will copy stubs as-is.

Robocopy /b is not recommended. If stubs are copied in this fashion, the following must be considered:

• for a copy from one server to another, both servers must have FileFly Agent installed
• this operation is essentially a backup and restore in one step, and thus inappropriately duplicates stubs which are intended to be unique
• after the duplication, one copy of the stubs should be deleted immediately
• run a Post-Restore Revalidate policy on the remaining copy
• this process will render the corresponding secondary storage files unscrubbable, even after they are demigrated

Windows Data Deduplication

If a Windows source server is configured to use migration policies and Windows Data Deduplication, it should be noted that a given file can either be deduplicated or migrated, but not both at the same time. FileFly^TM migration policies will automatically skip files that are already deduplicated. Similarly, Windows will skip FileFly^TM stubs when deduplicating.

When using both technologies, it is recommended to configure Data Deduplication and Migration based on file type such that the most efficacious strategy is chosen for each type of file.

Note: Microsoft’s legacy Single Instance Storage (SIS) feature is not supported. Do not use SIS on the same server as Caringo FileFly Agent.

Windows Shadow Copy

Windows Shadow Copy – also known as Volume Snapshot Service (VSS) – allows previous versions of files to be restored, e.g. from Windows Explorer. This mechanism cannot be used to restore a stub. Restore stubs from backup instead – see Chapter 7.

4.2.6 Behavioral Notes

Junction Points & Symlinks

With the exception of volume mount points, junction points will be skipped during traversal of the file system. Symlinks are also skipped. This ensures that files are not seen – and thus acted upon – multiple times during a single execution of a given policy. If it is intended that a policy should apply to files within a directory referred to by a junction point, either ensure that the Source encompasses the real location at the junction point’s destination, or specify the junction point itself as the Source.

Mount-DiskImage

On Windows 8 or above, VHD and ISO images may be mounted as normal drives using the PowerShell Mount-DiskImage cmdlet. This functionality can also be accessed via the Explorer context menu for an image file.

A known limitation of this cmdlet is that it does not permit sparse files to be mounted (see Microsoft KB2993573). Since migrated image files are always sparse, they must be demigrated prior to mounting. This can be achieved either by copying the file or by removing the sparse flag with the following command:

fsutil sparse setflag <file name> 0

4.3 NetApp Filer (Cluster-mode)

This section describes support for ‘Cluster-mode’ NetApp Filers. For ‘7-mode’ Filers (that is, 7.x Filers and 8.x Filers operating in ‘7-mode’), see §4.4.

4.3.1 Migration Support

Migration support for sources on NetApp Vservers (Storage Virtual Machines) is provided via NetApp FPolicy. This requires the use of a Caringo FileFly FPolicy Server. Client demigrations can be triggered via CIFS or NFS client access.

Please note that NetApp Filers currently support FPolicy for Vservers with FlexVol volumes but not Infinite volumes.

When accessed via CIFS on a Windows client, NetApp stub files can be identified by the ‘O’ (Offline) attribute in Explorer. Files with this flag may be displayed with an overlay icon. The icon may vary depending on the version of Windows on the client workstation.

4.3.2 Planning

Prerequisites

• NetApp Filer(s) must be licensed for the particular protocol(s) to be used (FPolicy requires a CIFS license)

Caringo FileFly FPolicy Servers require EXCLUSIVE use of CIFS connections to Vservers. This means Explorer windows must not be opened, drives must not be mapped, nor should any UNC paths to the filer be accessed from the FileFly FPolicy Server machine.

When creating a production deployment plan, please refer to §2.6 .

Filer System Requirements

Caringo FileFly FPolicy Server requires that the Filer is running:

• Data ONTAP version 9.1
• Data ONTAP version 9.0
• Data ONTAP version 8.2.2+

Network

Each FileFly FPolicy Server should have exactly one IP address.

Place the FPolicy Servers on the same subnet and same switch as their corresponding Vservers to minimize latency.

Antivirus Considerations

Ensure that Windows Defender or any other antivirus product installed on FileFly FPolicy Server machines is configured to omit scanning/screening NetApp shares.

Antivirus access to NetApp files will interfere with the correct operation of the FileFly FPolicy Server software. Antivirus protection should still be provided on client machines and/or the NetApp Vservers themselves as normal.

High-Availability for FileFly FPolicy Servers

It is strongly recommended to install Caringo FileFly FPolicy Servers in a High-Availability configuration. This configuration requires the installation of Caringo FileFly FPolicy Server on a group of machines which are addressed by a single FQDN. This provides High-Availability for migration and demigration operations on the associated Vservers.

Typically a pair of FileFly FPolicy Servers operating in HA will service all of the Vservers on a NetApp cluster.

Note: The servers that form the High-Availability FileFly FPolicy Server configuration must not be members of a Windows failover cluster.

DNS Configuration

All Active Directory Servers, Caringo FileFly FPolicy Servers, and NetApp Filers, must have both forward and reverse records in DNS.

All hostnames used in Filer and FileFly FPolicy Server configuration must be FQDNs.

4.3.3 Setup

Setup Parameters

Before starting the installation the following parameters must be considered:

• Management LIF IP Address: the address for management access to the Vserver

(not to be confused with cluster or node management addresses)

• CIFS Privileged User: a domain user for the exclusive use of FPolicy

Preparing Vserver Management Access

For each Vserver, ensure that ‘Management Access’ is allowed for at least one LIF. Check the LIF in OnCommand System Manager - if Management Access is not enabled, either add access to an existing LIF or create a new LIF just for Management Access.

Management authentication may be configured to use either passwords or client certificates. Management connections may be secured via TLS – this is mandatory when using certificate-based authentication.

For password-based authentication:

1. Select the Vserver in OnCommand System Manager and go to Configuration → Security → Users
2. Add a user for Application ‘ontapi’ with Role ‘vsadmin’
3. Record the username and password for later use on the ‘Management’ tab in Caringo FileFly NetApp Cluster-mode Config

Alternatively, for certificate-based authentication:

1. Create a client certificate with common name <Username>
2. Open a command line session to the cluster management address
3. Upload the CA Certificate (or the client certificate itself if self-signed):

(a) security certificate install -type client-ca -vserver

<vserver-name>

(b) Paste the contents of the CA Certificate at the prompt

1. security login create -username <Username> -application ontapi -authmethod cert -role vsadmin -vserver <vserver-name>

Configuring CIFS Privileged Data Access

If it has not already been created, create the CIFS Privileged User on the domain. Each FileFly FPolicy Server will use the same CIFS Privileged User for all Vservers that it will manage.

In OnCommand System Manager:

1. Navigate to the Vserver
2. Create a new local ‘Windows’ group with ALL available privileges
3. Add the CIFS Privileged User to this group
4. Allow a few minutes for the change to take effect (or FileFly FPolicy Server operations may fail with access denied errors)

Installation

On each FileFly FPolicy Server machine:

1. Close any CIFS sessions open to Vserver(s) before proceeding
2. Ensure the CIFS Privileged User has the ‘Log on as a service’ privilege
3. Run the Caringo FileFly NetApp FPolicy Server.exe
4. Follow the prompts to complete the installation
5. Follow the instructions to activate the installation as either a standalone server or High-Availability Caringo FileFly FPolicy Server

Installing ‘Caringo FileFly NetApp Cluster-mode Config’

• Run the installer:

Caringo FileFly NetApp Cluster-mode Config.exe

Configuring Components

Run Caringo FileFly NetApp Cluster-mode Config.

On the ‘FPolicy Config’ tab:

• Enter the FQDN used to register the FileFly FPolicy Server(s) in FileFly Admin

Portal

• Enter the CIFS Privileged User

On the ‘Management’ tab:

• Provide the credentials for management access (see above)

On the ‘Vservers’ tab:

• Click New...
• Enter the FQDN of the Vserver’s Data Access LIF
• Optionally, enter the FQDN of a different LIF for Vserver Management
• If using TLS for Management, click Get Server CA
• Click Apply to Filer

Once configuration is complete, click Save.

Apply Configuration to FileFly FPolicy Servers

1. Ensure the netapp clustered.cfg file has been copied to the correct location on all FileFly FPolicy Server machines

• C:\Program Files\Caringo FileFly\data\FileFly Agent\ netapp clustered.cfg

1. Restart the Caringo FileFly Agent service on each machine

4.3.4 Usage

URI Format

netapp://{FPolicy Server}/{NetApp Vserver}/{CIFS Share}/[{path}]

Where:

• FPolicy Server – FQDN alias that points to all FileFly^TM FileFly FPolicy Servers for the given Vserver
• NetApp Vserver – FQDN of the Vserver’s Data Access LIF
• CIFS Share – NetApp CIFS share name

Example:

netapp://fpol-svrs.example.com/vs1.example.com/data/

Note: The chosen CIFS share must be configured to Hide symbolic links. If symbolic link support is required for other CIFS clients, create a separate share just for FileFly^TM traversal that does hide links.

4.3.5 Snapshot Restore

Volume Restore

After an entire volume containing stubs is restored from snapshot, a Post-Restore Revalidate Policy must be run, as per the restore procedure described in Chapter 7.

Individual Stub Restore

Users cannot perform self-service restoration of stubs. However, an administrator may restore specific stubs or sets of stubs from snapshots by following the procedure outlined below. Be sure to provide this procedure to all administrators.

IMPORTANT: The following instructions mandate the use of Robocopy specifically. Other tools, such as Windows Explorer copy or the ‘Restore’ function in the Previous versions dialog, WILL NOT correctly restore stubs.

To restore one or more stubs from a snapshot-folder like:

\\<filer>\<share>\~snapshot\<snapshot-name>\<path>

to a restore folder on the same Filer like:

\\<filer>\<share>\<restore-path>

perform the following steps:

1. Go to an FileFly FPolicy Server machine
2. Open a command window
3. robocopy <snapshot-folder><folder> [<filename>...] [/b]
4. On a client machine (NOT the FileFly FPolicy Server), open all of the restored file(s) or demigrate them using a Demigrate Policy

• Check that the file has demigrated correctly

IMPORTANT: Until the demigration above is performed, the restored stub(s) may occupy space for the full size of the file.

As with any other FileFly^TM restore procedure, be sure to run a Post-Restore Revalidate Policy across the volume before the next Scrub – see Chapter 7.

4.3.6 Interoperability

Robocopy

Except when following the procedure in §4.3.5, Robocopy must not be used with the /b (backup mode) switch when copying FileFly^TM NetApp stubs.

When in backup mode, robocopy attempts to copy stub files as-is rather than demigrating them as they are read. This behavior is not supported.

Note: The /b switch requires Administrator privilege – it is not available to normal users.

4.3.7 Behavioral Notes

Unix Symbolic Links

Unix Symbolic links (also known as symlinks or softlinks) may be created on a Filer via an NFS mount. Symbolic links will not be seen during FileFly^TM Policy traversal of a NetApp file system (since only shares which hide symbolic links are supported for traversal). If it is intended that a policy should apply to files within a folder referred to by a symbolic link, ensure that the Source encompasses the real location at the link’s destination. A Source URI may NOT point to a symbolic link – use the real folder that the link points to instead.

Client-initiated demigrations via symbolic links will operate as expected.

QTree and User Quotas

NetApp QTree and user quotas are measured in terms of logical file size. Thus, migrating files has no effect on quota usage.

Snapshot Traversal

FileFly^TM will automatically skip snapshot directories when traversing shares using the netapp scheme.

4.3.8 Skipping Sparse Files

It is often undesirable to migrate files that are highly sparse since sparseness is not preserved by the migration process.

To enable sparse files to be skipped during migration policies, go to the Admin Portal ‘Settings Page’ and tick ‘Enable sparse file skipping’.

Skipping sparse files may then be configured per migration policy. On the ‘Policy Details’ page for Migrate and Simple Premigrate operations, tick ‘skip files more than 0% sparse’ and adjust the percentage as required using the drop-down box.

4.3.9 Advanced Configuration

Alternative Engine IP Addresses

Alternative engine IP addresses may be provided on the FileFly NetApp Cluster-mode Config ‘Advanced’ tab if filer communication is to be performed on a different IP address than that used for Admin Portal to FPolicy Server communication. This allows each node to have two IP addresses. Care must be taken that ALL communication – in both directions – between filer and FileFly FPolicy Server node occurs using the engine address.

Ordinarily, one IP address per server is sufficient. Contact Caringo Support if an advanced network configuration is required.

Cache First Block

When migrating files, the first block of the file may optionally be cached. This allows small reads to file headers to be completed immediately, without accessing secondary storage. By default this feature is disabled. This feature may be enabled on the ‘Advanced’ tab. The ‘Prefix size’ field allows the amount cached on disk after a migration to be tuned.

4.3.10 Troubleshooting

Troubleshooting Management Login

• Open a command line session to the cluster management address
• security login show -vserver <vserver-name>
• There should be an entry for the expected user for application ‘ontapi’ with role ‘vsadmin’
• Open a command line session to the cluster management address
• vserver context -vserver <vserver-name>
• security certificate show
• There should be a ‘server’ certificate for the Vserver management FQDN (NOT the bare hostname)
• If using certificate-based authentication, there should be a ‘client-ca’ entry
• security ssl show
• There should be an enabled entry for the Vserver management FQDN (NOT the bare hostname)

Troubleshooting TLS Management Access

Troubleshooting Vserver Configuration

Vserver configuration can be validated using Caringo FileFly NetApp Cluster-mode Config.

• Open the netapp clustered.cfg in FileFly NetApp Cluster-mode Config
• Go to the ‘Vservers’ tab • Select a Vserver
• Click Edit...
• Click Verify

Troubleshooting ‘ERR ADD PRIVILEGED SHARE NOT FOUND’

If the FileFly FPolicy Server reports privileged share not found, there is a misconfiguration or CIFS issue. Please attempt the following steps:

• Check all configuration using troubleshooting steps described above
• Ensure the FileFly FPolicy Server has no other CIFS sessions to Vservers
• run net use from Windows Command Prompt
• remove all mapped drives
• Reboot the server
• Retry the failed operation
• Check for new errors in agent.log

4.4 NetApp Filer (7-mode)

This section describes support for NetApp Filers 7.3 and above including 8.x Filers operating in ‘7-mode’. For version 9.x Filers and 8.x Filers running in ‘Cluster-mode’, see §4.3.

4.4.1 Migration Support

Migration support for sources on NetApp Filers is provided via NetApp FPolicy. This requires the use of a Caringo FileFly FPolicy Server. FileFly^TM supports the use of both physical Filers and vFilers as migration sources. Client demigrations can be triggered via CIFS or NFS client access.

When accessed via CIFS on a Windows client, NetApp stub files can be identified by the ‘O’ (Offline) attribute in Explorer. Files with this flag will be displayed with an overlay icon. The icon may vary depending on the version of Windows on the client workstation.

4.4.2 Planning

Prerequisites

• NetApp Filer(s) must be licensed for the particular protocol(s) to be used (FPolicy requires a CIFS license)

Caringo FileFly FPolicy Servers require EXCLUSIVE use of CIFS connections to their associated NetApp filers/vFilers. This means Explorer windows must not be opened, drives must not be mapped, nor should any UNC paths to the filer be accessed from the FileFly FPolicy Server machine.

Demigrations cannot be triggered by applications running locally on the FileFly FPolicy Servers since the Filer ignores these requests. This is an FPolicy restriction.

When creating a production deployment plan, please refer to §2.6 .

Filer System Requirements

Caringo FileFly FPolicy Server requires that the Filer is running Data ONTAP version 7.3 or above. Caringo recommends 7.3.6 or above.

Important: Place the FileFly FPolicy Servers on the same subnet and same switch as the Filers that they will serve to minimize latency.

Using the Filer on a Domain

If the NetApp Filer is joined to an Active Directory domain, check the following:

• All AD servers that the filer will communicate with are also DNS servers
• DNS contains the msdcs.<exampleDomain> subdomain (created automatically if DNS is set up as part of the Active Directory installation)
• Only the Active Directory DNS servers should be provided to the filer (check /etc/resolv.conf on the filer to confirm)

High-Availability for FPolicy Servers

It is strongly recommended to install Caringo FileFly FPolicy Servers in a High-Availability configuration. This configuration requires the installation of Caringo FileFly FPolicy Server on a group of machines which are all addressed by a single FQDN. This provides High-Availability for migration and demigration operations on the associated filers.

DNS Configuration

All Active Directory Servers, Caringo FileFly FPolicy Servers, and NetApp Filers, must have both forward and reverse records in DNS.

All hostnames used in Filer and FileFly FPolicy Server configuration must be FQDNs.

Incorrect DNS configuration or use of bare hostnames may lead to FileFly FPolicy Servers failing to register or disconnecting shortly after registration.

Using SMB2

If the target filer is configured to use the SMB2 protocol:

• Ensure that both of the following NetApp options are enabled:
• cifs.smb2.enable
• cifs.smb2.client.enable
• Using Local User Accounts to authenticate with the filer may cause connection issues, Active Directory domain authentication should be used instead

Unicode Filename Support

It is recommended that all volumes have UTF-8 support enabled (i.e. the volume language should be set to <lang>.UTF-8). Files with Unicode (non-ASCII) filenames cannot be accessed via NFS unless the UTF-8 option is enabled. To ensure maximal data accessibility, FileFly^TM will mark any file that would not be demigratable via both NFS and CIFS clients as ‘Do Not Migrate’.

4.4.3 Setup

Preinstallation Steps – NetApp Filers and vFilers

1. Enable HTTP servers

• From the console on each NetApp filer/vFiler:
• options httpd.admin.enable on

1. Create and enable FPolicy filefly on each NetApp filer/vFiler • Note: The name filefly must be used for the FPolicy

• On the NetApp filer console:
• netapp> options fpolicy.enable on
• netapp> fpolicy create filefly screen
• netapp> fpolicy options filefly required on
• netapp> fpolicy enable filefly

1. Create a NetApp administrator account:

• From the console on each NetApp filer/vFiler:
• netapp> useradmin domainuser add <username> -g administrators

Note: If the Filer is not on a domain, then a local user account may be created instead.

Preinstallation Steps – FileFly FPolicy Server Machine(s)

Ensure NetBIOS over TCP/IP is enabled to allow connections to and from the NetApp for FPolicy:

1. Determine which network interface(s) will be used to contact the filer(s)
2. Navigate to each Network interface’s Properties dialog box
3. Select Internet Protocol Version 4 (TCP/IPv4) → Properties → Advanced. . .
4. On the ‘WINS’ tab, select ‘Enable NetBIOS over TCP/IP’
5. Ensure the server firewall is configured to allow incoming NetBIOS traffic from the filer – e.g. enable the ‘File and Printer Sharing (NB-Session-In)’ rule in Windows Firewall

Installing Components

On each FileFly FPolicy Server machine:

1. Run the Caringo FileFly NetApp FPolicy Server.exe
2. Select install location
3. Enter the login credentials for an administrator user with the ‘Log on as a service’ privilege – this account MUST have the same username and password as an administrator level account on the Filer
4. Follow the instructions to activate the installation as either a ‘Standalone Server’ or High-Availability Caringo FileFly FPolicy Server
5. Edit netapp.cfg in the Caringo FileFly FPolicy Server data directory (e.g. C:\

Configuring Components

Program Files\Caringo FileFly\data\FileFly Agent):

• Set the netapp.filers property to a comma-delimited list of NetApp filer/vFiler FQDNs

1. Open Services → Caringo FileFly Agent
2. Restart the service

When using a High-Availability configuration, be sure to use the same netapp.cfg across all nodes and remember to restart each node’s service.

Cache First Block

When migrating files, the first block of the file may optionally be cached. This allows small reads to file headers to be completed immediately, without triggering a demigration from secondary storage. By default this feature is disabled. To enable it, set netapp.cacheFirstBlock to true in netapp.cfg.

4.4.4 Usage

URI Format

netapp://{FPolicy Server}/{NetApp Filer}/{CIFS Share}/[{path}]

Where:

• FPolicy Server – FQDN alias that points to all FileFly^TM FileFly FPolicy Servers for the given Filer
• NetApp Filer – FQDN of the Filer/vFiler
• CIFS Share – NetApp CIFS share name (FPolicy requires the use of CIFS)

Example:

netapp://fpol-svrs.example.com/netapp1.example.com/data/

4.4.5 Interoperability

Robocopy

Robocopy must not be used with the /b (backup mode) switch when copying FileFly^TM NetApp stubs.

When in backup mode, robocopy attempts to copy stub files as-is rather than demigrating them as they are read. This behavior is not supported.

Note: The /b switch requires Administrator privilege – it is not available to normal users.

4.4.6 Behavioral Notes

Unix Symbolic Links

Unix Symbolic links (also known as symlinks or softlinks) may be created on a Filer via an NFS mount. Symbolic links will be skipped during traversal of a NetApp file system. This ensures that files are not seen – and thus acted upon – multiple times during a single execution of a given policy. If it is intended that a policy should apply to files within a folder referred to by a symbolic link, ensure that the Source encompasses the real location at the link’s destination. A Source URI may NOT point to a symbolic link – use the real folder that the link points to instead.

QTree and User Quotas

NetApp QTree and user quotas are measured in terms of logical file size. Thus, migrating files has no effect on quota usage.

Snapshots

FileFly^TM will automatically skip snapshot directories when traversing NetApp Filer volumes using the netapp scheme.

CIFS Usage

Caringo FileFly FPolicy Servers require EXCLUSIVE use of CIFS connections to their associated NetApp filers/vFilers. This means Explorer windows must not be opened, drives must not be mapped, nor should any UNC paths to the filer be accessed from the FileFly FPolicy Server machine. Failure to observe this restriction will result in unpredictable FPolicy disconnections and interrupted service.

Demigrations cannot be triggered by applications running directly on the FileFly FPolicy Servers since the Filer ignores these requests. This is an FPolicy restriction.

4.4.7 Skipping Sparse Files

It is often undesirable to migrate files that are highly sparse since sparseness is not preserved by the migration process.

To enable sparse files to be skipped during migration policies, go to the Admin Portal ‘Settings Page’ and tick ‘Enable sparse file skipping’. The sparse file skipping option for migration policies requires at least Data ONTAP version 7.3.6.

Skipping sparse files may then be configured per migration policy. On the ‘Policy Details’ page for Migrate and Simple Premigrate operations, tick ‘skip files more than 0% sparse’ and adjust the percentage as required using the drop-down box.

4.4.8 Debug Status Monitoring

By default Caringo FileFly FPolicy Servers provide status information and statistics via a webpage located at http://127.0.0.1:8000 (accessible only from the FPolicy Server machine).

To run the webserver on a different TCP port, set netapp.web.port in netapp.cfg to the desired port number. To disable the webserver, set netapp.web.enable to false.

4.5 Caringo Swarm

4.5.1 Introduction

The swarm scheme should only be used when accessing Swarm storage nodes directly.

If accessing Swarm storage via a CloudScaler Gateway, the cloudscaler scheme must be used instead, see §4.6.

Note: FileFly^TM software does not support access to storage nodes via an SCSP Proxy.

4.5.2 Planning

Before proceeding with the installation, the following will be required:

• Swarm 8 or above

Firewall

The Swarm storage node port (TCP port 80 by default) must be allowed by any firewalls between the Caringo FileFly Swarm Plugin on the Caringo FileFly Gateway and the Swarm storage nodes. For further information regarding firewall configuration see Appendix A.

Domains and Endpoints

Swarm storage locations are accessed via a configured endpoint FQDN. Add several Swarm storage node IP addresses to DNS under a single endpoint FQDN (4-8 addresses are recommended). If Swarm domains are in use, the FQDN must be the name of the domain in which the FileFly^TM data will be stored. If domains are NOT in use (i.e. data will be stored in the default cluster domain), it is strongly recommended that the FQDN be the name of the cluster for best Swarm performance.

When using multiple Swarm domains, ensure that each domain FQDN is added to DNS as described above.

Buckets

Migrated files may be stored as either unnamed objects (accessed by UUID), or as named objects residing in a bucket. Bucket creation must be performed ahead of time, prior to configuring FileFly^TM.

FileFly Swarm Config will be used to create Destination URIs for use in the FileFly Admin Portal.

4.5.3 Setup

Installation

To perform a fresh installation:

1. Run the Caringo FileFly Agent.exe, select the FileFly Gateway role (see

§2.3.3 ) and select FileFly Swarm Plugin on the ‘Components’ page

1. Follow the prompts to complete the installation

Or, to add the FileFly Swarm Plugin to an existing FileFly Gateway or Agent:

1. Run the installer for the Caringo FileFly Swarm Plugin:

Caringo FileFly Swarm Plugin.exe

1. Follow the prompts to complete the installation

Installing ‘Caringo FileFly Swarm Config’

• Run the installer for Caringo FileFly Swarm Config: Caringo FileFly Swarm Config.exe

4.5.4 Plugin Configuration

Open ‘Caringo FileFly Swarm Config’ and complete the following configuration steps.

Create a FileFly^TM Encryption Key

If encryption-at-rest is to be used to protect FileFly^TM data on the destination, check Enable encryption. An encryption Key must be generated before FileFly^TM can be used to store encrypted data on a Swarm migration destination. FileFly^TM will encrypt all data migrated using the specified Encryption Key.

During the Encryption Key creation process, a copy of the information entered will be printed and it will be strongly recommended that a copy of the swarm.cfg file is stored in a safe location (e.g. written to a CD).

Do not continue unless able to print, and ensure a blank CD is available.

1. Click Generate in the ‘FileFly^TM Encryption’ section
2. Read the User Confirmation notice and click Yes to continue
3. Keep the suggested Key ID
4. Enter a passphrase from which to generate a new encryption key, and click OK

• an Encryption Key Details page will be printed

1. When prompted, enter the ‘Validation Code’ from the printed page

Set Metadata Options

Tick ‘Include metadata HTTP headers’ to store per-file metadata with the destination objects, such as original filename and location, content-type, owner and timestamps – see §4.5.7 for details. File extension to content-type mappings may be customized by editing the swarm-mimetypes file, found in C:\Program Files\Caringo FileFly\data\ swarm.data\.

Also tick ‘Include Content-Disposition’ to include original filename for use when downloading the target objects directly using a web browser.

Create an Index

Swarm Destinations require an index to be created prior to use.

In FileFly Swarm Config:

1. Click Create Index...
2. Follow the instructions
3. Use the resultant URI to create a Destination in the FileFly Admin Portal

Additional indexes can be added at a later date to further subdivide storage if required.

Important: Each FileFly Admin Portal must have its own destination indexes; DO NOT share indexes across multiple FileFly^TM implementations.

Apply Configuration to FileFly Gateways

1. Click Save to save all changes. Changes will be saved to swarm.cfg
2. Copy the swarm.cfg file to a blank CD to protect the encryption key
3. Copy swarm.cfg to the correct location on all FileFly Gateway machines:

• C:\Program Files\Caringo FileFly\data\FileFly Agent\swarm.cfg

1. Restart the Caringo FileFly Agent service on each machine

4.5.5 Usage

URI Format

Note: The following is informational only, FileFly Swarm Config should always be used to prepare Swarm URIs.

swarm://{gateway}/{endpoint}[:{port}]/?idx={index}

swarm://{gateway}/{endpoint}[:{port}]/{bucket}[:{partition}] Where:

• gateway – DNS alias for all Caringo Swarm Gateways
• endpoint – FQDN of the Swarm endpoint
• port – override the standard HTTP/HTTPS port
• index – index UUID, as created by FileFly Swarm Config
• bucket – bucket in which to store named objects
• partition – partition within bucket

Examples:

swarm://gw.example.com/data.example.com/?idx=968...

swarm://gw.example.com/data.example.com/myBucket

4.5.6 Disaster Recovery Considerations

During migration, each newly migrated file is recorded in the corresponding index. The index may be used in disaster scenarios where:

1. stubs have been lost, and
2. a Create DrTool File from Source file is not available, and
3. no current backup of the stubs exists

Index performance is optimized for migrations and demigrations, not for Create DrTool File from Destination queries.

Create DrTool File from Source policies are the recommended means to obtain a DrTool file for restoring stubs. This method provides better performance and the most up-todate stub location information.

It is recommended to regularly run Create DrTool File from Source policies following Migration policies.

4.5.7 Swarm Metadata Headers

The following metadata fields are supported:

• X-Alt-Meta-Name – the original source file’s filename (excluding directory path)
• X-Alt-Meta-Path – the original source file’s directory path (excluding the filename) in a platform-independent manner such that ‘/’ is used as the path separator and the path will start with ‘/’, followed by drive/volume/share if appropriate, but not end with ‘/’ (unless this path represents the root directory)
• X-FileFly-Meta-Partition – the Destination URI partition – if no partition is present, this header is omitted
• X-Source-Meta-Host – the FQDN of the original source file’s server
• X-Source-Meta-Owner – the owner of the original source file in a format appropriate to the source system (e.g. DOMAIN\username)
• X-Source-Meta-Modified – the Last Modified timestamp of the original source file at the time of migration in RFC3339 format
• X-Source-Meta-Created – the Created timestamp of the original source file in RFC3339 format
• X-Source-Meta-Attribs – a case-sensitive sequence of characters {AHRS} representing the original source file’s file flags: Archive, Hidden, Read-Only and

System

• all other characters are reserved for future use and should be ignored
• Content-Type – the MIME Type of the content, determined based on the fileextension of the original source filename

Note: Timestamps may be omitted if the source file timestamps are not set.

Non-ASCII characters will be be stored using RFC2047 encoding, as described in the Swarm documentation. Swarm will decode these values prior to indexing in Elasticsearch.

4.6 Caringo CloudScaler

4.6.1 Introduction

Caringo CloudScaler provides a multi-tenanted object storage platform built upon Swarm storage nodes. The FileFly^TM cloudscaler scheme must only be used when accessing the storage via CloudScaler. To store data on Swarm nodes directly, the swarm scheme must be used instead, see §4.5.

4.6.2 Planning

Before proceeding with the installation, the following will be required:

• Cloud Gateway 3.0.0 or above
• Swarm 8 or above
• a license that includes an entitlement for CloudScaler

Firewall

The TCP port used to access the CloudScaler Gateway via HTTP or HTTPS (possibly by way of a load-balancer) must be allowed by any firewalls between the FileFly CloudScaler Plugin on the FileFly Gateway and the CloudScaler Gateway endpoints. For further information regarding firewall configuration see Appendix A.

Domains and Buckets

CloudScaler domain names used with FileFly^TM must be valid FQDNs which resolve to one or more Cloud Gateways.

Migrated files may be stored as either unnamed objects (accessed by UUID), or as named objects residing in a bucket. Bucket creation must be performed ahead of time, prior to configuring FileFly^TM.

FileFly CloudScaler Config will assist in the creation of a Destination URI for use in the FileFly Admin Portal.

Authentication

When using buckets, it is a requirement that the configured credentials for accessing the bucket are also permitted to perform HEAD requests at the root of the domain in order to obtain domain information. This must be considered when provisioning buckets.

4.6.3 Setup

Installation

To perform a fresh installation:

1. Run the Caringo FileFly Agent.exe, select the FileFly Gateway role (see §2.3.3 ) and select FileFly CloudScaler Plugin on the ‘Components’ page

1. Follow the prompts to complete the installation

Or, to add the FileFly CloudScaler Plugin to an existing FileFly Gateway or Agent:

1. Run the installer for the Caringo FileFly CloudScaler Plugin:

Caringo FileFly CloudScaler Plugin.exe

1. Follow the prompts to complete the installation

Installing ‘Caringo FileFly CloudScaler Config’

• Run the installer for Caringo FileFly CloudScaler Config: Caringo FileFly CloudScaler Config.exe

4.6.4 Plugin Configuration

In ‘Caringo FileFly CloudScaler Config’:

1. Check ‘Use TLS’ if the CloudScaler endpoint will be accessed via HTTPS
2. Optionally, fill in the ‘HTTP Proxy’ section:

(a) Check Use Proxy if a proxy is required to access the endpoint

• Avoid using a proxy for best performance
• This feature is only supported for HTTPS endpoints

(b) Enter ‘Host’ and ‘Port’

1. Click New to add a new set of CloudScaler domain credentials

• If using named objects, supply the bucket name
• The bucket must already exist and be configured

1. Specify the CloudScaler storage domain, username and password

• The domain must already exist and be configured

1. Create an Index as described below
2. Create an Encryption Key as described below

Create an Index

CloudScaler Destinations require an index to be created prior to use.

In FileFly CloudScaler Config:

1. Select the domain in which to create the index
2. Click Create Index...
3. Follow the instructions
4. Use the resultant URI to create a Destination in the FileFly Admin Portal

Additional indexes can be added at a later date to further subdivide storage if required.

Important: Each FileFly Admin Portal must have its own destination indexes; DO NOT share indexes across multiple FileFly^TM implementations.

Create a FileFly^TM Encryption Key

If encryption-at-rest is to be used to protect FileFly^TM data on the destination, check Enable encryption. An encryption Key must be generated before FileFly^TMcan be used to store encrypted data on a CloudScaler migration destination. FileFly^TM will encrypt all data migrated using the specified Encryption Key.

During the Encryption Key creation process, a copy of the information entered will be printed and it will be strongly recommended that a copy of the cloudscaler.cfg file is stored in a safe location (e.g. written to a CD).

Do not continue unless able to print, and ensure a blank CD is available.

1. Click Generate in the ‘FileFly^TM Encryption’ section
2. Read the User Confirmation notice and click Yes to continue
3. Keep the suggested Key ID
4. Enter a passphrase from which to generate a new encryption key, and click OK

• an Encryption Key Details page will be printed

1. When prompted, enter the ‘Validation Code’ from the printed page
2. Click Save to save all changes. Changes will be saved to cloudscaler.cfg
3. Copy the cloudscaler.cfg file to a blank CD to protect the encryption key
4. Apply the configuration as described below

Set Metadata Options

Tick ‘Include metadata HTTP headers’ to store per-file metadata with the destination objects, such as original filename and location, content-type, owner and timestamps – see §4.6.7 for details. File extension to content-type mappings may be customized by editing the cloudscaler-mimetypes file, found in C:\Program Files\Caringo FileFly\data\ cloudscaler.data\.

Also tick ‘Include Content-Disposition’ to include original filename for use when downloading the target objects directly using a web browser.

Apply Configuration to FileFly Gateways

1. Copy cloudscaler.cfg to the correct location on all FileFly Gateway machines:

• C:\Program Files\Caringo FileFly\data\FileFly Agent\ cloudscaler.cfg

1. Restart the Caringo FileFly Agent service on each machine

4.6.5 Usage

URI Format

Note: The following is informational only, FileFly CloudScaler Config should always be used to prepare CloudScaler URIs.

cloudscaler://{gateway}/{endpoint}[:{port}]/?idx={index}

cloudscaler://{gateway}/{endpoint}[:{port}]/{bucket}[:{partition}] Where:

• gateway – DNS alias for all Caringo CloudScaler Gateways
• endpoint – FQDN of the CloudScaler endpoint
• port – override the standard HTTP/HTTPS port
• index – index UUID, as created by FileFly CloudScaler Config
• bucket – bucket in which to store named objects
• partition – partition within bucket

Examples:

cloudscaler://gw.example.com/data.example.com/?idx=968...

cloudscaler://gw.example.com/data.example.com/myBucket

4.6.6 Disaster Recovery Considerations

During migration, each newly migrated file is recorded in the corresponding index. The index may be used in disaster scenarios where:

1. stubs have been lost, and
2. a Create DrTool File from Source file is not available, and
3. no current backup of the stubs exists

Index performance is optimized for migrations and demigrations, not for Create DrTool File from Destination queries.

Create DrTool File from Source policies are the recommended means to obtain a DrTool file for restoring stubs. This method provides better performance and the most up-todate stub location information.

It is recommended to regularly run Create DrTool File from Source policies following Migration policies.

4.6.7 Swarm Metadata Headers

The following metadata fields are supported:

• X-Alt-Meta-Name – the original source file’s filename (excluding directory path)
• X-Alt-Meta-Path – the original source file’s directory path (excluding the filename) in a platform-independent manner such that ‘/’ is used as the path separator and the path will start with ‘/’, followed by drive/volume/share if appropriate, but not end with ‘/’ (unless this path represents the root directory)
• X-FileFly-Meta-Partition – the Destination URI partition – if no partition is present, this header is omitted
• X-Source-Meta-Host – the FQDN of the original source file’s server
• X-Source-Meta-Owner – the owner of the original source file in a format appropriate to the source system (e.g. DOMAIN\username)
• X-Source-Meta-Modified – the Last Modified timestamp of the original source file at the time of migration in RFC3339 format
• X-Source-Meta-Created – the Created timestamp of the original source file in RFC3339 format
• X-Source-Meta-Attribs – a case-sensitive sequence of characters {AHRS} representing the original source file’s file flags: Archive, Hidden, Read-Only and

System

• all other characters are reserved for future use and should be ignored
• Content-Type – the MIME Type of the content, determined based on the fileextension of the original source filename

Note: Timestamps may be omitted if the source file timestamps are not set.

Non-ASCII characters will be be stored using RFC2047 encoding, as described in the Swarm documentation. Swarm will decode these values prior to indexing in Elasticsearch.