FileFly 3.1 Community Edition
- 1 1 Introduction
- 2 2 Deployment
- 3 3 Policy Operations
- 3.1 3.1 Gather Statistics Operation
- 3.2 3.2 Migrate Operation
- 3.3 3.3 Quick-Remigrate Operation
- 3.4 3.4 Scrub Destination Operation
- 3.5 3.5 Post-Restore Revalidate Operation
- 3.6 3.6 Demigrate Operation
- 3.7 3.7 Advanced Demigrate Operation
- 3.8 3.8 Simple Premigrate Operation
- 3.9 3.9 Erase Cached Data Operation
- 4 4 Sources and Destinations
- 5 5 FileFly Admin Portal Reference
- 5.1 5.1 Introduction
- 5.2 5.2 Overview Tab
- 5.3 5.3 Servers
- 5.4 5.4 Sources
- 5.5 5.5 Destinations
- 5.6 5.6 Rules
- 5.7 5.7 Policies
- 5.8 5.8 Tasks
- 5.9 5.9 Task Execution
- 5.10 5.10 Settings Page
- 5.11 5.11 About Page
- 6 6 Configuration Backup
- 7 7 Storage Backup
- 8 8 System Upgrade
- 9 A Network Ports
- 10 B File and Directory Exclusion Examples
- 11 C Admin Portal Security Configuration
- 12 D Advanced FileFly Agent Configuration
- 13 1 Introduction
- 14 2 Deployment
- 15 3 Policy Operations
- 15.1 3.1 Gather Statistics Operation
- 15.2 3.2 Migrate Operation
- 15.3 3.3 Quick-Remigrate Operation
- 15.4 3.4 Scrub Destination Operation
- 15.5 3.5 Post-Restore Revalidate Operation
- 15.6 3.6 Demigrate Operation
- 15.7 3.7 Advanced Demigrate Operation
- 15.8 3.8 Simple Premigrate Operation
- 15.9 3.9 Erase Cached Data Operation
- 16 4 Sources and Destinations
- 17 5 FileFly Admin Portal Reference
- 17.1 5.1 Introduction
- 17.2 5.2 Overview Tab
- 17.3 5.3 Servers
- 17.4 5.4 Sources
- 17.5 5.5 Destinations
- 17.6 5.6 Rules
- 17.7 5.7 Policies
- 17.8 5.8 Tasks
- 17.9 5.9 Task Execution
- 17.10 5.10 Settings Page
- 17.11 5.11 About Page
- 18 6 Configuration Backup
- 19 7 Storage Backup
- 20 8 System Upgrade
- 21 A Network Ports
- 22 B File and Directory Exclusion Examples
- 23 C Admin Portal Security Configuration
- 24 D Advanced FileFly Agent Configuration
- 25 E Troubleshooting
- 25.1 E.1 Log Files
- 25.2 E.2 Interpreting Errors
1 Introduction
This guide pertains to FileFly Community Edition only. The full Administration Guide should be consulted for details of features that may be present in other product editions.
1.1 What is Caringo FileFly™?
Caringo FileFly is a heterogeneous Data Management System. It automates and manages the movement of data from primary storage locations to Caringo Swarm or CloudScaler object storage.
Files are migrated from primary storage locations to the object store. Files are demigrated transparently when accessed by a user or application.
What is Migration?
File migration can be summarized as follows: first, the file content and corresponding metadata are copied to secondary storage as an MWI file/object. Next, the original file is marked as a ‘stub’ and truncated to zero physical size (while retaining the original logical size for the benefit of users and the correct operation of applications). The resulting stub file will remain on primary storage in this state until such time as a user or application requests access to the file content, at which point the data will be automatically returned to primary storage.
Each stub encapsulates the location of the corresponding MWI data on secondary storage, without the need for a database or other centralized component.
1.2 Conventions used in this Book
References to labels, values and literals in the software are in ‘quoted italics’.
References to actions, such as clicking buttons, are in bold.
References to commands and text typed in are in fixed font.
Notes are denoted: Note: This is a note.
Important notes are denoted: Important: Important point here.
1.3 System Components
Figure 1.1 provides an overview of a FileFly system. All communication between FileFly components is secured with Transport Layer Security (TLS). The individual components are described below.
Figure 1.1: FileFly System Overview
Caringo FileFly Admin Portal
FileFly Admin Portal is the system’s policy manager. It provides a centralized web-based configuration interface, and is responsible for task scheduling, policy simulation, server monitoring and file reporting. It lies outside the data path for file transfers.
Caringo FileFly Agent
Caringo FileFly Agent performs file operations as directed by Admin Portal Policies.
FileFly Agent is also responsible for retrieving file data from secondary storage upon
user/application access.
Data is streamed directly between agents and storage without any intermediary staging on disk.
When installed in a Gateway configuration, FileFly Agent does not allow migration of files from that server.
Optionally, Gateways can be configured for High-Availability (HA).
Caringo FileFly FPolicy Server
FileFly FPolicy Server provides migration support for NetApp filers via the NetApp FPolicy protocol. This component is the equivalent of Caringo FileFly Agent for NetApp filers.
FileFly FPolicy Server may also be configured for High-Availability (HA).
Caringo FileFly DrTool
Caringo FileFly DrTool is an additional application that assists in Disaster Recovery scenarios.
Note: This functionality is not included with Community Edition licenses.
2 Deployment
This chapter will cover:
Installing Caringo FileFly Tools
Installing Caringo FileFly Agent on file servers
Installing Caringo FileFly Gateways as required
Getting started with FileFly policies
Production readiness
Refer to these instructions during initial deployment and when adding new components. For upgrade instructions, please refer to Chapter 8 instead.
For further details and usage instructions for each platform, refer to Chapter 4.
2.1 DNS Best Practice
In a production deployment, Fully Qualified Domain Names (FQDNs) should always be used in preference to bare IP addresses.
Storage locations in Caringo FileFly are referred to by URI. Relationships between files must be maintained over a long period of time. It is therefore advisable to take steps to ensure the FQDNs used in these URIs are valid long-term, even as individual server roles are changed or consolidated.
Create DNS aliases for each logical storage role for each server. Use different DNS aliases when storing your finance department’s data as opposed to your engineering department’s data – even if they initially reside on the same server.
2.2 Installing FileFly Tools
The Caringo FileFly Tools package consists of the FileFly Admin Portal and the FileFly DrTool application (not licensed for Community Edition users). The FileFly Admin Portal provides central management of policy execution while the FileFly DrTool is used in disaster recovery situations.
FileFly Tools must be installed before any other components.
2.2.1 System Requirements
A dedicated server with a supported operating system:
Windows Server 2016
Windows Server 2012 R2 (Apr 2014 update rollup)
Windows Server 2012
Windows Server 2008 R2 SP1
Minimum 4GB RAM
Minimum 2GB disk space for log files
Active clock synchronization (e.g. via NTP)
Internet Explorer 11 or higher (possibly on a separate workstation) will be required to access the FileFly Admin Portal web interface.
2.2.2 Setup
Run Caringo FileFly Tools.exe
Follow the instructions on screen
FileFly Tools is configured in the Admin Portal web interface after completing the installation process. The FileFly Admin Portal will be opened automatically and can be found later via the Start Menu.
The interface will lead you through the process for installing your license.
For production licensed installations, a ‘Backup & Scrub Grace Period’ setup page will be displayed. Please read the text carefully and set the minimum grace period as appropriate and after consulting with your backup plan – see also §7.2. This value may be revised later via the ‘Settings’ page.
2.3 Installing FileFly Agents
Proceed to install DataCore FileFly Agents as described below once the FileFly Tools installation completes. FileFly Agents perform file operations as directed by Admin Portal Policies. Also, in the case of user/application initiated demigration, agents retrieve the file data from secondary storage autonomously.
2.3.1 FileFly Agent Server Roles
Each FileFly Agent server may fulfill one of two roles, selected at installation time.
In the ‘FileFly Agent for migration’ role, an agent assists the operating system to migrate and demigrate files. It is essential for the agent to be installed on all machines from which files will be migrated.
The agent provides access to CloudScaler and Swarm destinations in the Gateway role.
2.3.2 High-Availability Gateway Configuration
A high-availability gateway configuration is recommended. Such FileFly Gateways must be activated as ‘High-Availability FileFly Gateways’.
High-Availability Gateway DNS Setup
At least two FileFly Gateways are required for High-Availability.
Add each FileFly Gateway server to DNS
Create a single alias that maps to each of the IP addresses
Use this alias in FileFly destination URIs, do not use for individual nodes:
gw-1.example.com→168.0.1
gw-2.example.com→168.0.2
example.com→192.168.0.1, 192.168.0.2
Note: The servers that form the High-Availability Gateway cluster must NOT be members of a Windows failover cluster.
For further DNS recommendations, refer to §2.1.
2.3.3 Installing FileFly Agent for Windows Servers
System Requirements
Supported Windows Server operating system:
Windows Server 2016
Windows Server 2012 R2 (Apr 2014 update rollup)
Windows Server 2012
Windows Server 2008 R2 SP1
Minimum 4GB RAM
Minimum 2GB disk space for log files
Active clock synchronization (e.g. via NTP)
Note: When installed in the Gateway role, a dedicated server is required, unless it is to be co-located on the FileFly Tools server. When co-locating, create separate DNS aliases to refer to the Gateway and the FileFly Admin Portal web interface.
Setup
Run the Caringo FileFly Agent.exe
Select install location
Select migration or Gateway role as appropriate, refer to 2.3.1
If installing a FileFly Gateway, select the desired plugins
Follow the instructions to activate the agent via FileFly Admin Portal
Activation
If no clustering is required, activate as a ‘Standalone Server’
If installing the FileFly Gateway for High-Availability, activate as a High-Availability FileFly Gateway
If the server is part of a Windows failover cluster, and this clustered resource is to be used as a FileFly Source, activate as a Windows failover cluster node
For further information see §5.3.1.
Important: If any type of clustering is used, ensure that FileFly Agent for Windows is installed on ALL cluster nodes.
2.3.4 Installing Caringo FileFly FPolicy Server for NetApp Filers
A Caringo FileFly FPolicy Server provides migration support for one or more NetApp
Filers through the FPolicy protocol. This component is the equivalent of Caringo FileFly Agent for NetApp Filers. Typically, FileFly FPolicy Servers are installed in a High Availability configuration.
System Requirements
A dedicated server with a supported operating system:
Windows Server 2016
Windows Server 2012 R2 (Apr 2014 update rollup)
Windows Server 2012
Windows Server 2008 R2 SP1
Minimum 4GB RAM
Minimum 2GB disk space for log files
Active clock synchronization (e.g. via NTP)
Setup
Installation of the FileFly FPolicy Server software requires careful preparation of the NetApp Filer and the FileFly FPolicy Server machines. Instructions are provided in §4.2.
Note: Legacy 7-Mode Filers require a different procedure at FileFly FPolicy Server installation time – see §4.3.
2.4 Installing Config Tools
In addition to the components described above, it may also be necessary to install one or more Config Tools. Full details are provided where required for each storage platform in Chapter 4.
2.5 Getting Started
2.5.1 Analyzing Volumes
The first step in a new DataCore deployment is to analyze the characteristics of the primary storage volumes once the software is installed. The following steps describe how to generate file statistics reports for each volume.
In the FileFly Admin Portal web interface (see Chapter 5 for full documentation):
Create Sources for each volume to analyze
Create a ‘Gather Statistics’ Policy and select all defined Sources
Create a Task for the ‘Gather Statistics’ Policy 4. On the ‘Overview’ tab, click Quick Run
Click on the Task’s name to run it immediately
When the Task has finished, expand the details by clicking on the Task name under ‘Recent Task History’
Click Go to Task to go to the ‘Task Details’ page
Access the report by clicking on View Last Stats
Pay particular attention to the ‘Last Modified % by size’ graph. This graph will help identify how much data would be affected by a migration policy based on the age of files.
Examine ‘File types by size’ to see if the data profile matches the expected usage of the volume.
2.5.2 Preparing for Migration
Using the information from the reports, create tasks to migrate files:
Prepare a destination for migrated files – see Create a Destination in FileFly Admin Portal
Create a Rule and a Migration Policy
A typical rule might limit migrations to files modified more than six months ago – do not use an ‘all files’ rule
To avoid unnecessary migration of active files, be conservative with your first Migration Policy
Create a Task for the new Policy
For now, disable the schedule
Save the task, then click on its name to open the ‘Task Details’ page
Click Simulate Now to