Open

1  Introduction

This guide pertains to FileFly Community Edition only. The full Administration Guide should be consulted for details of features that may be present in other product editions.

1.1  What is Caringo FileFly™?

Caringo FileFly is a heterogeneous Data Management System. It automates and manages the movement of data from primary storage locations to Caringo Swarm or CloudScaler object storage.

Files are migrated from primary storage locations to the object store. Files are demigrated transparently when accessed by a user or application.

What is Migration?

File migration can be summarized as follows: first, the file content and corresponding metadata are copied to secondary storage as an MWI file/object. Next, the original file is marked as a ‘stub’ and truncated to zero physical size (while retaining the original logical size for the benefit of users and the correct operation of applications). The resulting stub file will remain on primary storage in this state until such time as a user or application requests access to the file content, at which point the data will be automatically returned to primary storage.

Each stub encapsulates the location of the corresponding MWI data on secondary storage, without the need for a database or other centralized component.

1.2  Conventions used in this Book

References to labels, values and literals in the software are in ‘quoted italics’.

References to actions, such as clicking buttons, are in bold.

References to commands and text typed in are in fixed font.

Notes are denoted: Note: This is a note.

Important notes are denoted: Important: Important point here.

1.3  System Components

Figure 1.1 provides an overview of a FileFly system. All communication between FileFly components is secured with Transport Layer Security (TLS). The individual components are described below.

Figure 1.1: FileFly System Overview

Caringo FileFly Admin Portal

FileFly Admin Portal is the system’s policy manager. It provides a centralized web-based configuration interface, and is responsible for task scheduling, policy simulation, server monitoring and file reporting. It lies outside the data path for file transfers.

Caringo FileFly Agent

Caringo FileFly Agent performs file operations as directed by Admin Portal Policies.

FileFly Agent is also responsible for retrieving file data from secondary storage upon

user/application access.

Data is streamed directly between agents and storage without any intermediary staging on disk.

When installed in a Gateway configuration, FileFly Agent does not allow migration of files from that server.

Optionally, Gateways can be configured for High-Availability (HA).

Caringo FileFly FPolicy Server

FileFly FPolicy Server provides migration support for NetApp filers via the NetApp FPolicy protocol. This component is the equivalent of Caringo FileFly Agent for NetApp filers.

FileFly FPolicy Server may also be configured for High-Availability (HA).

Caringo FileFly DrTool

Caringo FileFly DrTool is an additional application that assists in Disaster Recovery scenarios.

Note: This functionality is not included with Community Edition licenses.

2  Deployment

This chapter will cover:

• Installing Caringo FileFly Tools

• Installing Caringo FileFly Agent on file servers

• Installing Caringo FileFly Gateways as required

• Getting started with FileFly policies

• Production readiness

Refer to these instructions during initial deployment and when adding new components. For upgrade instructions, please refer to Chapter 8 instead.

For further details and usage instructions for each platform, refer to Chapter 4.

2.1  DNS Best Practice

In a production deployment, Fully Qualified Domain Names (FQDNs) should always be used in preference to bare IP addresses.

Storage locations in Caringo FileFly are referred to by URI. Relationships between files must be maintained over a long period of time. It is therefore advisable to take steps to ensure the FQDNs used in these URIs are valid long-term, even as individual server roles are changed or consolidated.

Create DNS aliases for each logical storage role for each server. Use different DNS aliases when storing your finance department’s data as opposed to your engineering department’s data – even if they initially reside on the same server.

2.2  Installing FileFly Tools

The Caringo FileFly Tools package consists of the FileFly Admin Portal and the FileFly DrTool application (not licensed for Community Edition users). The FileFly Admin Portal provides central management of policy execution while the FileFly DrTool is used in disaster recovery situations.

FileFly Tools must be installed before any other components.

2.2.1  System Requirements

• A dedicated server with a supported operating system:

• Windows Server 2016

• Windows Server 2012 R2 (Apr 2014 update rollup)

• Windows Server 2012

• Windows Server 2008 R2 SP1

• Minimum 4GB RAM

• Minimum 2GB disk space for log files

• Active clock synchronization (e.g. via NTP)

Internet Explorer 11 or higher (possibly on a separate workstation) will be required to access the FileFly Admin Portal web interface.

2.2.2  Setup

1. Run Caringo FileFly Tools.exe

2. Follow the instructions on screen

FileFly Tools is configured in the Admin Portal web interface after completing the installation process. The FileFly Admin Portal will be opened automatically and can be found later via the Start Menu.

The interface will lead you through the process for installing your license.

For production licensed installations, a ‘Backup & Scrub Grace Period’ setup page will be displayed. Please read the text carefully and set the minimum grace period as appropriate and after consulting with your backup plan – see also §7.2. This value may be revised later via the ‘Settings’ page.

2.3  Installing FileFly Agents

Proceed to install DataCore FileFly Agents as described below once the FileFly Tools installation completes. FileFly Agents perform file operations as directed by Admin Portal Policies. Also, in the case of user/application initiated demigration, agents retrieve the file data from secondary storage autonomously.

2.3.1  FileFly Agent Server Roles

Each FileFly Agent server may fulfill one of two roles, selected at installation time.

In the ‘FileFly Agent for migration’ role, an agent assists the operating system to migrate and demigrate files. It is essential for the agent to be installed on all machines from which files will be migrated.

The agent provides access to CloudScaler and Swarm destinations in the Gateway role.

2.3.2  High-Availability Gateway Configuration

A high-availability gateway configuration is recommended. Such FileFly Gateways must be activated as ‘High-Availability FileFly Gateways’.

High-Availability Gateway DNS Setup

At least two FileFly Gateways are required for High-Availability.

1. Add each FileFly Gateway server to DNS

2. Create a single alias that maps to each of the IP addresses

3. Use this alias in FileFly destination URIs, do not use for individual nodes:

• gw-1.example.com→168.0.1

• gw-2.example.com→168.0.2

• example.com→192.168.0.1, 192.168.0.2

Note: The servers that form the High-Availability Gateway cluster must NOT be members of a Windows failover cluster.

For further DNS recommendations, refer to §2.1.

2.3.3  Installing FileFly Agent for Windows Servers

System Requirements

• Supported Windows Server operating system:

• Windows Server 2016

• Windows Server 2012 R2 (Apr 2014 update rollup)

• Windows Server 2012

• Windows Server 2008 R2 SP1

• Minimum 4GB RAM

• Minimum 2GB disk space for log files

• Active clock synchronization (e.g. via NTP)

Note: When installed in the Gateway role, a dedicated server is required, unless it is to be co-located on the FileFly Tools server. When co-locating, create separate DNS aliases to refer to the Gateway and the FileFly Admin Portal web interface.

Setup

1. Run the Caringo FileFly Agent.exe

2. Select install location

3. Select migration or Gateway role as appropriate, refer to 2.3.1

4. If installing a FileFly Gateway, select the desired plugins

5. Follow the instructions to activate the agent via FileFly Admin Portal

Activation

• If no clustering is required, activate as a ‘Standalone Server’

• If installing the FileFly Gateway for High-Availability, activate as a High-Availability FileFly Gateway

• If the server is part of a Windows failover cluster, and this clustered resource is to be used as a FileFly Source, activate as a Windows failover cluster node

For further information see §5.3.1.

Important: If any type of clustering is used, ensure that FileFly Agent for Windows is installed on ALL cluster nodes.

2.3.4  Installing Caringo FileFly FPolicy Server for NetApp Filers

A Caringo FileFly FPolicy Server provides migration support for one or more NetApp

Filers through the FPolicy protocol. This component is the equivalent of Caringo FileFly Agent for NetApp Filers. Typically, FileFly FPolicy Servers are installed in a High Availability configuration.

System Requirements

• A dedicated server with a supported operating system:

• Windows Server 2016

• Windows Server 2012 R2 (Apr 2014 update rollup)

• Windows Server 2012

• Windows Server 2008 R2 SP1

• Minimum 4GB RAM

• Minimum 2GB disk space for log files

• Active clock synchronization (e.g. via NTP)

Setup

Installation of the FileFly FPolicy Server software requires careful preparation of the NetApp Filer and the FileFly FPolicy Server machines. Instructions are provided in §4.2.

Note: Legacy 7-Mode Filers require a different procedure at FileFly FPolicy Server installation time – see §4.3.

2.4  Installing Config Tools

In addition to the components described above, it may also be necessary to install one or more Config Tools. Full details are provided where required for each storage platform in Chapter 4.

2.5  Getting Started

2.5.1  Analyzing Volumes

The first step in a new DataCore deployment is to analyze the characteristics of the primary storage volumes once the software is installed. The following steps describe how to generate file statistics reports for each volume.

In the FileFly Admin Portal web interface (see Chapter 5 for full documentation):

1. Create Sources for each volume to analyze

2. Create a ‘Gather Statistics’ Policy and select all defined Sources

3. Create a Task for the ‘Gather Statistics’ Policy 4. On the ‘Overview’ tab, click Quick Run

4. Click on the Task’s name to run it immediately

5. When the Task has finished, expand the details by clicking on the Task name under ‘Recent Task History’

6. Click Go to Task to go to the ‘Task Details’ page

7. Access the report by clicking on View Last Stats

Pay particular attention to the ‘Last Modified % by size’ graph. This graph will help identify how much data would be affected by a migration policy based on the age of files.

Examine ‘File types by size’ to see if the data profile matches the expected usage of the volume.

2.5.2  Preparing for Migration

Using the information from the reports, create tasks to migrate files:

1. Prepare a destination for migrated files – see Create a Destination in FileFly Admin Portal

2. Create a Rule and a Migration Policy

   • A typical rule might limit migrations to files modified more than six months ago – do not use an ‘all files’ rule

   • To avoid unnecessary migration of active files, be conservative with your first Migration Policy

3. Create a Task for the new Policy

   • For now, disable the schedule

4. Save the task, then click on its name to open the ‘Task Details’ page

5. Click Simulate Now to