...

...

...

...

...

...

...

...

The S3 Backup Restore Tool is the standalone utility for performing DR from the S3 backup bucket, either to the original cluster or to an empty cluster that is meant to replace the original. See S3 Backup Feeds.

Once the data is backed up in S3, the restore tool allows both examining a backup and control controlling how, what, and where it is restored:

• List all domains and buckets, or the buckets within a domain, with the logical space used for each.

• List all objects within a bucket or unnamed objects in a domain, optionally with sizes and paging.

• Restore either the complete cluster contents or else a list of domains, buckets, or individual objects to restore. 

• Rerun the restore, should any part of it fail to complete.

• Partition the restoration tasks across multiple instances of the command line tool, to run them in parallel.

...

Installing the Restore Tool

The S3 Backup Restore tool has a separate install package included in the Swarm download bundleavailable by Support request. Install it on one or more (for parallel restores) systems where the restore processes run.

Preparation (

...

One-

...

Time)

The swarmrestore package is delivered as a Python pip3 source distribution. Each machine needs to be prepared to be able to install this and future versions of swarmrestore.

1. As root, run the following command:

   Code Block
   language bash
   yum install python3

2. Verify version 3.6 is installed:

   Code Block
   python3 --version

3. Upgrade pip

pip3 install --upgrade pip

Installation

Uninstall Python 2 generation of the tool (caringo-swarmrestore-1.0.x.tar.gz) if installed:

...

Rerun this installation when a new version of swarmrestore is obtained:

1. Copy the latest version of the swarmrestore package to the server.

2. Run the following as root:

   Code Block
   language bash
   pip3 install caringo-swarmrestore-<version>.tar.gz

3. swarmrestore is likely in /usr/local/bin and is already in the path.

4. Repeat for any additional servers if planning to perform partitioning for parallel restores.

Restore Tool Settings

The tool uses a configuration file, .swarmrestore.cfg. Because , because the file contains sensitive passwords, the . The tool warns if the configuration file is not access-protected (chmod mode 600 or 400).

The configuration file follows the format of Swarm Storage settings files, using sections listing name = value pairs. These setting names map to the S3 Backup feed definition, where the values have the same meaning.

1. Locate the sample configuration file where it is installed:

   Code Block
   /usr/local/sample-.swarmrestore.cfg

2. Copy the file into the home directory and rename it, and open it for editing:

   Code Block
   language bash
   cp /usr/local/sample-.swarmrestore.cfg ~/.swarmrestore.cfg vi ~/.swarmrestore.cfg # Edit config settings

sample-.swarmrestore.cfg

# This is a sample configuration file for the swarmrestore utility.
# Save this file as ~/.swarmrestore.cfg and chmod 600 ~/.swarmrestore.cfg to keep passwords private.

# S3 host must be a fully qualified host name. The virtual host access style is supported if
# the host's first component is the bucket name.
# See https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region for Amazon S3 endpoints.
[s3]
host=s3.amazonaws.com
port=443
accessKeyID=<youraccesskeyid>
secretAccessKey=<yoursecretaccesskey>
bucketName=<yourbucketname>
region=us-east-1
# The option below uses HTTPS for access. For HTTP, set sslOption=none and adjust port.
sslOption=trusted

# The 4 options below are for swarmrestore initiating archival restore of content, such as GLACIER.
performArchiveRetrieval=false
retrievalTier=Standard
accountID=<ninedigitaccountid>
activeLifetimeDays=7

# Use these only if you need a forward proxy to reach the S3 service.
[forwardProxy]
host=
port=80
username=
password=

# The log file can be /dev/null, but logs are useful for diagnosing problems.
[log]
filename=swarmrestore.log
level=30

# The Swarm cluster must either be directly accessible or accessible via
# a proxy. The password below is the administrative password for the cluster.
[swarm]
host=<space separated list of swarm host IPs or gateway host>
password=ourpwdofchoicehere
cluster=<yourclustername>

Additional Restore Configuration

• Gateway — : Add the IP of the machine where the Restore tool runs to the Gateway configuration settingConfiguration scsp.allowSwarmAdminIP if communicating with a Swarm cluster via Gateway.

Using the Restore Tool

...

swarmrestore [<tool option>...] <command> [<command option> ...] [<objectspec> …]

Options:

• --help —  - Displays a summary of the current configuration. 

  Code Block
  language text
  >> swarmrestore --help usage: swarmrestore [-h] [-v] {ls,restore} ... Explore or restore objects stored in an S3 backup of a Swarm cluster. positional arguments: {ls,restore} ls list the contents of the S3 bucket, optionally recursively or using a long format restore restore the contents of the S3 bucket, optionally recursively or including prior versions optional arguments: -h, --help show this help message and exit -v, --version show program's version number and exit Uses ~/.swarmrestore.cfg for configuration.

• --version —  - Reports the version of the tool.

• ls --help — Displays - Displays help on the ls command, for listing and enumerating. 

  Code Block
  language text
  >> swarmrestore ls --help usage: swarmrestore [-h] [-v] {ls,restore} ... Explore or restore objects stored in an S3 backup of a Swarm cluster. positional arguments: {ls,restore} ls list the contents of the S3 bucket, optionally recursively or using a long format restore restore the contents of the S3 bucket, optionally recursively or including prior versions optional arguments: -h, --help show this help message and exit -v, --version show program's version number and exit Uses ~/.swarmrestore.cfg for configuration.

• restore --help — help - Displays help on the restore command, for selective restore and disaster recovery. 

  Code Block
  language text
  >> swarmrestore restore --help usage: swarmrestore restore [-h] [-R] [-v] [-n] [-p count/total] [-f FILE] [objectspec [objectspec ...]] positional arguments: objectspec any number of object specifications to restore optional arguments: -h, --help show this help message and exit -R, --recursive recursively traverse the objectspecs -v, --versions also restore prior versions -n, --noop perform checking but do not actually restore -p count/total, --partition count/total partition the work <count> from among <total> -f FILE, --file FILE use the specified file for objectspecs, one per line

ls

...

Subcommand

Enumeration and selection are handled by the ls command, which is modeled after the Linux command ls and whose results are captured with standard Linux stdout. Use the command to visualize what domains and buckets are backed up in S3 and are available to be restored. The output is sorted by name and interactively paginated to help manage large result sets by default.

...

Command options, which can be combined (for example, -Rvl): 

• -R or --recursive — - Recursively lists the given domain or bucket, or else the entire cluster. Without this option, the command lists the top-level contents of the object.

• -v or --versions — - List previous versions of versioned objects. Versions are not listed by default.

• -l or --long — - Lists details for each item returned in the output:

  • Creation date

  • Content length of the body

  • ETag

  • Archive status:

    • AN — - Archived; not available for restoration

    • AR — AR - Archived with an archive restore in progress; not available for restoration

    • AA — AA - Archived with a copy available for restoration

    • OK — OK - Not archived and fully available

  • Objectspec

  • Alias UUID, if the object is a domain or bucket

• <objectspec> —  - If none, the command runs across the entire contents of the S3 backup. If present, filters the command to a specific domain or bucket (context object) in Swarm. Use this format:

...

When running the command without any options, it returns the list of domains that are included in this S3 bucket for the Swarm cluster:

...

>>> swarmrestore ls -Rvl mydomain/ > mydomaincontents

restore

...

Subcommand

Object restoration and verification is handled by the restore subcommand, which has the following format:

restore [<command option> ...] [<objectspec> …]

<objectspec> —  - If none, applies the command to the entire cluster backup. If present, filters the command to a specific domain, bucket, object, or object version.
To target a command to a specific context (domain/bucket) or content object in Swarm, format the type of object as follows:

...

Any number of command options can be used, and the short forms may be combined with a single dash (-Rv). The <objectspecs>, -R, and -v options iterate over objects the same way as the ls command.

Options:

• -R or --recursive — - Recursively restore domains, buckets, or the entire cluster with an empty object spec. See above for what is iterated over when -R is not used.

• -v or --versions — - Include previous versions of versioned objects. They are not included by default.

• -f <file> or --file <file> — - Use objectspecs from a file instead of the command line.

• -p <count>/<total> or --partition <count>/<total> — - Partition work for a large restore job (but every instance restores buckets and domains before objects).

  • Example: To run 4 instances in parallel, configure each option to be one of the series: -p 1/4, -p -2/4, -p -3/4, -p -4/4

• -n or ---noop — - Perform the checking of a restore, but do not restore any objects.

  • Does not change the cluster state. The option can be used before and after a restore, as both a pre-check and a verification.

• <objectspecs> — <objectspecs> - Any number; newlines separate objects. If none, the top level of the cluster’s backup contents is the scope.

  • Using no object specification with the command options -Rv causes Swarm to restore all backed up objects in the entire cluster, including any historical versions of versioned objects.

What is restoredRestored: Restore copies an object from S3 to the cluster if the cluster object is missing or else older than the S3 object. Note: context objects restore before the content they contain: restore first restores any domains or buckets needed before restoring objects within them.

Output of Restore — : At the end of the restoration, the tool reports the number of objects restored and the number of objects skipped, for being either identical to or newer than the backed up copy. The command output lists each object spec with its status:

• current — - The object was not restored because the target cluster already has the same version of the object.

• older — - The object was not restored because it is older than the one in the target cluster.

• obsolete — - The object was not restored because the cluster does not allow the object to be written. Usually it means the object is deleted.

• needed — - The object needs restoration, but the -n option was used.

• restored — - The object was successfully restored.

• nocontext — - The object cannot be restored because its parent domain or bucket cannot be restored.

• failure — failure - The object cannot be restored. Consult the logs for details.

• archived — - The object is archived and the restore tool is not configured for archive restoration. This is a failure condition.

• initiated — - The object is archived and the tool has issued an object restoration request. See the Amazon S3 API RestoreObject Request Syntax. This is also a failure condition, but the object is counted in the archive retrieval initiated stats. It is these operations that incur expense to the bucket owner by the restore tool.

• ongoing — - The object is in archive and a restoration request has already been initiated. Restoration from archive is in progress. This is also a failure condition.

Rate of Restore — Restoration : Restoration may take a long time run, especially if recursion (-R) is used on domains or buckets. To boost the rate of restore, install the S3 Backup Restore tool on multiple servers and then run the restore command with partitioning parameters (-p) across all instances of the tool, which allows restoring faster in parallel, with minimal overlap.

Headers for Audit — : When the S3 Backup feed writes an object to the S3 bucket, it adds to the S3 copy a header (Castor-System-Tiered) that captures when and from where the object was tiered. When the S3 Backup Restore tool writes the S3 object back to Swarm, it includes that S3 header and then adds another one of the same, to capture when and from where the object was restored. These paired headers (both named Castor-System-Tiered) provide the audit trail of the object's movement to and from S3. Swarm persists these headers but does not include them in Entity-MD5 or Header-MD5 calculations. The dates are of the same format as Last-Modified (RFC 7232, section 2.2). See SCSP Headers. 

Audit

...

Headers

Castor-System-Tiered: <date-of-backup> <cluster-name>/<cluster-settings-uuid>
Castor-System-Tiered: <date-of-restore> <S3-service-host>/<bucket-name>

...