Configuring the "restic" backup utility for Gateway S3

Owned by Jamshid Afshar
Last updated: Jun 29, 2021 by Rumena Zlatkova (Deactivated)Version comment
2 min read

The backup software "restic" (https://restic.net) earns its tagline "Backups done right!" as a safe and easy cross-platform command-line utility for backing up file systems.

It is not a “single namespace” kind of client like s3cmd or rclone, where the named streams in Swarm match the names of the original files. Restic stores all directories and files in 5MB deduped and client-side encrypted blobs. While it uses the S3 protocol it does not use S3 multipart uploads. It does not support compression (https://github.com/restic/restic/issues/21) or adjustable block sizes (https://github.com/restic/restic/issues/1071).

As you’d expect restic performs incremental backups and lets you make and recover snapshots and tags.

By default the encryption keys are kept on the server alongside the data, so you don’t have to worry about managing those keys. But the keys are encrypted with a password you enter when you run "restic init". Remember this is warrant-proof storage — if you forget the password you won’t be able to recover any backups!

A killer feature on Linux and macOS is that restic lets you mount your backups to a local virtual (FUSE) directory. This lets you e.g. verify your backup by mounting it then doing a simple recursive diff.

Step-by-step guide

Restic configuration is via environment variables for the access key, secret key, and domain endpoint. 

  1. Install "restic" on the machine with access to the directory you will be backing up. Restic should be available in your operating system's package manager or see https://restic.readthedocs.io/en/stable/020_installation.html.

    $ brew install restic  # or apt-get install -y restic or yum install -y restic

  2. Create an S3 token via the Content UI or a curl to your domain endpoint:

    $ curl -i -u caringoadmin -X POST -H 'X-User-Secret-Key-Meta: secret' \
         -H 'X-User-Token-Expires-Meta: +365' 'https://support.cloud.caringo.com/.TOKEN/'
    Enter host password for user 'caringoadmin': ...
    HTTP/1.1 201 Created
    Date: Tue, 03 Jul 2018 18:10:18 GMT
    Gateway-Request-Id: 2FE9C3C985129777
    Server: CAStor Cluster/9.6.0
    Via: 1.1 support.cloud.caringo.com (Cloud Gateway SCSP/5.3.0)
    Gateway-Protocol: scsp
    Set-Cookie: token=ce1a7e2f8ee53b035ab6d2f3979946a4; expires=Wed, 03-Jul-2019 18:10:18 GMT; path=/
    Content-Type: text/plain;charset=utf-8
    Content-Length: 87
    Token ce1a7e2f8ee53b035ab6d2f3979946a4 issued for caringoadmin in [root] with secret secret
  3. Export the configuration variables:

    $ export AWS_ACCESS_KEY_ID=ce1a7e2f8ee53b035ab6d2f3979946a4
    $ export AWS_SECRET_ACCESS_KEY=secret
    $ export RESTIC_REPOSITORY=s3:support.cloud.caringo.com/mybucket

  4. Initialize the restic repo the first time you create a backup. This will also create the bucket if it does not exist.

    $ restic init
    enter password for new repository:
    enter password again:
    created restic repository 07617a1b05 at s3:support.cloud.caringo.com/mybucket

    Please note that knowledge of your password is required to access
    the repository. Losing your password means that your data is
    irrecoverably lost.

  5. Now you're ready to backup a directory, using the same password you used with "restic init". This initial backup of 230GB took ~2 hours.

    $ restic backup /var/jenkins_home/
    enter password for repository:
    repository 07617a1b opened successfully, password is correct

    Files: 461496 new, 0 changed, 0 unmodified
    Dirs: 1 new, 0 changed, 0 unmodified
    Added: 228.495 GiB

    processed 461496 files, 277.912 GiB in 1:50:50
    snapshot f6f0e61f saved

  6. On Linux or macOS with FUSE support you can mount the backup as a read-only virtual directory to directly access backed up files. Enter the same password you used with "restic init".

    $ restic mount /mnt/restic-remote  # run this in a separate window

  7. Now you can access snapshots directly, eg..

    $ diff -u -r /var/jenkins_home /mnt/restic-remote/snapshots/latest/var/jenkins_home


DataCore does not directly support or guarantee third-party utilities but let us know if you have questions or run into problems.



© DataCore Software Corporation. · https://www.datacore.com · All rights reserved.