How to set up a Docker Registry with a Swarm S3 storage target

Owned by Tony Lokko (Unlicensed)
Last updated: Feb 05, 2024 by Jamshid AfsharVersion comment
5 min read

Containers are a great way to build small services and run multiple sandboxed services in a single host. One of the most popular tools to run containers is a service called Docker.

Docker distributes containers using a Registry, a server-side storage application for Docker images. It works much like yum or apt package managers, except that it is for full images.

When you issue a docker pull image command, it will contact the primary public registry and try to locate and download that image. However, if you need tight control over your images, your organization can run its own registry to pull from.

The registry comes from Docker and is a small container. For its storage back end, it defaults to the file system, but you can configure any of several storage back ends, such as S3.

The S3 storage back end supports S3-compatible services. This allows the registry service to be stateless, with no volumes to be backed up or preserved.

To set up a registry

Prerequisites

  • A running Docker environment with an internet connection.
  • A running Swarm cluster with S3 exposed.

Note that you can choose between two methods for setting up the container:

  1. Docker run with an add-in config file
  2. Docker run with environment variables

Pull the registry image

First, you need to pull the docker image for the current version of the registry (which, at the time of writing, is version 2.7):

docker pull registry:2.7

2.7: Pulling from library/registry

Digest: sha256:8004747f1e8cd820a148fb7499d71a76d45ff66bac6a29129bfdbfdc0154d146

Status: Image is up to date for registry:2.7

This pulls down the docker container from the public registry to your local docker host.

The host will then be able run the container via a docker run command:

docker run -d -p 5000:5000 --restart always --name registry:2.7

This command would run a docker registry with local storage bound to port 5000 on the host.

Option 1: Configure with YAML file

You can set up S3 using a YAML config file. The Docker configuration documentation starts with this example:

https://raw.githubusercontent.com/docker/distribution/master/cmd/registry/config-example.yml

version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
auth:
  htpasswd:
    realm: basic-realm
    path: /etc/registry
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3

The elements to change are in the storage section.

Replace storage:with S3:
storage: 
    cache: 
        blobdescriptor: inmemory 
    filesystem: 
        rootdirectory: /var/lib/registry


  s3:
    accesskey: awsaccesskey
    secretkey: awssecretkey
    region: us-west-1
    regionendpoint: http://myobjects.local
    bucket: bucketname
    encrypt: true
    keyid: mykeyid
    secure: true
    v4auth: true
    chunksize: 5242880
    multipartcopychunksize: 33554432
    multipartcopymaxconcurrency: 100
    multipartcopythresholdsize: 33554432
    rootdirectory: /s3/object/name/prefix



Example of a full config.yml:

version: 0.1
log:
    fields:
        service: registry
storage:
    cache:
        blobdescriptor: inmemory
    s3:
        accesskey: 480ee4b88c3380b70d957a3fb2d69054
        secretkey: test
        region: us-west-1
        regionendpoint: http://dockertest.caringo.com
        bucket: registry1
        secure: false
        v4auth: true
        chunksize: 5242880
        multipartcopychunksize: 33554432
        multipartcopymaxconcurrency: 100
        multipartcopythresholdsize: 33554432
http:
    addr: :5000
    headers:
        X-Content-Type-Options: [nosniff]

Save the config file and do a run to apply it:

docker run -d -p 5000:5000 --restart=always --name registryS3t2 -v `pwd`/config.yml:/etc/docker/registry/config.yml registry:2.7

This causes docker to start a container named registryS3t2, pull the config file from /config.yml, and overwrite the one in the container.

Option 2: Configure with environment variables

You could do the same configuration using environment variables. Run docker and supply all of these values:

docker run -d -p 5000:5000 --name registry --restart=always \
	-e REGISTRY_STORAGE=s3 \
	-e REGISTRY_STORAGE_S3_REGION=us-east-1 \
	-e REGISTRY_STORAGE_S3_BUCKET=your.bucket.example.com \
	-e REGISTRY_STORAGE_S3_ROOTDIRECTORY=docker-registry-exp1 \
	-e REGISTRY_STORAGE_S3_V4AUTH=false \
	-e REGISTRY_STORAGE_S3_ACCESSKEY=480ee4b88c3380b70d957a3fb2d69054 \
	-e REGISTRY_STORAGE_S3_SECRETKEY=secret registry:2.7

Push containers to registry

Once your registry is up and running, you can download containers and push them to your local registry.

docker pull ubuntu

Using default tag: latest

latest: Pulling from library/ubuntu

Digest: sha256:b88f8848e9a1a4e4558ba7cfc4acc5879e1d0e7ac06401409062ad2627e6fb58

Status: Downloaded newer image for ubuntu:latest

Next, tag the pulled image with info for the local registry:

docker image tag ubuntu localhost:5000/myfirstimage

Then push the image to the S3-enabled Swarm cluster:

docker push localhost:5000/myfirstimage

e80c789bc6ac: Mounted from my-ubuntu

6c3332381368: Mounted from my-ubuntu

ef1a1ec5bba9: Mounted from my-ubuntu

a1aa3da2a80a: Mounted from my-ubuntu

latest: digest: sha256:1bbdea4846231d91cce6c7ff3907d26fca444fd6b7e3c282b90c7fe4251f9f86 size: 1152

The push refers to the repository [localhost:5000/myfirstimage].

Use docker ps to list your containers:

docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

d3b5f0c552e4 registry:2.7 "/entrypoint.sh /etc…" 7 days ago Up 7 days 0.0.0.0:5000->5000/tcp registryS3t2

With that done, your local developers will be able to push containers from their local machines to this registry using the forward-facing IP and port.

Important

The config file example shows how to interact with the storage and is sufficient only for a small lab environment. To put the registry into production, more configuration is required.

Follow the Docker registry guidance:

Option 3: Configure with YAML file

Here is a docker-compose.yml for our internal registry, of course with a Swarm S3 backend.

# /var/permanent/s3-registry/docker-compose.yml
#
# $ cd /var/permanent/s3-registry
# $ docker compose up -d
# $ docker compose ps
# $ docker compose logs -f
#
# Assumes directory /var/permanent/certs exists on the docker
# server containing a valid cert for the docker server hostname.

version: '3.7'

services:

  s3registry:
    restart: always
    image: registry:2
    deploy:
      resources:
        limits:
          memory: 2g
    ports:
      - "3333:5000"
    secrets:
      - docker-repo.tx.caringo.com.crt
      - docker-repo.tx.caringo.com.key
      - s3_accesskey
      - s3_secretkey
    entrypoint: ["sh", "-xc", "REGISTRY_STORAGE_S3_ACCESSKEY=`cat /run/secrets/s3_accesskey` REGISTRY_STORAGE_S3_SECRETKEY=`cat /run/secrets/s3_secretkey` registry serve /etc/docker/registry/config.yml"]
    environment:
      - REGISTRY_LOG_LEVEL=debug
      - REGISTRY_LOG_FIELDS_SERVICE=registry
      - REGISTRY_LOG_FIELDS_ENVIRONMENT=development
      
      - REGISTRY_STORAGE_DELETE_ENABLED=true
      - REGISTRY_HTTP_TLS_CERTIFICATE=/run/secrets/docker-repo.tx.caringo.com.crt
      - REGISTRY_HTTP_TLS_KEY=/run/secrets/docker-repo.tx.caringo.com.key
      
      - REGISTRY_STORAGE=s3
      - REGISTRY_STORAGE_S3_SECURE=true
      - REGISTRY_STORAGE_S3_REGION=generic
      - REGISTRY_STORAGE_S3_REGIONENDPOINT=https://registry-blobs.cloud.caringo.com
      - REGISTRY_STORAGE_S3_ENCRYPT=false
      - REGISTRY_STORAGE_S3_SKIPVERIFY=false
      - REGISTRY_STORAGE_S3_BUCKET=docker-repo
      - REGISTRY_STORAGE_S3_ROOTDIRECTORY=
      - REGISTRY_STORAGE_S3_ACCESSKEY
      - REGISTRY_STORAGE_S3_SECRETKEY

      - REGISTRY_STORAGE_S3_CHUNKSIZE=104857600
      - REGISTRY_HTTP_HOST=https://docker-repo.tx.caringo.com:3333

      # Normally docker is redirected to the S3 endpoint for blob downloads. 
      # Uncomment this if that endpoint is not exposed to docker.
      # - REGISTRY_STORAGE_REDIRECT_DISABLE=true

secrets:
  s3_accesskey:
    file: /home/build/s3-access-key.txt
  s3_secretkey:
    file: /home/build/s3-secret-key.txt
  docker-repo.tx.caringo.com.crt:
    file: /var/permanent/certs/docker-repo.tx.caringo.com.crt
  docker-repo.tx.caringo.com.key:
    file: /var/permanent/certs/docker-repo.tx.caringo.com.key


© DataCore Software Corporation. · https://www.datacore.com · All rights reserved.