Configuring haproxy SSL offloading with a Self Signed Certificate on CentOS7/8

CentOS 8 / RHEL 8 / RockyLinux 8 : yum install haproxy

CentOS 7/RHEL 7 follow instructions provided here CentOS Security Update - [CentOS-announce] Announcing release of...

Note: On CentOS7/RHEL7 the active haproxy.cfg will be in '/etc/opt/rh/rh-haproxy18/haproxy'

Haproxy is already pre-installed on the Swarm Cluster Installer provisioned - and VMware bundle provided Gateway VM

The following configuration steps are needed to configure HAProxy as an SSL offloader for Content Gateway.

Step-by-step guide

Verify Content Gateway is listening on port 8080 for SCSP, 8090 for S3 and 8091 for Service Proxy:

The example below are not the default gateway ports, this is done on purpose to avoid port conflicts.

/etc/caringo/cloudgateway/gateway.cfg

[scsp]
enabled = true
bindAddress = 0.0.0.0
bindPort = 8080
externalHTTPPort = 80
externalHTTPSPort = 443

[s3]
enabled = true
bindAddress = 0.0.0.0
bindPort = 8090

[cluster_admin]
enabled = true
bindAddress = 0.0.0.0
bindPort = 8091
externalHTTPSPort = 91

Setup and install HAProxy. This package is part of the EPEL repository.

Use the following example configuration for /etc/haproxy/haproxy.cfg or on CentOS7 with haproxy1.8 /etc/opt/rh/rh-haproxy18-haproxy/haproxy.cfg

global
    log 127.0.0.1 local2
    chroot /var/lib/haproxy
    stats socket /var/lib/haproxy/stats mode 660 level admin
    user haproxy
    group haproxy
    daemon

    ca-base /etc/pki/tls/certs
    crt-base /etc/pki/tls/private

    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
    ssl-default-bind-options no-sslv3
    maxconn 2048
    tune.ssl.default-dh-param 2048

defaults
    log     global
    mode    http
    option  forwardfor
    # Do not use "option  http-server-close", it causes S3 PUT incompatibility with some clients including FileFly!
    option  httplog
    option  dontlognull
    timeout connect 5000
    timeout client  50000
    # This timeout should always be larger than gateway.cfg's [storage_cluster] indexerSocketTimeout
    # and large enough to not timeout an S3 CopyObject or an SCSP COPY in a versioned bucket.
    timeout server  130000

frontend www-http
    bind 0.0.0.0:80
    http-request set-header X-Forwarded-Proto http
    http-request set-header X-Forwarded-Port 80
    default_backend www-backend-scsp
    acl iss3 hdr_sub(Authorization) AWS
    acl iss3 url_reg [?&](AWSAccessKeyId|X-Amz-Credential)=
    use_backend www-backend-s3 if iss3

frontend www-https
    bind 0.0.0.0:443 ssl crt /etc/pki/tls/certs/YOUR_DOMAIN.pem
    http-request set-header X-Forwarded-Proto https
    http-request set-header X-Forwarded-Port 443
    default_backend www-backend-scsp
    acl iss3 hdr_sub(Authorization) AWS
    acl iss3 url_reg [?&](AWSAccessKeyId|X-Amz-Credential)=
    use_backend www-backend-s3 if iss3

frontend www-https-svc
    bind 0.0.0.0:91 ssl crt /etc/pki/tls/certs/YOUR_DOMAIN.pem
    http-request set-header X-Forwarded-Proto https
    http-request set-header X-Forwarded-Port 91
    default_backend www-backend-svc

backend www-backend-scsp
    balance leastconn
    # On HAPROXY < 2.2
    option httpchk HEAD /_admin/manage/version HTTP/1.1\r\nHost:\ haproxy-healthcheck
    http-check expect status 200
    # On HAPROXY > 2.2
    #option httpchk
    #http-check connect
    #http-check send meth HEAD uri /_admin/manage/version ver HTTP/1.1 hdr Host haproxy-healthcheck
    #http-check expect status 200
  
    #redirect scheme https if !{ ssl_fc }   <--- Uncomment this line if you want to force HTTPS
    server gw1 YOUR_GATEWAY1_IP:8080 check inter 10s fall 3 rise 2
    server gw2 YOUR_GATEWAY2_IP:8080 check inter 10s fall 3 rise 2

backend www-backend-s3
    balance leastconn
    # On HAPROXY < 2.2
    option httpchk HEAD / HTTP/1.1\r\nHost:\ haproxy-healthcheck
    http-check expect status 403
    # On HAPROXY > 2.2
    #option httpchk
    #http-check connect
    #http-check send meth HEAD uri / ver HTTP/1.1 hdr Host haproxy-healthcheck
    #http-check expect status 403
    
    #redirect scheme https if !{ ssl_fc }    <--- Uncomment this line if you want to force HTTPS
    server gw1 YOUR_GATEWAY1_IP:8090 check inter 10s fall 3 rise 2
    server gw2 YOUR_GATEWAY2_IP:8090 check inter 10s fall 3 rise 2

backend www-backend-svc
    balance leastconn
    # On HAPROXY < 2.2
    option httpchk HEAD / HTTP/1.1\r\nHost:\ haproxy-healthcheck
    http-check expect status 401
    # On HAPROXY > 2.2
    #option httpchk
    #http-check connect
    #http-check send meth HEAD uri / ver HTTP/1.1 hdr Host haproxy-healthcheck
    #http-check expect status 401
    
    # This rule rewrites CORS header to add the port number used on frontend
    http-request replace-value Access-Control-Allow-Origin (.*) \1:91
    #redirect scheme https if !{ ssl_fc } <--- Uncomment this line if you want to force HTTPS
    server gw1 YOUR_GATEWAY1_IP:8091 check inter 10s fall 3 rise 2
    server gw2 YOUR_GATEWAY2_IP:8091 check inter 10s fall 3 rise 2

Start HAProxy:
systemctl restart haproxy

Loadbalancing healthcheck

If you have 2 or more gateways, it is recommended to use the http-check directives to verify the service is live.

Each protocol requires different healthcheck configuration.

SCSP Protocol

HAPROXY < 2.2

    option httpchk HEAD /_admin/manage/version HTTP/1.1\r\nHost:\ haproxy-healthcheck
    http-check expect status 200

HAPROXY > 2.2

S3 Protocol

HAPROXY < 2.2

HAPROXY > 2.2

Admin Protocol ( cluster_admin )

HAPROXY < 2.2

HAPROXY > 2.2

Backend section for all protocols

Example Logging with SCSP health check

In the following example I ran a while loop, to read an object every second… and while doing it shutdown 1 of the 2 gateways waited a few seconds then started it back up.

Create a Self-Signed SSL Certificate

First we need to make an openssl.conf file, in this example YOUR_DOMAIN = swarm.example.com

Generate the self-signed CA private key

Generate the self-signed CA root certificate

Generate wildcard custom domain private key

Generate certificate signing request for your domain

Generate the final domain certificate

Generate the certificate pem file.

Place the YOUR_DOMAIN.pem file where you configured it in haproxy.cfg example here put it in /etc/pki/tls/certs

Copy the selfsignCA.crt to /etc/pki/ca-trust/source/anchors and run update-ca-trust to tell CentOS to trust your self-signed root certificate.

Restart haproxy to activate the changes.

Now you can copy the selfsignCA.crt file to your clients and follow the browser specific procedures to install and accept it.

The host <contentgatewayIP>:8091 needs to be used in the login page to connect for Swarm UI in the configuration example above.

Replication Feed configuration

The following setting must appear and be set properly in the /etc/caringo/cloudgateway/gateway.cfg file if the content gateway is going to be used as the destination for a remote replication feed:

In the example above, replicate "172.30" with the IP addresses (or prefix) of clients sending administrative requests to the gateway.

The most common example is the IP addresses (or prefix) of the nodes in a cluster using a remote replication feed with the gateway as the destination.

Troubleshooting tips

If you occasionally see the error SEC_ERROR_UNKNOWN_ISSUER , verify you don't have lingering old haproxy processes , and kill them manually.

To check if your selfsigned CA certificate is trusted run the following command:

for CentOS7 and haproxy 1.8 by default require additional log settings to see the output in a file, add the following line in /etc/rsyslog.conf after the boot.log

then run

you should now see haproxy logging in /var/log/haproxy.log

Do not forget to enable the ports on your firewall