Configuring haproxy SSL offloading with a Self Signed Certificate on CentOS7/8

 

CentOS 8 / RHEL 8 / RockyLinux 8 : yum install haproxy

CentOS 7/RHEL 7 follow instructions provided here CentOS Security Update - [CentOS-announce] Announcing release of...

Note: On CentOS7/RHEL7 the active haproxy.cfg will be in '/etc/opt/rh/rh-haproxy18/haproxy'

Haproxy is already pre-installed on the Swarm Cluster Installer provisioned - and VMware bundle provided Gateway VM

The following configuration steps are needed to configure HAProxy as an SSL offloader for Content Gateway.

Step-by-step guide

  • Verify Content Gateway is listening on port 8080 for SCSP, 8090 for S3 and 8091 for Service Proxy:

The example below are not the default gateway ports, this is done on purpose to avoid port conflicts.

/etc/caringo/cloudgateway/gateway.cfg [scsp] enabled = true bindAddress = 0.0.0.0 bindPort = 8080 externalHTTPPort = 80 externalHTTPSPort = 443 [s3] enabled = true bindAddress = 0.0.0.0 bindPort = 8090 [cluster_admin] enabled = true bindAddress = 0.0.0.0 bindPort = 8091 externalHTTPSPort = 91
  • Setup and install HAProxy. This package is part of the EPEL repository.

  • Use the following example configuration for /etc/haproxy/haproxy.cfg or on CentOS7 with haproxy1.8 /etc/opt/rh/rh-haproxy18-haproxy/haproxy.cfg

global log 127.0.0.1 local2 chroot /var/lib/haproxy stats socket /var/lib/haproxy/stats mode 660 level admin user haproxy group haproxy daemon ca-base /etc/pki/tls/certs crt-base /etc/pki/tls/private ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS ssl-default-bind-options no-sslv3 maxconn 2048 tune.ssl.default-dh-param 2048 defaults log global mode http option forwardfor # Do not use "option http-server-close", it causes S3 PUT incompatibility with some clients including FileFly! option httplog option dontlognull timeout connect 5000 timeout client 50000 # This timeout should always be larger than gateway.cfg's [storage_cluster] indexerSocketTimeout # and large enough to not timeout an S3 CopyObject or an SCSP COPY in a versioned bucket. timeout server 130000 frontend www-http bind 0.0.0.0:80 http-request set-header X-Forwarded-Proto http http-request set-header X-Forwarded-Port 80 default_backend www-backend-scsp acl iss3 hdr_sub(Authorization) AWS acl iss3 url_reg [?&](AWSAccessKeyId|X-Amz-Credential)= use_backend www-backend-s3 if iss3 frontend www-https bind 0.0.0.0:443 ssl crt /etc/pki/tls/certs/YOUR_DOMAIN.pem http-request set-header X-Forwarded-Proto https http-request set-header X-Forwarded-Port 443 default_backend www-backend-scsp acl iss3 hdr_sub(Authorization) AWS acl iss3 url_reg [?&](AWSAccessKeyId|X-Amz-Credential)= use_backend www-backend-s3 if iss3 frontend www-https-svc bind 0.0.0.0:91 ssl crt /etc/pki/tls/certs/YOUR_DOMAIN.pem http-request set-header X-Forwarded-Proto https http-request set-header X-Forwarded-Port 91 default_backend www-backend-svc backend www-backend-scsp balance leastconn # On HAPROXY < 2.2 option httpchk HEAD /_admin/manage/version HTTP/1.1\r\nHost:\ haproxy-healthcheck http-check expect status 200 # On HAPROXY > 2.2 #option httpchk #http-check connect #http-check send meth HEAD uri /_admin/manage/version ver HTTP/1.1 hdr Host haproxy-healthcheck #http-check expect status 200 #redirect scheme https if !{ ssl_fc } <--- Uncomment this line if you want to force HTTPS server gw1 YOUR_GATEWAY1_IP:8080 check inter 10s fall 3 rise 2 server gw2 YOUR_GATEWAY2_IP:8080 check inter 10s fall 3 rise 2 backend www-backend-s3 balance leastconn # On HAPROXY < 2.2 option httpchk HEAD / HTTP/1.1\r\nHost:\ haproxy-healthcheck http-check expect status 403 # On HAPROXY > 2.2 #option httpchk #http-check connect #http-check send meth HEAD uri / ver HTTP/1.1 hdr Host haproxy-healthcheck #http-check expect status 403 #redirect scheme https if !{ ssl_fc } <--- Uncomment this line if you want to force HTTPS server gw1 YOUR_GATEWAY1_IP:8090 check inter 10s fall 3 rise 2 server gw2 YOUR_GATEWAY2_IP:8090 check inter 10s fall 3 rise 2 backend www-backend-svc balance leastconn # On HAPROXY < 2.2 option httpchk HEAD / HTTP/1.1\r\nHost:\ haproxy-healthcheck http-check expect status 401 # On HAPROXY > 2.2 #option httpchk #http-check connect #http-check send meth HEAD uri / ver HTTP/1.1 hdr Host haproxy-healthcheck #http-check expect status 401 # This rule rewrites CORS header to add the port number used on frontend http-request replace-value Access-Control-Allow-Origin (.*) \1:91 #redirect scheme https if !{ ssl_fc } <--- Uncomment this line if you want to force HTTPS server gw1 YOUR_GATEWAY1_IP:8091 check inter 10s fall 3 rise 2 server gw2 YOUR_GATEWAY2_IP:8091 check inter 10s fall 3 rise 2
  • Start HAProxy:
    systemctl restart haproxy

Loadbalancing healthcheck

If you have 2 or more gateways, it is recommended to use the http-check directives to verify the service is live.

Each protocol requires different healthcheck configuration.

SCSP Protocol

HAPROXY < 2.2

option httpchk HEAD /_admin/manage/version HTTP/1.1\r\nHost:\ haproxy-healthcheck http-check expect status 200

HAPROXY > 2.2

S3 Protocol

HAPROXY < 2.2

HAPROXY > 2.2

Admin Protocol ( cluster_admin )

HAPROXY < 2.2

HAPROXY > 2.2

Backend section for all protocols

Example Logging with SCSP health check

In the following example I ran a while loop, to read an object every second… and while doing it shutdown 1 of the 2 gateways waited a few seconds then started it back up.

Create a Self-Signed SSL Certificate

First we need to make an openssl.conf file, in this example YOUR_DOMAIN = swarm.example.com

Generate the self-signed CA private key

Generate the self-signed CA root certificate

Generate wildcard custom domain private key

Generate certificate signing request for your domain

Generate the final domain certificate

Generate the certificate pem file.

Place the YOUR_DOMAIN.pem file where you configured it in haproxy.cfg example here put it in /etc/pki/tls/certs

Copy the selfsignCA.crt to /etc/pki/ca-trust/source/anchors and run update-ca-trust to tell CentOS to trust your self-signed root certificate.

Restart haproxy to activate the changes.

Now you can copy the selfsignCA.crt file to your clients and follow the browser specific procedures to install and accept it.

 

The host <contentgatewayIP>:8091 needs to be used in the login page to connect for Swarm UI in the configuration example above.

Replication Feed configuration

The following setting must appear and be set properly in the /etc/caringo/cloudgateway/gateway.cfg file if the content gateway is going to be used as the destination for a remote replication feed:

In the example above, replicate "172.30" with the IP addresses (or prefix) of clients sending administrative requests to the gateway.

The most common example is the IP addresses (or prefix) of the nodes in a cluster using a remote replication feed with the gateway as the destination.

Troubleshooting tips

If you occasionally see the error SEC_ERROR_UNKNOWN_ISSUER , verify you don't have lingering old haproxy processes , and kill them manually.

To check if your selfsigned CA certificate is trusted run the following command:

for CentOS7 and haproxy 1.8 by default require additional log settings to see the output in a file, add the following line in /etc/rsyslog.conf after the boot.log

then run

you should now see haproxy logging in /var/log/haproxy.log

 

Do not forget to enable the ports on your firewall