CentOS 8 / RHEL 8 / RockyLinux 8 : yum install haproxy
CentOS 7/RHEL 7 follow instructions provided here https://www.linux.org/threads/centos-announce-announcing-release-of-haproxy-1-8-on-centos-7-x86_64.18168/
Note: On CentOS7/RHEL7 the active haproxy.cfg will be in '/etc/opt/rh/rh-haproxy18/haproxy'
Haproxy is already pre-installed on the Swarm Cluster Installer provisioned - and VMware bundle provided Gateway VM
The following configuration steps are needed to configure HAProxy as an SSL offloader for Content Gateway.
Step-by-step guide
Verify Content Gateway is listening on port 8080 for SCSP, 8090 for S3 and 8091 for Service Proxy:
The example below are not the default gateway ports, this is done on purpose to avoid port conflicts.
/etc/caringo/cloudgateway/gateway.cfg [scsp] enabled = true bindAddress = 0.0.0.0 bindPort = 8080 externalHTTPPort = 80 externalHTTPSPort = 443 [s3] enabled = true bindAddress = 0.0.0.0 bindPort = 8090 [cluster_admin] enabled = true bindAddress = 0.0.0.0 bindPort = 8091 externalHTTPSPort = 91
Note: In this example HTTP access to those personalities is provided. The bind address needs to be modified to 127.0.0.1 for all 3 personalities if security hardening is desired and HTTPS needs to be forced.
Setup and install HAProxy. This package is part of the EPEL repository.
HAproxy is pre-installed on the SwarmContentGateway VM and SCI deployed gateway VM’s.
Use the following example configuration for
/etc/haproxy/haproxy.cfg
or on CentOS7 with haproxy1.8 /etc/opt/rh/rh-haproxy18-haproxy/haproxy.cfg
global log 127.0.0.1 local2 chroot /var/lib/haproxy stats socket /var/lib/haproxy/stats mode 660 level admin user haproxy group haproxy daemon ca-base /etc/pki/tls/certs crt-base /etc/pki/tls/private ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS ssl-default-bind-options no-sslv3 maxconn 2048 tune.ssl.default-dh-param 2048 defaults log global mode http option forwardfor # Do not use "option http-server-close", it causes S3 PUT incompatibility with some clients including FileFly! option httplog option dontlognull timeout connect 5000 timeout client 50000 # This timeout should always be larger than gateway.cfg's [storage_cluster] indexerSocketTimeout. timeout server 130000 frontend www-http bind 0.0.0.0:80 http-request set-header X-Forwarded-Proto http http-request set-header X-Forwarded-Port 80 default_backend www-backend-scsp acl iss3 hdr_sub(Authorization) AWS acl iss3 url_reg [?&](AWSAccessKeyId|X-Amz-Credential)= use_backend www-backend-s3 if iss3 frontend www-https bind 0.0.0.0:443 ssl crt /etc/pki/tls/certs/YOUR_DOMAIN.pem http-request set-header X-Forwarded-Proto https http-request set-header X-Forwarded-Port 443 default_backend www-backend-scsp acl iss3 hdr_sub(Authorization) AWS acl iss3 url_reg [?&](AWSAccessKeyId|X-Amz-Credential)= use_backend www-backend-s3 if iss3 frontend www-https-svc bind 0.0.0.0:91 ssl crt /etc/pki/tls/certs/YOUR_DOMAIN.pem http-request set-header X-Forwarded-Proto https http-request set-header X-Forwarded-Port 91 default_backend www-backend-svc backend www-backend-scsp balance leastconn option httpchk http-check connect http-check send meth HEAD uri /_admin/manage/version http-check expect status 200 #redirect scheme https if !{ ssl_fc } <--- Uncomment this line if you want to force HTTPS server gw1 YOUR_GATEWAY1_IP:8080 check inter 10s fall 3 rise 2 server gw2 YOUR_GATEWAY2_IP:8080 check inter 10s fall 3 rise 2 backend www-backend-s3 balance leastconn option httpchk http-check connect http-check send meth HEAD uri /_admin/manage/version http-check expect status 200 #redirect scheme https if !{ ssl_fc } <--- Uncomment this line if you want to force HTTPS server gw1 YOUR_GATEWAY1_IP:8090 check inter 10s fall 3 rise 2 server gw2 YOUR_GATEWAY2_IP:8090 check inter 10s fall 3 rise 2 backend www-backend-svc balance leastconn option httpchk http-check connect http-check send meth HEAD uri /_admin/manage/version http-check expect status 200 # This rule rewrites CORS header to add the port number used on frontend http-request replace-value Access-Control-Allow-Origin (.*) \1:91 #redirect scheme https if !{ ssl_fc } <--- Uncomment this line if you want to force HTTPS server gw1 YOUR_GATEWAY1_IP:8091 check inter 10s fall 3 rise 2 server gw2 YOUR_GATEWAY2_IP:8091 check inter 10s fall 3 rise 2
Start HAProxy:
systemctl restart haproxy
If when restarting HAProxy this error is thrown “Starting frontend www-https-svc: cannot bind socket [0.0.0.0:91]”, either disable SELinux or run this command:
setsebool -P haproxy_connect_any=1
Loadbalancing healtcheck
If you have 2 or more gateways, it is recommended to use the http-check directives to verify the service is live.
option httpchk http-check connect http-check send meth HEAD uri /_admin/manage/version http-check expect status 200 server gw1 YOUR_GATEWAY1_IP:8090 check inter 10s fall 3 rise 2 server gw2 YOUR_GATEWAY2_IP:8090 check inter 10s fall 3 rise 2
In the following example I ran a while loop, to read an object every second… and while doing it shutdown 1 of the 2 gateways waited a few seconds then started it back up.
Mar 15 08:01:54 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45870 [15/Mar/2024:08:01:54.044] www-http swarm-scsp/gw1 0/0/0/20/29 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.0|} "GET /test14/test2 HTTP/1.1" Mar 15 08:01:55 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45876 [15/Mar/2024:08:01:55.095] www-http swarm-scsp/gw2 0/0/0/20/28 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1" Mar 15 08:01:56 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45880 [15/Mar/2024:08:01:56.144] www-http swarm-scsp/gw1 0/0/0/16/22 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.0|} "GET /test14/test2 HTTP/1.1" Mar 15 08:01:57 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45890 [15/Mar/2024:08:01:57.187] www-http swarm-scsp/gw2 0/0/0/20/29 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1" **** Here I stop gw1 service... which produces 503's until haproxy detects it Mar 15 08:02:01 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45894 [15/Mar/2024:08:01:58.236] www-http swarm-scsp/gw1 0/0/-1/-1/3003 503 216 - - SC-- 1/1/0/0/3 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {|} "GET /test14/test2 HTTP/1.1" Mar 15 08:02:02 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45908 [15/Mar/2024:08:02:02.260] www-http swarm-scsp/gw2 0/0/0/20/29 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1" Mar 15 08:02:06 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45914 [15/Mar/2024:08:02:03.310] www-http swarm-scsp/gw1 0/0/-1/-1/3006 503 216 - - SC-- 1/1/0/0/3 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {|} "GET /test14/test2 HTTP/1.1" Mar 15 08:02:07 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45928 [15/Mar/2024:08:02:07.338] www-http swarm-scsp/gw2 0/0/0/19/27 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1" Mar 15 08:02:11 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45932 [15/Mar/2024:08:02:08.387] www-http swarm-scsp/gw1 0/0/-1/-1/3006 503 216 - - SC-- 1/1/0/0/3 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {|} "GET /test14/test2 HTTP/1.1" Mar 15 08:02:12 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45946 [15/Mar/2024:08:02:12.415] www-http swarm-scsp/gw2 0/0/0/14/21 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1" Mar 15 08:02:16 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45954 [15/Mar/2024:08:02:13.456] www-http swarm-scsp/gw1 0/0/-1/-1/3006 503 216 - - SC-- 1/1/0/0/3 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {|} "GET /test14/test2 HTTP/1.1" **** HAproxy figures out gw1 is down Mar 15 08:02:16 cert-prod-haproxy haproxy[1411077]: [WARNING] (1411077) : Server swarm-scsp/gw1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Mar 15 08:02:16 cert-prod-haproxy haproxy[1411077]: Server swarm-scsp/gw1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Mar 15 08:02:16 cert-prod-haproxy haproxy[1411077]: Server swarm-scsp/gw1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. **** At this point all calls go to gw2 only Mar 15 08:02:17 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45968 [15/Mar/2024:08:02:17.483] www-http swarm-scsp/gw2 0/0/0/19/28 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1" Mar 15 08:02:18 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45974 [15/Mar/2024:08:02:18.532] www-http swarm-scsp/gw2 0/0/0/18/24 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1" Mar 15 08:02:19 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45978 [15/Mar/2024:08:02:19.577] www-http swarm-scsp/gw2 0/0/0/17/24 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1" Mar 15 08:02:20 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45984 [15/Mar/2024:08:02:20.623] www-http swarm-scsp/gw2 0/0/0/16/23 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1" Mar 15 08:02:21 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45988 [15/Mar/2024:08:02:21.667] www-http swarm-scsp/gw2 0/0/0/16/25 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1" *** Here I start gw1 back up, and Haproxy detects it. Mar 15 08:02:36 cert-prod-haproxy haproxy[1411077]: [WARNING] (1411077) : Server swarm-scsp/gw1 is UP, reason: Layer7 check passed, code: 200, check duration: 18ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. Mar 15 08:02:36 cert-prod-haproxy haproxy[1411077]: Server swarm-scsp/gw1 is UP, reason: Layer7 check passed, code: 200, check duration: 18ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. Mar 15 08:02:36 cert-prod-haproxy haproxy[1411077]: Server swarm-scsp/gw1 is UP, reason: Layer7 check passed, code: 200, check duration: 18ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. *** Here we are back using both gateways Mar 15 08:02:37 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:46086 [15/Mar/2024:08:02:37.293] www-http swarm-scsp/gw2 0/0/0/13/21 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1" Mar 15 08:02:39 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:46090 [15/Mar/2024:08:02:38.336] www-http swarm-scsp/gw1 0/0/0/894/899 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.0|} "GET /test14/test2 HTTP/1.1" Mar 15 08:02:40 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:46100 [15/Mar/2024:08:02:40.256] www-http swarm-scsp/gw2 0/0/0/14/21 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1" Mar 15 08:02:41 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:46106 [15/Mar/2024:08:02:41.299] www-http swarm-scsp/gw1 0/0/0/18/23 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.0|} "GET /test14/test2 HTTP/1.1"
Create a Self-Signed SSL Certificate
First we need to make an openssl.conf file, in this example YOUR_DOMAIN = swarm.example.com
[ req ] prompt = no distinguished_name = server_distinguished_name req_extensions = v3_req default_md = sha256 [ server_distinguished_name ] commonName = *.swarm.example.com stateOrProvinceName = Texas countryName = US emailAddress = admin@example.com organizationName = Example Inc. localityName = Austin [ v3_req ] keyUsage = nonRepudiation, digitalSignature, keyEncipherment extendedKeyUsage = serverAuth subjectAltName = @alt_names [ alt_names ] DNS.1=swarm.example.com DNS.2=*.swarm.example.com
Generate the self-signed CA private key
openssl genrsa -out selfsignCA.key 4096
Generate the self-signed CA root certificate
openssl req -new -x509 -nodes -days 365 -sha256 -key selfsignCA.key -out selfsignCA.crt -subj "/C=US/O=_DEV CA/CN=SelfSigned certificates"
Generate wildcard custom domain private key
openssl genrsa -out YOUR_DOMAIN.key 4096
Generate certificate signing request for your domain
openssl req -new -nodes -key YOUR_DOMAIN.key -config openssl.conf -out YOUR_DOMAIN.csr
Generate the final domain certificate
openssl x509 -req -in YOUR_DOMAIN.csr -CA selfsignCA.crt -CAkey selfsignCA.key -CAcreateserial -out YOUR_DOMAIN.crt -days 365 -sha256 -extfile openssl.conf -extensions v3_req
Generate the certificate pem file.
cat YOUR_DOMAIN.crt YOUR_DOMAIN.key > YOUR_DOMAIN.pem
Place the YOUR_DOMAIN.pem file where you configured it in haproxy.cfg example here put it in /etc/pki/tls/certs
bind 0.0.0.0:443 ssl crt /etc/pki/tls/certs/YOUR_DOMAIN.pem
Copy the selfsignCA.crt to /etc/pki/ca-trust/source/anchors
and run update-ca-trust to tell CentOS to trust your self-signed root certificate.
This is an important step for haproxy as by default it is configured to look for trusted certificates in the /etc/pki/tls/certs
folder. ( see ca-base parameter )
Restart haproxy to activate the changes.
Now you can copy the selfsignCA.crt file to your clients and follow the browser specific procedures to install and accept it.
The host <contentgatewayIP>:8091 needs to be used in the login page to connect for Swarm UI in the configuration example above.
Replication Feed configuration
The following setting must appear and be set properly in the /etc/caringo/cloudgateway/gateway.cfg
file if the content gateway is going to be used as the destination for a remote replication feed:
[scsp] ... allowSwarmAdminIP=172.30
In the example above, replicate "172.30" with the IP addresses (or prefix) of clients sending administrative requests to the gateway.
The most common example is the IP addresses (or prefix) of the nodes in a cluster using a remote replication feed with the gateway as the destination.
Troubleshooting tips
If you occasionally see the error SEC_ERROR_UNKNOWN_ISSUER , verify you don't have lingering old haproxy processes , and kill them manually.
To check if your selfsigned CA certificate is trusted run the following command:
trust list --filter=ca-anchors | grep Self -i -A2 -B4 Example Output you should see: pkcs11:id=%f4%a2%f6%c6%e4%db%bc%c8%a1%23%83%d3%67%14%7c%51%c1%8d%bd%ba;type=cert type: certificate label: SelfSigned certificates trust: anchor category: authority
for CentOS7 and haproxy 1.8 by default require additional log settings to see the output in a file, add the following line in /etc/rsyslog.conf after the boot.log
local2.* /var/log/haproxy.log
then run
systemctl restart rsyslog
you should now see haproxy logging in /var/log/haproxy.log
Do not forget to enable the ports on your firewall
firewall-cmd --add-port=443/tcp --permanent firewall-cmd --add-port=80/tcp --permanent firewall-cmd --reload
Keep in mind that normally you have multiple zone’s , since gateway has 2 adapters, the above example only adds the ports to the Default firewall zone