Overview

CentOS 7 and Rocky Linux 8 both reached end-of-life (EOL) in May/June 2024. As a result, migrating from CentOS 7 (C7) to a supported distribution is required. DataCore has standardized on Rocky Linux 9 (RL9) for future deployments.

This document describes the migration process for all components currently running on CentOS 7 to Rocky Linux 9.

Rocky Linux 9 provides:

• General support until 31 May 2027.

• Security patch support until 31 May 2032.

General Notes

• In this document, CentOS 7 has been abbreviated to C7, and Rocky Linux 9 to RL9.

• The guide below assumes that the components are all running as virtual machines. If you are running with physical servers, contact DataCore Support for further information.

• These processes typically assume the reuse of the same IPs and hostnames for both Frontend (also known as Public or External) and Backend (also known as Swarm) networks. The frontend IPs and hostnames can be changed if desired, but remember to change DNS and/or load-balancers to point to the new frontend IPs. Backend IPs can also be changed in all components EXCEPT Elasticsearch, but pay attention to any specific notes in the individual process. These IPs must be in the existing backend subnet, but outside of the desired DHCP range.

• These processes only copy across configuration files that are created by Swarm installation procedures. It is up to the user to ensure that the process is adapted to account for any additional packages or other server configurations that have been performed outside of the standard Swarm installation.

• The processes require that the C7 instance MUST be upgraded to the same major.minor version of Swarm component that will be deployed. This is further described in each section.

• The guide below assumes that either a minimal RL9 server has been installed, or that the respective RL9 OVA has been deployed from the DataCore VM bundle.

Prerequisites

• A (temporary) frontend IP. This allows copying relevant configuration files and an easy transition from C7 to RL9 instance. This IP can either be temporary, DHCP-assigned, or the desired final frontend IP.

• The rsync must be available on both the C7 and the RL9 instances.

• If deploying by installing a minimal RL9 instance, this MUST be internet-connected. There is no offline version of this guide at present. Therefore, it is necessary to deploy using the OVA method if internet connection is not possible.

• It is recommended that a valid backup and/or VM snapshot are made prior to commencing any upgrades or changes.

SCS Server

Prepare the (Old) C7 SCS Server

Upgrade SCS to the same version as the packages to be installed on RL9 or the OVA images. It is assumed that you are currently at v1.7.1-el7.
If the existing SCS server is less than 1.7.1, perform the upgrade to 1.7.1. Full instructions for the upgrade can be found here.

Gather Information

On the C7 instance, verify that the support bundle will aid DataCore Support in case of an issue or a requirement to revert changes.

Note down the IPs, Site Name, and Administrator credentials, which will be entered EXACTLY the same during the init wizard phase.

Create a backup of the SCS system configuration and Swarm software repo.

Prepare the (New) RL9 SCS Server

1. Install a minimal instance of Rocky Linux 9 or deploy the SCS RL9 OVA (recommended).

2. Prepare the basic configuration:

   1. Set the hostname: hostnamectl set-hostname <HOSTNAME>

   2. Set the timezone, if required: timedatectl set-timezone <CONTINENT>/<CITY>

   3. Configure the frontend (external) IP, if not configured during installation:
      nmcli con mod <NIC_NAME> <SETTING> <SETTING_VALUE>

3. Ensure that a second virtual disk of at least 100GB is installed and mounted to /var/log. This is automatically created and configured during the OVA deployment.

4. Ensure the OS is patched to the latest releases (assuming internet or repo connectivity): dnf update -y.

Configure the (New) RL9 SCS server

1. Copy the SCS backup to this server. ( if using the OVA template its not necessary to copy the node_exporter )

   C7_SCS_FRONTEND_IP=<IP_of_existing_C7_SCS_server> rsync -av -R --ignore-missing-args \ ${C7_SCS_FRONTEND_IP}:/etc/hosts \ ${C7_SCS_FRONTEND_IP}:/etc/chrony.conf \ ${C7_SCS_FRONTEND_IP}:/root/mig2RL9 \ ${C7_SCS_FRONTEND_IP}:/usr/local/bin/node_exporter \ ${C7_SCS_FRONTEND_IP}:/usr/lib/systemd/system/node_exporter.service \ /

2. RL9 does not automatically enable chronyd, therefore, this needs to be manually started. The configuration should have been copied from the old C7 server in the previous set.

   systemctl enable chronyd && systemctl start chronyd # And to check it is working correctly chronyc sources

3. Disable the SELinux ( Skip this step if you are using the OVA template )

   1. Check if SELinux is enabled or disabled. In a default RL9 installation, it will be Enforcing. However, if it is already disabled, skip to step 4.

      getenforce

   2. Disable SELinux, by editing the /etc/selinux/config file, commenting out the line SELINUX=enforcing or SELINUX=permissive, and adding the line SELINUX=disabled. Then reboot the server after saving the file.

      vi /etc/selinux/config ... #SELINUX=enforcing SELINUX=disabled ... reboot

4. Download the installation package from DataCore, and transfer the SCS installation package to /root. If you need the latest version for SCS use this step. ( You can skip this step if you are using the OVA template )

   cd ${PATH_TO_INSTALLER_PACKAGES} rpm --import RPM-GPG-KEY dnf install -y swarm-scs-VERSION.el9.x86_64.rpm

5. Initialize the SCS server containers and configurations with basic defaults.

   scsctl init wizard -a

   1. For Site_Name, use EXACTLY the same name as the old C7 SCS server, and as recorded at the beginning of this process.

   2. For Swarm Administrator Password, use EXACTLY the same password as the old C7 SCS server, and as recorded at the beginning of this process

   3. For the Swarm network IP, you can either specify the same Swarm IP as the existing C7 SCS server or specify a new IP.
      ** If using a new IP, this must still be in the same subnet and outside of the DHCP range, and you MUST make changes to the Swarm cluster detailed later in this guide.

6. If admin.username is not “admin”, run the following command:

   scsctl platform config set -d "admin.userName=${EXISTING_ADMIN.USERNAME}"

7. Restore the SCS backup.

   scsctl backup restore /root/mig2RL9/scs_backup_full.YYYYMMDD.tgz

8. Ensure that the platform and network_boot components have the correct version activated. This is likely to be the highest version number listed.

9. Initialize the DHCP server using the same parameters as the old C7 SCS instance and as recorded at the beginning of this process.

   scsctl init dhcp --dhcp-reserve-lower ${EXISTING_DHCP_LOWER} --dhcp-reserve-upper ${EXISTING_DHCP_UPPER}

10. If required, re-enable SELinux (i.e. reverting step 3). This may take some time to complete.

    Re-enable SELinux, by editing the /etc/selinux/config file, removing the line SELINUX=disabled, and un-commenting out the line SELINUX=enforcing or SELINUX=permissive.
    Save the file, then reboot the server.

    vi /etc/selinux/config ... SELINUX=enforcing ... reboot

11. If the system was deployed with a minimal installation, install the Swarm support tools.

    cd /root curl -O https://support.cloud.datacore.com/tools/updateBundle.sh bash updateBundle.sh fresh rm updateBundle.sh

    If the system was deployed with the OVA, the support tools are already installed, but update with the command below:

    cd /root/dist && ./updateBundle.sh

Migrate to the (New) RL9 SCS Server

At this point, the C7 SCS instance is the live server. This section of the process will finalize the process by disconnecting the C7 instance and making the RL9 server the primary instance.

1. If you are re-using the same Swarm IP on the backend network, disconnect the backend (Swarm) interface at the VM level for the C7 SCS instance, and connect the second NIC on the RL9 SCS VM to the Swarm private network VLAN.

2. Verify the following:

   1. You are receiving Swarm cluster logs.

      tail -F /var/log/datacore/castor.log

   2. You can see the storage nodes and can talk to them as expected.

      /root/dist/swarmctl -d ${ANY_STORAGE_NODE_IP}

   3. The settings have migrated correctly.

      scsctl storage software list # check current version is active scsctl storage instance list # check it is showing correct scsctl platform config show -d admin.userName # check it is same scsctl platform config show -d admin.password # check it is same

3. [Optional] Set static swarm storage IP addresses if you wish to retain the same ip addresses as before:

   scsctl storage instance list # will show your current configuration scsctl storage config set -d -i {chassis ID} "network.ipV4Address={your.static.address.here}"

4. Re-enable and restart the node_exporter, if present. (Skip this step if using the OVA template)

   [[ -e /usr/lib/systemd/system/node_exporter.service ]] && { \ systemctl enable --now node_exporter; }

5. If the old C7 SCS backend (Swarm) IP has been retained, it is not strictly necessary to reboot storage nodes, though, it is recommended that you reboot one or all nodes at this time to check that they successfully boot after this migration.
   If a new backend (Swarm) IP has been used, it is mandatory to reboot all the storage nodes, to propagate this new IP to the cluster configuration.

   # To test one storage node /root/dist/swarmctl -d ${ANY_STORAGE_NODE_IP} -Rchassis -p ${SWARM_ADMIN_USER}:${SWARM_PASSWORD} # To perform a rolling reboot on the storage nodes /root/dist/swarmrestart -d ${ANY_STORAGE_NODE_IP} -p ${SWARM_ADMIN_USER}:${SWARM_PASSWORD}

6. Now, new RL9 SCS is considered “Live”, so the C7 SCS server can be retired or powered off.

7. Assuming the frontend (public) IP is intended to be the same as the old C7 SCS server; this needs to be reconfigured accordingly, and the interface “bounced” for this to take effect.

   nmcli con mod ${SCS_FRONTEND_NIC_NAME} ipv4.addresses ${SCS_FRONTEND_IP}/${FRONTEND_PREFIX} nmcli con mod ${SCS_FRONTEND_NIC_NAME} ipv4.gateway ${GATEWAY} nmcli con mod ${SCS_FRONTEND_NIC_NAME} ipv4.dns ${DNS_SERVER_1},${DNS_SERVER_2} nmcli con mod ${SCS_FRONTEND_NIC_NAME} ipv4.dns-search ${DNS_DOMAIN} nmcli con mod ${SCS_FRONTEND_NIC_NAME} ipv4.method manual nmcli con mod ${SCS_FRONTEND_NIC_NAME} connection.autoconnect yes nmcli con reload nmcli con down ${SCS_FRONTEND_NIC_NAME} && nmcli con up ${SCS_FRONTEND_NIC_NAME}

8. Fix SCSL-487

   firewall-cmd --new-policy swarm-public --permanent firewall-cmd --reload firewall-cmd --policy swarm-public --add-ingress-zone=swarm_site --permanent firewall-cmd --policy swarm-public --add-egress-zone=public --permanent firewall-cmd --policy swarm-public --set-target=ACCEPT --permanent firewall-cmd --reload

9. Make sure to disable the IPv4 default route on the backend network interface

   nmcli con mod ${SCS_BACKEND_NIC_NAME} ipv4.never-default yes nmcli con up ${SCS_BACKEND_NIC_NAME}

Optional: Upgrade to the Latest Storage Software

1. Add the new storage bundle to the repo.

   scsctl repo component add -f [storage bundle file name]

2. Activate the new storage bundle version.

   scsctl storage software list scsctl storage software activate [storage software version]

3. Reboot the cluster.

   # To perform a rolling reboot on the storage nodes /root/dist/swarmrestart -d ${ANY_STORAGE_NODE_IP} -p ${SWARM_ADMIN_USER}:${SWARM_PASSWORD}

Gateway Server

This process aims to create a new gateway instance and to copy over the configuration files. Then, choose either option:

• Option A - adding the (new) RL9 server to the DNS and load-balancer configuration.

• Option B - transferring the (old) frontend (public) C7 Gateway IP to the (new) RL9 server.

In a production environment, there will likely be multiple gateway instances that need to be migrated. Therefore, repeat this procedure for each gateway, then test each instance thoroughly before proceeding to the next gateway instance.

Gather Information

On the C7 instance, make a support bundle that will aid DataCore Support in case of an issue or a requirement to revert changes.

Note the existing network configuration and plan any new IPs, if required.

Prepare the (New) RL9 Gateway Server

1. Install a minimal instance of Rocky Linux 9 or deploy the Gateway RL9 OVA.

2. Prepare the basic configuration:

   1. Set the hostname: hostnamectl set-hostname <HOSTNAME>

   2. Set the timezone, if required: timedatectl set-timezone <CONTINENT>/<CITY>

   3. Configure the frontend (external) IP, if not configured during installation.
      nmcli con mod <NIC_NAME> <SETTING> <SETTING_VALUE>

3. Verify the OS is patched to the latest releases (assuming internet or repo connectivity): dnf update

4. If you have deployed using the Gateway OVA, the software packages are already installed so you can skip this step.

   cd ${PATH_TO_INSTALLER_PACKAGES} rpm --import RPM-GPG-KEY dnf install -y caringo-gateway-VERSION.noarch.rpm dnf install -y caringo-gateway-webui-VERSION.noarch.rpm dnf install -y caringo-storage-webui-VERSION.noarch.rpm

5. If the system was deployed with a minimal installation, install the Swarm support tools.

   cd /root curl -O https://support.cloud.datacore.com/tools/updateBundle.sh bash updateBundle.sh fresh rm updateBundle.sh

   If the system was deployed with the OVA, the support tools should already be installed, but update the following:

   cd /root/dist && ./updateBundle.sh

Prepare the (Old) C7 Gateway Server

1. Upgrade Gateway to the same version as the packages to be installed on RL9 or the OVA images. For the purposes of this guide, the versions are assumed to be 8.3.0, 7.10.0, and 3.5.0.
   Comprehensive upgrade instructions are available on the https://perifery.atlassian.net/wiki/spaces/public/pages/2443817319 documentation. A summarized procedure is provided below:

   1. Download the installation package from DataCore, and transfer the GW installation package to /root.

   2. Upgrade the software and perform the relevant upgrade steps.

      cd ${PATH_TO_INSTALLER_PACKAGES} yum install -y caringo-gateway-VERSION.noarch.rpm yum install -y caringo-gateway-webui-VERSION.noarch.rpm yum install -y caringo-storage-webui-VERSION.noarch.rpm

2. Create a backup of the GW user changes.

   mkdir /root/mig2RL9 awk -F: '$3>=1000 && $3<65000 {print $0}' /etc/passwd > /root/mig2RL9/user1000 awk -F: '$3>=1000 {print $0}' /etc/group > /root/mig2RL9/group1000 for USER in $(awk -F: '{print $1}' /root/mig2RL9/user1000); do grep "^$USER:" /etc/shadow; done > /root/mig2RL9/shadow1000

Configure the (New) RL9 Gateway Server

1. Copy the configuration files and the GW backup to this server.

   C7_GW_FRONTEND_IP=<IP_of_existing_C7_GW_server> rsync -av -R --ignore-missing-args \ ${C7_GW_FRONTEND_IP}:/etc/hosts \ ${C7_GW_FRONTEND_IP}:/etc/chrony.conf \ ${C7_GW_FRONTEND_IP}:/root/mig2RL9 \ ${C7_GW_FRONTEND_IP}:/etc/caringo/cloudgateway \ ${C7_GW_FRONTEND_IP}:/etc/sysconfig/cloudgateway \ ${C7_GW_FRONTEND_IP}:/usr/local/bin/node_exporter \ ${C7_GW_FRONTEND_IP}:/usr/lib/systemd/system/node_exporter.service \ /

2. RL9 does not automatically enable chronyd, therefore, start this manually. The configuration should have been copied from the old C7 server in the previous set.

   systemctl enable chronyd && systemctl start chronyd # And to check it is working correctly chronyc sources

3. Recreate the frontend (public) firewall configurations.

   firewall-cmd --zone=public --add-port=${SCSP_PORT}/tcp --permanent firewall-cmd --zone=public --add-port=${S3_PORT}/tcp --permanent firewall-cmd --zone=public --add-port=${SWARMUI_PORT}/tcp --permanent firewall-cmd --zone=public --add-port=9095/tcp --permanent firewall-cmd --zone=public --add-port=9100/tcp --permanent firewall-cmd --reload

4. [Optional] Create a backend (swarm) firewall configuration.

   firewall-cmd --new-zone swarm --permanent firewall-cmd --reload firewall-cmd --zone=swarm --add-service=http --permanent firewall-cmd --zone=swarm --add-service=ssh --permanent firewall-cmd --zone=swarm --add-port=9095/tcp --permanent firewall-cmd --zone=swarm --add-port=9100/tcp --permanent firewall-cmd --reload

5. Ensures that only new users are appended by filtering out existing ones:

   grep -v -f <(awk -F: '{print $1}' /etc/passwd) /root/mig2RL9/user1000 > /root/mig2RL9/user1000_new grep -v -f <(awk -F: '{print $1}' /etc/group) /root/mig2RL9/group1000 > /root/mig2RL9/group1000_new grep -v -f <(awk -F: '{print $1}' /etc/shadow) /root/mig2RL9/shadow1000 > /root/mig2RL9/shadow1000_new

6. [Optional] Logs all existing duplicate users to a file (/root/mig2RL9/duplicate_users.log) for manual review:

   awk -F: '{print $1}' /root/mig2RL9/user1000 | while read user; do if id "$user" &>/dev/null; then echo "Duplicate user found: $user (UID: $(id -u $user))" >> /root/mig2RL9/duplicate_users.log fi done

7. Append only non-existing PAM user configurations.

   cat /root/mig2RL9/user1000 >> /etc/passwd cat /root/mig2RL9/group1000 >> /etc/group cat /root/mig2RL9/shadow1000 >> /etc/shadow

Migrate to the (New) RL9 Gateway Server

At this point, the C7 GW instance is still a live server. This section of the process will finalize the process by either adding the new server to the DNS/load-balancer or disconnecting the C7 instance and making the RL9 server the primary instance.

Option A - Adding to DNS/Load-Balancer

The frontend (external) interface was already configured during the initial preparations, so there is nothing that needs to be done here.

1. Configure the networking for the Backend (Swarm) network. The IP address must be distinct from the current C7 instance, reside within the Swarm subnet, fall outside the DHCP range, and be unique.

   nmcli con mod ${GW_BACKEND_NIC_NAME} ipv4.addresses ${GW_BACKEND_IP}/${BACKEND_PREFIX} nmcli con mod ${GW_BACKEND_NIC_NAME} ipv4.method manual nmcli con mod ${GW_BACKEND_NIC_NAME} connection.autoconnect yes nmcli con mod ${GW_BACKEND_NIC_NAME} connection.zone swarm nmcli con reload nmcli con down ${GW_BACKEND_NIC_NAME} && nmcli con up ${GW_BACKEND_NIC_NAME}

2. Proceed to test the gateway.

Option B - Replacing the Existing C7 Instance

1. Remove (or comment) the C7 gateway’s frontend (external) IP from the DNS and/or Load-Balancer configuration. This is not mandatory, but failure to do so may result in application-level issues trying to access an unresponsive gateway.

2. For the C7 Gateway instance, disconnect both the frontend and backend network ports, at the VM level.

3. Re-configure the frontend (external) interface to match the configuration of the old C7 gateway, and the interface will need to be “bounced” for this to take effect.

   nmcli con mod ${GW_FRONTEND_NIC_NAME} ipv4.addresses ${GW_FRONTEND_IP}/${FRONTEND_PREFIX} nmcli con mod ${GW_FRONTEND_NIC_NAME} ipv4.gateway ${GATEWAY} nmcli con mod ${GW_FRONTEND_NIC_NAME} ipv4.dns ${DNS_SERVER_1},${DNS_SERVER_2} nmcli con mod ${GW_FRONTEND_NIC_NAME} ipv4.dns-search ${DNS_DOMAIN} nmcli con mod ${GW_FRONTEND_NIC_NAME} ipv4.method manual nmcli con mod ${GW_FRONTEND_NIC_NAME} connection.autoconnect yes nmcli con reload nmcli con down ${GW_FRONTEND_NIC_NAME} && nmcli con up ${GW_FRONTEND_NIC_NAME}

4. Set up the networking for the Backend (Swarm) network. This will be the same as the current C7 instance.

   nmcli con mod ${GW_BACKEND_NIC_NAME} ipv4.addresses ${GW_BACKEND_IP}/${BACKEND_PREFIX} nmcli con mod ${GW_BACKEND_NIC_NAME} ipv4.method manual nmcli con mod ${GW_BACKEND_NIC_NAME} connection.autoconnect yes nmcli con mod ${GW_BACKEND_NIC_NAME} connection.zone swarm nmcli con reload nmcli con down ${GW_BACKEND_NIC_NAME} && nmcli con up ${GW_BACKEND_NIC_NAME}

Testing

1. Verify that the storage nodes are present and talk to them as expected.

   /root/dist/swarmctl -d ${ANY_STORAGE_NODE_IP} curl -I http://${ANY_STORAGE_NODE_IP}

2. Restart the cloudgateway services, and validate they start correctly.

   systemctl restart cloudgateway systemctl status cloudgateway

3. Verify that the gateway is performing correctly.

   # This should be unauthorized, but returning the relevant "Cloud Gateway" and "CAStor Cluster" header curl -I http://${GW_FRONTEND_IP}:${SCSP_PORT} # This should return a full set of header information curl -I -u ${VALID_USER}:${VALID_PASSWORD} http://${GW_FRONTEND_IP}:${SCSP_PORT}

4. Verify access via a web browser by navigating to http://${GW_FRONTEND_IP}:${SCSP_PORT}/_admin/portal and ensure that you can log in as usual.

5. Perform other tests, as applicable and able, to the gateway’s frontend IP.

6. Recommend rebooting the server and verifying all services come up as expected following a reboot.

Finalization

Now, this gateway is ready to be made available to the applications again.

1. Add (or uncomment) the RL9 gateway’s frontend (external) IP into the DNS and/or Load-Balancer configuration.

2. Remove the C7 gateway’s frontend (external) IP from the DNS and/or Load-Balancer configuration if necessaary.

3. Power off the C7 gateway server.

Elasticsearch Server

There are two approaches to this upgrade:

• Suspending a single node, transferring the (old) C7 Elasticsearch IP, configuration, and data to a (new) RL9 server, and then resuming with the RL9 server.

• Adding a (new) RL9 Elasticsearch server to the Elasticsearch cluster, allowing data to balance across, and then retiring an (old) C7 Elasticsearch server.

In a production environment, there will likely be multiple Elasticsearch nodes, that all need to be migrated. Therefore, repeat this procedure for each node, waiting for each node to complete before proceeding to the next Elasticsearch instance.

Gather Information

On the C7 instance, make a support bundle that will aid DataCore Support in the event of an issue or a requirement to revert changes.

Note the existing network configuration and plan any new IPs, if required.

Upgrade the (Old) C7 Elasticsearch Cluster

Upgrade Elasticsearch cluster to the same version as the packages to be installed on RL9 or the OVA images. For the purpose of this guide, it is assumed to be 7.17.14.
Full instructions for the upgrade can be found here https://perifery.atlassian.net/wiki/spaces/public/pages/2443809756

Upgrade the Elasticsearch cluster to match the version of the packages to be installed on RL9 or the OVA images. For the purposes of this guide, the version is assumed to be 7.17.14.

Complete upgrade instructions are available on the https://perifery.atlassian.net/wiki/spaces/public/pages/2443809756 documentation.

Prepare the (New) RL9 Elasticsearch Server

1. Install a minimal instance of Rocky Linux 8 or deploy the Elasticsearch RL9 OVA.

2. Prepare the basic configuration:

   1. Set the hostname: hostnamectl set-hostname <HOSTNAME>

   2. Set the timezone, if required: timedatectl set-timezone <CONTINENT>/<CITY>

   3. Configure the Frontend (External) IP, if not configured during installation: nmcli con mod <NIC_NAME> <SETTING> <SETTING_VALUE>.