New Swarm installations using SCS implementation

Reference the Swarm Cluster Services (SCS) implementation KB article for any new Swarm installation using Swarm version 14 or greater.

Hardware

The following table lists the recommended configuration for a CSN.

Parameter	Description
Node	x86 with 64-bit Intel® Xeon® or AMD® Opteron™ (or equivalent) CPUs Virtualization Swarm supports virtualization via VMware ESXi, Linux KVM, and Microsoft Hyper-V (contact Support for details).
RAM	6 GB minimum, 12 GB recommended
OS	Red Hat® Enterprise Linux® (RHEL) 6 (release 6.8, 6.9, 6.10) Server 64-bit (English version) CentOS 6 (release 6.8, 6.9, 6.10) 64-bit Server Edition (English version)
Disk Storage	RAID configuration, for redundancy A separate partition with at least 16GB of available space configured for `/var/log`, to prevent logging from over-running other critical processes in `/var`.
Network	2 Gigabit Ethernet NICs minimum, 4 Gigabit Ethernet NICs recommended Single-network: Dedicated subnet that the CSN can control to allocate storage node IP addresses Dual-network: Dedicated switch or VLAN for the internal storage network Managed switch (recommended) Either a disabled or properly configured switch spanning tree with a rapidly converging algorithm when using a secondary CSN

Operating System

Swarm CSN was developed and tested with the U.S. English versions of RHEL 6 Server 64-bit and CentOS 6 64-bit Server Edition with default repositories. Other versions/languages or Linux distributions are not currently supported. The CSN installation will fail if the RHEL/CentOS version is anything other than the English version of release 6.8+.

Note

When you install RHEL 6 Server 64-bit or CentOS 6 64-bit Server Edition, Swarm requires selecting the Basic Server installation type. Do not select Minimal, as this option may cause the CSN installation to fail.

Subsequent installation instructions will assume a pre-installed RHEL Linux environment with either internet connectivity to a default repository or an alternatively configured default yum repository for use in installing required third party packages. The CSN installation process is dependent on additional rpm packages from a default repository that are not installed on the system by default. Non-default repositories do not always contain the required version of all packages and are therefore not supported. Specifically, the tftp- server package required for network booting Swarm nodes must come from a default repository; the alternate atftp- server package is not supported.

If you are installing RHEL/CentOS from an ISO image, you must obtain the latest security fixes from a repository prior to installing or upgrading CSN. Specifically, for the GHOST security vulnerability you should update the glibc library (yum -y update glibc). If you install or upgrade RHEL/CentOS from an internet repository, the latest glibc library should already be included.

Upgrade warning

Upgrading to RHEL/CentOS 6.8+ while the EPEL repo is enabled will upgrade monit to version 5.14.1, which breaks the firstcsnboot script. Do not enable the EPEL repo until after the CSN has been installed.

Networking

The CSN provides two options for network configuration:

Dual-network — Isolates the Swarm cluster on a dedicated network so that the external network is isolated from both the PXE network boot server (including DHCP) and cluster multicast traffic. The CSN itself with a dual-network configuration is 'dual-homed' with half its network interfaces cabled to the external network and the other half cabled to the internal network. The storage nodes the CSN manages are cabled to the internal network only. This internal private network ensures the Swarm cluster traffic is protected from unauthorized access. In this network configuration, the CSN owns network interface detection and configuration, and it assumes there will not be any external manipulation of networking files or services. This configuration is ideal for installations without a lot of networking expertise and for those where cluster isolation is desirable.

Single-network — Makes the Swarm nodes directly addressable on the same network as the CSN without requiring the SCSP Proxy to proxy requests. Both the CSN and the storage nodes it manages are cabled to the same network with single-network. In single network configurations, the CSN assumes a network administrator familiar with the environment in which the CSN runs will configure and maintain the CSN's network interfaces as well as basic network configuration, like configuring a static IP identity for the CSN. A single-network configuration is ideal for installations that have some networking expertise and want the CSN and Swarm nodes' IP addresses to reside in a larger subnet range.

Both network options allow definition of what servers the CSN will network boot as Swarm nodes. Single-network enables netboot protection by default to prevent accidental format of servers not intended as storage nodes on a larger network. Dual-network does not require netboot protection by default, but it can be enabled.

Note

The CSN does not support migration from one type of network configuration to another without reinstalling the CSN.

The following table summarizes the pros and cons of each networking option:

Feature	Dual-network	Single-network
Isolate cluster multicast traffic on a separate internal network, using SCSP Proxy to proxy requests to and from the external network	X
Allow storage nodes to be directly addressable on a larger network without a proxy or an additional router connected to the private network		X
Requires a dedicated switch or VLAN for internal storage network	X
Requires a dedicated routable subnet that the CSN can control for allocation of storage node IP Addresses		X
Automate detection and configuration of network interfaces and bonding	X
Allow manual configuration of CSN networking and bonding by an experienced Linux administrator		X
Enable netboot protection to only allow network boot of Swarm nodes for explicitly defined servers	X (After enabling in the CSN console)	X (By default)

The following sections discuss additional details about each type of configuration to aid you in deciding which is more appropriate for your environment.

Dual-network

A dual-network CSN requires access to both the external network as well as a dedicated internal network. Allocation of the address space in the internal network is broken down as follows, depending on the network size selected during initial configuration (small or large):

Network Size	CSN	External Applications	DHCP	Swarm Netboot
small (/24)	x.y.z.0-16	x.y.z.17-32	x.y.z.33-83	x.y.z.84-254
large (/16)	x.y.0.0-254	x.y.1.0-254	x.y.2.0-254	x.y.3.0 - x.y.255.254

CSN range — provides IP Addresses for the various services on the Primary and Secondary CSNs.
External Applications range — provides for third-party applications that need to run on the internal network to interface with the Swarm cluster. All IP addresses in the third-party range must be static. If you are using CFS or FileFly with Swarm, the IP Address should be assigned in the third-party range. Best practice is to keep a list of all IP Addresses that have been allocated to different applications in the third-party range to prevent IP collisions.
DHCP range — provides an initial, temporary IP Address to Swarm nodes during their initial boot until permanent addresses can be assigned to each Swarm process by the CSN. Other applications that used the CSN's DHCP server on the internal network would reduce the number of Swarm nodes that could be booted at the same time, which is why a separate third-party range has been provided. For a small network, the maximum number of servers that can be simultaneously booted from the provided DHCP range is 50. Once booted, there are a total of 170 IP Addresses in the Netboot range for a small network so 50 servers could run roughly 3 multi-server Swarm processes per node. Swarm nodes that support higher multi-server process counts require additional IP Addresses in the Netboot range, decreasing the number of nodes that can be booted. In a large network, the maximum number of Swarm nodes that can be booted at one time is 254.
Netboot range — provides the IP Addresses seen in the Admin Console for all Swarm processes.

Cabling Dual-network Interfaces

A dual-network CSN requires at least one network interface each for the Internal and External networks. Additional available NICs would ideally be split between the two networks, alternating each NIC port's cabled connection across all network cards for redundancy (External, Internal, External, Internal, etc). The CSN's initial configuration script will detect how all NICs are cabled based on whether or not the specified external gateway is reachable from each NIC. The configuration script will print what it detects to allow you to verify correct cabling; see "Dual-network Primary Configuration" for details. Once confirmed during initial configuration, the CSN will bond all NICs assigned to each interface into a single bonded interface with balance-alb bonding.

Single-network

A single network CSN requires access to a dedicated subnet. The subnet must be at least a Class C range with 256 IP addresses and may be as large as a full Class B network. Important: The CSN itself must be configured with a static IP Address, subnet mask and gateway prior to CSN installation. The CSN's IP Address must be within the x.y.z.1 - x.y.z.16 range for the configured subnet. The gateway configured for a single-network CSN will also be used for the Swarm nodes when they are configured. See the RHEL/CentOS documentation for complete instructions on configuring networking. As an example, the following demonstrates what the manually configured em3 interface with a static 172.20.20.11 IP Address, 255.255.255.0 subnet mask and 172.20.20.1 gateway would look like in /etc/sysconfig/network-scripts/ifcfg-em3:

# CSN interface 
DEVICE=em3 
IPADDR=172.20.20.11 
NETMASK=255.255.255.0 
GATEWAY=172.20.20.1 
BOOTPROTO=none 
ONBOOT=yes 
USERCTL=no 
MTU=1500 
NM_CONTROLLED=no

Be sure to restart the network after configuring the IP Address (/etc/init.d/network restart). Similar to a dual-network configuration, a single-network CSN will divide up the allocated address space into ranges for the CSN, DHCP, and Netboot. The allocated ranges will vary depending on the size of the dedicated subnet and will be printed for confirmation during single-network installation.

Cabling Single-network Interfaces

All network interfaces on a single network CSN should be cabled to the same broadcast domain (VLAN) on one or more network switches. A minimum of two cabled network interfaces is still recommended for redundancy. In addition to configuring the CSN's static IP address, any desired bonding configuration (for either redundancy or throughput) should be completed by the administrator prior to starting single-network CSN installation. See the RHEL/CentOS documentation for complete instructions on configuring network bonding.

Pre-Installation Checklist

Before you install CSN, confirm all of the following:

You have valid IP addresses for your DNS and NTP servers.
For dual-network configurations: You must have two available external IP addresses: one to access the Primary CSN and another to access the CSN "cluster" (that is, the Primary CSN (required) and Secondary CSN (optional)). If you set up a Primary and Secondary CSN, you access the CSN console using the CSN cluster IP address in the event the Primary CSN is not available. For more information, see "Primary vs. Secondary" .

Before you continue, make sure these IP addresses are not already in use.

For single-network configurations: You must have statically configured the IP address for the CSN as described in previous sections.
All network interface adapters are connected properly. At least one network interface must be enabled during RHEL/CentOS installation so that it is available for use during CSN installation.

About Primary vs. Secondary

With two CSNs on an internal network, one CSN must be designated as the primary. A primary CSN is responsible for responding to DHCP requests and also for listening to all communication on the well-known IP addresses for the CSN's network(s). A secondary CSN can provide redundancy for all services on the primary CSN in the event of a disaster and can also provide scalability for incoming SCSP requests to the Swarm SCSP Proxy in dual-network configurations. To do this, you can either:

Use the /wiki/spaces/DOCS/pages/2443822872 to instantiate an SCSPClient that "load balances" across two or more Swarm SCSP Proxy(s).
Use standard web load-balancing software or hardware in front of the SCSP Proxy(s).

Important

In the event of a failover from a Primary CSN to a Secondary CSN, the Secondary takes on the complete network identify of the Primary, which is why the Primary and Secondary CSNs must be installed on the same client VLAN. If the networks are completely separate, the Secondary will not be able to restore the Primary's configuration, as the restored IP Addresses will not be reachable on the Secondary's network.

About monit

The CSN installation script installs the watchdog process monit, which performs a variety of functions, including starting and restarting services if they fail unexpectedly. If monit is not already installed on your system, CSN installs and configures it automatically. When monit is installed, the following is added to your existing configuration:

Changes /etc/monit.conf by adding the following:

#START Caringo additions 
set daemon 15                  # check services at 15 second intervals 
set httpd port 2812 and 
    use address localhost      # only accept connection from localhost 
    allow localhost            # allow localhost to connect to server 
set logfile /var/log/monit.log # logs to local file 
##END Caringo Additions

Installs several product-specific configuration files under /etc/monit.d. The file name ends with .monitrc
Adds a product-specific file under /etc/init/ to enable the upstart service to monitor monit so it runs when the machine restarts or if monit itself fails.

Note

Manual editing of monit configuration files is not supported.

Browser not supported