Swarm Cluster Services (SCS) 1.6

SCS will remember the IP addresses of storage nodes once they have been booted, as it did in SCS 1.5 (.1).

Assigning Pre-Defined IP Addresses to Storage Nodes

In version 1.6, static IP address assignment is by default enabled. SCS supports the same automatic assignment of static IP addresses to nodes but adds the additional ability to manually assign static IP addresses.

New Installations

Follow the normal installation procedures for both SCS and the Swarm Storage bundle for SCS, including a run of scsctl diagnostics config scan_missing at the end.

1. Initialize DHCP with the desired lower and upper reserved ranges.

• scsctl init dhcp --dhcp-reserve-lower {size_of_lower_range} --dhcp-reserve-upper {size_of_upper_range}

2. Verify the following:

• dhcpd.conf only mentions a single pool of IP addresses

• scsctl storage config show -d "network.useStaticAddresses" should show a value of true. This is required for both automatic and manual assignment of static IP addresses

Assigning Pre-Defined Static IP Addresses

Before assigning static IP addresses, you need to list the storage nodes and record their chassis IDs.

• scsctl storage instance list

Once you have made a list of the storage node chassis and their desired IP assignments, use the following command to set the static IP address for each node:

• scsctl storage config set -d -i {chassis ID} "network.ipV4Address={your.static.address.here}"

The final step is to reboot the cluster and confirm that the storage nodes have the desired IP addresses.

• scsctl storage instance list

SCS 1.5.1

Static IP address assignment is by default enabled, beginning with SCS 1.5 and Swarm 15.3. However, the process to ensure correct behavior differs somewhat based on whether the environment is being upgraded or is a fresh installation.

Architecture

From 1.5 onward, SCS no longer uses a dual-pool system of address allocation. Older versions used a “transient” DHCP pool to assign addresses while nodes were first booting, with the expectation that the nodes would later acquire an address from the more permanent “storage” pool.

Now, SCS uses a single pool of addresses, bounded by the upper and lower reserved ranges. Once a node begins booting and reports its IP address to SCS, SCS will associate that IP address with the node’s chassis ID and embed the IP address within the node’s node.cfg configuration file. This ensures that the node ends up with the same IP address, regardless of how DHCP responds during the initial boot process on subsequent boots.

Fresh Installations

The assignment of static IP addresses is by default enabled. To ensure this is the case, follow the normal installation procedures for both SCS and the Swarm Storage bundle for SCS, including a run of scsctl diagnostics config scan_missing at the end and initializing DHCP (scsctl init dhcp ...).

Verify the following:

• dhcpd.conf only mentions a single pool of IP addresses

• scsctl storage config show -d "network.useStaticAddresses" should show a value of true

Once these have been confirmed, the configuration is complete for static IP address assignments.

Upgrading Environments

From Dynamic IP Assignments

If the current environment relies on dynamic IP addresses for storage nodes, then no extra steps are required.

Follow the normal upgrade procedures for both SCS and the Swarm Storage bundle for SCS. When upgrading SCS, be sure to run scsctl diagnostics upgrade_check -a to ensure that all SCS-related software components are properly updated, verifying that all steps are completed successfully. Once all upgrade steps have been completed, including a run of scsctl diagnostics config scan_missing at the end, re-initialize DHCP (scsctl init dhcp ...) using the same upper and lower reserved ranges as in the current configuration.

Follow the verification steps mentioned in Fresh Installations. Once done, configuration is complete for static IP address assignments.

From Static IP Assignments

In order to set up static IP assignments in a pre-upgrade environment, several manual steps had to be taken:

• Changes to reserved ranges to hold static IP assignments

• Manual updates to node.cfg templates for each running node

Both of these should be undone when upgrading.

Remove Custom Templates

First, obtain a list of nodes that SCS is aware of:

scsctl storage instance list -d

For each node in this list, get the instance ID (chassis ID):

scsctl storage config file unset -d -i "{instance ID}" node.cfg

DHCP Updates

It is likely that in the initial setup of static IP address support, either: (1) an upper reserved range was defined, or (2) the lower reserved range was extended. Neither is needed, but it is not harmful to leave either in place other than a reduced number of IP addresses available for booting nodes. In most cases, this is not a significant impact.

If these addresses should be added back to the pool, record the desired lower and upper range sizes and save them for later in the upgrade process.

Upgrade

1. Remove any custom node.cfg templates, as described above.

2. Note the desired lower and upper reserved ranges for DHCP

3. Follow the normal upgrade procedures for both SCS and the Swarm Storage bundle for SCS.

   1. When upgrading SCS, be sure to run scsctl diagnostics upgrade_check -a to ensure that all SCS-related software components are properly updated, verifying that all steps complete successfully.

   2. Run scsctl diagnostics config scan_missing at the end.

4. Using the saved values for lower and upper reserved ranges, initialize DHCP: scsctl init dhcp --dhcp-reserve-lower {size_of_lower_range} --dhcp-reserve-upper {size_of_upper_range}

5. Follow the verification steps mentioned in Fresh Installations.

Once done, configuration is complete for static IP address assignments.

SCS 1.4 and Earlier

The Basics

To support static IP assignments, two main updates are needed: DHCP configuration and node.cfg for each booting node. The DHCP configuration updates are needed to ensure the static IP addresses used are not part of the DHCP pool. The changes to node.cfg are needed so the Swarm Storage nodes know what network information to use.

When making these changes, the DHCP updates should be made first, followed by the node.cfg changes. Once all the changes are complete, the Swarm Storage nodes can be (re-)booted to use the static IP assignments.

Updating DHCP Configuration

SCS DHCP configuration has three major ranges that it manages:

• A lower reserved range

• Dynamic (DHCP) allocation range, which is further subdivided

• An upper reserved range

The lower and upper reserved ranges are optional, though one of them MUST be defined. In practice, this is usually the lower range, and it is used for things like the SCS interface itself, attached Gateway instances, etc. The upper range typically goes undefined and unused (a size of zero).

For static IP assignments, this upper reserved range is typically the part of the network carved out for statically-addressed nodes. Alternatively, the lower range may be extended to allow room for the static IP addresses. However this may cause IP address conflicts with existing address assignments unless all nodes are booted together.

Using the Upper Reserved Range

The upper reserved range is determined not by providing specific IP address boundaries, but by specifying the number of addresses within the subnet that should be reserved. One address per storage node is sufficient, though adding a buffer for future expansion may be desired.

Once the size of the upper reserved range has been determined, use the following command:

scsctl init dhcp --dhcp-reserve-lower {prior_size_of_lower_range} --dhcp-reserve-upper {size_of_upper_range}

This command updates the DHCP configuration and restarts the dhcpd service.

Extending the Lower Reserved Range

Generally a lower reserved range has already been defined for an SCS installation. The first step in extending it is to make note of the current size of the lower reserved range. Then determine the size of the extension. One address per storage node is sufficient, though adding a buffer for future expansion may be desired. The new size of the lower reserved range is the sum of the current size and the size of the extension.

Use the following command to update the DHCP configuration and restart the dhcpd service:

scsctl init dhcp --dhcp-reserve-lower {new_size_of_lower_range}

Updating node.cfg Templates

SCS does not manage configuration files in the traditional sense. Rather, it uses templates that are rendered when needed, with values populated from settings. Typically the template for node.cfg is inherited from a generic template provided by the storage service within SCS, but it may be overridden for the entire cluster or for individual nodes. This method relies on the latter.

Download the Base Template

This is the file that needs to be customized for each node booting with a static IP address. To obtain the template:

scsctl storage config file show -d node.cfg > "./node.cfg.base"

Verify no extraneous line breaks are present in the template as this prevents nodes from booting properly. These can be introduced if a terminal session wraps the lines of text in the template. Example: Using :set list in vi displays unprintable characters and line endings.

List the Nodes

Obtain a list of nodes (or “instances” in SCS terminology) that need to be updated:

scsctl storage instance list -d

Make note of all of the instance IDs that are listed; they are needed for future commands.

Update Each Node

A custom node.cfg template needs to be created with the node’s networking information for each instance ID. That updated template then needs to be set for that node.

Customize the Template

Create a copy of the base template specific to the node:

cp "./node.cfg.base" "./node.cfg.{instance_id}"

Use a text editor such as vi or nano to add the following to the end of the node-specific template ('./node.cfg.{instance_id}' in the above command), and save the changes. The blank line at the end is required otherwise the node does not boot properly.

[network]
ipAddress = {static IP address}
netmask = {Swarm internal network netmask}
gateway = {internal IP address of SCS}

Upload the Template

Use the following command to tell SCS to use the customized template when booting:

scsctl storage config file set -d -i {instance_id} -f "./node.cfg.{instance_id}" node.cfg

Reboot the Cluster

Reboot the entire cluster once both DHCP and the node.cfg templates have been updated. The nodes use the static IP addresses assigned to them in the custom node.cfg templates once they are online.