Swarm configuration parameters- persistence and configuration

Within any Swarm cluster, there are broadly 2 ways to define configuration parameter values.

First, you can specify configuration values that are read via a startup configuration file. This file can reside on a PXE boot server, the node.cfg file on a USB stick, a CSN’s cluster.cfg file, or via etcd as defined on an SCS. How this is configured of course depends on what method your cluster is using to boot. In the majority of cases in 2024, the SCS is the system used to boot a Swarm cluster.
Second, you can configure parameters where that configuration file exists within the cluster itself. This configuration is stored in a stream called the Persistent Settings Stream (PSS) or Cluster Settings UUID.

How you configure these specific configuration parameters/ values depends largely on the parameter in question. Some configuration values are valid cluster wide, meaning if you change the value, it affects every node. Some parameters are node-specific. Some parameters are available to be changed real-time. Some parameters require a reboot to change them. You may have been asked to set values via the swarmctl script from the Support tools bundle, from the scsctl command on an SCS, or even from the Swarm UI.

If you ever want to see what a parameter's value is on a node right now in the cluster, the best way to do that is to use the swarmctl command from the Support tools bundle. In the examples below, I am going to set SCSP_HOST to any Swarm IP. swarmctl will pick up that value and so you don’t have to then specify the IP within the swarmctl command. I am presuming your support tools are deployed in /root/dist as that is the default, but of course adjust the path as necessary. For example, you can use:
export SCSP_HOST=<any swarm node IP>
cd /root/dist
./swarmctl -p admin:<admin password> -C <any configuration parameter>

like so:
./swarmctl -p admin:<admin password> -C network.dnsDomain
./swarmctl -p admin:<admin password> -C network.dnsServers

You might have done similar commands, but let’s clarify what the output means.

If you take a look at the last columns, you will see Level and Readonly which are particularly useful. If the "level" says "cluster" and the Readonly is false, that means that the parameter and its value are stored in the Persistent Settings Stream. Any time you use swarmctl to change a value like this, this value applies to every node in the cluster. When you boot a node, it will read this value in the PSS as it comes online. Every node will have this same value for that parameter. Every value in the PSS that shows False in the Readonly column can be changed real-time with swarmctl.

If you see "node" in the Level column, that means specifically that the value is NOT in the PSS. Some of these node level parameters can be changed real-time and some can't. You can tell by the Readonly column. An example of a node level parameter that is NOT readonly is log.nodeLogLevel. That means you can change the log level of one node leaving the other nodes with the default log level.

If you see a value like 0 here, that means that the value is not set. If you change this value to 10, for example, and want to set it back to the default log level, you could set log.nodeLogLevel to 0 which effectively unsets the node level value.

So it makes sense that you don't have to use -a for any cluster level parameters as they are all the same.

Now, how does this relate to scsctl? swarmctl output shows real-time what's in the cluster, both cluster-wide and node-specific values. scsctl affects what is stored in the etcd database in a container on the SCS. When you run scsctl <component> config show -d, you are actually querying etcd. This may or may not be consistent with what's in the cluster real-time.

You have to set a node level, readonly value with scsctl because you can't set it real-time (hence the readonly designation). That means that when you set a node level, readonly parameter with scsctl, you have to reboot the nodes to pick up that value as that is when the node reads the configuration from etcd.

When you use scsctl to set readonly FALSE values, the SCS does it's best to set the value real-time in the cluster along with updating its etcd database. In our experience, it is not 100% perfect. Meaning, if you set a value with scsctl, it is best to double check that parameter's value with swarmctl to verify the change was made real-time in the cluster. I typically use swarmctl for everything cluster level. I use scsctl for anything that is node level since node level values, even if changed with swarmctl (like the above log.nodeLogLevel parameter) will NOT persist through a node reboot.

To summarize:

swarmctl affects a running cluster's parameter values. Some of these parameters are persistent, some aren't. Some are node-level, some aren't. Any node level parameter set with swarmctl will be reset on node reboot.
scsctl affects the etcd database and you have to use it to affect any node level readonly parameters. Those parameters will require a reboot to take effect. scsctl can set node and cluster level parameters, but if setting cluster level parameters, I would double check any desired real-time changes using swarmctl.