Overview

After a storage node is assigned a static IP address and rebooted, SCS may continue to show the node's original IP address, and in some cases the same chassis ID may appear more than once in the storage instance inventory. The storage node itself comes online correctly in Swarm with the new static IP; the discrepancy is limited to SCS's own inventory records.

This is a known SCS inventory/registration issue. Rather than editing etcd manually, use the scs_ip_reconciler.py tool described below to detect and correct affected records.

Symptoms

• Running scsctl storage instance list -d shows the original (DHCP or previous) IP address instead of the newly configured static IP.

• The same chassis ID appears two or more times in the instance list.

• Duplicate entries may show the raw chassis ID as the Name instead of the expected chassis-<ID> format.

• Swarm (swarmctl) correctly shows the storage node online with the new static IP, confirming the static IP configuration itself applied successfully.

Prerequisites

• Run the tool on the SCS host.

• Requires Python 3, and uses podman and scsctl, which are already available on the SCS host.

Files Written by This Tool

By default, the tool writes to /var/lib/scs-ip-reconciler/:

The base directory can be overridden with the SCS_RECONCILER_DIR environment variable. For disaster recovery, copy snapshots off-box; scsctl backup remains the canonical full SCS backup.

Usage

Dry run (report only, no changes):

Apply the fix (snapshot, reconcile, rebuild cache, verify). The pod is restarted so the corrected IPs appear immediately:

Check current instance-list health only, without making changes:

Limit the operation to one component:

What the Tool Changes

For each node whose static IP override (network.ipV4Address) differs from its current SCS IP assignment, and which is reachable at that static IP, the tool:

• Points ip_assignments/<static-ip> at the node.

• Removes the stale ip_assignments/<old-ip> entry.

• Aligns the node's last_known_ip.

The tool never modifies node identities or GIDs, so it cannot remove a live node. It skips nodes that are not reachable at the override IP, container-network IPs, and IPs already owned by another node. Running the tool again on a healthy cluster makes no changes.

Overlapping or Chained Reassignments

When an entire cluster is renumbered, one node's new static IP is often another node's current (still-assigned) IP. The tool will not point a node at an IP that another node still holds; that node is skipped for the moment.

To resolve this without repeated manual runs, --apply loops internally: it re-reads etcd and reconciles again, pass by pass, until nothing is left to resolve. Freeing an IP in one pass unblocks the node that needed it in the next. This appears in the output as Pass 1, Pass 2, and so on. A single --apply --yes run fully converges, with one cache rebuild at the end; there is no need to run it a second time.

If something genuinely cannot be reconciled, for example a node that is not reachable at its static IP, or a direct A-to-B/B-to-A IP swap with no free intermediate address, it is reported at the end under a NOTE rather than skipped silently. Bring the node online (or free or stage an address), then re-run the tool.

Rollback

Every apply run writes an audit log. To undo a specific run, point --rollback at its audit file.

Preview the rollback:

Perform the rollback (snapshots are taken first, then the cache is rebuilt):

Full Restore From a Snapshot

Safety Summary

• Dry run by default; changes require --apply --yes.

• An etcd snapshot is taken and verified on the host before any apply or rollback.

• A per-change audit log records previous values for every change.

• The tool is idempotent, skips ambiguous cases, and only modifies IP-assignment records.