...

...

The storage cluster is designed to automatically adapt when a volume (hard drive) or node fails for any reason. Swarm checks every storage cluster volume during the node startup procedure, and it tracks any gaps in service that would trigger a status change:

• If a

  A volume is

  disconnected from the cluster for more than 2 weeks, it is

  considered "stale" and

  its

  the contents cannot be used unless an administrator

  specifically

  overrides this process

  .If a node is shut down

  if disconnected from the cluster for more than 2 weeks

  , all of its

  .

• A node and all volumes are considered stale and cannot be used if it is shut down for more than 2 weeks.

The "stale" status is triggered by a service gap of 2 weeks, which is the default value for the disk.obsoleteTimeout setting.

You can force Force a volume remount by modifying the disk.volumes setting and adding the :k (keep) policy option. You can also return Return them to service dynamically (either remounting or reformatting) using SNMP. (v9.3)

Reformatting

...

Volumes (

...

Recommended)

Reformatting the volume allows it to be filled by the health processor (HP) in an orderly fashion. Doing so Performing this prevents creating excessive work for the health processor and prevents generating trapped space that will need to be reclaimedneeded to be reclaimed.

With support tools updated, you can use swarmctl from your support tools directory to return a disk to service by reformatting the disk. This command does not work when run from CentOS 6 (like the CSN) but will work from any SCS, Content Gateway, or Elasticsearch node. You can specify the disk or, if left blank, it will reformat all stale disks on that node.

./swarmctl -d 192.168.4.173 -F /dev/sda -p "admin:datacore"

On any CentOS 6 based system, use SNMP for this procedure using the MIBs in our support tools directory using the read/write SNMP string as shown here. In this case, the MIBs are present in the current directory.

snmpset -v2c -c ourpwdofchoicehere -m ./CARINGO-MIB.txt:./CARINGO-CASTOR-MIB.txt  192.168.99.100 castorFormatStaleVolumeAction s "/dev/sda"

Remounting

...

Volumes

It is rarely desirable to remount a volume that has stale content. The volume's missing content will have been is recovered by this time and so the cluster will have has its full complement of replicas of the cluster's content. Adding extra replicas will create creates work for the health processor to sift through the replicas, cleaning up redundant and obsolete copies. This cleanup will create creates trapped space in the cluster that will take several HP cycles to reclaim.

...

...

...

...

With support tools updated, you can use swarmctl from your support tools directory to return a disk to service by re-mounting the disk. This command does not work when run from CentOS 6 (like the CSN) but will work from any SCS, Content Gateway, or Elasticsearch node. You can specify the disk or, if left blank, it will remount all stale disks on that node.

./swarmctl -d 192.168.4.173 -I /dev/sda -p "admin:datacore"

You will see these ANNOUNCE level messages in the castor.log file:

192.168.4.173 <177>2023-11-20T18:24:03.091Z 192.168.4.173 MAINPROC RESTAPI_STORAGE ANNOUNCE: AMV18 Remounting stale volume 78c8ebabc7ffa5a9d044793222d0d897(/dev/sda) (via Management API)
192.168.4.173 <177>2023-11-20T18:24:03.092Z 192.168.4.173 MAINPROC ADMIN ANNOUNCE: AAD14 Remounting stale /dev/sda volume by administrator request
192.168.4.173 <177>2023-11-20T18:24:03.093Z 192.168.4.173 MAINPROC NODE ANNOUNCE: AND02 Removed /dev/sda, volumeID was 78c8ebabc7ffa5a9d044793222d0d897
192.168.4.173 <177>2023-11-20T18:24:12.754Z 192.168.4.173 VOL-SDA-1 DISK ANNOUNCE: ADD06 Mounted /dev/sda(mechanical), volumeID is 78c8ebabc7ffa5a9d044793222d0d897

On a CSN or other CentOS server, you must use SNMP for this procedure using the MIBs using the read/write SNMP string as shown here. In this case, the MIBs are present in the current directory.

snmpset -v2c -c ourpwdofchoicehere -m ./CARINGO-MIB.txt:./CARINGO-CASTOR-MIB.txt  192.168.99.100 castorRemountStaleVolumeAction s "/dev/sda"

...