The storage cluster is designed to automatically adapt when a volume (hard drive) or node fails for any reason. Swarm checks every storage cluster volume during the node startup procedure, and it tracks any gaps in service that would trigger a status change:
- If a volume is disconnected from the cluster for more than 2 weeks, it is considered "stale" and its contents cannot be used unless an administrator specifically overrides this process.
- If a node is shut down for more than 2 weeks, all of its volumes are considered stale and cannot be used.
The "stale" status is triggered by a service gap of 2 weeks, which is the default value for the disk.obsoleteTimeout setting.
You can force a volume remount by modifying the disk.volumes setting and adding the :k (keep) policy option. You can also return them to service dynamically (either remounting or reformatting) using SNMP. (v9.3)
Reformatting volumes (recommended)
Reformatting the volume allows it to be filled by the health processor in an orderly fashion. Doing so prevents creating excessive work for the health processor and prevents generating trapped space that will need to be reclaimed.
snmpset -v2c -c ourpwdofchoicehere -m ./CARINGO-MIB.txt:./CARINGO-CASTOR-MIB.txt 192.168.99.100 castorFormatStaleVolumeAction s "/dev/sda"
Important
Remounting volumes
It is rarely desirable to remount a volume that has stale content. The volume's missing content will have been recovered by this time and so the cluster will have its full complement of replicas of the cluster's content. Adding extra replicas will create work for the health processor to sift through the replicas, cleaning up redundant and obsolete copies. This cleanup will create trapped space in the cluster that will take several HP cycles to reclaim.
Note
snmpset -v2c -c ourpwdofchoicehere -m ./CARINGO-MIB.txt:./CARINGO-CASTOR-MIB.txt 192.168.99.100 castorRemountStaleVolumeAction s "/dev/sda"