Identifying the Slot Number of a Defective Drive in Swarm

Although the hard drive ID feature built into Swarm can usually be used for drive identification, there are some instances where the hardware in place is not supported by it. The scope of this article is to assist with drive identification in a cluster when normal means to do so cannot be used.

Identify the Disk

1. Log into Swarm UIS, then select the storage node with the defective drive and determine the SCSI device name (e.g. /dev/sdac) of the failed or failing disk:

Open

2. Next, log into a system where the Swarm Support Tools bundle is installed (typically for the ‘root’ user under /root/dist) for cluster management (usually a SCS or CSN system). From there, use the ‘swarmctl’ utility to export ‘dmesg -T’ output of the storage node with the defective drive (e.g. 100.126.4.94).

Output:

3. From the dmesg output file generated, look for the the Bus ID of the failed or failing disk:

Output:

From this output, we can see that the Bus ID of the failed disk is 14:0:28:0.

4. Now, identify the Slot Number of the failed disk by Bus ID:

Output:

With this , we see that the location of the drive in question is enclosure “0x500605b0000273bf" at slot 28.

5. The slot numbers, by convention, should start at value '0'. This can be confirmed as follows:

Output:

Confirmed in this example, the slot numbering scheme starts at '0'.

Replace defective disk

1. Cross reference the slot number determined above with the drive enclosure specification of the storage node. This will allow you to identify the drive bay number for the failed drive in the enclosure / chassis.
   For the example below, we use the disk slot designations on a Huawei 5288 V5 server enclosure:
   Slot of front disks

   Open

   

   Slot of rear disks

2. With the physical location & drive determined, you can now replace it. First, suspend volume recovery in the storage cluster from either Swarm UIS or using the ‘swarmctl’ CLI tool (we use swarmctl below):

    

3. Next, remove the failed disk from the chassis according the slot number identified from previous section. You can verify the serial number on the drive vs. that displayed in Swarm UIS confirm the correct drive has been removed. If the the serial numbers don’t match, simply insert the disk back into the enclosure.

4. With the failed drive removed, insert your replacement drive into the empty slot.

5. You can now re-enable volume recovery in the cluster by turning off volume recovery suspension:

6. Verify the new drive appears in Swarm UIS. After a few minutes have passed, it should show it has a non-zero stream count (which means it’s actively taking data).