Managing Chassis and Drives - Legacy Admin Console

Viewing the Node Status Page

To view the status of a node, click its IP address on the left side of the Swarm Admin Console. If the cluster is configured to use subclusters, expand a subcluster node name to display IP addresses and then click an IP address to display the node information.

To find a particular node, see Finding Nodes in the Cluster.

The top row of the Node Status page provides summary information about the node and its associated volumes, such as up-time and storage usage statistics:

• Streams: Counts the total number of managed data components (such as replicas and segments), not logical objects (such as video files).
• Trapped: Calculates the space that is pending reclamation by the Swarm defragmentation process. This process is controlled by several Swarm parameters (see the Settings Reference).

Shutting Down or Restarting a Node

To shut down or restart a node,

• Click Shutdown Node or Restart Node in the Swarm Admin Console.

A node that is shutdown or rebooted by an Administrator will appear with a Maintenance state on other nodes in the cluster.

See Finding Nodes in the Cluster.

Identifying a Drive

You can identify one or all volumes on a node using the links on the right side of the Swarm Admin Console under Restart Node.

The Identify function allows you to select a particular volume and enable the corresponding LED drive light, which can be helpful in identifying a failed or failing drive. Simply select the targeted volume and the amount of time that the light will be enabled.

On the Node Status page, an Identify light displays next to the targeted volume for easy identification.

See Drive Identification Plugin for how to enable the drive light.

If a hardware-specific API is not used, Swarm will revert to a default process to flash the light.

Retiring a Drive

You can retire one or all volumes on a node using the links on the right side of the Swarm Admin Console under Restart Node.

On occasion, you may need to replace Swarm volumes for regular maintenance or to upgrade the cluster nodes with higher capacity drives. If multiple volumes need to be replaced across multiple Swarm nodes, the volumes should be retired one at a time. When you initiate a retire, you choose if you would like a minimally disruptive retire that is limited to just the volume(s) being retired, or an accelerated retire that uses all nodes in the cluster to replicate objects on the retiring volume(s) as quickly as possible. Note that the cluster-wide retire may impact performance as it does put additional load on the cluster.

Clicking Retire Node retires all volumes on the node, at the same time. Clicking Retire next to a volume retires only the selected volume. A volume is also retired automatically if a configurable number of errors occur.

See Retiring Volumes.

Before you retire a node or volume, make sure that the cluster:

• Has enough capacity for the objects on the retiring node to replicate elsewhere.
• Has enough unique nodes to replicate the objects with only one replica on any given node.

A retiring node or volume accepts no new or updated objects. Retiring a node or volume means all of its objects, including replicas, are copied to other nodes in the cluster.

On the Swarm Admin Console's Node Status page, the Node Operations section includes a Retire Rate that tracks the number of objects per hour that were removed from a retiring volume. The SNMP MIB includes this same value in the retireRatePerHour MIB entry. If no volumes on the node are retiring, the value is 0.

After all objects are copied, the node or volume's state changes to Retired and Swarm no longer uses the node or volume. At this point, you should remove and repair the volume or discard it.

Errors and Announcements

The last 10 errors and announcements appear on the Node Status page. If there are no errors or announcements, the page is blank. The error count in the node summary grid corresponds to the list of errors in the error section.

Messages display in the node status area if you remove or insert a drive into a running node. This feature, referred to as hot plugging (adding a new drive) or hot swapping (replacing a failed drive), lets you remove failed drives for analysis or to add storage capacity to a node at any time.

For example, when you add a volume, the following message appears:

mounted /dev/sdb, volumeID is 561479FB832DCC526B1D7EDCD06B83E1

When you remove a volume, the following message appears:

removed /dev/sdb, volumeID was 561479FB832DCC526B1D7EDCD06B83E1

Node Status Reporting

You can troubleshoot node errors and announcements by viewing the reporting sections in the Node Status page. You can access these sections at the bottom of the Node Status page. The information in each section can be helpful when working with Swarm Support to resolve an issue.

Node Info section

The Node Info status section contains general information about the hardware installed on the node, as well as time server information and current uptime. Use this status information to determine if your node requires additional hardware resources.

For example, if the Index Utilization and Buffer Utilization values rise to 80% or more, the Swarm Admin Console generates an alert that indicates the node may require additional RAM to maintain cluster performance. Additionally, if the Time value does not match the same value in the remaining cluster nodes, the node may not be communicating properly with an NTP server.

Additional Node Info reports

Scroll to the bottom of the Node Info section to access these links to additional reports:

• SNMP Repository (the SNMP repository dump)

• Object Counts (the Python classes in use)
• Uncollectable Garbage
• HTML Templates
• Loggers... (the settings window for changing the logging levels)
• Dmesg dump (the last 1000 messages logged by the Linux kernel reading buffer, for diagnosing a Swarm issue when a system panic or error occurs)
• Hwinfo dump (the Linux hardware detection tool output)

Node Configuration section

The Node Configuration status section contains the cluster and network configuration settings assigned to the node. Use this status information to quickly verify your system configuration without using SNMP commands.

Node Operations section

The Node Operations status section describes the state of the node. If you encounter a problem in your storage cluster, a Swarm Support representative can use the information in this page to help you determine if the node is communicating effectively with other nodes and resources in the cluster.

For example, some cluster features (such as the Capacity column value in the Swarm Admin Console) will not update until the HP cycles are completed separately on each node. The HP Cycle time parameter increases exponentially as the number of objects increase on the node. Additionally, if the SCSP Last read bid and SCSP Last write bid parameters are high, the node may not be servicing new requests.

Hardware Status section

The Hardware Status section contains status and operational reporting (if available) for various hardware components installed on the node. Use this status information to retrieve node system data, such as the serial number and BIOS version.

Hardware status reporting is dependent on hardware that supports and populates IPMI sensors, SMART status, and, in some cases, manufacturer-specific components such as SAS. Depending on your hardware, not all status fields are populated. The hardware status values are independently scanned and populated for each node, allowing variations in supported utilities on a node-by-node basis.

Additional Hardware Status reports

Scroll to the bottom of the Hardware Status section to access these links to additional reports:

• Test Network - Pings all nodes in the cluster to ensure that all nodes can communicate with each other using TCP/IP and UDP (see details below).

• Test Volumes - Pings the volumes on your local hard drives and provides a response time (in milliseconds).
• Dmesg Dump - Displays the last 1000 messages logged by the Linux kernel reading buffer. These messages can help you troubleshoot and diagnose a Swarm issue when a system panic or error occurs.
• Hwinfo dump (the Linux hardware detection tool output)
• Send Health Report (script that sends the hardware health report to the configured destination)

Test Network

Test Network performs two sets of tests:

• First, it sends 100 UDP multicasts to the cluster and computes the results:
• Which nodes responded
• How many responses returned
• How long the responses took, on average
• Next, it fetches the status page (port 80) via TCP for all of those responding nodes (only once for each node). It tracks the total time for each of those round trips.

The data in the Network Test Results window lets you compare the responding nodes with the list of nodes that you expected to see in the cluster. You can also evaluate UDP packet loss and TCP connectivity within the cluster.