Viewing the Node Status Page

Click the IP address on the left side of the Swarm Admin Console to view the status of a node. Expand a subcluster node name to display IP addresses and then click an IP address to display the node information if the cluster is configured to use subclusters.

See Finding Nodes in the Cluster to find a particular node.

The top row of the Node Status page provides summary information about the node and the associated volumes, such as up-time and storage usage statistics:

Open

• Streams: Counts the total number of managed data components (such as replicas and segments), not logical objects (such as video files).

• Trapped: Calculates the space pending reclamation by the Swarm defragmentation process. This process is controlled by several Swarm parameters (See Settings Reference).

Shutting Down or Restarting a Node

Click Shutdown Node or Restart Node in the Swarm Admin Console to shut down or restart a node.

A node shutdown or rebooted by an Administrator appears with a Maintenance state on other nodes in the cluster.

See Finding Nodes in the Cluster.

Identifying a Drive

Identify one or all volumes on a node using the links on the right side of the Swarm Admin Console under Restart Node.

The Identify function allows selection of a particular volume and enable the corresponding LED drive light, which can be helpful in identifying a failed or failing drive. Select the targeted volume and the amount of time the light is enabled.

On the Node Status page, an Identify light displays next to the targeted volume for easy identification.

See Drive Identification Plugin for how to enable the drive light.

Swarm reverts to a default process to flash the light if a hardware-specific API is not used.

Open

Retiring a Drive

Retire one or all volumes on a node using the links on the right side of the Swarm Admin Console under Restart Node.

On occasion, replacing Swarm volumes is required for regular maintenance or to upgrade the cluster nodes with higher capacity drives. Best practice is to retire volumes one at a time if multiple volumes need to be replaced across multiple Swarm nodes. Either choose a minimally disruptive retire limited to the volume(s) being retired, or an accelerated retire using all nodes in the cluster to replicate objects on the retiring volume(s) as quickly as possible when initiating a retire. Note the cluster-wide retire may impact performance as it does put additional load on the cluster.

Clicking Retire Node retires all volumes on the node, at the same time. Clicking Retire next to a volume retires the selected volume. A volume is also retired automatically if a configurable number of errors occur.

See Retiring Volumesarchived.

Verify the cluster meets the following conditions before retiring a node or volume:

• Has enough capacity for the objects on the retiring node to replicate elsewhere.

• Has enough unique nodes to replicate the objects with one replica on any given node.

A retiring node or volume accepts no new or updated objects. Retiring a node or volume means all objects, including replicas, are copied to other nodes in the cluster.

On the Swarm Admin Console's Node Status page, the Node Operations section includes a Retire Rate tracking the number of objects per hour removed from a retiring volume. The SNMP MIB includes this same value in the retireRatePerHour MIB entry. The value is 0 if no volumes on the node are retiring.

The node or volume's state changes to Retired and Swarm no longer uses the node or volume after all objects are copied. At this point, remove and repair the volume or discard it.

Errors and Announcements

The last 10 errors and announcements appear on the Node Status page. The page is blank if there are no errors or announcements. The error count in the node summary grid corresponds to the list of errors in the error section.

Messages display in the node status area if removing or inserting a drive into a running node. This feature, referred to as hot plugging (adding a new drive) or hot swapping (replacing a failed drive), allows removal of failed drives for analysis or to add storage capacity to a node at any time.

The following message appears when adding a volume:

The following message appears when removing a volume:

Node Status Reporting

Troubleshoot node errors and announcements by viewing the reporting sections in the Node Status page. Access these sections at the bottom of the Node Status page. The information in each section can be helpful when working with Swarm Support to resolve an issue.

Node Info section

The Node Info status section contains general information about the hardware installed on the node, as well as time server information and current uptime. Use this status information to determine if a node requires additional hardware resources.

The Swarm Admin Console generates an alert indicating the node may require additional RAM to maintain cluster performance if the Index Utilization and Buffer Utilization values rise to 80% or more. The node may not be communicating properly with an NTP server if the Time value does not match the same value in the remaining cluster nodes.

Additional Node Info Reports

Scroll to the bottom of the Node Info section to access these links to additional reports:

• SNMP Repository (the SNMP repository dump)

• Object Counts (the Python classes in use)

• Uncollectable Garbage

• HTML Templates

• Loggers... (the settings window for changing the logging levels)

• Dmesg dump (the last 1000 messages logged by the Linux kernel reading buffer, for diagnosing a Swarm issue when a system panic or error occurs)

• Hwinfo dump (the Linux hardware detection tool output)

Node Configuration Section

The Node Configuration status section contains the cluster and network configuration settings assigned to the node. Use this status information to quickly verify the system configuration without using SNMP commands.

Node Operations Section

The Node Operations status section describes the state of the node. A Swarm Support representative can use the information in this page to assist in determining if the node is communicating effectively with other nodes and resources in the cluster if a problem is encountered a problem in a storage cluster.

Some cluster features (such as the Capacity column value in the Swarm Admin Console) do not update until the HP cycles are completed separately on each node. The HP Cycle time parameter increases exponentially as the number of objects increase on the node. The node may not be servicing new requests if the SCSP Last read bid and SCSP Last write bid parameters are high.

Hardware Status Section

The Hardware Status section contains status and operational reporting (if available) for various hardware components installed on the node. Use this status information to retrieve node system data, such as the serial number and BIOS version.

Hardware status reporting is dependent on hardware supporting and populating IPMI sensors, SMART status, and, in some cases, manufacturer-specific components such as SAS. Not all status fields are populated depending on the hardware. The hardware status values are independently scanned and populated for each node, allowing variations in supported utilities on a node-by-node basis.

Additional Hardware Status Reports

Scroll to the bottom of the Hardware Status section to access these links to additional reports:

• Test Network - Pings all nodes in the cluster to verify all nodes can communicate with each other using TCP/IP and UDP (see details below).

• Test Volumes - Pings the volumes on the local hard drives and provides a response time (in milliseconds).

• Dmesg Dump - Displays the last 1000 messages logged by the Linux kernel reading buffer. These messages can help troubleshoot and diagnose a Swarm issue when a system panic or error occurs.

• Hwinfo dump (the Linux hardware detection tool output)

• Send Health Report (script that sends the hardware health report to the configured destination)

Test Network

Test Network performs two sets of tests:

• First, it sends 100 UDP multicasts to the cluster and computes the results:

• Which nodes responded

• How many responses returned

• How long the responses took, on average

• Next, it fetches the status page (port 80) via TCP for all responding nodes (once for each node). It tracks the total time for each of those round trips.

The data in the Network Test Results window allows comparing the responding nodes with the list of expected nodes in the cluster. Evaluate UDP packet loss and TCP connectivity within the cluster.