How to add iperf3 to a Swarm probe build

The Swarm probe build is the same kernel and fsimage used to run Swarm storage nodes on but without the storage part.

It is a very useful tool when troubleshooting driver issues, or other hardware issues as it includes SSH access.

The support team uses this tool to troubleshoot and a client is provided a download link to use on their CSN or other PXE server.

Unlike a usual swarm upgrade image it exists in a zip format and the zip contains both the fsimage and kernel for the version of Swarm it is built for.

Swarm images load into memory so the base OS is not editable once started. Navigating to ‘/etc/’ and making lasting changes on a config file are not possible. It is not possible to add packages to the probe build using tools like yum or apt or even dpkg / rpm.

This article shows how to drop a precompiled binary in with the associated library.

Setup

Install probe-build

This example probe build is for version 12 and the CSN has the IP 10.0.1.26.

scp castor-12.0.0-x86_64-probe.zip root@10.0.1.26:/root/

Transfer over using WinSCP or FTP or another tool altogether but this is the easiest method when running on a linux/unix client.

There is no need to unzip the file when it is transferred over; move it to /root/

[root@swarmservicenode ~]# ll
total 234776
-rw-r--r--  1 root root 118058980 Feb 22 10:06 caringo-castor-12.0.0-1-x86_64(1).rpm
-rw-r--r--. 1 root root        71 Aug 20  2018 caringo_csn_backup.disabled
-rw-r--r--  1 root root 122143502 Feb 22 10:10 castor-12.0.0-x86_64-probe.zip
drwxr-xr-x  4 root root      4096 Dec 10 05:22 dist
-rw-r--r--  1 root root    184320 Feb 22 10:58 iperf.tar
-rw-r--r--  1 root root       966 May 31  2019 metrics.cfg
drwxr-xr-x. 2 root root      4096 Dec 10 05:19 Platform-8.3.2

On this example CSN the support tools are installed.

Use a handy script there to add the probe build.

cd dist
[root@swarmservicenode ~]# cd dist/
You have new mail in /var/spool/mail/root
[root@swarmservicenode dist]# ls
add-bashrcforcustomers.sh               csn-check-backups.sh           indexerConfig171.sh                     platform-send-health-reports.sh
bashrcforcustomers                      csn-create-nodeconfigs.sh      indexerConfig233.sh                     platform-update-mibs.sh
CARINGO-CASTOR-MIB.txt                  csn-install-from-zip.sh        indexer-enumerator.sh                   proxy-set-fw-nat.sh
caringo-content-gateway-audit           csn-modify-saveset.sh          indexer-grab.sh                         README.TXT
caringo-content-gateway-server          csn-ntpd-watch.sh              legacy-tools                            REVISION.txt
CARINGO-MIB.txt                         csn-patch-sosreport702.sh      logging.yaml.syslog                     sendretrieve.sh
CentOS-Base-68.repo                     csn-read-pss.sh                logging.yml                             settings_checker.py
CentOS-Base-6.repo                      csn-rename-cluster.sh          logging.yml-2017_0823                   SimpleCAS.zip
CHANGELOG                               csn-send-health-reports.sh     logrotate-elasticsearch                 snmp-castor-tool.sh
check-siblings.sh                       csn_settings_checker           logrotate-elasticsearch-2017_0823       swarmctl
checkuuidsAllNodes.sh                   csn-update-mibs.sh             make-immutable-streams-mutable.sh       swarmrestart
cipperf.py                              csn-yum-68-upgrade.sh          mcast-tester.py                         techsupport-bundle-grab.sh
cns-dig                                 DD_DISKIO.sh                   ntpd-instantaneous-collect.sh           Tech-Support-Scripts-Bundle.pdf
collect_health_reports.sh               demoParallelWriteToSwarm.sh    parseJsonBuckets.py                     test-volumes-network.sh
copy-streams-to-new-cluster.sh          generateMutableTestStreams.sh  Pcurl.sh                                tmpwatch
createNODEScsv.sh                       generateTestData.sh            performance-profiler.sh                 updateBundle.sh
csn-add-sncfg-to-ip-assignments.sh      getretrieve.sh                 platform-delete-csn-backups.sh          updateTestData.sh
csn-add-to-ip-assignments-from-file.sh  gw-change-log-level.sh         platform-enumerate-vols.sh              uploaddir.sh
csn-allow-68.sh                         hwinfo-dmesg-grab.sh           platform-get-default-search-feed-ip.py  uploader.html
csn-assign-ips.sh                       igmp-querier.pl                platform-read-pss.sh

The probe is unzipped and the kernel and fsimage are placed in the correct location creating a named folder indicating what it is.

The net result is:

Select the probe from your netboot config and then select update.

Any node rebooted now loads that image.

Get Iperf3

A version of iperf3 is needed from here.

https://iperf.fr/iperf-download.php

The version for this example is:

iPerf 3.1.3 - DEB package (8 jun 2016 - 8.6 KiB) + libiperf0 3.1.3 - DEB package (53.9 KiB)

Newer versions should work as well. It will be the
Ubuntu 64 bits / Debian 64 bits / Mint 64 bits (AMD64) by Raoul Gunnar Borenius and Roberto LuMiBreras. (sha256) packages. As this closely matches the Debian release Swarm uses.

The deb packages are meant to be used with a package manager but here we just want the pre-compiled binaries. To get them we can do the following

Unpack the deb files

First transfer the files to an ubuntu/debian based machine.

make sure dpkg-deb is installed

Then run

For each file that you have

You’ll see a directory structure like this

What we need is the iperf3 binary and the lib files.

So to make it easy we can transfer the files over in a tar bundle.

Then transfer it to our CSN in my case thats on 10.0.1.26

When we have the packages over on the CSN we can SSH there and do another transfer

in this case my username is root : caringo is the password

here we’re transferring the iperf tarball to a storage node ips /dev/shm ‘shared memory’

https://www.cyberciti.biz/tips/what-is-devshm-and-its-practical-usage.html

Running Iperf

Now that we have all of the packages in place we can ssh to a storage node and run the tool

cd to /dev/shm and untar your iperf

Then we can go to usr/bin/iperf3 and try to run it

it tells us that its missing the libiperf library. But thats fine cause we brought our own

This command will only be valid for this tty session but it tells the kernel where to find the library it needs for iperf.

Now if we go back:

We have our iperf server running on port 5201

To test that it works we can run iperf in client mode from another host.

And this is what it will look like on the storage node side.

You can then use iperf3 to validate that the network is giving the expected throughput in both directions.

Knowledge Base