When troubleshooting various problems on the Content Gateway one might need to capture a Java thread dump. This KB article explains how to do that using jstack.
Prerequisites
Note
CentOS/RHEL 7 is EOL and yum commands on CentOS 7 now fail with "Could not retrieve mirrorlist http://mirrorlist.centos.org/". There is a workaround, but please plan a migration to RockyLinux/RHEL 8.
jstack needs to be installed, and may not be installed by default. To check whether jstack is installed or not please run the following command:
if jstack -h 2>/dev/null; then echo "jstack is Installed"; else echo "jstack is Not Installed"; fi
If jstack is installed, progress to the "Instructions" section below. If not please install the OpenJDK development package according to the current version of Cloud Gateway.
Check your existing CloudGateway server level:
rpm -qa | grep caringo-gateway-[0-9]
If you are currently running Gateway version 7.10.7 or earlier, then please install using:
yum install java-1.8.0-openjdk-devel
if you are currently running Gateway version 8.0 or later, then please install using:
yum install java-11-openjdk-devel
Instructions
Collect out the PID of the Content Gateway process by running this command
PID=$(ps aux | grep [c]loudgateway.pid | awk '{ print $2 }')Capture the thread dump using jstack and the open file descriptors using lsof by running these commands using the PID determined at step 1
jstack $PID > thread_dump.txt lsof -p $PID > lsof.txt
If the above command does not capture anything, try running this to force a thread dump from a non-responsive process:
jstack -F $PID > thread_dump.txt
If you are using JDK 11+, forcing a thread dump requires this command
jhsdb jstack --locks --pid $PID > thread_dump.txt
Repeat the jstack/lsof commands 3 times total a few seconds apart like this:
for i in $(seq 3); do jstack $PID > thread_dump-$i.txt; lsof -p $PID > lsof-$i.txt; sleep 60;done
or ( for JDK 8 )
for i in $(seq 3); do jstack -F $PID > thread_dump-$i.txt; lsof -p $PID > lsof-$i.txt; sleep 60;done
or (for JDK11+)
for i in $(seq 3); do jstack --locks --pid $PID > thread_dump-$i.txt; lsof -p $PID > lsof-$i.txt; sleep 60;done