Page Comparison

When troubleshooting various problems on the Content Gateway one might need to capture a Java thread dump. This KB article explains how to do that using jstack.

Prerequisites

jstack needs to be installed, and may not be installed by default. Install jstack as part of the openjdk package with the following example, replacing the version as necessary. To check whether jstack is installed or not please run the following command:

if jstack -h 2>/dev/null; then echo "jstack is Installed"; else echo "jstack is Not Installed"; fi

If jstack is installed, progress to the "Instructions" section below. If not please install the OpenJDK development package according to the current version of Cloud Gateway.

Check your existing CloudGateway server level:

rpm -qa | grep caringo-gateway-[0-9]

If you are currently running Gateway version 7.10.7 or earlier, then please install using:

yum install java-1.8.0-openjdk-devel

Instructions

...

if you are currently running Gateway version 8.0 or later, then please install using:

yum install java-11-openjdk-devel

Instructions

1. Collect out the PID of the Content Gateway process by running this command

   Code Block
   PID=$(ps aux | grep [c]loudgateway.pid | awk '{ print $2 }')

2. Capture the thread dump using jstack and the open file descriptors using lsof by running this command these commands using the PID determined at step 1

   Code Block
   jstack $PID > thread_dump.txt lsof -p $PID > lsof.txt

3. If the above command does not capture anything, try running this to force a thread dump from a non-responsive process:

   Code Block
   jstack -F $PID > thread_dump.txt

4. If you are using JDK 11+, forcing a thread dump requires this command

   Code Block
   jhsdb jstack --locks --pid $PID > thread_dump.txt

5. Repeat the jstack/lsof commands 3 times total a few seconds apart like this:

   Code Block
   for i in $(seq 3); do jstack $PID > thread_dump-$i.txt; lsof -p $PID > lsof-$i.txt; sleep 360;done

   or ( for JDK 8 )

   Code Block
   for i in $(seq 3); do jstack -F $PID > thread_dump-$i.txt; lsof -p $PID > lsof-$i.txt; sleep 60;done

or (for JDK11+)

for i in $(seq 3); do jstack --locks --pid $PID > thread_dump-$i.txt; lsof -p $PID > lsof-$i.txt; sleep 60;done

Related articles

...