Agnus Dei (jackal) wrote,
Agnus Dei
jackal

A simple script to grab the tomcat thread dump if the load on the box is too high

Below is a script I wrote that run every 2 minutes out of cron and checks the load on the tomcat server and checks the number of threads in tomcat, and if they are too high it calls another command to do a thread dump so we can see what it was doing when it got out of control.

The external script it's calling ("get_thread_dump.pl") is a simple perl script that just gets the tomcat pid and does a "kill -s SIGQUIT $tomcat_pid" which causes tomcat to dump it's threads to catalina.out. It doesn't actually kill tomcat... all it does is causes it to print it's threads to catalina.out.

#!/bin/bash
#
# A simple script to grab the tomcat thread dump if the load on the box is too high
#
# Created: Wed Aug  7 15:20:12 UTC 2013
# Version: 0.2
####################################################################################

MAX_LOAD=3        # threshold if load (truncated) is greater than this integer 
MAX_THREADS=150   # threshold if tomcat threads go beyond this value
THREAD_DUMP="/usr/local/bin/get_thread_dump.pl"

#check load
LOAD=`uptime | awk -F'average:' '{ print $2 }' | cut -f1 -d,`
LOAD_TRUNCATED="$(echo $LOAD | sed 's/\.[0-9]*//')"

# get thread count
# TOMCAT_THREADS=`ps -utomcat -onlwp |grep -v NLWP`
TOMCAT_THREADS=`ps -utomcat -onlwp,cmd |grep catalina | awk '{print $1}'`


# Get thread dump if load is too high or if threads are too high
if [[ $LOAD_TRUNCATED -gt $MAX_LOAD ]] || [[ $TOMCAT_THREADS -gt $MAX_THREADS ]] ; then
	DATE=`/bin/date +%F" "%T`
	echo "$DATE:LOAD=$LOAD;TOMCAT_THREADS=$TOMCAT_THREADS"
	if [ -x $THREAD_DUMP ] ; then
		$THREAD_DUMP 
	fi
fi
Subscribe
  • Post a new comment

    Error

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

  • 0 comments