Simple Process Snipers

Process snipers are used to kill errant processes

I’ve used process snipers and watchdog’s to handle realistic process management in large clusters in the past but never felt any of them were very elegant in terms of code simplicity.

It dawned on me the other day that this would work out perfectly:

for proc in `find /proc -iregex "/proc/[0-9]+" -maxdepth 1 -cmin +15`; do

    if [ "$(cat $proc/cmdline |grep -E '^/usr/share/munin/munin-update')" != "" ]; then
        pid=$(basename $proc)
        cmdline=$(cat $proc/cmdline)
        echo "Killing process: $pid with: $cmdline"
        kill -9 $pid
    fi 

done

Fix fixes a bug in Munin where the background update task won’t ever finish and end up building breaking our stats.




%d bloggers like this: