Looking thru the running sims jobs .. there were two stuck on node 4, which is down (more busted hardware I think). I rescheduled them to run. The job on node 13 has a humongous single query sequence: olson@bio-ppc-13:~$ ls -l /vol/metagenome-48-hour/Jobs.prod/214/proc/ sims.seed/sims.in/317.fa total 3520 -rw-rw-rw- 1 olson metag48 3598632 Nov 29 10:06 in.1 this seems to be a bug ... the blast has been running since nov 29 and has written zero bytes of output: olson@bio-ppc-13:~$ ps uww14844 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND olson 14844 99.8 28.8 1736452 1164836 ? R Nov29 20243:40 /vol/ metagenome-48-hour/FIGdisk.backend.prod/env/cee/bin/blastall -m 8 -e 0.1 -p blastx -i /vol/metagenome-48-hour/Jobs.prod/214/proc/sims.seed/ sims.in/317.fa/in.1 -d /scratch/metagenome-48-hour/Data/ SEED_2006_07_01 -o /vol/metagenome-48-hour/Jobs.prod/214/proc/ sims.seed/sims.raw/317.fa/out.1 olson@bio-ppc-13:~$ ls -l /vol/metagenome-48-hour/Jobs.prod/214/proc/ sims.seed/sims.raw/317.fa/out.1 -rw-r--r-- 1 olson metag48 0 Nov 29 10:06 /vol/metagenome-48-hour/ Jobs.prod/214/proc/sims.seed/sims.raw/317.fa/out.1 I have killed that process. It appears that the blast for job 215 is similarly operating on a huge single input file: olson@bio-ppc-26:~$ ps wwuw14054 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND olson 14054 99.7 9.9 1483480 401176 ? R Dec11 2747:32 /vol/ metagenome-48-hour/FIGdisk.backend.prod/env/cee/bin/blastall -m 8 -e 0.1 -p blastx -i /vol/metagenome-48-hour/Jobs.prod/215/proc/sims.seed/ sims.in/334.fa/in.1 -d /scratch/metagenome-48-hour/Data/ SEED_2006_07_01 -o /vol/metagenome-48-hour/Jobs.prod/215/proc/ sims.seed/sims.raw/334.fa/out.1 olson@bio-ppc-26:~$ ls -l /vol/metagenome-48-hour/Jobs.prod/215/proc/ sims.seed/sims.in/334.fa/in.1 -rw-rw-rw- 1 olson metag48 2847768 Dec 6 11:42 /vol/metagenome-48- hour/Jobs.prod/215/proc/sims.seed/sims.in/334.fa/in.1 olson@bio-ppc-26:~$ ls -l /vol/metagenome-48-hour/Jobs.prod/215/proc/ sims.seed/sims.raw/334.fa/out.1 -rw-r--r-- 1 olson metag48 0 Dec 11 14:04 /vol/metagenome-48-hour/ Jobs.prod/215/proc/sims.seed/sims.raw/334.fa/out.1 and should probably be killed and the job fixed to split the input more appropriately.