Pavan Yes, that's where I compiled and installed the nightly snapshot to. (I did a make clean and recompile of the app, though I see that only hydra needed changing ) - I assumed that the mpich snapshot did include the updated hydra - if not, I should redo it with the fixed hydra. I'll wipe everything and rebuild one more time just in case I messed up. Does it matter that I'm installing to a non standard location? Does it need to be on any system paths or have special privileges? JB -----Original Message----- From: Pavan Balaji [mailto:[email protected]] Sent: 07 June 2013 22:08 To: Biddiscombe, John A. Cc: [email protected] Subject: Re: [mpich-discuss] Problems running MPICH jobs under SLURM Are you using the correct mpiexec? Your submission script is using the mpiexec from this directory: /home/biddisco/apps/mpich-3.0.4/bin/mpiexec -- Pavan On 06/07/2013 03:40 AM, Biddiscombe, John A. wrote:
I downloaded the nightly tarball and recompiled/installed mpich (used mpich-master-v3.0.4-259-gf322ce79)
I still get this (output below) with a simple hello world program.
Now you must understand that I have no idea what I'm doing (really). I wanted to test some debugging features under slurm so installed slurm myself on a workstation with just 2 cores and have the bare minimum setup. I'm doing the following
sudo munged &
sudo slurmd &
sudo slurmctld -D
and then I can run jobs on the local machine and it seems to be ok, except that mpi jobs always give the double free error as below when run under slurm, but are just fine when run from the command line.
My suspicion is that slurm is not actually using the hydra pm that I just compiled. I installed slurm from rpms. Should I recompile slurm myself and somehow tell it which mpi to use?
My job script looks as follows
######################
#!/bin/bash
#
# Create the job script from the supplied parameters
#
#SBATCH --job-name=pvserver
#SBATCH --time=00:04:00
#SBATCH --nodes=1
#SBATCH --partition=normal
#SBATCH --output=/home/biddisco/slurm.out
#SBATCH --error=/home/biddisco/slurm.err
#SBATCH --mem=2048MB
#export
# echo "Path is $PATH"
# echo "LD_LIBRARY_PATH is " $LD_LIBRARY_PATH
# cd /home/biddisco/build/pv-38/bin/
#export PMI_DEBUG=9
#ulimit -s unlimited
#ulimit -c 0
/home/biddisco/apps/mpich-3.0.4/bin/mpiexec -rmk slurm -n 2 /home/biddisco/build/hello/hello
######################
It gives the same result with or without the -rmk slurm and the #ulimit settings.
Apologies for wasting your time, I'm certain I'm doing something wrong - I just don't know what.
JB
biddisco@breno2 ~ $ more ~/slurm.err
*** glibc detected *** /home/biddisco/build/hello/hello: double free or corruption (fasttop): 0x0000000001896340 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7f9a1695cb96]
/home/biddisco/build/hello/hello(MPIDI_Populate_vc_node_ids+0x3f9)[0x4 27c89]
/home/biddisco/build/hello/hello(MPID_Init+0x136)[0x4253f6]
/home/biddisco/build/hello/hello(MPIR_Init_thread+0x22f)[0x414cbf]
/home/biddisco/build/hello/hello(MPI_Init+0xae)[0x4146ee]
/home/biddisco/build/hello/hello(main+0x22)[0x413f2e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f9a168ff76d ]
/home/biddisco/build/hello/hello[0x413e31]
======= Memory map: ========
00400000-0051a000 r-xp 00000000 08:01 8661191 /home/biddisco/build/hello/hello
0071a000-00727000 r--p 0011a000 08:01 8661191 /home/biddisco/build/hello/hello
00727000-00729000 rw-p 00127000 08:01 8661191 /home/biddisco/build/hello/hello
00729000-00751000 rw-p 00000000 00:00 0
01895000-018b6000 rw-p 00000000 00:00 0 [heap]
7f9a166c8000-7f9a166dd000 r-xp 00000000 08:01 9047556 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f9a166dd000-7f9a168dc000 ---p 00015000 08:01 9047556 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f9a168dc000-7f9a168dd000 r--p 00014000 08:01 9047556 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f9a168dd000-7f9a168de000 rw-p 00015000 08:01 9047556 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f9a168de000-7f9a16a93000 r-xp 00000000 08:01 9050358 /lib/x86_64-linux-gnu/libc-2.15.so
7f9a16a93000-7f9a16c92000 ---p 001b5000 08:01 9050358 /lib/x86_64-linux-gnu/libc-2.15.so
7f9a16c92000-7f9a16c96000 r--p 001b4000 08:01 9050358 /lib/x86_64-linux-gnu/libc-2.15.so
7f9a16c96000-7f9a16c98000 rw-p 001b8000 08:01 9050358 /lib/x86_64-linux-gnu/libc-2.15.so
7f9a16c98000-7f9a16c9d000 rw-p 00000000 00:00 0
7f9a16c9d000-7f9a16cb5000 r-xp 00000000 08:01 9050338 /lib/x86_64-linux-gnu/libpthread-2.15.so
7f9a16cb5000-7f9a16eb4000 ---p 00018000 08:01 9050338 /lib/x86_64-linux-gnu/libpthread-2.15.so
7f9a16eb4000-7f9a16eb5000 r--p 00017000 08:01 9050338 /lib/x86_64-linux-gnu/libpthread-2.15.so
7f9a16eb5000-7f9a16eb6000 rw-p 00018000 08:01 9050338 /lib/x86_64-linux-gnu/libpthread-2.15.so
7f9a16eb6000-7f9a16eba000 rw-p 00000000 00:00 0
7f9a16eba000-7f9a16edc000 r-xp 00000000 08:01 9050344 /lib/x86_64-linux-gnu/ld-2.15.so
7f9a170c1000-7f9a170c4000 rw-p 00000000 00:00 0
7f9a170d9000-7f9a170dc000 rw-p 00000000 00:00 0
7f9a170dc000-7f9a170dd000 r--p 00022000 08:01 9050344 /lib/x86_64-linux-gnu/ld-2.15.so
7f9a170dd000-7f9a170df000 rw-p 00023000 08:01 9050344 /lib/x86_64-linux-gnu/ld-2.15.so
7fff52f27000-7fff52f48000 rw-p 00000000 00:00 0 [stack]
7fff52fff000-7fff53000000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
-- Pavan Balaji http://www.mcs.anl.gov/~balaji