- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Problems running MPICH jobs under SLURM
by Pavan Balaji 07 Jun '13

07 Jun '13

John, On 06/07/2013 04:37 PM, Biddiscombe, John A. wrote: > I did a complete rebuild and sometimes error appears twice, apart from that, no change > > I'll play some more next week, but for now it isn't urgent. I still think I've done something wrong > > For reference, my mpich compile is (no slurm options present - I bet that's what's wrong) > ./configure --prefix=/home/biddisco/apps/mpich-3.0.4 --enable-static --with-pic Your naming is pretty confusing. You are using "mpich-3.0.4" for a build that is not really mpich-3.0.4. Try this: % wget http://www.mpich.org/static/tarballs/nightly/master/mpich/mpich-master-v3.0… % tar -xzvf mpich-master-v3.0.4-259-gf322ce79.tar.gz % cd mpich-master-v3.0.4-259-gf322ce79 % ./configure --prefix=`pwd`/install CC=gcc CXX=g++ F77=gfortran FC=gfortran && make && make install % salloc -N 2 -n 4 % ./install/bin/mpicc hello.c -o hello % ./install/bin/mpiexec -n 4 ./hello -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Problems running MPICH jobs under SLURM
by Biddiscombe, John A. 07 Jun '13

07 Jun '13

I did a complete rebuild and sometimes error appears twice, apart from that, no change I'll play some more next week, but for now it isn't urgent. I still think I've done something wrong For reference, my mpich compile is (no slurm options present - I bet that's what's wrong) ./configure --prefix=/home/biddisco/apps/mpich-3.0.4 --enable-static --with-pic JB breno2 biddisco # more slurm.err *** glibc detected *** /home/biddisco/build/hello/hello: double free or corruption (fasttop): 0x0000000001cf8340 *** *** glibc detected *** /home/biddisco/build/hello/hello: double free or corruption (fasttop): 0x00000000010b7340 *** ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7f11c18f0b96] /home/biddisco/build/hello/hello(MPIDI_Populate_vc_node_ids+0x3f9)[0x427c89] /home/biddisco/build/hello/hello(MPID_Init+0x136)[0x4253f6] /home/biddisco/build/hello/hello(MPIR_Init_thread+0x22f)[0x414cbf] /home/biddisco/build/hello/hello(MPI_Init+0xae)[0x4146ee] /home/biddisco/build/hello/hello(main+0x22)[0x413f2e] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f11c189376d] /home/biddisco/build/hello/hello[0x413e31] ======= Memory map: ======== 00400000-0051a000 r-xp 00000000 08:01 8661128 /home/biddisco/build/hello/hello 0071a000-00727000 r--p 0011a000 08:01 8661128 /home/biddisco/build/hello/hello 00727000-00729000 rw-p 00127000 08:01 8661128 /home/biddisco/build/hello/hello 00729000-00751000 rw-p 00000000 00:00 0 01cf7000-01d18000 rw-p 00000000 00:00 0 [heap] 7f11c165c000-7f11c1671000 r-xp 00000000 08:01 9047556 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f11c1671000-7f11c1870000 ---p 00015000 08:01 9047556 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f11c1870000-7f11c1871000 r--p 00014000 08:01 9047556 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f11c1871000-7f11c1872000 rw-p 00015000 08:01 9047556 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f11c1872000-7f11c1a27000 r-xp 00000000 08:01 9050358 /lib/x86_64-linux-gnu/libc-2.15.so 7f11c1a27000-7f11c1c26000 ---p 001b5000 08:01 9050358 /lib/x86_64-linux-gnu/libc-2.15.so 7f11c1c26000-7f11c1c2a000 r--p 001b4000 08:01 9050358 /lib/x86_64-linux-gnu/libc-2.15.so 7f11c1c2a000-7f11c1c2c000 rw-p 001b8000 08:01 9050358 /lib/x86_64-linux-gnu/libc-2.15.so 7f11c1c2c000-7f11c1c31000 rw-p 00000000 00:00 0 7f11c1c31000-7f11c1c49000 r-xp 00000000 08:01 9050338 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f11c1c49000-7f11c1e48000 ---p 00018000 08:01 9050338 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f11c1e48000-7f11c1e49000 r--p 00017000 08:01 9050338 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f11c1e49000-7f11c1e4a000 rw-p 00018000 08:01 9050338 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f11c1e4a000-7f11c1e4e000 rw-p 00000000 00:00 0 7f11c1e4e000-7f11c1e70000 r-xp 00000000 08:01 9050344 /lib/x86_64-linux-gnu/ld-2.15.so 7f11c2053000-7f11c2056000 rw-p 00000000 00:00 0 7f11c206d000-7f11c2070000 rw-p 00000000 00:00 0 7f11c2070000-7f11c2071000 r--p 00022000 08:01 9050344 /lib/x86_64-linux-gnu/ld-2.15.so 7f11c2071000-7f11c2073000 rw-p 00023000 08:01 9050344 /lib/x86_64-linux-gnu/ld-2.15.so 7fff0da4c000-7fff0da6d000 rw-p 00000000 00:00 0 [stack] 7fff0db6a000-7fff0db6b000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7f27e69f2b96] /home/biddisco/build/hello/hello(MPIDI_Populate_vc_node_ids+0x3f9)[0x427c89] /home/biddisco/build/hello/hello(MPID_Init+0x136)[0x4253f6] /home/biddisco/build/hello/hello(MPIR_Init_thread+0x22f)[0x414cbf] /home/biddisco/build/hello/hello(MPI_Init+0xae)[0x4146ee] /home/biddisco/build/hello/hello(main+0x22)[0x413f2e] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f27e699576d] /home/biddisco/build/hello/hello[0x413e31] ======= Memory map: ======== 00400000-0051a000 r-xp 00000000 08:01 8661128 /home/biddisco/build/hello/hello 0071a000-00727000 r--p 0011a000 08:01 8661128 /home/biddisco/build/hello/hello 00727000-00729000 rw-p 00127000 08:01 8661128 /home/biddisco/build/hello/hello 00729000-00751000 rw-p 00000000 00:00 0 010b6000-010d7000 rw-p 00000000 00:00 0 [heap] 7f27e675e000-7f27e6773000 r-xp 00000000 08:01 9047556 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f27e6773000-7f27e6972000 ---p 00015000 08:01 9047556 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f27e6972000-7f27e6973000 r--p 00014000 08:01 9047556 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f27e6973000-7f27e6974000 rw-p 00015000 08:01 9047556 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f27e6974000-7f27e6b29000 r-xp 00000000 08:01 9050358 /lib/x86_64-linux-gnu/libc-2.15.so 7f27e6b29000-7f27e6d28000 ---p 001b5000 08:01 9050358 /lib/x86_64-linux-gnu/libc-2.15.so 7f27e6d28000-7f27e6d2c000 r--p 001b4000 08:01 9050358 /lib/x86_64-linux-gnu/libc-2.15.so 7f27e6d2c000-7f27e6d2e000 rw-p 001b8000 08:01 9050358 /lib/x86_64-linux-gnu/libc-2.15.so 7f27e6d2e000-7f27e6d33000 rw-p 00000000 00:00 0 7f27e6d33000-7f27e6d4b000 r-xp 00000000 08:01 9050338 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f27e6d4b000-7f27e6f4a000 ---p 00018000 08:01 9050338 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f27e6f4a000-7f27e6f4b000 r--p 00017000 08:01 9050338 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f27e6f4b000-7f27e6f4c000 rw-p 00018000 08:01 9050338 /lib/x86_64-linux-gnu/libpthread-2.15.so 7f27e6f4c000-7f27e6f50000 rw-p 00000000 00:00 0 7f27e6f50000-7f27e6f72000 r-xp 00000000 08:01 9050344 /lib/x86_64-linux-gnu/ld-2.15.so 7f27e7155000-7f27e7158000 rw-p 00000000 00:00 0 7f27e716f000-7f27e7172000 rw-p 00000000 00:00 0 7f27e7172000-7f27e7173000 r--p 00022000 08:01 9050344 /lib/x86_64-linux-gnu/ld-2.15.so 7f27e7173000-7f27e7175000 rw-p 00023000 08:01 9050344 /lib/x86_64-linux-gnu/ld-2.15.so 7fff08d40000-7fff08d61000 rw-p 00000000 00:00 0 [stack] 7fff08dff000-7fff08e00000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

1 0

Re: [mpich-discuss] MPE is available in mpich-3.0.4 ? rlog doesn't work in configure process.
by Antonio J. Peña 07 Jun '13

07 Jun '13

Hi Fernando, Your log file seems to indicate that configure is not finding the directories you indicated in the following options: --with-mpilibs=/home/fernando_luz/usr/mpich/lib/ --with-mpiinc=/home/fernando_luz/usr/mpich/include/ Could you please check these paths are correct and contain the corresponding libraries and include files? Antonio On Friday, June 07, 2013 05:04:42 PM fernando_luz wrote: > In attachment. > > Fernando > > On 06/07/2013 04:56 PM, Jeff Hammond wrote: > > please attach config.log. > > > > jeff > > > > On Fri, Jun 7, 2013 at 1:07 PM, fernando_luz <fernando_luz(a)tpn.usp.br> wrote: > >> Hi Rajeev, > >> > >> Thanks for the answers. > >> > >> I get the source code in repository, but I didn't succeed in the compile > >> process. > >> I ran the autogen.sh and after this I tried to configure my installation > >> and I received the following error message. > >> > >> > >> fernando_luz@TPN000300:~/git/mpe$ ./configure > >> --prefix=/home/fernando_luz/usr/mpe > >> --with-mpicc=/home/fernando_luz/usr/mpich/bin/mpicc > >> --with-mpilibs=/home/fernando_luz/usr/mpich/lib/ > >> --with-mpiinc=/home/fernando_luz/usr/mpich/include/ > >> Configuring MPE Profiling System with > >> '--prefix=/home/fernando_luz/usr/mpe' > >> '--with-mpicc=/home/fernando_luz/usr/mpich/bin/mpicc' > >> '--with-mpilibs=/home/fernando_luz/usr/mpich/lib/' > >> '--with-mpiinc=/home/fernando_luz/usr/mpich/include/' > >> 'MPI_CC=/home/fernando_luz/usr/mpich/bin/mpicc' > >> 'MPI_INC=/home/fernando_luz/usr/mpich/include' > >> 'MPI_LIBS=/home/fernando_luz/usr/mpich/lib' > >> checking for current directory name... /home/fernando_luz/git/mpe > >> checking gnumake... yes using --no-print-directory > >> checking BSD 4.4 make... no - whew > >> checking OSF V3 make... no > >> checking for virtual path format... VPATH > >> User supplied MPI implmentation (Good Luck!) > >> checking for leftover Makefiles in subpackages ... none > >> checking for gcc... cc > >> checking whether the C compiler works... yes > >> checking for C compiler default output file name... a.out > >> checking for suffix of executables... > >> checking whether we are cross compiling... no > >> checking for suffix of object files... o > >> checking whether we are using the GNU C compiler... yes > >> checking whether cc accepts -g... yes > >> checking for cc option to accept ISO C89... none needed > >> checking whether MPI_CC has been set ... > >> /home/fernando_luz/usr/mpich/bin/mpicc > >> checking whether we are using the GNU Fortran 77 compiler... no > >> checking whether f77 accepts -g... no > >> checking whether MPI_F77 has been set ... f77 > >> checking for the linkage of the supplied MPI C definitions ... no > >> configure: error: Cannot link with basic MPI C program! > >> > >> Check your MPI include paths, MPI libraries and MPI CC compiler > >> > >> Where /home/fernando_luz/usr/mpich/ is my mpi installation (MPICH-3.0.4). > >> > >> I prefer to use the mpe in repository because in the site, the last > >> version > >> was dated in 2010 and in the git repository the last commit was in 2012. > >> > >> Regards > >> > >> Fernando > >> > >> On 06/04/2013 03:05 PM, Rajeev Thakur wrote: > >>> It can be downloaded from > >>> http://www.mcs.anl.gov/research/projects/perfvis/download/index.htm. > >>> > >>> The source repository is at http://git.mpich.org/mpe.git/ > >>> > >>> Rajeev > >>> > >>> On Jun 4, 2013, at 12:48 PM, Jeff Hammond wrote: > >>>> MPE isn't actively developed and should sit strictly on top of any MPI > >>>> implementation so you can just grab MPE from an older release of > >>>> MPICH. > >>>> > >>>> My guess is that MPE will be a standalone download at some point in the > >>>> future. > >>>> > >>>> Jeff > >>>> > >>>> On Tue, Jun 4, 2013 at 10:59 AM, fernando_luz <fernando_luz(a)tpn.usp.br> > >>>> > >>>> wrote: > >>>>> Hi, > >>>>> > >>>>> I didn't find the MPE source in mpich-3.0.4 package. Where I can > >>>>> download > >>>>> the source? It is still compatible with mpich? > >>>>> > >>>>> And I tried to install the logging support available in this release, > >>>>> but my > >>>>> try didn't was successful. I received the follow error: > >>>>> > >>>>> /home/fernando_luz/software/mpich-3.0.4/src/util/logging/rlog/configur > >>>>> e: > >>>>> line 3694: PAC_CC_SUBDIR_SHLIBS: command not found > >>>>> configure: creating ./config.status > >>>>> config.status: error: cannot find input file: `Makefile.in' > >>>>> configure: error: src/util/logging/rlog configure failed > >>>>> > >>>>> I attached the c.txt file used in the configuration. > >>>>> > >>>>> Regards > >>>>> > >>>>> Fernando > >>>>> > >>>>> _______________________________________________ > >>>>> discuss mailing list discuss(a)mpich.org > >>>>> To manage subscription options or unsubscribe: > >>>>> https://lists.mpich.org/mailman/listinfo/discuss > >>>> > >>>> -- > >>>> Jeff Hammond > >>>> Argonne Leadership Computing Facility > >>>> University of Chicago Computation Institute > >>>> jhammond(a)alcf.anl.gov / (630) 252-5381 > >>>> http://www.linkedin.com/in/jeffhammond > >>>> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > >>>> ALCF docs: http://www.alcf.anl.gov/user-guides > >>>> _______________________________________________ > >>>> discuss mailing list discuss(a)mpich.org > >>>> To manage subscription options or unsubscribe: > >>>> https://lists.mpich.org/mailman/listinfo/discuss > >>> > >>> _______________________________________________ > >>> discuss mailing list discuss(a)mpich.org > >>> To manage subscription options or unsubscribe: > >>> https://lists.mpich.org/mailman/listinfo/discuss > >> > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss -- Antonio J. Peña Postdoctoral Appointee Mathematics and Computer Science Division Argonne National Laboratory 9700 South Cass Avenue, Bldg. 240, Of. 3148 Argonne, IL 60439-4847 (+1) 630-252-7928 apenya(a)mcs.anl.gov

1 0

Re: [mpich-discuss] Problems running MPICH jobs under SLURM
by Pavan Balaji 07 Jun '13

07 Jun '13

On 06/07/2013 03:50 PM, Biddiscombe, John A. wrote: > Yes, that's where I compiled and installed the nightly snapshot to. (I did a make clean and recompile of the app, though I see that only hydra needed changing ) - > I assumed that the mpich snapshot did include the updated hydra - if not, I should redo it with the fixed hydra. > > I'll wipe everything and rebuild one more time just in case I messed up. Does it matter that I'm installing to a non standard location? Does it need to be on any system paths or have special privileges? Non-standard locations are fine, as long as you use full paths to the executables. Just untar the nightly tarball and build from scratch. Technically you only need to get a new mpiexec, not the entire mpich, but it only takes a couple of minutes to build, so it's not a big deal either way. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Problems running MPICH jobs under SLURM
by Biddiscombe, John A. 07 Jun '13

07 Jun '13

Pavan Yes, that's where I compiled and installed the nightly snapshot to. (I did a make clean and recompile of the app, though I see that only hydra needed changing ) - I assumed that the mpich snapshot did include the updated hydra - if not, I should redo it with the fixed hydra. I'll wipe everything and rebuild one more time just in case I messed up. Does it matter that I'm installing to a non standard location? Does it need to be on any system paths or have special privileges? JB -----Original Message----- From: Pavan Balaji [mailto:[email protected]] Sent: 07 June 2013 22:08 To: Biddiscombe, John A. Cc: discuss(a)mpich.org Subject: Re: [mpich-discuss] Problems running MPICH jobs under SLURM Are you using the correct mpiexec? Your submission script is using the mpiexec from this directory: /home/biddisco/apps/mpich-3.0.4/bin/mpiexec -- Pavan On 06/07/2013 03:40 AM, Biddiscombe, John A. wrote: > I downloaded the nightly tarball and recompiled/installed mpich (used > mpich-master-v3.0.4-259-gf322ce79) > > I still get this (output below) with a simple hello world program. > > Now you must understand that I have no idea what I'm doing (really). I > wanted to test some debugging features under slurm so installed slurm > myself on a workstation with just 2 cores and have the bare minimum > setup. I'm doing the following > > sudo munged & > > sudo slurmd & > > sudo slurmctld -D > > and then I can run jobs on the local machine and it seems to be ok, > except that mpi jobs always give the double free error as below when > run under slurm, but are just fine when run from the command line. > > My suspicion is that slurm is not actually using the hydra pm that I > just compiled. I installed slurm from rpms. Should I recompile slurm > myself and somehow tell it which mpi to use? > > My job script looks as follows > > ###################### > > #!/bin/bash > > # > > # Create the job script from the supplied parameters > > # > > #SBATCH --job-name=pvserver > > #SBATCH --time=00:04:00 > > #SBATCH --nodes=1 > > #SBATCH --partition=normal > > #SBATCH --output=/home/biddisco/slurm.out > > #SBATCH --error=/home/biddisco/slurm.err > > #SBATCH --mem=2048MB > > #export > > # echo "Path is $PATH" > > # echo "LD_LIBRARY_PATH is " $LD_LIBRARY_PATH > > # cd /home/biddisco/build/pv-38/bin/ > > #export PMI_DEBUG=9 > > #ulimit -s unlimited > > #ulimit -c 0 > > /home/biddisco/apps/mpich-3.0.4/bin/mpiexec -rmk slurm -n 2 > /home/biddisco/build/hello/hello > > ###################### > > It gives the same result with or without the -rmk slurm and the > #ulimit settings. > > Apologies for wasting your time, I'm certain I'm doing something wrong > - I just don't know what. > > JB > > biddisco@breno2 ~ $ more ~/slurm.err > > *** glibc detected *** /home/biddisco/build/hello/hello: double free > or corruption (fasttop): 0x0000000001896340 *** > > ======= Backtrace: ========= > > /lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7f9a1695cb96] > > /home/biddisco/build/hello/hello(MPIDI_Populate_vc_node_ids+0x3f9)[0x4 > 27c89] > > /home/biddisco/build/hello/hello(MPID_Init+0x136)[0x4253f6] > > /home/biddisco/build/hello/hello(MPIR_Init_thread+0x22f)[0x414cbf] > > /home/biddisco/build/hello/hello(MPI_Init+0xae)[0x4146ee] > > /home/biddisco/build/hello/hello(main+0x22)[0x413f2e] > > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f9a168ff76d > ] > > /home/biddisco/build/hello/hello[0x413e31] > > ======= Memory map: ======== > > 00400000-0051a000 r-xp 00000000 08:01 8661191 > /home/biddisco/build/hello/hello > > 0071a000-00727000 r--p 0011a000 08:01 8661191 > /home/biddisco/build/hello/hello > > 00727000-00729000 rw-p 00127000 08:01 8661191 > /home/biddisco/build/hello/hello > > 00729000-00751000 rw-p 00000000 00:00 0 > > 01895000-018b6000 rw-p 00000000 00:00 0 > [heap] > > 7f9a166c8000-7f9a166dd000 r-xp 00000000 08:01 9047556 > /lib/x86_64-linux-gnu/libgcc_s.so.1 > > 7f9a166dd000-7f9a168dc000 ---p 00015000 08:01 9047556 > /lib/x86_64-linux-gnu/libgcc_s.so.1 > > 7f9a168dc000-7f9a168dd000 r--p 00014000 08:01 9047556 > /lib/x86_64-linux-gnu/libgcc_s.so.1 > > 7f9a168dd000-7f9a168de000 rw-p 00015000 08:01 9047556 > /lib/x86_64-linux-gnu/libgcc_s.so.1 > > 7f9a168de000-7f9a16a93000 r-xp 00000000 08:01 9050358 > /lib/x86_64-linux-gnu/libc-2.15.so > > 7f9a16a93000-7f9a16c92000 ---p 001b5000 08:01 9050358 > /lib/x86_64-linux-gnu/libc-2.15.so > > 7f9a16c92000-7f9a16c96000 r--p 001b4000 08:01 9050358 > /lib/x86_64-linux-gnu/libc-2.15.so > > 7f9a16c96000-7f9a16c98000 rw-p 001b8000 08:01 9050358 > /lib/x86_64-linux-gnu/libc-2.15.so > > 7f9a16c98000-7f9a16c9d000 rw-p 00000000 00:00 0 > > 7f9a16c9d000-7f9a16cb5000 r-xp 00000000 08:01 9050338 > /lib/x86_64-linux-gnu/libpthread-2.15.so > > 7f9a16cb5000-7f9a16eb4000 ---p 00018000 08:01 9050338 > /lib/x86_64-linux-gnu/libpthread-2.15.so > > 7f9a16eb4000-7f9a16eb5000 r--p 00017000 08:01 9050338 > /lib/x86_64-linux-gnu/libpthread-2.15.so > > 7f9a16eb5000-7f9a16eb6000 rw-p 00018000 08:01 9050338 > /lib/x86_64-linux-gnu/libpthread-2.15.so > > 7f9a16eb6000-7f9a16eba000 rw-p 00000000 00:00 0 > > 7f9a16eba000-7f9a16edc000 r-xp 00000000 08:01 9050344 > /lib/x86_64-linux-gnu/ld-2.15.so > > 7f9a170c1000-7f9a170c4000 rw-p 00000000 00:00 0 > > 7f9a170d9000-7f9a170dc000 rw-p 00000000 00:00 0 > > 7f9a170dc000-7f9a170dd000 r--p 00022000 08:01 9050344 > /lib/x86_64-linux-gnu/ld-2.15.so > > 7f9a170dd000-7f9a170df000 rw-p 00023000 08:01 9050344 > /lib/x86_64-linux-gnu/ld-2.15.so > > 7fff52f27000-7fff52f48000 rw-p 00000000 00:00 0 [stack] > > 7fff52fff000-7fff53000000 r-xp 00000000 00:00 0 [vdso] > > ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Problems running MPICH jobs under SLURM
by Pavan Balaji 07 Jun '13

07 Jun '13

Are you using the correct mpiexec? Your submission script is using the mpiexec from this directory: /home/biddisco/apps/mpich-3.0.4/bin/mpiexec -- Pavan On 06/07/2013 03:40 AM, Biddiscombe, John A. wrote: > I downloaded the nightly tarball and recompiled/installed mpich (used > mpich-master-v3.0.4-259-gf322ce79) > > I still get this (output below) with a simple hello world program. > > Now you must understand that I have no idea what Iï¿½m doing (really). I > wanted to test some debugging features under slurm so installed slurm > myself on a workstation with just 2 cores and have the bare minimum > setup. Iï¿½m doing the following > > sudo munged & > > sudo slurmd & > > sudo slurmctld -D > > and then I can run jobs on the local machine and it seems to be ok, > except that mpi jobs always give the double free error as below when run > under slurm, but are just fine when run from the command line. > > My suspicion is that slurm is not actually using the hydra pm that I > just compiled. I installed slurm from rpms. Should I recompile slurm > myself and somehow tell it which mpi to use? > > My job script looks as follows > > ###################### > > #!/bin/bash > > # > > # Create the job script from the supplied parameters > > # > > #SBATCH --job-name=pvserver > > #SBATCH --time=00:04:00 > > #SBATCH --nodes=1 > > #SBATCH --partition=normal > > #SBATCH --output=/home/biddisco/slurm.out > > #SBATCH --error=/home/biddisco/slurm.err > > #SBATCH --mem=2048MB > > #export > > # echo "Path is $PATH" > > # echo "LD_LIBRARY_PATH is " $LD_LIBRARY_PATH > > # cd /home/biddisco/build/pv-38/bin/ > > #export PMI_DEBUG=9 > > #ulimit -s unlimited > > #ulimit -c 0 > > /home/biddisco/apps/mpich-3.0.4/bin/mpiexec -rmk slurm -n 2 > /home/biddisco/build/hello/hello > > ###################### > > It gives the same result with or without the ï¿½rmk slurm and the #ulimit > settings. > > Apologies for wasting your time, Iï¿½m certain Iï¿½m doing something wrong ï¿½ > I just donï¿½t know what. > > JB > > biddisco@breno2 ~ $ more ~/slurm.err > > *** glibc detected *** /home/biddisco/build/hello/hello: double free or > corruption (fasttop): 0x0000000001896340 *** > > ======= Backtrace: ========= > > /lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7f9a1695cb96] > > /home/biddisco/build/hello/hello(MPIDI_Populate_vc_node_ids+0x3f9)[0x427c89] > > /home/biddisco/build/hello/hello(MPID_Init+0x136)[0x4253f6] > > /home/biddisco/build/hello/hello(MPIR_Init_thread+0x22f)[0x414cbf] > > /home/biddisco/build/hello/hello(MPI_Init+0xae)[0x4146ee] > > /home/biddisco/build/hello/hello(main+0x22)[0x413f2e] > > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f9a168ff76d] > > /home/biddisco/build/hello/hello[0x413e31] > > ======= Memory map: ======== > > 00400000-0051a000 r-xp 00000000 08:01 8661191 > /home/biddisco/build/hello/hello > > 0071a000-00727000 r--p 0011a000 08:01 8661191 > /home/biddisco/build/hello/hello > > 00727000-00729000 rw-p 00127000 08:01 8661191 > /home/biddisco/build/hello/hello > > 00729000-00751000 rw-p 00000000 00:00 0 > > 01895000-018b6000 rw-p 00000000 00:00 0 > [heap] > > 7f9a166c8000-7f9a166dd000 r-xp 00000000 08:01 9047556 > /lib/x86_64-linux-gnu/libgcc_s.so.1 > > 7f9a166dd000-7f9a168dc000 ---p 00015000 08:01 9047556 > /lib/x86_64-linux-gnu/libgcc_s.so.1 > > 7f9a168dc000-7f9a168dd000 r--p 00014000 08:01 9047556 > /lib/x86_64-linux-gnu/libgcc_s.so.1 > > 7f9a168dd000-7f9a168de000 rw-p 00015000 08:01 9047556 > /lib/x86_64-linux-gnu/libgcc_s.so.1 > > 7f9a168de000-7f9a16a93000 r-xp 00000000 08:01 9050358 > /lib/x86_64-linux-gnu/libc-2.15.so > > 7f9a16a93000-7f9a16c92000 ---p 001b5000 08:01 9050358 > /lib/x86_64-linux-gnu/libc-2.15.so > > 7f9a16c92000-7f9a16c96000 r--p 001b4000 08:01 9050358 > /lib/x86_64-linux-gnu/libc-2.15.so > > 7f9a16c96000-7f9a16c98000 rw-p 001b8000 08:01 9050358 > /lib/x86_64-linux-gnu/libc-2.15.so > > 7f9a16c98000-7f9a16c9d000 rw-p 00000000 00:00 0 > > 7f9a16c9d000-7f9a16cb5000 r-xp 00000000 08:01 9050338 > /lib/x86_64-linux-gnu/libpthread-2.15.so > > 7f9a16cb5000-7f9a16eb4000 ---p 00018000 08:01 9050338 > /lib/x86_64-linux-gnu/libpthread-2.15.so > > 7f9a16eb4000-7f9a16eb5000 r--p 00017000 08:01 9050338 > /lib/x86_64-linux-gnu/libpthread-2.15.so > > 7f9a16eb5000-7f9a16eb6000 rw-p 00018000 08:01 9050338 > /lib/x86_64-linux-gnu/libpthread-2.15.so > > 7f9a16eb6000-7f9a16eba000 rw-p 00000000 00:00 0 > > 7f9a16eba000-7f9a16edc000 r-xp 00000000 08:01 9050344 > /lib/x86_64-linux-gnu/ld-2.15.so > > 7f9a170c1000-7f9a170c4000 rw-p 00000000 00:00 0 > > 7f9a170d9000-7f9a170dc000 rw-p 00000000 00:00 0 > > 7f9a170dc000-7f9a170dd000 r--p 00022000 08:01 9050344 > /lib/x86_64-linux-gnu/ld-2.15.so > > 7f9a170dd000-7f9a170df000 rw-p 00023000 08:01 9050344 > /lib/x86_64-linux-gnu/ld-2.15.so > > 7fff52f27000-7fff52f48000 rw-p 00000000 00:00 0 > [stack] > > 7fff52fff000-7fff53000000 r-xp 00000000 00:00 0 > [vdso] > > ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 > [vsyscall] > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] MPE is available in mpich-3.0.4 ? rlog doesn't work in configure process.
by fernando_luz 07 Jun '13

07 Jun '13

In attachment. Fernando On 06/07/2013 04:56 PM, Jeff Hammond wrote: > please attach config.log. > > jeff > > On Fri, Jun 7, 2013 at 1:07 PM, fernando_luz <fernando_luz(a)tpn.usp.br> wrote: >> Hi Rajeev, >> >> Thanks for the answers. >> >> I get the source code in repository, but I didn't succeed in the compile >> process. >> I ran the autogen.sh and after this I tried to configure my installation and >> I received the following error message. >> >> >> fernando_luz@TPN000300:~/git/mpe$ ./configure >> --prefix=/home/fernando_luz/usr/mpe >> --with-mpicc=/home/fernando_luz/usr/mpich/bin/mpicc >> --with-mpilibs=/home/fernando_luz/usr/mpich/lib/ >> --with-mpiinc=/home/fernando_luz/usr/mpich/include/ >> Configuring MPE Profiling System with '--prefix=/home/fernando_luz/usr/mpe' >> '--with-mpicc=/home/fernando_luz/usr/mpich/bin/mpicc' >> '--with-mpilibs=/home/fernando_luz/usr/mpich/lib/' >> '--with-mpiinc=/home/fernando_luz/usr/mpich/include/' >> 'MPI_CC=/home/fernando_luz/usr/mpich/bin/mpicc' >> 'MPI_INC=/home/fernando_luz/usr/mpich/include' >> 'MPI_LIBS=/home/fernando_luz/usr/mpich/lib' >> checking for current directory name... /home/fernando_luz/git/mpe >> checking gnumake... yes using --no-print-directory >> checking BSD 4.4 make... no - whew >> checking OSF V3 make... no >> checking for virtual path format... VPATH >> User supplied MPI implmentation (Good Luck!) >> checking for leftover Makefiles in subpackages ... none >> checking for gcc... cc >> checking whether the C compiler works... yes >> checking for C compiler default output file name... a.out >> checking for suffix of executables... >> checking whether we are cross compiling... no >> checking for suffix of object files... o >> checking whether we are using the GNU C compiler... yes >> checking whether cc accepts -g... yes >> checking for cc option to accept ISO C89... none needed >> checking whether MPI_CC has been set ... >> /home/fernando_luz/usr/mpich/bin/mpicc >> checking whether we are using the GNU Fortran 77 compiler... no >> checking whether f77 accepts -g... no >> checking whether MPI_F77 has been set ... f77 >> checking for the linkage of the supplied MPI C definitions ... no >> configure: error: Cannot link with basic MPI C program! >> Check your MPI include paths, MPI libraries and MPI CC compiler >> >> Where /home/fernando_luz/usr/mpich/ is my mpi installation (MPICH-3.0.4). >> >> I prefer to use the mpe in repository because in the site, the last version >> was dated in 2010 and in the git repository the last commit was in 2012. >> >> Regards >> >> Fernando >> >> >> >> On 06/04/2013 03:05 PM, Rajeev Thakur wrote: >>> It can be downloaded from >>> http://www.mcs.anl.gov/research/projects/perfvis/download/index.htm. >>> >>> The source repository is at http://git.mpich.org/mpe.git/ >>> >>> Rajeev >>> >>> On Jun 4, 2013, at 12:48 PM, Jeff Hammond wrote: >>> >>>> MPE isn't actively developed and should sit strictly on top of any MPI >>>> implementation so you can just grab MPE from an older release of >>>> MPICH. >>>> >>>> My guess is that MPE will be a standalone download at some point in the >>>> future. >>>> >>>> Jeff >>>> >>>> On Tue, Jun 4, 2013 at 10:59 AM, fernando_luz <fernando_luz(a)tpn.usp.br> >>>> wrote: >>>>> Hi, >>>>> >>>>> I didn't find the MPE source in mpich-3.0.4 package. Where I can >>>>> download >>>>> the source? It is still compatible with mpich? >>>>> >>>>> And I tried to install the logging support available in this release, >>>>> but my >>>>> try didn't was successful. I received the follow error: >>>>> >>>>> /home/fernando_luz/software/mpich-3.0.4/src/util/logging/rlog/configure: >>>>> line 3694: PAC_CC_SUBDIR_SHLIBS: command not found >>>>> configure: creating ./config.status >>>>> config.status: error: cannot find input file: `Makefile.in' >>>>> configure: error: src/util/logging/rlog configure failed >>>>> >>>>> I attached the c.txt file used in the configuration. >>>>> >>>>> Regards >>>>> >>>>> Fernando >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> >>>> -- >>>> Jeff Hammond >>>> Argonne Leadership Computing Facility >>>> University of Chicago Computation Institute >>>> jhammond(a)alcf.anl.gov / (630) 252-5381 >>>> http://www.linkedin.com/in/jeffhammond >>>> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >>>> ALCF docs: http://www.alcf.anl.gov/user-guides >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > >

1 0

Re: [mpich-discuss] MPE is available in mpich-3.0.4 ? rlog doesn't work in configure process.
by Jeff Hammond 07 Jun '13

07 Jun '13

please attach config.log. jeff On Fri, Jun 7, 2013 at 1:07 PM, fernando_luz <fernando_luz(a)tpn.usp.br> wrote: > Hi Rajeev, > > Thanks for the answers. > > I get the source code in repository, but I didn't succeed in the compile > process. > I ran the autogen.sh and after this I tried to configure my installation and > I received the following error message. > > > fernando_luz@TPN000300:~/git/mpe$ ./configure > --prefix=/home/fernando_luz/usr/mpe > --with-mpicc=/home/fernando_luz/usr/mpich/bin/mpicc > --with-mpilibs=/home/fernando_luz/usr/mpich/lib/ > --with-mpiinc=/home/fernando_luz/usr/mpich/include/ > Configuring MPE Profiling System with '--prefix=/home/fernando_luz/usr/mpe' > '--with-mpicc=/home/fernando_luz/usr/mpich/bin/mpicc' > '--with-mpilibs=/home/fernando_luz/usr/mpich/lib/' > '--with-mpiinc=/home/fernando_luz/usr/mpich/include/' > 'MPI_CC=/home/fernando_luz/usr/mpich/bin/mpicc' > 'MPI_INC=/home/fernando_luz/usr/mpich/include' > 'MPI_LIBS=/home/fernando_luz/usr/mpich/lib' > checking for current directory name... /home/fernando_luz/git/mpe > checking gnumake... yes using --no-print-directory > checking BSD 4.4 make... no - whew > checking OSF V3 make... no > checking for virtual path format... VPATH > User supplied MPI implmentation (Good Luck!) > checking for leftover Makefiles in subpackages ... none > checking for gcc... cc > checking whether the C compiler works... yes > checking for C compiler default output file name... a.out > checking for suffix of executables... > checking whether we are cross compiling... no > checking for suffix of object files... o > checking whether we are using the GNU C compiler... yes > checking whether cc accepts -g... yes > checking for cc option to accept ISO C89... none needed > checking whether MPI_CC has been set ... > /home/fernando_luz/usr/mpich/bin/mpicc > checking whether we are using the GNU Fortran 77 compiler... no > checking whether f77 accepts -g... no > checking whether MPI_F77 has been set ... f77 > checking for the linkage of the supplied MPI C definitions ... no > configure: error: Cannot link with basic MPI C program! > Check your MPI include paths, MPI libraries and MPI CC compiler > > Where /home/fernando_luz/usr/mpich/ is my mpi installation (MPICH-3.0.4). > > I prefer to use the mpe in repository because in the site, the last version > was dated in 2010 and in the git repository the last commit was in 2012. > > Regards > > Fernando > > > > On 06/04/2013 03:05 PM, Rajeev Thakur wrote: >> >> It can be downloaded from >> http://www.mcs.anl.gov/research/projects/perfvis/download/index.htm. >> >> The source repository is at http://git.mpich.org/mpe.git/ >> >> Rajeev >> >> On Jun 4, 2013, at 12:48 PM, Jeff Hammond wrote: >> >>> MPE isn't actively developed and should sit strictly on top of any MPI >>> implementation so you can just grab MPE from an older release of >>> MPICH. >>> >>> My guess is that MPE will be a standalone download at some point in the >>> future. >>> >>> Jeff >>> >>> On Tue, Jun 4, 2013 at 10:59 AM, fernando_luz <fernando_luz(a)tpn.usp.br> >>> wrote: >>>> >>>> Hi, >>>> >>>> I didn't find the MPE source in mpich-3.0.4 package. Where I can >>>> download >>>> the source? It is still compatible with mpich? >>>> >>>> And I tried to install the logging support available in this release, >>>> but my >>>> try didn't was successful. I received the follow error: >>>> >>>> /home/fernando_luz/software/mpich-3.0.4/src/util/logging/rlog/configure: >>>> line 3694: PAC_CC_SUBDIR_SHLIBS: command not found >>>> configure: creating ./config.status >>>> config.status: error: cannot find input file: `Makefile.in' >>>> configure: error: src/util/logging/rlog configure failed >>>> >>>> I attached the c.txt file used in the configuration. >>>> >>>> Regards >>>> >>>> Fernando >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> >>> >>> -- >>> Jeff Hammond >>> Argonne Leadership Computing Facility >>> University of Chicago Computation Institute >>> jhammond(a)alcf.anl.gov / (630) 252-5381 >>> http://www.linkedin.com/in/jeffhammond >>> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >>> ALCF docs: http://www.alcf.anl.gov/user-guides >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [mpich-discuss] MPE is available in mpich-3.0.4 ? rlog doesn't work in configure process.
by fernando_luz 07 Jun '13

07 Jun '13

Hi Rajeev, Thanks for the answers. I get the source code in repository, but I didn't succeed in the compile process. I ran the autogen.sh and after this I tried to configure my installation and I received the following error message. fernando_luz@TPN000300:~/git/mpe$ ./configure --prefix=/home/fernando_luz/usr/mpe --with-mpicc=/home/fernando_luz/usr/mpich/bin/mpicc --with-mpilibs=/home/fernando_luz/usr/mpich/lib/ --with-mpiinc=/home/fernando_luz/usr/mpich/include/ Configuring MPE Profiling System with '--prefix=/home/fernando_luz/usr/mpe' '--with-mpicc=/home/fernando_luz/usr/mpich/bin/mpicc' '--with-mpilibs=/home/fernando_luz/usr/mpich/lib/' '--with-mpiinc=/home/fernando_luz/usr/mpich/include/' 'MPI_CC=/home/fernando_luz/usr/mpich/bin/mpicc' 'MPI_INC=/home/fernando_luz/usr/mpich/include' 'MPI_LIBS=/home/fernando_luz/usr/mpich/lib' checking for current directory name... /home/fernando_luz/git/mpe checking gnumake... yes using --no-print-directory checking BSD 4.4 make... no - whew checking OSF V3 make... no checking for virtual path format... VPATH User supplied MPI implmentation (Good Luck!) checking for leftover Makefiles in subpackages ... none checking for gcc... cc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether cc accepts -g... yes checking for cc option to accept ISO C89... none needed checking whether MPI_CC has been set ... /home/fernando_luz/usr/mpich/bin/mpicc checking whether we are using the GNU Fortran 77 compiler... no checking whether f77 accepts -g... no checking whether MPI_F77 has been set ... f77 checking for the linkage of the supplied MPI C definitions ... no configure: error: Cannot link with basic MPI C program! Check your MPI include paths, MPI libraries and MPI CC compiler Where /home/fernando_luz/usr/mpich/ is my mpi installation (MPICH-3.0.4). I prefer to use the mpe in repository because in the site, the last version was dated in 2010 and in the git repository the last commit was in 2012. Regards Fernando On 06/04/2013 03:05 PM, Rajeev Thakur wrote: > It can be downloaded from http://www.mcs.anl.gov/research/projects/perfvis/download/index.htm. > > The source repository is at http://git.mpich.org/mpe.git/ > > Rajeev > > On Jun 4, 2013, at 12:48 PM, Jeff Hammond wrote: > >> MPE isn't actively developed and should sit strictly on top of any MPI >> implementation so you can just grab MPE from an older release of >> MPICH. >> >> My guess is that MPE will be a standalone download at some point in the future. >> >> Jeff >> >> On Tue, Jun 4, 2013 at 10:59 AM, fernando_luz <fernando_luz(a)tpn.usp.br> wrote: >>> Hi, >>> >>> I didn't find the MPE source in mpich-3.0.4 package. Where I can download >>> the source? It is still compatible with mpich? >>> >>> And I tried to install the logging support available in this release, but my >>> try didn't was successful. I received the follow error: >>> >>> /home/fernando_luz/software/mpich-3.0.4/src/util/logging/rlog/configure: >>> line 3694: PAC_CC_SUBDIR_SHLIBS: command not found >>> configure: creating ./config.status >>> config.status: error: cannot find input file: `Makefile.in' >>> configure: error: src/util/logging/rlog configure failed >>> >>> I attached the c.txt file used in the configuration. >>> >>> Regards >>> >>> Fernando >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> -- >> Jeff Hammond >> Argonne Leadership Computing Facility >> University of Chicago Computation Institute >> jhammond(a)alcf.anl.gov / (630) 252-5381 >> http://www.linkedin.com/in/jeffhammond >> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >> ALCF docs: http://www.alcf.anl.gov/user-guides >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Problems running MPICH jobs under SLURM
by Pavan Balaji 07 Jun '13

07 Jun '13

On 06/07/2013 09:02 AM, Markus Geimer wrote: > The nightly snapshot fixes the issue for me. Many thanks! > > @John: Did you also recompile your sample application? From what > I understand, the issue is not in hydra but in the MPI library > (please correct me if I'm wrong, Pavan). No, my initial diagnosis was incorrect. The problem was completely inside hydra (mpiexec), so the application doesn't need to be recompiled. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0