- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Problems running MPICH jobs under SLURM
by Rajeev Thakur 01 Jun '13

01 Jun '13

I have created a ticket for it: http://trac.mpich.org/projects/mpich/ticket/1871 Rajeev On Jun 1, 2013, at 10:29 AM, Markus Geimer wrote: > Jeff, > >> Have you tried MPICH 3.0.4? Hydra has been improved a great deal since >> the 2.4 release. > > As I wrote: We first tried the latest and greatest, i.e., MPICH 3.0.4. > Only afterwards we tried MPICH2 1.5 to see whether previous versions > behave the same. Unfortunately, both show the same behavior. > > The predecessor version MPICH2 1.4.1p1, however, works. Therefore we > assume that some bug sneaked in between 1.4.1p1 and 1.5. > > Markus > > -- > Dr. Markus Geimer > Juelich Supercomputing Centre > Institute for Advanced Simulation > Forschungszentrum Juelich GmbH > 52425 Juelich, Germany > > Phone: +49-2461-61-1773 > Fax: +49-2461-61-6656 > E-mail: m.geimer(a)fz-juelich.de > WWW: http://www.fz-juelich.de/jsc/ > > > ------------------------------------------------------------------------------------------------ > ------------------------------------------------------------------------------------------------ > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > ------------------------------------------------------------------------------------------------ > ------------------------------------------------------------------------------------------------ > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Problems running MPICH jobs under SLURM
by Markus Geimer 01 Jun '13

01 Jun '13

Jeff, > Have you tried MPICH 3.0.4? Hydra has been improved a great deal since > the 2.4 release. As I wrote: We first tried the latest and greatest, i.e., MPICH 3.0.4. Only afterwards we tried MPICH2 1.5 to see whether previous versions behave the same. Unfortunately, both show the same behavior. The predecessor version MPICH2 1.4.1p1, however, works. Therefore we assume that some bug sneaked in between 1.4.1p1 and 1.5. Markus -- Dr. Markus Geimer Juelich Supercomputing Centre Institute for Advanced Simulation Forschungszentrum Juelich GmbH 52425 Juelich, Germany Phone: +49-2461-61-1773 Fax: +49-2461-61-6656 E-mail: m.geimer(a)fz-juelich.de WWW: http://www.fz-juelich.de/jsc/ ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

1 0

Re: [mpich-discuss] Problems running MPICH jobs under SLURM
by Jeff Hammond 01 Jun '13

01 Jun '13

Have you tried MPICH 3.0.4? Hydra has been improved a great deal since the 2.4 release. Jeff Sent from my iPhone On Jun 1, 2013, at 9:21 AM, Markus Geimer <m.geimer(a)fz-juelich.de> wrote: > Dear MPICH developers, > > We are experiencing some problems getting MPICH jobs to run under > SLURM (Debian package slurm-llnl 2.3.4-2+b1) on our small test > cluster. When starting an MPI job with more than one rank, the > program crashes immediately with the following output: > > ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- > > *** glibc detected *** ./hello: double free or corruption (fasttop): > 0x00000000014c9680 *** > ======= Backtrace: ========= > /lib/x86_64-linux-gnu/libc.so.6(+0x76d76)[0x7f4ee51fed76] > /lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7f4ee5203aac] > /opt/mpich/3.0.4-gcc/lib/libmpich.so.10(MPIDI_Populate_vc_node_ids+0x3f9)[0x7f4ee5dec5d9] > /opt/mpich/3.0.4-gcc/lib/libmpich.so.10(MPID_Init+0x136)[0x7f4ee5de6dd6] > /opt/mpich/3.0.4-gcc/lib/libmpich.so.10(MPIR_Init_thread+0x23f)[0x7f4ee5e9cf7f] > /opt/mpich/3.0.4-gcc/lib/libmpich.so.10(MPI_Init+0xae)[0x7f4ee5e9c90e] > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7fa889cddead] > ./hello[0x400799] > > ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 ----- > > Single rank jobs run fine, but they are obviously of little interest ;-) > The test program is a simple 'hello world' printing the rank, started > using a minimal batch script: > > ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- > > #!/bin/sh > #SBATCH -n 4 > mpiexec ./hello > > ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 ----- > > We have tried with MPICH 3.0.4 as well as MPICH2 1.5, configured using > only > > --prefix=... --enable-shared --enable-debuginfo > > Both are showing the same symptoms. MPICH2 1.4.1p1, however, works > without problems. Any idea what's going wrong in the newer versions? > > Thanks, > Markus > > -- > Dr. Markus Geimer > Juelich Supercomputing Centre > Institute for Advanced Simulation > Forschungszentrum Juelich GmbH > 52425 Juelich, Germany > > Phone: +49-2461-61-1773 > Fax: +49-2461-61-6656 > E-mail: m.geimer(a)fz-juelich.de > WWW: http://www.fz-juelich.de/jsc/ > > > ------------------------------------------------------------------------------------------------ > ------------------------------------------------------------------------------------------------ > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > ------------------------------------------------------------------------------------------------ > ------------------------------------------------------------------------------------------------ > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Problems running MPICH jobs under SLURM
by Markus Geimer 01 Jun '13

01 Jun '13

Dear MPICH developers, We are experiencing some problems getting MPICH jobs to run under SLURM (Debian package slurm-llnl 2.3.4-2+b1) on our small test cluster. When starting an MPI job with more than one rank, the program crashes immediately with the following output: ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- *** glibc detected *** ./hello: double free or corruption (fasttop): 0x00000000014c9680 *** ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x76d76)[0x7f4ee51fed76] /lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7f4ee5203aac] /opt/mpich/3.0.4-gcc/lib/libmpich.so.10(MPIDI_Populate_vc_node_ids+0x3f9)[0x7f4ee5dec5d9] /opt/mpich/3.0.4-gcc/lib/libmpich.so.10(MPID_Init+0x136)[0x7f4ee5de6dd6] /opt/mpich/3.0.4-gcc/lib/libmpich.so.10(MPIR_Init_thread+0x23f)[0x7f4ee5e9cf7f] /opt/mpich/3.0.4-gcc/lib/libmpich.so.10(MPI_Init+0xae)[0x7f4ee5e9c90e] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7fa889cddead] ./hello[0x400799] ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 ----- Single rank jobs run fine, but they are obviously of little interest ;-) The test program is a simple 'hello world' printing the rank, started using a minimal batch script: ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- #!/bin/sh #SBATCH -n 4 mpiexec ./hello ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 ----- We have tried with MPICH 3.0.4 as well as MPICH2 1.5, configured using only --prefix=... --enable-shared --enable-debuginfo Both are showing the same symptoms. MPICH2 1.4.1p1, however, works without problems. Any idea what's going wrong in the newer versions? Thanks, Markus -- Dr. Markus Geimer Juelich Supercomputing Centre Institute for Advanced Simulation Forschungszentrum Juelich GmbH 52425 Juelich, Germany Phone: +49-2461-61-1773 Fax: +49-2461-61-6656 E-mail: m.geimer(a)fz-juelich.de WWW: http://www.fz-juelich.de/jsc/ ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

1 0

Re: [mpich-discuss] MPI_COMM_Spawn leave children processes in active wait
by Yauheni Zelenko 30 May '13

30 May '13

Hi, Edscott! I think you need to call MPI_Comm_disconnect() in child processes. Eugene. ________________________________________ From: discuss-bounces(a)mpich.org [discuss-bounces(a)mpich.org] On Behalf Of Edscott Wilson [edscott.wilson.garcia(a)gmail.com] Sent: Thursday, May 30, 2013 12:48 PM To: discuss(a)mpich.org Subject: [mpich-discuss] MPI_COMM_Spawn leave children processes in active wait MPI_COMM_Spawn leave children processes in active wait until parent process is done. I've just checked latest ompi and behavior is the same as in mpich. So be it. -- ------------------------------------------------------------------------------------ Dr. Edscott Wilson Garcia Applied Mathematics and Computing Mexican Petroleum Institute

1 0

MPI_COMM_Spawn leave children processes in active wait
by Edscott Wilson 30 May '13

30 May '13

MPI_COMM_Spawn leave children processes in active wait until parent process is done. I've just checked latest ompi and behavior is the same as in mpich. So be it. -- ------------------------------------------------------------------------------------ Dr. Edscott Wilson Garcia Applied Mathematics and Computing Mexican Petroleum Institute

1 0

Re: [mpich-discuss] weird behavior with mpiexe (3.0.4)
by Edscott Wilson 29 May '13

29 May '13

2013/5/29 Jeff Hammond <jhammond(a)alcf.anl.gov> > > > > > > > Wouldn't a message such as "`pwd` directory does not exist on node > velascoj" > > be more illustrative? > > Yes. However, the set of improper uses of MPI that could generate > helpful error messages is uncountable. Do you not think it is a good > use of finite developer effort to implement an infinitesimal fraction > of such warnings? There has to be a minimum requirement placed upon > the user. I personally think that it should include running in a > directory that actually exists. > Certainly! But then again, some developer must have thought it a good idea, since under different circumstances, I get: /bin/bash -c mpiexec -n 1 -hosts tauro,velascoj gmandel [proxy:0:0@tauro] launch_procs (./pm/pmiserv/pmip_cb.c:648): unable to change wdir to /tmp/edscott/mnt/tauro-home/GIT/gmandel (No such file or directory) [proxy:0:0@tauro] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:893): launch_procs returned error [proxy:0:0@tauro] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status [proxy:0:0@tauro] main (./pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@velascoj] control_cb (./pm/pmiserv/pmiserv_cb.c:202): assert (!closed) failed [mpiexec@velascoj] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status [mpiexec@velascoj] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:197): error waiting for event [mpiexec@velascoj] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion Which is inconsistent with the previous behavior. Anyways, its no big deal. BTW, would you happen to know why a process which is started with MPI_Comm_spawn will go into what seems like an active wait after MPI_Comm_disconnect and MPI_Finalize has been called? These spawned processed will hog up CPU until the parent process exits. Curious enough, this behavior is not mirrored in openmpi. Edscott ------------------------------- Dr. Edscott Wilson Garcia Applied Mathematics and Computing Mexican Petroleum Institute

1 0

Re: [mpich-discuss] weird behavior with mpiexe (3.0.4)
by Jeff Hammond 29 May '13

29 May '13

> This is weird. If I chdir to /home/edscott/tmp and execute the following > command I get: > > edscott@tauro ~/tmp $ mpiexec -n 2 -hosts tauro,velascoj /bin/hostname > tauro > velascoj > > But if I chdir to /home/edscott/temp and do the same I get: > > edscott@tauro ~/temp $ mpiexec -n 2 -hosts tauro,velascoj /bin/hostname > tauro > [mpiexec@tauro] control_cb (./pm/pmiserv/pmiserv_cb.c:202): assert (!closed) > failed > [mpiexec@tauro] HYDT_dmxu_poll_wait_for_event > (./tools/demux/demux_poll.c:77): callback returned error status > [mpiexec@tauro] HYD_pmci_wait_for_completion > (./pm/pmiserv/pmiserv_pmci.c:197): error waiting for event > [mpiexec@tauro] main (./ui/mpich/mpiexec.c:331): process manager error > waiting for completion > > > Wouldn't a message such as "`pwd` directory does not exist on node velascoj" > be more illustrative? Yes. However, the set of improper uses of MPI that could generate helpful error messages is uncountable. Do you not think it is a good use of finite developer effort to implement an infinitesimal fraction of such warnings? There has to be a minimum requirement placed upon the user. I personally think that it should include running in a directory that actually exists. Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

weird behavior with mpiexe (3.0.4)
by Edscott Wilson 29 May '13

29 May '13

This is weird. If I chdir to /home/edscott/tmp and execute the following command I get: edscott@tauro ~/tmp $ mpiexec -n 2 -hosts tauro,velascoj /bin/hostname tauro velascoj But if I chdir to /home/edscott/temp and do the same I get: edscott@tauro ~/temp $ mpiexec -n 2 -hosts tauro,velascoj /bin/hostname tauro [mpiexec@tauro] control_cb (./pm/pmiserv/pmiserv_cb.c:202): assert (!closed) failed [mpiexec@tauro] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status [mpiexec@tauro] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:197): error waiting for event [mpiexec@tauro] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion Wouldn't a message such as "`pwd` directory does not exist on node velascoj" be more illustrative? ------------------------------- Dr. Edscott Wilson Garcia Applied Mathematics and Computing Mexican Petroleum Institute

1 0

Re: [mpich-discuss] [PATCH v2] test: add attrdeleteget, MPI_Attr_get called from delete_fn
by Pavan Balaji 21 May '13

21 May '13

On 05/21/2013 08:58 AM US Central Time, Jeff Squyres (jsquyres) wrote: > You're talking in addition to the 2 hour hybrid time on Wednesday, > right? (i.e., assuming that that will be taken up with init/finalize > discussions) Correct. -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0