We certainly want to give better error messages where possible.
I've created a ticket for it:
https://trac.mpich.org/projects/mpich/ticket/1872
-- Pavan
On 05/29/2013 04:37 PM, Edscott Wilson wrote:
>
>
>
> 2013/5/29 Jeff Hammond <jhammond(a)alcf.anl.gov
> <mailto:[email protected]>>
>
>
> >
> >
> > Wouldn't a message such as "`pwd` directory does not exist on
> node velascoj"
> > be more illustrative?
>
> Yes. However, the set of improper uses of MPI that could generate
> helpful error messages is uncountable. Do you not think it is a good
> use of finite developer effort to implement an infinitesimal fraction
> of such warnings? There has to be a minimum requirement placed upon
> the user. I personally think that it should include running in a
> directory that actually exists.
>
>
> Certainly! But then again, some developer must have thought it a good
> idea, since under different circumstances, I get:
>
> /bin/bash -c mpiexec -n 1 -hosts tauro,velascoj gmandel
> [proxy:0:0@tauro] launch_procs (./pm/pmiserv/pmip_cb.c:648): unable to
> change wdir to /tmp/edscott/mnt/tauro-home/GIT/gmandel (No such file or
> directory)
> [proxy:0:0@tauro] HYD_pmcd_pmip_control_cmd_cb
> (./pm/pmiserv/pmip_cb.c:893): launch_procs returned error
> [proxy:0:0@tauro] HYDT_dmxu_poll_wait_for_event
> (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:0@tauro] main (./pm/pmiserv/pmip.c:206): demux engine error
> waiting for event
> [mpiexec@velascoj] control_cb (./pm/pmiserv/pmiserv_cb.c:202): assert
> (!closed) failed
> [mpiexec@velascoj] HYDT_dmxu_poll_wait_for_event
> (./tools/demux/demux_poll.c:77): callback returned error status
> [mpiexec@velascoj] HYD_pmci_wait_for_completion
> (./pm/pmiserv/pmiserv_pmci.c:197): error waiting for event
> [mpiexec@velascoj] main (./ui/mpich/mpiexec.c:331): process manager
> error waiting for completion
> Which is inconsistent with the previous behavior. Anyways, its no big deal.
>
> BTW, would you happen to know why a process which is started with
> MPI_Comm_spawn will go into what seems like an active wait after
> MPI_Comm_disconnect and MPI_Finalize has been called? These spawned
> processed will hog up CPU until the parent process exits. Curious
> enough, this behavior is not mirrored in openmpi.
>
> Edscott
>
>
>
> -------------------------------
> Dr. Edscott Wilson Garcia
> Applied Mathematics and Computing
> Mexican Petroleum Institute
>
>
> _______________________________________________
> discuss mailing list discuss(a)mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss
>
--
Pavan Balaji
http://www.mcs.anl.gov/~balaji
Thanks. I'll look into this.
Markus: does this happen only with SLURM or can you reproduce this
without SLURM as well?
-- Pavan
On 06/01/2013 02:39 PM, Rajeev Thakur wrote:
> I have created a ticket for it: http://trac.mpich.org/projects/mpich/ticket/1871
>
> Rajeev
>
> On Jun 1, 2013, at 10:29 AM, Markus Geimer wrote:
>
>> Jeff,
>>
>>> Have you tried MPICH 3.0.4? Hydra has been improved a great deal since
>>> the 2.4 release.
>>
>> As I wrote: We first tried the latest and greatest, i.e., MPICH 3.0.4.
>> Only afterwards we tried MPICH2 1.5 to see whether previous versions
>> behave the same. Unfortunately, both show the same behavior.
>>
>> The predecessor version MPICH2 1.4.1p1, however, works. Therefore we
>> assume that some bug sneaked in between 1.4.1p1 and 1.5.
>>
>> Markus
>>
>> --
>> Dr. Markus Geimer
>> Juelich Supercomputing Centre
>> Institute for Advanced Simulation
>> Forschungszentrum Juelich GmbH
>> 52425 Juelich, Germany
>>
>> Phone: +49-2461-61-1773
>> Fax: +49-2461-61-6656
>> E-mail: m.geimer(a)fz-juelich.de
>> WWW: http://www.fz-juelich.de/jsc/
>>
>>
>> ------------------------------------------------------------------------------------------------
>> ------------------------------------------------------------------------------------------------
>> Forschungszentrum Juelich GmbH
>> 52425 Juelich
>> Sitz der Gesellschaft: Juelich
>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
>> Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
>> Prof. Dr. Sebastian M. Schmidt
>> ------------------------------------------------------------------------------------------------
>> ------------------------------------------------------------------------------------------------
>> _______________________________________________
>> discuss mailing list discuss(a)mpich.org
>> To manage subscription options or unsubscribe:
>> https://lists.mpich.org/mailman/listinfo/discuss
>
> _______________________________________________
> discuss mailing list discuss(a)mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss
>
--
Pavan Balaji
http://www.mcs.anl.gov/~balaji
I have created a ticket for it: http://trac.mpich.org/projects/mpich/ticket/1871
Rajeev
On Jun 1, 2013, at 10:29 AM, Markus Geimer wrote:
> Jeff,
>
>> Have you tried MPICH 3.0.4? Hydra has been improved a great deal since
>> the 2.4 release.
>
> As I wrote: We first tried the latest and greatest, i.e., MPICH 3.0.4.
> Only afterwards we tried MPICH2 1.5 to see whether previous versions
> behave the same. Unfortunately, both show the same behavior.
>
> The predecessor version MPICH2 1.4.1p1, however, works. Therefore we
> assume that some bug sneaked in between 1.4.1p1 and 1.5.
>
> Markus
>
> --
> Dr. Markus Geimer
> Juelich Supercomputing Centre
> Institute for Advanced Simulation
> Forschungszentrum Juelich GmbH
> 52425 Juelich, Germany
>
> Phone: +49-2461-61-1773
> Fax: +49-2461-61-6656
> E-mail: m.geimer(a)fz-juelich.de
> WWW: http://www.fz-juelich.de/jsc/
>
>
> ------------------------------------------------------------------------------------------------
> ------------------------------------------------------------------------------------------------
> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Prof. Dr. Sebastian M. Schmidt
> ------------------------------------------------------------------------------------------------
> ------------------------------------------------------------------------------------------------
> _______________________________________________
> discuss mailing list discuss(a)mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss
Jeff,
> Have you tried MPICH 3.0.4? Hydra has been improved a great deal since
> the 2.4 release.
As I wrote: We first tried the latest and greatest, i.e., MPICH 3.0.4.
Only afterwards we tried MPICH2 1.5 to see whether previous versions
behave the same. Unfortunately, both show the same behavior.
The predecessor version MPICH2 1.4.1p1, however, works. Therefore we
assume that some bug sneaked in between 1.4.1p1 and 1.5.
Markus
--
Dr. Markus Geimer
Juelich Supercomputing Centre
Institute for Advanced Simulation
Forschungszentrum Juelich GmbH
52425 Juelich, Germany
Phone: +49-2461-61-1773
Fax: +49-2461-61-6656
E-mail: m.geimer(a)fz-juelich.de
WWW: http://www.fz-juelich.de/jsc/
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Have you tried MPICH 3.0.4? Hydra has been improved a great deal since
the 2.4 release.
Jeff
Sent from my iPhone
On Jun 1, 2013, at 9:21 AM, Markus Geimer <m.geimer(a)fz-juelich.de> wrote:
> Dear MPICH developers,
>
> We are experiencing some problems getting MPICH jobs to run under
> SLURM (Debian package slurm-llnl 2.3.4-2+b1) on our small test
> cluster. When starting an MPI job with more than one rank, the
> program crashes immediately with the following output:
>
> ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< -----
>
> *** glibc detected *** ./hello: double free or corruption (fasttop):
> 0x00000000014c9680 ***
> ======= Backtrace: =========
> /lib/x86_64-linux-gnu/libc.so.6(+0x76d76)[0x7f4ee51fed76]
> /lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7f4ee5203aac]
> /opt/mpich/3.0.4-gcc/lib/libmpich.so.10(MPIDI_Populate_vc_node_ids+0x3f9)[0x7f4ee5dec5d9]
> /opt/mpich/3.0.4-gcc/lib/libmpich.so.10(MPID_Init+0x136)[0x7f4ee5de6dd6]
> /opt/mpich/3.0.4-gcc/lib/libmpich.so.10(MPIR_Init_thread+0x23f)[0x7f4ee5e9cf7f]
> /opt/mpich/3.0.4-gcc/lib/libmpich.so.10(MPI_Init+0xae)[0x7f4ee5e9c90e]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7fa889cddead]
> ./hello[0x400799]
>
> ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 -----
>
> Single rank jobs run fine, but they are obviously of little interest ;-)
> The test program is a simple 'hello world' printing the rank, started
> using a minimal batch script:
>
> ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< -----
>
> #!/bin/sh
> #SBATCH -n 4
> mpiexec ./hello
>
> ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 -----
>
> We have tried with MPICH 3.0.4 as well as MPICH2 1.5, configured using
> only
>
> --prefix=... --enable-shared --enable-debuginfo
>
> Both are showing the same symptoms. MPICH2 1.4.1p1, however, works
> without problems. Any idea what's going wrong in the newer versions?
>
> Thanks,
> Markus
>
> --
> Dr. Markus Geimer
> Juelich Supercomputing Centre
> Institute for Advanced Simulation
> Forschungszentrum Juelich GmbH
> 52425 Juelich, Germany
>
> Phone: +49-2461-61-1773
> Fax: +49-2461-61-6656
> E-mail: m.geimer(a)fz-juelich.de
> WWW: http://www.fz-juelich.de/jsc/
>
>
> ------------------------------------------------------------------------------------------------
> ------------------------------------------------------------------------------------------------
> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Prof. Dr. Sebastian M. Schmidt
> ------------------------------------------------------------------------------------------------
> ------------------------------------------------------------------------------------------------
> _______________________________________________
> discuss mailing list discuss(a)mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss
Dear MPICH developers,
We are experiencing some problems getting MPICH jobs to run under
SLURM (Debian package slurm-llnl 2.3.4-2+b1) on our small test
cluster. When starting an MPI job with more than one rank, the
program crashes immediately with the following output:
----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< -----
*** glibc detected *** ./hello: double free or corruption (fasttop):
0x00000000014c9680 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x76d76)[0x7f4ee51fed76]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7f4ee5203aac]
/opt/mpich/3.0.4-gcc/lib/libmpich.so.10(MPIDI_Populate_vc_node_ids+0x3f9)[0x7f4ee5dec5d9]
/opt/mpich/3.0.4-gcc/lib/libmpich.so.10(MPID_Init+0x136)[0x7f4ee5de6dd6]
/opt/mpich/3.0.4-gcc/lib/libmpich.so.10(MPIR_Init_thread+0x23f)[0x7f4ee5e9cf7f]
/opt/mpich/3.0.4-gcc/lib/libmpich.so.10(MPI_Init+0xae)[0x7f4ee5e9c90e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7fa889cddead]
./hello[0x400799]
----- >8 ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 -----
Single rank jobs run fine, but they are obviously of little interest ;-)
The test program is a simple 'hello world' printing the rank, started
using a minimal batch script:
----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< -----
#!/bin/sh
#SBATCH -n 4
mpiexec ./hello
----- >8 ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 -----
We have tried with MPICH 3.0.4 as well as MPICH2 1.5, configured using
only
--prefix=... --enable-shared --enable-debuginfo
Both are showing the same symptoms. MPICH2 1.4.1p1, however, works
without problems. Any idea what's going wrong in the newer versions?
Thanks,
Markus
--
Dr. Markus Geimer
Juelich Supercomputing Centre
Institute for Advanced Simulation
Forschungszentrum Juelich GmbH
52425 Juelich, Germany
Phone: +49-2461-61-1773
Fax: +49-2461-61-6656
E-mail: m.geimer(a)fz-juelich.de
WWW: http://www.fz-juelich.de/jsc/
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
