i used mpiexec with checkpoint and created two checkpoint files:
mpiexec -ckpointlib blcr -ckpoint-prefix /home/john/ckpts/app.ckpoint
-ckpoint-interval 4 -n 4 /home/john/app/md
context-num0-0-0
context-num1-0-0
i am trying to make a restart
mpiexec -ckpointlib blcr -ckpoint-prefix /home/john/ckpts/app.ckpoint -n 4
-ckpoint-num 1
but nothing happened it just hangs
i also tried:
mpiexec -ckpointlib blcr -ckpoint-prefix /home/john/ckpts/app.ckpoint -n 4
-ckpoint-num 0-0-0
also hangs
Pavan,
> 1. Can you run your application processes using "ddd" or some other
> debugger to see where the double free is coming from? You might have to
> build mpich with --enable-g=dbg to get the debug symbols in.
Here is the full stack backtrace:
----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< -----
#0 0x00007ffff6deb475 in *__GI_raise (sig=<optimized out>)
at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x00007ffff6dee6f0 in *__GI_abort () at abort.c:92
#2 0x00007ffff6e2652b in __libc_message (do_abort=<optimized out>,
fmt=<optimized out>) at ../sysdeps/unix/sysv/linux/libc_fatal.c:189
#3 0x00007ffff6e2fd76 in malloc_printerr (action=3,
str=0x7ffff6f081e0 "double free or corruption (fasttop)",
ptr=<optimized out>) at malloc.c:6283
#4 0x00007ffff6e34aac in *__GI___libc_free (mem=<optimized out>)
at malloc.c:3738
#5 0x00007ffff7a1d5d9 in populate_ids_from_mapping (
did_map=<synthetic pointer>, num_nodes=<synthetic pointer>,
mapping=<optimized out>, pg=<optimized out>)
at src/mpid/ch3/src/mpid_vc.c:1063
#6 MPIDI_Populate_vc_node_ids (pg=pg@entry=0x604910,
our_pg_rank=our_pg_rank@entry=0) at src/mpid/ch3/src/mpid_vc.c:1193
#7 0x00007ffff7a17dd6 in MPID_Init (argc=argc@entry=0x7fffffffd97c,
argv=argv@entry=0x7fffffffd970, requested=requested@entry=0,
provided=provided@entry=0x7fffffffd8e8,
has_args=has_args@entry=0x7fffffffd8e0,
has_env=has_env@entry=0x7fffffffd8e4) at
src/mpid/ch3/src/mpid_init.c:156
#8 0x00007ffff7acdf7f in MPIR_Init_thread (argc=argc@entry=0x7fffffffd97c,
argv=argv@entry=0x7fffffffd970, required=required@entry=0,
provided=provided@entry=0x7fffffffd944) at src/mpi/init/initthread.c:431
#9 0x00007ffff7acd90e in PMPI_Init (argc=0x7fffffffd97c,
argv=0x7fffffffd970)
at src/mpi/init/init.c:136
#10 0x000000000040086d in main ()
----- >8 ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 -----
> 2. Can you send me the output with the ssh launcher as well?
See mail sent off-list.
Thanks,
Markus
--
Dr. Markus Geimer
Juelich Supercomputing Centre
Institute for Advanced Simulation
Forschungszentrum Juelich GmbH
52425 Juelich, Germany
Phone: +49-2461-61-1773
Fax: +49-2461-61-6656
E-mail: m.geimer(a)fz-juelich.de
WWW: http://www.fz-juelich.de/jsc/
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
On 06/03/2013 01:57 AM, Markus Geimer wrote:
> (reply intentionally not sent to the list -- I don't like such logs
> to show up in mailing list archives...)
Ok, I'm cc'ing discuss(a)mpich.org back.
>> Just to make sure, are these the only configure options you are using:
>>
>> --prefix=... --enable-shared --enable-debuginfo
>
> Yes, these are the only options (besides explicitly specifying the
> GNU compiler, but this shouldn't do any harm). Please find the full
> configure log attached.
Thanks.
>> Also, can you run mpiexec with the -verbose option for one of the
>> failing tests (probably just mpiexec -n 4 ./hello) and send me the output?
>
> Output attached.
Hmm. The double free error seems to be coming from the executable,
rather than from mpiexec or the proxy. So we might be looking in the
wrong place.
1. Can you run your application processes using "ddd" or some other
debugger to see where the double free is coming from? You might have to
build mpich with --enable-g=dbg to get the debug symbols in.
2. Can you send me the output with the ssh launcher as well? I want to
see if there are any critical differences in the environment variables
being propagated (e.g., LD_LIBRARY_PATH/LD_PRELOAD) that might affect
shared library builds.
Feel free to send me the logs off-list.
Thanks,
-- Pavan
--
Pavan Balaji
http://www.mcs.anl.gov/~balaji
Markus,
I'm not able to reproduce this at all. Looking through the code didn't
give any information either. Just to make sure, are these the only
configure options you are using:
--prefix=... --enable-shared --enable-debuginfo
Also, can you run mpiexec with the -verbose option for one of the
failing tests (probably just mpiexec -n 4 ./hello) and send me the output?
-- Pavan
On 06/02/2013 02:46 PM, Markus Geimer wrote:
> Hi Pavan,
>
>> Can you try the following:
>>
>> # Use the slurm launcher, and user-specified resources
>> % mpiexec -f hostfile -launcher slurm -n 4 ./hello
>
> Works.
>
>> # Use the ssh launcher, and user-specified resources
>> % mpiexec -f hostfile -launcher ssh -n 4 ./hello
>
> Works.
>
>> # Use the ssh launcher, and slurm-specified resources
>> % mpiexec -rmk slurm -launcher ssh -n 4 ./hello
>
> Fails.
>
>> # Use the slurm launcher, and slurm-specified resources
>> % mpiexec -rmk slurm -launcher slurm -n 4 ./hello
>
> Fails.
>
> So it seems as if the SLURM-provided host list is somehow causing
> trouble...
>
> Markus
>
> --
> Dr. Markus Geimer
> Juelich Supercomputing Centre
> Institute for Advanced Simulation
> Forschungszentrum Juelich GmbH
> 52425 Juelich, Germany
>
> Phone: +49-2461-61-1773
> Fax: +49-2461-61-6656
> E-Mail: m.geimer(a)fz-juelich.de
> WWW: http://www.fz-juelich.de/ias/jsc
>
>
> ------------------------------------------------------------------------------------------------
> ------------------------------------------------------------------------------------------------
> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Prof. Dr. Sebastian M. Schmidt
> ------------------------------------------------------------------------------------------------
> ------------------------------------------------------------------------------------------------
>
--
Pavan Balaji
http://www.mcs.anl.gov/~balaji
Hi Pavan,
> Can you try the following:
>
> # Use the slurm launcher, and user-specified resources
> % mpiexec -f hostfile -launcher slurm -n 4 ./hello
Works.
> # Use the ssh launcher, and user-specified resources
> % mpiexec -f hostfile -launcher ssh -n 4 ./hello
Works.
> # Use the ssh launcher, and slurm-specified resources
> % mpiexec -rmk slurm -launcher ssh -n 4 ./hello
Fails.
> # Use the slurm launcher, and slurm-specified resources
> % mpiexec -rmk slurm -launcher slurm -n 4 ./hello
Fails.
So it seems as if the SLURM-provided host list is somehow causing
trouble...
Markus
--
Dr. Markus Geimer
Juelich Supercomputing Centre
Institute for Advanced Simulation
Forschungszentrum Juelich GmbH
52425 Juelich, Germany
Phone: +49-2461-61-1773
Fax: +49-2461-61-6656
E-Mail: m.geimer(a)fz-juelich.de
WWW: http://www.fz-juelich.de/ias/jsc
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Hi Markus,
On 06/02/2013 04:31 AM, Markus Geimer wrote:
>> Markus: does this happen only with SLURM or can you reproduce this
>> without SLURM as well?
>
> It seems to happen only when hydra queries the host list from SLURM.
> I tried executing two different setups on the head node, both listing
> two compute nodes in a hostfile:
>
> 1) mpiexec -f hostfile -n 4 ./hello
>
> Since the SLURM PAM module was disabled, SSH login to the nodes
> was possible and the job run as expected, with two ranks on each
> node. SLURM's sinfo showed both nodes as state 'idle' and the
> HYDRA_DEBUG output said '--rmk user --launcher ssh'.
That sounds good.
> 2) mpiexec -f hostfile -rmk slurm -n 4 ./hello
>
> This job ran as well, with the nodes allocated via SLURM and
> shown as 'alloc'. Debug output: '--rmk slurm --launcher slurm'.
>
> Specifying a hostfile with mpiexec within a SLURM batch job also
> worked, but that's obviously not what you normally want to do...
Hmm. -f hostfile and -rmk slurm are contradictory options, since both
are just ways to get the host list. This should throw an error. I'll
add that into Hydra.
> Hope this helps. If there is anything else I should try out to help
> tracking down the issue, please let me know.
I'm still trying to find the exact option that fails. Can you try the
following:
# Use the slurm launcher, and user-specified resources
% mpiexec -f hostfile -launcher slurm -n 4 ./hello
# Use the ssh launcher, and user-specified resources
% mpiexec -f hostfile -launcher ssh -n 4 ./hello
# Use the ssh launcher, and slurm-specified resources
% mpiexec -rmk slurm -launcher ssh -n 4 ./hello
# Use the slurm launcher, and slurm-specified resources
% mpiexec -rmk slurm -launcher slurm -n 4 ./hello
At least one of them should throw the error you reported.
-- Pavan
--
Pavan Balaji
http://www.mcs.anl.gov/~balaji
i used mpiexec with checkpoint and created two checkpoint files:
mpiexec -ckpointlib blcr -ckpoint-prefix /home/john/ckpts/app.ckpoint
-ckpoint-interval 4 -n 4 /home/john/app/md
context-num0-0-0
context-num1-0-0
i am trying to make a restart
mpiexec -ckpointlib blcr -ckpoint-prefix /home/john/ckpts/app.ckpoint -n 4
-ckpoint-num 1
but nothing happened it just hangs
i also tried:
mpiexec -ckpointlib blcr -ckpoint-prefix /home/john/ckpts/app.ckpoint -n 4
-ckpoint-num 0-0-0
also hangs
Hi Pavan,
> Markus: does this happen only with SLURM or can you reproduce this
> without SLURM as well?
It seems to happen only when hydra queries the host list from SLURM.
I tried executing two different setups on the head node, both listing
two compute nodes in a hostfile:
1) mpiexec -f hostfile -n 4 ./hello
Since the SLURM PAM module was disabled, SSH login to the nodes
was possible and the job run as expected, with two ranks on each
node. SLURM's sinfo showed both nodes as state 'idle' and the
HYDRA_DEBUG output said '--rmk user --launcher ssh'.
2) mpiexec -f hostfile -rmk slurm -n 4 ./hello
This job ran as well, with the nodes allocated via SLURM and
shown as 'alloc'. Debug output: '--rmk slurm --launcher slurm'.
Specifying a hostfile with mpiexec within a SLURM batch job also
worked, but that's obviously not what you normally want to do...
Hope this helps. If there is anything else I should try out to help
tracking down the issue, please let me know.
Thanks,
Markus
--
Dr. Markus Geimer
Juelich Supercomputing Centre
Institute for Advanced Simulation
Forschungszentrum Juelich GmbH
52425 Juelich, Germany
Phone: +49-2461-61-1773
Fax: +49-2461-61-6656
E-mail: m.geimer(a)fz-juelich.de
WWW: http://www.fz-juelich.de/jsc/
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
We certainly want to give better error messages where possible.
I've created a ticket for it:
https://trac.mpich.org/projects/mpich/ticket/1872
-- Pavan
On 05/29/2013 04:37 PM, Edscott Wilson wrote:
>
>
>
> 2013/5/29 Jeff Hammond <jhammond(a)alcf.anl.gov
> <mailto:[email protected]>>
>
>
> >
> >
> > Wouldn't a message such as "`pwd` directory does not exist on
> node velascoj"
> > be more illustrative?
>
> Yes. However, the set of improper uses of MPI that could generate
> helpful error messages is uncountable. Do you not think it is a good
> use of finite developer effort to implement an infinitesimal fraction
> of such warnings? There has to be a minimum requirement placed upon
> the user. I personally think that it should include running in a
> directory that actually exists.
>
>
> Certainly! But then again, some developer must have thought it a good
> idea, since under different circumstances, I get:
>
> /bin/bash -c mpiexec -n 1 -hosts tauro,velascoj gmandel
> [proxy:0:0@tauro] launch_procs (./pm/pmiserv/pmip_cb.c:648): unable to
> change wdir to /tmp/edscott/mnt/tauro-home/GIT/gmandel (No such file or
> directory)
> [proxy:0:0@tauro] HYD_pmcd_pmip_control_cmd_cb
> (./pm/pmiserv/pmip_cb.c:893): launch_procs returned error
> [proxy:0:0@tauro] HYDT_dmxu_poll_wait_for_event
> (./tools/demux/demux_poll.c:77): callback returned error status
> [proxy:0:0@tauro] main (./pm/pmiserv/pmip.c:206): demux engine error
> waiting for event
> [mpiexec@velascoj] control_cb (./pm/pmiserv/pmiserv_cb.c:202): assert
> (!closed) failed
> [mpiexec@velascoj] HYDT_dmxu_poll_wait_for_event
> (./tools/demux/demux_poll.c:77): callback returned error status
> [mpiexec@velascoj] HYD_pmci_wait_for_completion
> (./pm/pmiserv/pmiserv_pmci.c:197): error waiting for event
> [mpiexec@velascoj] main (./ui/mpich/mpiexec.c:331): process manager
> error waiting for completion
> Which is inconsistent with the previous behavior. Anyways, its no big deal.
>
> BTW, would you happen to know why a process which is started with
> MPI_Comm_spawn will go into what seems like an active wait after
> MPI_Comm_disconnect and MPI_Finalize has been called? These spawned
> processed will hog up CPU until the parent process exits. Curious
> enough, this behavior is not mirrored in openmpi.
>
> Edscott
>
>
>
> -------------------------------
> Dr. Edscott Wilson Garcia
> Applied Mathematics and Computing
> Mexican Petroleum Institute
>
>
> _______________________________________________
> discuss mailing list discuss(a)mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss
>
--
Pavan Balaji
http://www.mcs.anl.gov/~balaji
Thanks. I'll look into this.
Markus: does this happen only with SLURM or can you reproduce this
without SLURM as well?
-- Pavan
On 06/01/2013 02:39 PM, Rajeev Thakur wrote:
> I have created a ticket for it: http://trac.mpich.org/projects/mpich/ticket/1871
>
> Rajeev
>
> On Jun 1, 2013, at 10:29 AM, Markus Geimer wrote:
>
>> Jeff,
>>
>>> Have you tried MPICH 3.0.4? Hydra has been improved a great deal since
>>> the 2.4 release.
>>
>> As I wrote: We first tried the latest and greatest, i.e., MPICH 3.0.4.
>> Only afterwards we tried MPICH2 1.5 to see whether previous versions
>> behave the same. Unfortunately, both show the same behavior.
>>
>> The predecessor version MPICH2 1.4.1p1, however, works. Therefore we
>> assume that some bug sneaked in between 1.4.1p1 and 1.5.
>>
>> Markus
>>
>> --
>> Dr. Markus Geimer
>> Juelich Supercomputing Centre
>> Institute for Advanced Simulation
>> Forschungszentrum Juelich GmbH
>> 52425 Juelich, Germany
>>
>> Phone: +49-2461-61-1773
>> Fax: +49-2461-61-6656
>> E-mail: m.geimer(a)fz-juelich.de
>> WWW: http://www.fz-juelich.de/jsc/
>>
>>
>> ------------------------------------------------------------------------------------------------
>> ------------------------------------------------------------------------------------------------
>> Forschungszentrum Juelich GmbH
>> 52425 Juelich
>> Sitz der Gesellschaft: Juelich
>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
>> Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
>> Prof. Dr. Sebastian M. Schmidt
>> ------------------------------------------------------------------------------------------------
>> ------------------------------------------------------------------------------------------------
>> _______________________________________________
>> discuss mailing list discuss(a)mpich.org
>> To manage subscription options or unsubscribe:
>> https://lists.mpich.org/mailman/listinfo/discuss
>
> _______________________________________________
> discuss mailing list discuss(a)mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss
>
--
Pavan Balaji
http://www.mcs.anl.gov/~balaji
