June 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPICH2 logging on windows
by Jayesh Krishna 04 Jun '13

04 Jun '13

Hi, Can you send us a simple test case (extracted from your code) that fails? Regards, Jayesh ----- Original Message ----- From: "bruno guerraz" <bruno.guerraz(a)orange.com> To: discuss(a)mpich.org Sent: Tuesday, June 4, 2013 5:10:49 AM Subject: [mpich-discuss] MPICH2 logging on windows Hi, I try to use MPE logging on my mpi program on Windows. It works fine on the cpi example. I have only linked my program with mpe.lib and run on a single machine with –localonly –log options. I use dynamic process via MPI_Spawn. The program crashed on MPI_Intercomm_merge with the log ------------------------------------------------------- .\src\logging\src\clog_commset.c:CLOG_CommSet_get_IDs() - PMPI_Comm_get_attr() fails! Backtrace of the callstack at rank 0: unable to read the cmd header on the pmi context, Error = -1 I really appreciate any help Bruno _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, France Telecom - Orange is not liable for messages that have been modified, changed or falsified. Thank you. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPICH2 logging on windows
by bruno.guerraz＠orange.com 04 Jun '13

04 Jun '13

Hi, I try to use MPE logging on my mpi program on Windows. It works fine on the cpi example. I have only linked my program with mpe.lib and run on a single machine with -localonly -log options. I use dynamic process via MPI_Spawn. The program crashed on MPI_Intercomm_merge with the log ------------------------------------------------------- .\src\logging\src\clog_commset.c:CLOG_CommSet_get_IDs() - PMPI_Comm_get_attr() fails! Backtrace of the callstack at rank 0: unable to read the cmd header on the pmi context, Error = -1 I really appreciate any help Bruno _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, France Telecom - Orange is not liable for messages that have been modified, changed or falsified. Thank you.

1 0

ckpoint-num error
by john donald 03 Jun '13

03 Jun '13

i used mpiexec with checkpoint and created two checkpoint files: mpiexec -ckpointlib blcr -ckpoint-prefix /home/john/ckpts/app.ckpoint -ckpoint-interval 4 -n 4 /home/john/app/md context-num0-0-0 context-num1-0-0 i am trying to make a restart mpiexec -ckpointlib blcr -ckpoint-prefix /home/john/ckpts/app.ckpoint -n 4 -ckpoint-num 1 but nothing happened it just hangs i also tried: mpiexec -ckpointlib blcr -ckpoint-prefix /home/john/ckpts/app.ckpoint -n 4 -ckpoint-num 0-0-0 also hangs

1 0

Re: [mpich-discuss] Problems running MPICH jobs under SLURM
by Markus Geimer 03 Jun '13

03 Jun '13

Pavan, > 1. Can you run your application processes using "ddd" or some other > debugger to see where the double free is coming from? You might have to > build mpich with --enable-g=dbg to get the debug symbols in. Here is the full stack backtrace: ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- 8< ----- #0 0x00007ffff6deb475 in *__GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x00007ffff6dee6f0 in *__GI_abort () at abort.c:92 #2 0x00007ffff6e2652b in __libc_message (do_abort=<optimized out>, fmt=<optimized out>) at ../sysdeps/unix/sysv/linux/libc_fatal.c:189 #3 0x00007ffff6e2fd76 in malloc_printerr (action=3, str=0x7ffff6f081e0 "double free or corruption (fasttop)", ptr=<optimized out>) at malloc.c:6283 #4 0x00007ffff6e34aac in *__GI___libc_free (mem=<optimized out>) at malloc.c:3738 #5 0x00007ffff7a1d5d9 in populate_ids_from_mapping ( did_map=<synthetic pointer>, num_nodes=<synthetic pointer>, mapping=<optimized out>, pg=<optimized out>) at src/mpid/ch3/src/mpid_vc.c:1063 #6 MPIDI_Populate_vc_node_ids (pg=pg@entry=0x604910, our_pg_rank=our_pg_rank@entry=0) at src/mpid/ch3/src/mpid_vc.c:1193 #7 0x00007ffff7a17dd6 in MPID_Init (argc=argc@entry=0x7fffffffd97c, argv=argv@entry=0x7fffffffd970, requested=requested@entry=0, provided=provided@entry=0x7fffffffd8e8, has_args=has_args@entry=0x7fffffffd8e0, has_env=has_env@entry=0x7fffffffd8e4) at src/mpid/ch3/src/mpid_init.c:156 #8 0x00007ffff7acdf7f in MPIR_Init_thread (argc=argc@entry=0x7fffffffd97c, argv=argv@entry=0x7fffffffd970, required=required@entry=0, provided=provided@entry=0x7fffffffd944) at src/mpi/init/initthread.c:431 #9 0x00007ffff7acd90e in PMPI_Init (argc=0x7fffffffd97c, argv=0x7fffffffd970) at src/mpi/init/init.c:136 #10 0x000000000040086d in main () ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 ----- >8 ----- > 2. Can you send me the output with the ssh launcher as well? See mail sent off-list. Thanks, Markus -- Dr. Markus Geimer Juelich Supercomputing Centre Institute for Advanced Simulation Forschungszentrum Juelich GmbH 52425 Juelich, Germany Phone: +49-2461-61-1773 Fax: +49-2461-61-6656 E-mail: m.geimer(a)fz-juelich.de WWW: http://www.fz-juelich.de/jsc/ ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

1 0

Re: [mpich-discuss] Problems running MPICH jobs under SLURM
by Pavan Balaji 03 Jun '13

03 Jun '13

On 06/03/2013 01:57 AM, Markus Geimer wrote: > (reply intentionally not sent to the list -- I don't like such logs > to show up in mailing list archives...) Ok, I'm cc'ing discuss(a)mpich.org back. >> Just to make sure, are these the only configure options you are using: >> >> --prefix=... --enable-shared --enable-debuginfo > > Yes, these are the only options (besides explicitly specifying the > GNU compiler, but this shouldn't do any harm). Please find the full > configure log attached. Thanks. >> Also, can you run mpiexec with the -verbose option for one of the >> failing tests (probably just mpiexec -n 4 ./hello) and send me the output? > > Output attached. Hmm. The double free error seems to be coming from the executable, rather than from mpiexec or the proxy. So we might be looking in the wrong place. 1. Can you run your application processes using "ddd" or some other debugger to see where the double free is coming from? You might have to build mpich with --enable-g=dbg to get the debug symbols in. 2. Can you send me the output with the ssh launcher as well? I want to see if there are any critical differences in the environment variables being propagated (e.g., LD_LIBRARY_PATH/LD_PRELOAD) that might affect shared library builds. Feel free to send me the logs off-list. Thanks, -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Problems running MPICH jobs under SLURM
by Pavan Balaji 02 Jun '13

02 Jun '13

Markus, I'm not able to reproduce this at all. Looking through the code didn't give any information either. Just to make sure, are these the only configure options you are using: --prefix=... --enable-shared --enable-debuginfo Also, can you run mpiexec with the -verbose option for one of the failing tests (probably just mpiexec -n 4 ./hello) and send me the output? -- Pavan On 06/02/2013 02:46 PM, Markus Geimer wrote: > Hi Pavan, > >> Can you try the following: >> >> # Use the slurm launcher, and user-specified resources >> % mpiexec -f hostfile -launcher slurm -n 4 ./hello > > Works. > >> # Use the ssh launcher, and user-specified resources >> % mpiexec -f hostfile -launcher ssh -n 4 ./hello > > Works. > >> # Use the ssh launcher, and slurm-specified resources >> % mpiexec -rmk slurm -launcher ssh -n 4 ./hello > > Fails. > >> # Use the slurm launcher, and slurm-specified resources >> % mpiexec -rmk slurm -launcher slurm -n 4 ./hello > > Fails. > > So it seems as if the SLURM-provided host list is somehow causing > trouble... > > Markus > > -- > Dr. Markus Geimer > Juelich Supercomputing Centre > Institute for Advanced Simulation > Forschungszentrum Juelich GmbH > 52425 Juelich, Germany > > Phone: +49-2461-61-1773 > Fax: +49-2461-61-6656 > E-Mail: m.geimer(a)fz-juelich.de > WWW: http://www.fz-juelich.de/ias/jsc > > > ------------------------------------------------------------------------------------------------ > ------------------------------------------------------------------------------------------------ > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > ------------------------------------------------------------------------------------------------ > ------------------------------------------------------------------------------------------------ > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Problems running MPICH jobs under SLURM
by Markus Geimer 02 Jun '13

02 Jun '13

Hi Pavan, > Can you try the following: > > # Use the slurm launcher, and user-specified resources > % mpiexec -f hostfile -launcher slurm -n 4 ./hello Works. > # Use the ssh launcher, and user-specified resources > % mpiexec -f hostfile -launcher ssh -n 4 ./hello Works. > # Use the ssh launcher, and slurm-specified resources > % mpiexec -rmk slurm -launcher ssh -n 4 ./hello Fails. > # Use the slurm launcher, and slurm-specified resources > % mpiexec -rmk slurm -launcher slurm -n 4 ./hello Fails. So it seems as if the SLURM-provided host list is somehow causing trouble... Markus -- Dr. Markus Geimer Juelich Supercomputing Centre Institute for Advanced Simulation Forschungszentrum Juelich GmbH 52425 Juelich, Germany Phone: +49-2461-61-1773 Fax: +49-2461-61-6656 E-Mail: m.geimer(a)fz-juelich.de WWW: http://www.fz-juelich.de/ias/jsc ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

1 0

Re: [mpich-discuss] Problems running MPICH jobs under SLURM
by Pavan Balaji 02 Jun '13

02 Jun '13

Hi Markus, On 06/02/2013 04:31 AM, Markus Geimer wrote: >> Markus: does this happen only with SLURM or can you reproduce this >> without SLURM as well? > > It seems to happen only when hydra queries the host list from SLURM. > I tried executing two different setups on the head node, both listing > two compute nodes in a hostfile: > > 1) mpiexec -f hostfile -n 4 ./hello > > Since the SLURM PAM module was disabled, SSH login to the nodes > was possible and the job run as expected, with two ranks on each > node. SLURM's sinfo showed both nodes as state 'idle' and the > HYDRA_DEBUG output said '--rmk user --launcher ssh'. That sounds good. > 2) mpiexec -f hostfile -rmk slurm -n 4 ./hello > > This job ran as well, with the nodes allocated via SLURM and > shown as 'alloc'. Debug output: '--rmk slurm --launcher slurm'. > > Specifying a hostfile with mpiexec within a SLURM batch job also > worked, but that's obviously not what you normally want to do... Hmm. -f hostfile and -rmk slurm are contradictory options, since both are just ways to get the host list. This should throw an error. I'll add that into Hydra. > Hope this helps. If there is anything else I should try out to help > tracking down the issue, please let me know. I'm still trying to find the exact option that fails. Can you try the following: # Use the slurm launcher, and user-specified resources % mpiexec -f hostfile -launcher slurm -n 4 ./hello # Use the ssh launcher, and user-specified resources % mpiexec -f hostfile -launcher ssh -n 4 ./hello # Use the ssh launcher, and slurm-specified resources % mpiexec -rmk slurm -launcher ssh -n 4 ./hello # Use the slurm launcher, and slurm-specified resources % mpiexec -rmk slurm -launcher slurm -n 4 ./hello At least one of them should throw the error you reported. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

ckpoint-num error
by john donald 02 Jun '13

02 Jun '13

i used mpiexec with checkpoint and created two checkpoint files: mpiexec -ckpointlib blcr -ckpoint-prefix /home/john/ckpts/app.ckpoint -ckpoint-interval 4 -n 4 /home/john/app/md context-num0-0-0 context-num1-0-0 i am trying to make a restart mpiexec -ckpointlib blcr -ckpoint-prefix /home/john/ckpts/app.ckpoint -n 4 -ckpoint-num 1 but nothing happened it just hangs i also tried: mpiexec -ckpointlib blcr -ckpoint-prefix /home/john/ckpts/app.ckpoint -n 4 -ckpoint-num 0-0-0 also hangs

1 0

Re: [mpich-discuss] Problems running MPICH jobs under SLURM
by Markus Geimer 02 Jun '13

02 Jun '13

Hi Pavan, > Markus: does this happen only with SLURM or can you reproduce this > without SLURM as well? It seems to happen only when hydra queries the host list from SLURM. I tried executing two different setups on the head node, both listing two compute nodes in a hostfile: 1) mpiexec -f hostfile -n 4 ./hello Since the SLURM PAM module was disabled, SSH login to the nodes was possible and the job run as expected, with two ranks on each node. SLURM's sinfo showed both nodes as state 'idle' and the HYDRA_DEBUG output said '--rmk user --launcher ssh'. 2) mpiexec -f hostfile -rmk slurm -n 4 ./hello This job ran as well, with the nodes allocated via SLURM and shown as 'alloc'. Debug output: '--rmk slurm --launcher slurm'. Specifying a hostfile with mpiexec within a SLURM batch job also worked, but that's obviously not what you normally want to do... Hope this helps. If there is anything else I should try out to help tracking down the issue, please let me know. Thanks, Markus -- Dr. Markus Geimer Juelich Supercomputing Centre Institute for Advanced Simulation Forschungszentrum Juelich GmbH 52425 Juelich, Germany Phone: +49-2461-61-1773 Fax: +49-2461-61-6656 E-mail: m.geimer(a)fz-juelich.de WWW: http://www.fz-juelich.de/jsc/ ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

1 0