July 2015 - discuss - mailman3.cels.anl.gov

hydra, stdin close(), and SLURM
by Aaron Knister 25 Jul '15

25 Jul '15

I sent this off to the mvapich list yesterday and it was suggested I raise it here since this is the upstream project: This is a bit of a cross post from a thread I started on the slurm dev list: http://article.gmane.org/gmane.comp.distributed.slurm.devel/8176 I'd like to get feedback on the idea that "--input none" be passed to srun when using the SLURM hydra bootstrap mechanism. I figured it would be inserted somewhere around here http://trac.mpich.org/projects/mpich/browser/src/pm/hydra/tools/bootstrap/e…. Without this argument I'm getting spurious job aborts and confusing errors. The gist of it is mpiexec.hydra closes stdin before it exec's srun. srun then (possibly via the munge libraries) calls some function that does a look up via nss. We use sssd for AAA so libnss_sssd will handle this request. Part of the caching mechanism sssd uses will cause the library to open() the cache file. The lowest fd available is 0 so the cache file is opened on fd 0. srun then believes it's got stdin attached and it causes the issues outlined in the slurm dev post. I think passing "--input none" is the right thing to do here since hydra has in fact closed stdin to srun. I tested this via the HYDRA_LAUNCHER_EXTRA_ARGS environment variable and it does resolve the errors I described. Thanks! -Aaron -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Announcing the availability of mpich-3.2b4
by Balaji, Pavan 24 Jul '15

24 Jul '15

The MPICH team is pleased to announce the availability of a new preview release, mpich-3.2b4. This represents the final beta release leading up to mpich-3.2. The 3.2b4 release includes a large number of bug fixes compared to the previous 3.2b3 release. Nevertheless, it is still considered a preview release and is not intended for production systems at this time. More information about the release can be found on the MPICH website: http://www.mpich.org A list of changes in this release is included at the end of this email. Regards, The MPICH Team =============================================================================== Changes in 3.2 =============================================================================== # Added support for MPI-3.1 features including nonblocking collective I/O, address manipulation routines, thread-safety for MPI initialization, pre-init functionality, and new MPI_T routines to look up variables by name. # Fortran 2008 bindings are enabled by default and fully supported. # Added support for the Mellanox MXM InfiniBand interface. (thanks to Mellanox for the code contribution). # Added support for the Mellanox HCOLL interface for collectives. (thanks to Mellanox for the code contribution). # Significant stability improvements to the MPICH/portals4 implementation. # Completely revamped RMA infrastructure including several scalability improvements, performance improvements, and bug fixes. # Added experimental support for Open Fabrics Interfaces (OFI) version 1.0.0. https://github.com/ofiwg/libfabric (thanks to Intel for code contribution) # The Myrinet MX network module, which had a life cyle from 1.1 till 3.1.2, has now been deleted. # Several other minor bug fixes, memory leak fixes, and code cleanup. A full list of changes is available at the following link: http://git.mpich.org/mpich.git/shortlog/v3.1.3..v3.2b4 A full list of bugs that have been fixed is available at the following link: https://trac.mpich.org/projects/mpich/query?status=closed&group=resolution&… _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] different processes' output
by Balaji, Pavan 22 Jul '15

22 Jul '15

You can direct them to different files. Try "mpiexec -outfile–pattern –help" - Pavan From: A student from china Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" Date: Wednesday, July 22, 2015 at 9:11 AM To: "discuss(a)mpich.org<mailto:[email protected]>" Subject: [mpich-discuss] different processes' output If we run more than one processes in different computers , every process is forked as a daemon process , but their standard output is redirect to the terminal we execute "mpiexec" command. Is there any way to control each processes's output ? for example, cancel the redirect. thanks _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

different processes' output
by A student from china 22 Jul '15

22 Jul '15

If we run more than one processes in different computers , every process is forked as a daemon process , but their standard output is redirect to the terminal we execute "mpiexec" command. Is there any way to control each processes's output ? for example, cancel the redirect. thanks _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpich hangs in MPI_Send() - multi threaded environment
by Balaji, Pavan 20 Jul '15

20 Jul '15

After reviewing Nenad's code, it looks like this was indeed the issue — the code was assuming that the send would complete before the receive is called. That's obviously an incorrect code, not an issue with MPICH. Bsend could help, but it has other problems. Most importantly, the buffer attach is not communicator-safe. So if another library is using Bsends as well, it might use up your buffer space, and you'll be back in this deadlock mode. A better approach is to not make this assumption if possible. If you are sending to yourself, can you use MPI_Sendrecv instead? Or prepost the receive before calling the send? -- Pavan From: Nenad Vukicevic Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" Date: Monday, July 20, 2015 at 3:41 PM To: Jeff Hammond, "discuss(a)mpich.org<mailto:[email protected]>" Cc: Gary Funck Subject: Re: [mpich-discuss] mpich hangs in MPI_Send() - multi threaded environment Yes. Send did not have a matching tag receive as thread was waiting for something else. Switching to Bsend should help? On 7/17/2015 9:03 PM, Jeff Hammond wrote: Blocking Send to self will hang if Irecv not pre-posted. You doing that? Jeff On Friday, July 17, 2015, Nenad Vukicevic <<mailto:[email protected]>nenad(a)intrepid.com<mailto:[email protected]>> wrote: I will via separate mail. But, I see that I made a mistake in my description as pthread does MPI_Send () to itself and not to another rank. I'll try reversing the order (send to others then to yourself). On 7/17/2015 4:26 PM, Balaji, Pavan wrote: You should not need to pass any additional configuration options. MPICH is thread-safe by default. Can you send us a simple program that reproduces the error? -- Pavan On 7/17/15, 6:16 PM, "Nenad Vukicevic" <nenad(a)intrepid.com<mailto:[email protected]>> wrote: I am having a problem where the system locks up inside the MPI_Send() routine. In my test, each MPI rank has an additional pthread and system locks up when: - main thread does MPI_Recv from ANY rank - pthread does MPI_Send to another rank I verified with MPI_Init_thread() that I can run MPI_THREAD_MULTIPLE environment. The same thing happened on MPICH with Fedora 20 (3.0.4) and the one build from 3.2b3 sources. When building from sources I provided '--enable-threads=multiple' option. I also tried to play with '--enable-thread-cs' option but got build failure when 'per-object' was selected. Is this supposed to work? Thanks, Nenad I am attaching traces form GDB for the rank that locks up. (gdb) info thread Id Target Id Frame 2 Thread 0x7ffff6a5a700 (LWP 29570) "barrier_test" 0x0000003ef040bca0 in pthread_cond_wait@@GLIBC_2.3.2<mailto:pthread_cond_wait@@GLIBC_2.3.2> () from /usr/lib64/libpthread.so.0 * 1 Thread 0x7ffff7a64b80 (LWP 29568) "barrier_test" 0x00007ffff7c58717 in MPIU_Thread_CS_yield_lockname_recursive_impl_ ( lockname=0x7ffff7cdc8b1 "global_mutex", mutex=<optimized out>, kind=MPIU_Nest_global_mutex) at ../src/src/include/mpiimplthreadpost.h:190 (gdb) where #0 0x00007ffff7c58717 in MPIU_Thread_CS_yield_lockname_recursive_impl_ (lockname=0x7ffff7cdc8b1 "global_mutex", mutex=<optimized out>, kind=MPIU_Nest_global_mutex) at ../src/src/include/mpiimplthreadpost.h:190 #1 0x00007ffff7c5db42 in MPIDI_CH3I_Progress (progress_state=progress_state@entry=0x7fffffffd2c0, is_blocking=is_blocking@entry=1) at ../src/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:507 #2 0x00007ffff7b5e795 in PMPI_Recv (buf=0x7fffffffd61c, count=1, datatype=1275069445, source=-2, tag=299, comm=1140850688, status=0x7fffffffd620) at ../src/src/mpi/pt2pt/recv.c:157 #3 0x0000000000401732 in receive_int () at comm.c:52 #4 0x0000000000400bf2 in main (argc=1, argv=0x7fffffffd758) at barrier_test.c:39 (gdb) thread 2 [Switching to thread 2 (Thread 0x7ffff6a5a700 (LWP 29570))] #0 0x0000003ef040bca0 in pthread_cond_wait@@GLIBC_2.3.2<mailto:pthread_cond_wait@@GLIBC_2.3.2> () from /usr/lib64/libpthread.so.0 (gdb) where #0 0x0000003ef040bca0 in pthread_cond_wait@@GLIBC_2.3.2<mailto:pthread_cond_wait@@GLIBC_2.3.2> () from /usr/lib64/libpthread.so.0 #1 0x00007ffff7c5d614 in MPIDI_CH3I_Progress_delay (completion_count=<optimized out>) at ../src/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:566 #2 MPIDI_CH3I_Progress (progress_state=progress_state@entry=0x7ffff6a59710, is_blocking=is_blocking@entry=1) at ../src/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:347 #3 0x00007ffff7b632ec in PMPI_Send (buf=0x7ffff6a5985c, count=1, datatype=1275069445, dest=0, tag=199, comm=1140850688) at ../src/src/mpi/pt2pt/send.c:145 #4 0x0000000000400e42 in barrier_thread_release (id=0) at barrier.c:115 #5 0x0000000000401098 in barrier_helper (arg=0x0) at barrier.c:186 #6 0x0000003ef0407ee5 in start_thread () from /usr/lib64/libpthread.so.0 #7 0x0000003eef8f4d1d in clone () from /usr/lib64/libc.so.6 _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpich hangs in MPI_Send() - multi threaded environment
by Nenad Vukicevic 20 Jul '15

20 Jul '15

Yes. Send did not have a matching tag receive as thread was waiting for something else. Switching to Bsend should help? On 7/17/2015 9:03 PM, Jeff Hammond wrote: > Blocking Send to self will hang if Irecv not pre-posted. You doing that? > > Jeff > > On Friday, July 17, 2015, Nenad Vukicevic <nenad(a)intrepid.com > <mailto:[email protected]>> wrote: > > I will via separate mail. But, I see that I made a mistake in my > description as pthread does MPI_Send () to itself and not to > another rank. > I'll try reversing the order (send to others then to yourself). > > On 7/17/2015 4:26 PM, Balaji, Pavan wrote: > > You should not need to pass any additional configuration > options. MPICH is thread-safe by default. > > Can you send us a simple program that reproduces the error? > > -- Pavan > > > > > > On 7/17/15, 6:16 PM, "Nenad Vukicevic" <nenad(a)intrepid.com> wrote: > > I am having a problem where the system locks up inside the > MPI_Send() > routine. In my test, each MPI rank has an additional > pthread and system > locks up when: > > - main thread does MPI_Recv from ANY rank > - pthread does MPI_Send to another rank > > I verified with MPI_Init_thread() that I can run > MPI_THREAD_MULTIPLE > environment. > > The same thing happened on MPICH with Fedora 20 (3.0.4) > and the one > build from 3.2b3 sources. When building from sources I > provided > '--enable-threads=multiple' option. I also tried to play with > '--enable-thread-cs' option but got build failure when > 'per-object' was > selected. > > Is this supposed to work? > > Thanks, > Nenad > > > I am attaching traces form GDB for the rank that locks up. > > > (gdb) info thread > Id Target Id Frame > 2 Thread 0x7ffff6a5a700 (LWP 29570) "barrier_test" > 0x0000003ef040bca0 in pthread_cond_wait@@GLIBC_2.3.2 () from > /usr/lib64/libpthread.so.0 > * 1 Thread 0x7ffff7a64b80 (LWP 29568) "barrier_test" > 0x00007ffff7c58717 in > MPIU_Thread_CS_yield_lockname_recursive_impl_ ( > lockname=0x7ffff7cdc8b1 "global_mutex", > mutex=<optimized out>, > kind=MPIU_Nest_global_mutex) at > ../src/src/include/mpiimplthreadpost.h:190 > (gdb) where > #0 0x00007ffff7c58717 in > MPIU_Thread_CS_yield_lockname_recursive_impl_ > (lockname=0x7ffff7cdc8b1 "global_mutex", mutex=<optimized > out>, > kind=MPIU_Nest_global_mutex) at > ../src/src/include/mpiimplthreadpost.h:190 > #1 0x00007ffff7c5db42 in MPIDI_CH3I_Progress > (progress_state=progress_state@entry=0x7fffffffd2c0, > is_blocking=is_blocking@entry=1) > at > ../src/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:507 > #2 0x00007ffff7b5e795 in PMPI_Recv (buf=0x7fffffffd61c, > count=1, > datatype=1275069445, source=-2, tag=299, comm=1140850688, > status=0x7fffffffd620) > at ../src/src/mpi/pt2pt/recv.c:157 > #3 0x0000000000401732 in receive_int () at comm.c:52 > #4 0x0000000000400bf2 in main (argc=1, > argv=0x7fffffffd758) at > barrier_test.c:39 > (gdb) thread 2 > [Switching to thread 2 (Thread 0x7ffff6a5a700 (LWP 29570))] > #0 0x0000003ef040bca0 in pthread_cond_wait@@GLIBC_2.3.2 > () from > /usr/lib64/libpthread.so.0 > (gdb) where > #0 0x0000003ef040bca0 in pthread_cond_wait@@GLIBC_2.3.2 > () from > /usr/lib64/libpthread.so.0 > #1 0x00007ffff7c5d614 in MPIDI_CH3I_Progress_delay > (completion_count=<optimized out>) > at > ../src/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:566 > #2 MPIDI_CH3I_Progress > (progress_state=progress_state@entry=0x7ffff6a59710, > is_blocking=is_blocking@entry=1) > at > ../src/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:347 > #3 0x00007ffff7b632ec in PMPI_Send (buf=0x7ffff6a5985c, > count=1, > datatype=1275069445, dest=0, tag=199, comm=1140850688) > at ../src/src/mpi/pt2pt/send.c:145 > #4 0x0000000000400e42 in barrier_thread_release (id=0) at > barrier.c:115 > #5 0x0000000000401098 in barrier_helper (arg=0x0) at > barrier.c:186 > #6 0x0000003ef0407ee5 in start_thread () from > /usr/lib64/libpthread.so.0 > #7 0x0000003eef8f4d1d in clone () from /usr/lib64/libc.so.6 > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Jeff Hammond > jeff.science(a)gmail.com <mailto:[email protected]> > http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpich hangs in MPI_Send() - multi threaded environment
by Balaji, Pavan 18 Jul '15

18 Jul '15

Jeff, That's not needed if the receive is being called from a different thread, like the case described below. -- Pavan From: Jeff Hammond Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" Date: Friday, July 17, 2015 at 11:03 PM To: "discuss(a)mpich.org<mailto:[email protected]>" Cc: Gary Funck Subject: Re: [mpich-discuss] mpich hangs in MPI_Send() - multi threaded environment Blocking Send to self will hang if Irecv not pre-posted. You doing that? Jeff On Friday, July 17, 2015, Nenad Vukicevic <nenad(a)intrepid.com<mailto:[email protected]>> wrote: I will via separate mail. But, I see that I made a mistake in my description as pthread does MPI_Send () to itself and not to another rank. I'll try reversing the order (send to others then to yourself). On 7/17/2015 4:26 PM, Balaji, Pavan wrote: You should not need to pass any additional configuration options. MPICH is thread-safe by default. Can you send us a simple program that reproduces the error? -- Pavan On 7/17/15, 6:16 PM, "Nenad Vukicevic" <nenad(a)intrepid.com> wrote: I am having a problem where the system locks up inside the MPI_Send() routine. In my test, each MPI rank has an additional pthread and system locks up when: - main thread does MPI_Recv from ANY rank - pthread does MPI_Send to another rank I verified with MPI_Init_thread() that I can run MPI_THREAD_MULTIPLE environment. The same thing happened on MPICH with Fedora 20 (3.0.4) and the one build from 3.2b3 sources. When building from sources I provided '--enable-threads=multiple' option. I also tried to play with '--enable-thread-cs' option but got build failure when 'per-object' was selected. Is this supposed to work? Thanks, Nenad I am attaching traces form GDB for the rank that locks up. (gdb) info thread Id Target Id Frame 2 Thread 0x7ffff6a5a700 (LWP 29570) "barrier_test" 0x0000003ef040bca0 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0 * 1 Thread 0x7ffff7a64b80 (LWP 29568) "barrier_test" 0x00007ffff7c58717 in MPIU_Thread_CS_yield_lockname_recursive_impl_ ( lockname=0x7ffff7cdc8b1 "global_mutex", mutex=<optimized out>, kind=MPIU_Nest_global_mutex) at ../src/src/include/mpiimplthreadpost.h:190 (gdb) where #0 0x00007ffff7c58717 in MPIU_Thread_CS_yield_lockname_recursive_impl_ (lockname=0x7ffff7cdc8b1 "global_mutex", mutex=<optimized out>, kind=MPIU_Nest_global_mutex) at ../src/src/include/mpiimplthreadpost.h:190 #1 0x00007ffff7c5db42 in MPIDI_CH3I_Progress (progress_state=progress_state@entry=0x7fffffffd2c0, is_blocking=is_blocking@entry=1) at ../src/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:507 #2 0x00007ffff7b5e795 in PMPI_Recv (buf=0x7fffffffd61c, count=1, datatype=1275069445, source=-2, tag=299, comm=1140850688, status=0x7fffffffd620) at ../src/src/mpi/pt2pt/recv.c:157 #3 0x0000000000401732 in receive_int () at comm.c:52 #4 0x0000000000400bf2 in main (argc=1, argv=0x7fffffffd758) at barrier_test.c:39 (gdb) thread 2 [Switching to thread 2 (Thread 0x7ffff6a5a700 (LWP 29570))] #0 0x0000003ef040bca0 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0 (gdb) where #0 0x0000003ef040bca0 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0 #1 0x00007ffff7c5d614 in MPIDI_CH3I_Progress_delay (completion_count=<optimized out>) at ../src/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:566 #2 MPIDI_CH3I_Progress (progress_state=progress_state@entry=0x7ffff6a59710, is_blocking=is_blocking@entry=1) at ../src/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:347 #3 0x00007ffff7b632ec in PMPI_Send (buf=0x7ffff6a5985c, count=1, datatype=1275069445, dest=0, tag=199, comm=1140850688) at ../src/src/mpi/pt2pt/send.c:145 #4 0x0000000000400e42 in barrier_thread_release (id=0) at barrier.c:115 #5 0x0000000000401098 in barrier_helper (arg=0x0) at barrier.c:186 #6 0x0000003ef0407ee5 in start_thread () from /usr/lib64/libpthread.so.0 #7 0x0000003eef8f4d1d in clone () from /usr/lib64/libc.so.6 _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpich hangs in MPI_Send() - multi threaded environment
by Jeff Hammond 17 Jul '15

17 Jul '15

Blocking Send to self will hang if Irecv not pre-posted. You doing that? Jeff On Friday, July 17, 2015, Nenad Vukicevic <nenad(a)intrepid.com> wrote: > I will via separate mail. But, I see that I made a mistake in my > description as pthread does MPI_Send () to itself and not to another rank. > I'll try reversing the order (send to others then to yourself). > > On 7/17/2015 4:26 PM, Balaji, Pavan wrote: > >> You should not need to pass any additional configuration options. MPICH >> is thread-safe by default. >> >> Can you send us a simple program that reproduces the error? >> >> -- Pavan >> >> >> >> >> >> On 7/17/15, 6:16 PM, "Nenad Vukicevic" <nenad(a)intrepid.com> wrote: >> >> I am having a problem where the system locks up inside the MPI_Send() >>> routine. In my test, each MPI rank has an additional pthread and system >>> locks up when: >>> >>> - main thread does MPI_Recv from ANY rank >>> - pthread does MPI_Send to another rank >>> >>> I verified with MPI_Init_thread() that I can run MPI_THREAD_MULTIPLE >>> environment. >>> >>> The same thing happened on MPICH with Fedora 20 (3.0.4) and the one >>> build from 3.2b3 sources. When building from sources I provided >>> '--enable-threads=multiple' option. I also tried to play with >>> '--enable-thread-cs' option but got build failure when 'per-object' was >>> selected. >>> >>> Is this supposed to work? >>> >>> Thanks, >>> Nenad >>> >>> >>> I am attaching traces form GDB for the rank that locks up. >>> >>> >>> (gdb) info thread >>> Id Target Id Frame >>> 2 Thread 0x7ffff6a5a700 (LWP 29570) "barrier_test" >>> 0x0000003ef040bca0 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /usr/lib64/libpthread.so.0 >>> * 1 Thread 0x7ffff7a64b80 (LWP 29568) "barrier_test" >>> 0x00007ffff7c58717 in MPIU_Thread_CS_yield_lockname_recursive_impl_ ( >>> lockname=0x7ffff7cdc8b1 "global_mutex", mutex=<optimized out>, >>> kind=MPIU_Nest_global_mutex) at >>> ../src/src/include/mpiimplthreadpost.h:190 >>> (gdb) where >>> #0 0x00007ffff7c58717 in MPIU_Thread_CS_yield_lockname_recursive_impl_ >>> (lockname=0x7ffff7cdc8b1 "global_mutex", mutex=<optimized out>, >>> kind=MPIU_Nest_global_mutex) at >>> ../src/src/include/mpiimplthreadpost.h:190 >>> #1 0x00007ffff7c5db42 in MPIDI_CH3I_Progress >>> (progress_state=progress_state@entry=0x7fffffffd2c0, >>> is_blocking=is_blocking@entry=1) >>> at ../src/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:507 >>> #2 0x00007ffff7b5e795 in PMPI_Recv (buf=0x7fffffffd61c, count=1, >>> datatype=1275069445, source=-2, tag=299, comm=1140850688, >>> status=0x7fffffffd620) >>> at ../src/src/mpi/pt2pt/recv.c:157 >>> #3 0x0000000000401732 in receive_int () at comm.c:52 >>> #4 0x0000000000400bf2 in main (argc=1, argv=0x7fffffffd758) at >>> barrier_test.c:39 >>> (gdb) thread 2 >>> [Switching to thread 2 (Thread 0x7ffff6a5a700 (LWP 29570))] >>> #0 0x0000003ef040bca0 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /usr/lib64/libpthread.so.0 >>> (gdb) where >>> #0 0x0000003ef040bca0 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /usr/lib64/libpthread.so.0 >>> #1 0x00007ffff7c5d614 in MPIDI_CH3I_Progress_delay >>> (completion_count=<optimized out>) >>> at ../src/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:566 >>> #2 MPIDI_CH3I_Progress >>> (progress_state=progress_state@entry=0x7ffff6a59710, >>> is_blocking=is_blocking@entry=1) >>> at ../src/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:347 >>> #3 0x00007ffff7b632ec in PMPI_Send (buf=0x7ffff6a5985c, count=1, >>> datatype=1275069445, dest=0, tag=199, comm=1140850688) >>> at ../src/src/mpi/pt2pt/send.c:145 >>> #4 0x0000000000400e42 in barrier_thread_release (id=0) at barrier.c:115 >>> #5 0x0000000000401098 in barrier_helper (arg=0x0) at barrier.c:186 >>> #6 0x0000003ef0407ee5 in start_thread () from /usr/lib64/libpthread.so.0 >>> #7 0x0000003eef8f4d1d in clone () from /usr/lib64/libc.so.6 >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpich hangs in MPI_Send() - multi threaded environment
by Nenad Vukicevic 17 Jul '15

17 Jul '15

I will via separate mail. But, I see that I made a mistake in my description as pthread does MPI_Send () to itself and not to another rank. I'll try reversing the order (send to others then to yourself). On 7/17/2015 4:26 PM, Balaji, Pavan wrote: > You should not need to pass any additional configuration options. MPICH is thread-safe by default. > > Can you send us a simple program that reproduces the error? > > -- Pavan > > > > > > On 7/17/15, 6:16 PM, "Nenad Vukicevic" <nenad(a)intrepid.com> wrote: > >> I am having a problem where the system locks up inside the MPI_Send() >> routine. In my test, each MPI rank has an additional pthread and system >> locks up when: >> >> - main thread does MPI_Recv from ANY rank >> - pthread does MPI_Send to another rank >> >> I verified with MPI_Init_thread() that I can run MPI_THREAD_MULTIPLE >> environment. >> >> The same thing happened on MPICH with Fedora 20 (3.0.4) and the one >> build from 3.2b3 sources. When building from sources I provided >> '--enable-threads=multiple' option. I also tried to play with >> '--enable-thread-cs' option but got build failure when 'per-object' was >> selected. >> >> Is this supposed to work? >> >> Thanks, >> Nenad >> >> >> I am attaching traces form GDB for the rank that locks up. >> >> >> (gdb) info thread >> Id Target Id Frame >> 2 Thread 0x7ffff6a5a700 (LWP 29570) "barrier_test" >> 0x0000003ef040bca0 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /usr/lib64/libpthread.so.0 >> * 1 Thread 0x7ffff7a64b80 (LWP 29568) "barrier_test" >> 0x00007ffff7c58717 in MPIU_Thread_CS_yield_lockname_recursive_impl_ ( >> lockname=0x7ffff7cdc8b1 "global_mutex", mutex=<optimized out>, >> kind=MPIU_Nest_global_mutex) at ../src/src/include/mpiimplthreadpost.h:190 >> (gdb) where >> #0 0x00007ffff7c58717 in MPIU_Thread_CS_yield_lockname_recursive_impl_ >> (lockname=0x7ffff7cdc8b1 "global_mutex", mutex=<optimized out>, >> kind=MPIU_Nest_global_mutex) at >> ../src/src/include/mpiimplthreadpost.h:190 >> #1 0x00007ffff7c5db42 in MPIDI_CH3I_Progress >> (progress_state=progress_state@entry=0x7fffffffd2c0, >> is_blocking=is_blocking@entry=1) >> at ../src/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:507 >> #2 0x00007ffff7b5e795 in PMPI_Recv (buf=0x7fffffffd61c, count=1, >> datatype=1275069445, source=-2, tag=299, comm=1140850688, >> status=0x7fffffffd620) >> at ../src/src/mpi/pt2pt/recv.c:157 >> #3 0x0000000000401732 in receive_int () at comm.c:52 >> #4 0x0000000000400bf2 in main (argc=1, argv=0x7fffffffd758) at >> barrier_test.c:39 >> (gdb) thread 2 >> [Switching to thread 2 (Thread 0x7ffff6a5a700 (LWP 29570))] >> #0 0x0000003ef040bca0 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /usr/lib64/libpthread.so.0 >> (gdb) where >> #0 0x0000003ef040bca0 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /usr/lib64/libpthread.so.0 >> #1 0x00007ffff7c5d614 in MPIDI_CH3I_Progress_delay >> (completion_count=<optimized out>) >> at ../src/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:566 >> #2 MPIDI_CH3I_Progress >> (progress_state=progress_state@entry=0x7ffff6a59710, >> is_blocking=is_blocking@entry=1) >> at ../src/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:347 >> #3 0x00007ffff7b632ec in PMPI_Send (buf=0x7ffff6a5985c, count=1, >> datatype=1275069445, dest=0, tag=199, comm=1140850688) >> at ../src/src/mpi/pt2pt/send.c:145 >> #4 0x0000000000400e42 in barrier_thread_release (id=0) at barrier.c:115 >> #5 0x0000000000401098 in barrier_helper (arg=0x0) at barrier.c:186 >> #6 0x0000003ef0407ee5 in start_thread () from /usr/lib64/libpthread.so.0 >> #7 0x0000003eef8f4d1d in clone () from /usr/lib64/libc.so.6 >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpich hangs in MPI_Send() - multi threaded environment
by Balaji, Pavan 17 Jul '15

17 Jul '15

You should not need to pass any additional configuration options. MPICH is thread-safe by default. Can you send us a simple program that reproduces the error? -- Pavan On 7/17/15, 6:16 PM, "Nenad Vukicevic" <nenad(a)intrepid.com> wrote: >I am having a problem where the system locks up inside the MPI_Send() >routine. In my test, each MPI rank has an additional pthread and system >locks up when: > >- main thread does MPI_Recv from ANY rank >- pthread does MPI_Send to another rank > >I verified with MPI_Init_thread() that I can run MPI_THREAD_MULTIPLE >environment. > >The same thing happened on MPICH with Fedora 20 (3.0.4) and the one >build from 3.2b3 sources. When building from sources I provided >'--enable-threads=multiple' option. I also tried to play with >'--enable-thread-cs' option but got build failure when 'per-object' was >selected. > >Is this supposed to work? > >Thanks, >Nenad > > >I am attaching traces form GDB for the rank that locks up. > > >(gdb) info thread > Id Target Id Frame > 2 Thread 0x7ffff6a5a700 (LWP 29570) "barrier_test" >0x0000003ef040bca0 in pthread_cond_wait@@GLIBC_2.3.2 () from >/usr/lib64/libpthread.so.0 >* 1 Thread 0x7ffff7a64b80 (LWP 29568) "barrier_test" >0x00007ffff7c58717 in MPIU_Thread_CS_yield_lockname_recursive_impl_ ( > lockname=0x7ffff7cdc8b1 "global_mutex", mutex=<optimized out>, >kind=MPIU_Nest_global_mutex) at ../src/src/include/mpiimplthreadpost.h:190 >(gdb) where >#0 0x00007ffff7c58717 in MPIU_Thread_CS_yield_lockname_recursive_impl_ >(lockname=0x7ffff7cdc8b1 "global_mutex", mutex=<optimized out>, > kind=MPIU_Nest_global_mutex) at >../src/src/include/mpiimplthreadpost.h:190 >#1 0x00007ffff7c5db42 in MPIDI_CH3I_Progress >(progress_state=progress_state@entry=0x7fffffffd2c0, >is_blocking=is_blocking@entry=1) > at ../src/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:507 >#2 0x00007ffff7b5e795 in PMPI_Recv (buf=0x7fffffffd61c, count=1, >datatype=1275069445, source=-2, tag=299, comm=1140850688, >status=0x7fffffffd620) > at ../src/src/mpi/pt2pt/recv.c:157 >#3 0x0000000000401732 in receive_int () at comm.c:52 >#4 0x0000000000400bf2 in main (argc=1, argv=0x7fffffffd758) at >barrier_test.c:39 >(gdb) thread 2 >[Switching to thread 2 (Thread 0x7ffff6a5a700 (LWP 29570))] >#0 0x0000003ef040bca0 in pthread_cond_wait@@GLIBC_2.3.2 () from >/usr/lib64/libpthread.so.0 >(gdb) where >#0 0x0000003ef040bca0 in pthread_cond_wait@@GLIBC_2.3.2 () from >/usr/lib64/libpthread.so.0 >#1 0x00007ffff7c5d614 in MPIDI_CH3I_Progress_delay >(completion_count=<optimized out>) > at ../src/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:566 >#2 MPIDI_CH3I_Progress >(progress_state=progress_state@entry=0x7ffff6a59710, >is_blocking=is_blocking@entry=1) > at ../src/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:347 >#3 0x00007ffff7b632ec in PMPI_Send (buf=0x7ffff6a5985c, count=1, >datatype=1275069445, dest=0, tag=199, comm=1140850688) > at ../src/src/mpi/pt2pt/send.c:145 >#4 0x0000000000400e42 in barrier_thread_release (id=0) at barrier.c:115 >#5 0x0000000000401098 in barrier_helper (arg=0x0) at barrier.c:186 >#6 0x0000003ef0407ee5 in start_thread () from /usr/lib64/libpthread.so.0 >#7 0x0000003eef8f4d1d in clone () from /usr/lib64/libc.so.6 >_______________________________________________ >discuss mailing list discuss(a)mpich.org >To manage subscription options or unsubscribe: >https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0