On Tue, Aug 16, 2016 at 7:18 AM, Mark Davis <[email protected]> wrote:

Hello, I'm hitting a data race and a deconstruction error when using MPI_Bcast in an MPI application that also uses multiple threads per process. I'll go into detail below, but the fundamental question I have is: what can I assume, if anything, about the state of an MPI buffer during an MPI call? (I do realize that the buffers are not declared const in MPI_Bcast, whereas in other situations, such as MPI_Send, they are.)

Nothing, unless the buffer pointer has the const attribute on it. At the non-root processes, MPI_Bcast behaves like a MPI_Recv call, and thus you cannot touch the buffer until the function returned. Multiple threads cannot use the same buffer to multiple MPI_Bcast calls. Jeff

Background: running on Linux 3.13.0-52 / Ubuntu x86_64. gcc 5.3. MPICH 3.2 release version with thread support enabled. I am MPI_Initing with thread multiple support. I'm also using thread sanitizer on my application (although not on MPICH itself). C++11 standard.

Problem description: I'm running into a SIGABRT at the end of the program when MPICH seems to be running its deconstructors. My program is running NPROC MPI processes, each with T C++11 threads (so, NPROC * T threads total). This simplified application (which I've created to exhibit this problem) simply has one root looping through H times, doing a MPI_Bcast to other processes each time. Only one thread per process participates in the MPI_Bcast. Thread sanitizer is also detecting a data race. In fact, thread sanitizer is detecting that MPI_Bcast is *writing* to the root's buffer, even though in a broadcast, at least semantically, the root should only ever be sending data, not receiving data. Taking a quick look at MPI_Bcast (which ultimately calls to MPIDI_CH3U_Receive_data_found which invokes the memcpy), it does seem that depending on the message size (in my case, it's just 10 MPI_INTs per message), scatter and allgather can be used or a binomial tree algorithm. I haven't dug in to see which one is being used in my case, but this indicates that there's at least a possibility that the root's buffer can be received into during a MPI_Bcast. My program is producing the "right answer" but could just be luck.

Here's the basic structure of the program:

if root process if root thread A: signal using condition variable to (T-1) non-root threads that it can read the buffer (via shared memory) B: MPI_Bcast to other procs else if not root thread wait on condition variable until it's ok to read buffer read buffer else if not root process // AFAIK there are no problems with non-root processes MPI_Bcast as a non-root

Thread sanitizer is detecting that on the root process, MPI_Bcast is writing to the buffer that is being broadcast. Simultaneously, non-root threads in the root process are reading the buffer being broadcast *while the MPI_Bcast is happening*. When I change the order of statements A and B above, the data race goes away. So, it seems to me that my assumption (that on the root node, the buffer being broadcast is being written, or at least it's a possibility that the MPI call will write the buffer) is incorrect. (FYI, the "writing" that's happening in MPI_Bcast is coming from MPIDI_CH3U_Receive_data_found src/mpid/ch3/src/ch3u_handle_recv_pkt.c:152.)

Also, at the end of the program, I'm hitting a SIGABRT during the deconstruction of something in libmpi (__do_global_dtors_aux). Full backtrace below. This issue also goes away when I reverse the order of statements A and B. I imagine I'm corrupting some state in MPICH, but I'm not sure.

I should point out that I'd prefer to have the ordering of A and B as above, so some threads can make progress while the MPI_Bcast is happening.

So, my questions are:

1. what assumptions can be made about buffers in general in MPI during MPI operations? Does the standard specify anything about this? No reading non-const buffers during MPI operations, or can I do this in some situations?

2. Is there any way for me to achieve what I'm trying to do (above, where the reading of the buffer is happening simultaneously with an MPI operation that shouldn't need to write the user buffer but does)? This would be very helpful to know.

Thank you.

Program received signal SIGABRT, Aborted. [Switching to Thread 0x7ff937bf8700 (LWP 32001)] 0x00007ff93fd17c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56

(gdb) bt full #0 0x00007ff93fd17c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 resultvar = 0 pid = 31987 selftid = 32001 #1 0x00007ff93fd1b028 in __GI_abort () at abort.c:89 save_stage = 2 act = {__sigaction_handler = {sa_handler = 0xbea6495, sa_sigaction = 0xbea6495}, sa_mask = {__val = {140708540140376, 140708494113584, 134272, 0, 140708494597155, 134240, 140708497979320, 131072, 140708494598025, 140708358448000, 140708521085690, 140708539470768, 0, 140708540266640, 140708551504800, 1}}, sa_flags = 934869088, sa_restorer = 0x1} sigs = {__val = {32, 0 <repeats 15 times>}} #2 0x00007ff93fd62dfa in malloc_printerr (ptr=<optimized out>, str=<optimized out>, action=<optimized out>) at malloc.c:5000 No locals. #3 free_check (mem=<optimized out>, caller=<optimized out>) at hooks.c:298 p = <optimized out> #4 0x00007ff93fd1d53a in __cxa_finalize (d=0x7ff941207708) at cxa_finalize.c:56 check = 97 cxafn = <optimized out> cxaarg = <optimized out> f = 0x7ff9400a1230 <initial+944> funcs = 0x7ff9400a0e80 <initial> #5 0x00007ff940ff7123 in __do_global_dtors_aux () from mpich/debug/lib/libmpicxx.so.12 No symbol table info available. #6 0x00007ff937b8e410 in ?? () No symbol table info available. #7 0x00007ff9426e270a in _dl_fini () at dl-fini.c:252 array = 0x7ff941205238 i = 0 nmaps = 32761 nloaded = <optimized out> i = 4 l = 0x7ff9428d7a00 ns = 140708494300474 maps = 0x7ff937b8e330 maps_size = 140708497985152 do_audit = 1116568064 __PRETTY_FUNCTION__ = "_dl_fini" _______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

-- Jeff Hammond [email protected] http://jeffhammond.github.io/ _______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss