At the non-root processes, MPI_Bcast behaves like a MPI_Recv call, and thus you cannot touch the buffer until the function returned.
That part makes sense. I'm not allowing the buffer to be read or otherwise used on non-root threads. It makes sense to me that this acts as a MPI_Recv call. The thing that I'm confused by is on the root process, as it seems that the root process' buffer is also written to during the course of the MPI_Bcast; it should act like an MPI_Send. It seems like this is just an implementation detail, and as you pointed out, since the MPI_Bcast buffer is not marked const, anything could happen.
Background: running on Linux 3.13.0-52 / Ubuntu x86_64. gcc 5.3. MPICH 3.2 release version with thread support enabled. I am MPI_Initing with thread multiple support. I'm also using thread sanitizer on my application (although not on MPICH itself). C++11 standard.
Problem description: I'm running into a SIGABRT at the end of the program when MPICH seems to be running its deconstructors. My program is running NPROC MPI processes, each with T C++11 threads (so, NPROC * T threads total). This simplified application (which I've created to exhibit this problem) simply has one root looping through H times, doing a MPI_Bcast to other processes each time. Only one thread per process participates in the MPI_Bcast. Thread sanitizer is also detecting a data race. In fact, thread sanitizer is detecting that MPI_Bcast is *writing* to the root's buffer, even though in a broadcast, at least semantically, the root should only ever be sending data, not receiving data. Taking a quick look at MPI_Bcast (which ultimately calls to MPIDI_CH3U_Receive_data_found which invokes the memcpy), it does seem that depending on the message size (in my case, it's just 10 MPI_INTs per message), scatter and allgather can be used or a binomial tree algorithm. I haven't dug in to see which one is being used in my case, but this indicates that there's at least a possibility that the root's buffer can be received into during a MPI_Bcast. My program is producing the "right answer" but could just be luck.
Here's the basic structure of the program:
if root process if root thread A: signal using condition variable to (T-1) non-root threads that it can read the buffer (via shared memory) B: MPI_Bcast to other procs else if not root thread wait on condition variable until it's ok to read buffer read buffer else if not root process // AFAIK there are no problems with non-root processes MPI_Bcast as a non-root
Thread sanitizer is detecting that on the root process, MPI_Bcast is writing to the buffer that is being broadcast. Simultaneously, non-root threads in the root process are reading the buffer being broadcast *while the MPI_Bcast is happening*. When I change the order of statements A and B above, the data race goes away. So, it seems to me that my assumption (that on the root node, the buffer being broadcast is being written, or at least it's a possibility that the MPI call will write the buffer) is incorrect. (FYI, the "writing" that's happening in MPI_Bcast is coming from MPIDI_CH3U_Receive_data_found src/mpid/ch3/src/ch3u_handle_recv_pkt.c:152.)
Also, at the end of the program, I'm hitting a SIGABRT during the deconstruction of something in libmpi (__do_global_dtors_aux). Full backtrace below. This issue also goes away when I reverse the order of statements A and B. I imagine I'm corrupting some state in MPICH, but I'm not sure.
I should point out that I'd prefer to have the ordering of A and B as above, so some threads can make progress while the MPI_Bcast is happening.
So, my questions are:
1. what assumptions can be made about buffers in general in MPI during MPI operations? Does the standard specify anything about this? No reading non-const buffers during MPI operations, or can I do this in some situations?
2. Is there any way for me to achieve what I'm trying to do (above, where the reading of the buffer is happening simultaneously with an MPI operation that shouldn't need to write the user buffer but does)? This would be very helpful to know.
Thank you.
Program received signal SIGABRT, Aborted. [Switching to Thread 0x7ff937bf8700 (LWP 32001)] 0x00007ff93fd17c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
(gdb) bt full #0 0x00007ff93fd17c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 resultvar = 0 pid = 31987 selftid = 32001 #1 0x00007ff93fd1b028 in __GI_abort () at abort.c:89 save_stage = 2 act = {__sigaction_handler = {sa_handler = 0xbea6495, sa_sigaction = 0xbea6495}, sa_mask = {__val = {140708540140376, 140708494113584, 134272, 0, 140708494597155, 134240, 140708497979320, 131072, 140708494598025, 140708358448000, 140708521085690, 140708539470768, 0, 140708540266640, 140708551504800, 1}}, sa_flags = 934869088, sa_restorer = 0x1} sigs = {__val = {32, 0 <repeats 15 times>}} #2 0x00007ff93fd62dfa in malloc_printerr (ptr=<optimized out>, str=<optimized out>, action=<optimized out>) at malloc.c:5000 No locals. #3 free_check (mem=<optimized out>, caller=<optimized out>) at hooks.c:298 p = <optimized out> #4 0x00007ff93fd1d53a in __cxa_finalize (d=0x7ff941207708) at cxa_finalize.c:56 check = 97 cxafn = <optimized out> cxaarg = <optimized out> f = 0x7ff9400a1230 <initial+944> funcs = 0x7ff9400a0e80 <initial> #5 0x00007ff940ff7123 in __do_global_dtors_aux () from mpich/debug/lib/libmpicxx.so.12 No symbol table info available. #6 0x00007ff937b8e410 in ?? () No symbol table info available. #7 0x00007ff9426e270a in _dl_fini () at dl-fini.c:252 array = 0x7ff941205238 i = 0 nmaps = 32761 nloaded = <optimized out> i = 4 l = 0x7ff9428d7a00 ns = 140708494300474 maps = 0x7ff937b8e330 maps_size = 140708497985152 do_audit = 1116568064 __PRETTY_FUNCTION__ = "_dl_fini" _______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
-- Jeff Hammond [email protected] http://jeffhammond.github.io/
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss