August 2016 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Reading buffers during MPI call in multithreaded application
by Thakur, Rajeev 16 Aug '16

16 Aug '16

I guess what is happening is the allgather in the scatter-allgather algorithm is happening everywhere, even at the root. The root must participate in the allgather, but the copying into the local buffer at the root can be avoided since the data is already there. Since all this is within a blocking MPI_Bcast operation, it was not considered a problem. I don’t know if the buffer at the root is modified during the operation or just over-written with the same values. If the latter, I don’t know whey there would be a problem even in the multithreaded case. Rajeev > On Aug 16, 2016, at 3:47 PM, Mark Davis <markdavisinboston(a)gmail.com> wrote: > > I'm glad that is the intention of MPI_Bcast. That agrees with my > intuition. However, unless I'm mistaken, there may be a bug here as it > seems that the buffer is being written to on the root process (see my > original post for more details). > > Given that there's only one buffer argument in MPI_Bcast, it does seem > that it would be impossible to mark it as const as it would only be > const on the root and non-const on the non-roots. But, it would be > nice if it was effectively const for the root process. > > On Tue, Aug 16, 2016 at 4:34 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: >> >>> On Aug 16, 2016, at 3:12 PM, Mark Davis <markdavisinboston(a)gmail.com> wrote: >>> >>>> At the non-root processes, MPI_Bcast behaves like a MPI_Recv call, and thus >>>> you cannot touch the buffer until the function returned. >>> >>> That part makes sense. I'm not allowing the buffer to be read or >>> otherwise used on non-root threads. It makes sense to me that this >>> acts as a MPI_Recv call. >>> >>> The thing that I'm confused by is on the root process, as it seems >>> that the root process' buffer is also written to during the course of >>> the MPI_Bcast; it should act like an MPI_Send. It seems like this is >>> just an implementation detail, and as you pointed out, since the >>> MPI_Bcast buffer is not marked const, anything could happen. >> >> As of MPI-2.2, we allowed the root process to read from the buffer while communication is going on for point-to-point and collective operations. In MPI-3.0, we removed that wording when we added "const" attributes, and might have accidentally removed the wording in collectives also even though we didn't add const in all cases. I didn't get a chance to look through the MPI-3.1 standard, but if we missed that, we should fix it. >> >> The intention is that MPI_Bcast allows the user to read from the buffer at the root process, but not write to it. So, for example, you can do multiple MPI_Ibcasts from the same buffer. >> >> -- Pavan >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Reading buffers during MPI call in multithreaded application
by Mark Davis 16 Aug '16

16 Aug '16

I'm glad that is the intention of MPI_Bcast. That agrees with my intuition. However, unless I'm mistaken, there may be a bug here as it seems that the buffer is being written to on the root process (see my original post for more details). Given that there's only one buffer argument in MPI_Bcast, it does seem that it would be impossible to mark it as const as it would only be const on the root and non-const on the non-roots. But, it would be nice if it was effectively const for the root process. On Tue, Aug 16, 2016 at 4:34 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > >> On Aug 16, 2016, at 3:12 PM, Mark Davis <markdavisinboston(a)gmail.com> wrote: >> >>> At the non-root processes, MPI_Bcast behaves like a MPI_Recv call, and thus >>> you cannot touch the buffer until the function returned. >> >> That part makes sense. I'm not allowing the buffer to be read or >> otherwise used on non-root threads. It makes sense to me that this >> acts as a MPI_Recv call. >> >> The thing that I'm confused by is on the root process, as it seems >> that the root process' buffer is also written to during the course of >> the MPI_Bcast; it should act like an MPI_Send. It seems like this is >> just an implementation detail, and as you pointed out, since the >> MPI_Bcast buffer is not marked const, anything could happen. > > As of MPI-2.2, we allowed the root process to read from the buffer while communication is going on for point-to-point and collective operations. In MPI-3.0, we removed that wording when we added "const" attributes, and might have accidentally removed the wording in collectives also even though we didn't add const in all cases. I didn't get a chance to look through the MPI-3.1 standard, but if we missed that, we should fix it. > > The intention is that MPI_Bcast allows the user to read from the buffer at the root process, but not write to it. So, for example, you can do multiple MPI_Ibcasts from the same buffer. > > -- Pavan > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Reading buffers during MPI call in multithreaded application
by Balaji, Pavan 16 Aug '16

16 Aug '16

> On Aug 16, 2016, at 3:12 PM, Mark Davis <markdavisinboston(a)gmail.com> wrote: > >> At the non-root processes, MPI_Bcast behaves like a MPI_Recv call, and thus >> you cannot touch the buffer until the function returned. > > That part makes sense. I'm not allowing the buffer to be read or > otherwise used on non-root threads. It makes sense to me that this > acts as a MPI_Recv call. > > The thing that I'm confused by is on the root process, as it seems > that the root process' buffer is also written to during the course of > the MPI_Bcast; it should act like an MPI_Send. It seems like this is > just an implementation detail, and as you pointed out, since the > MPI_Bcast buffer is not marked const, anything could happen. As of MPI-2.2, we allowed the root process to read from the buffer while communication is going on for point-to-point and collective operations. In MPI-3.0, we removed that wording when we added "const" attributes, and might have accidentally removed the wording in collectives also even though we didn't add const in all cases. I didn't get a chance to look through the MPI-3.1 standard, but if we missed that, we should fix it. The intention is that MPI_Bcast allows the user to read from the buffer at the root process, but not write to it. So, for example, you can do multiple MPI_Ibcasts from the same buffer. -- Pavan _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Reading buffers during MPI call in multithreaded application
by Mark Davis 16 Aug '16

16 Aug '16

> At the non-root processes, MPI_Bcast behaves like a MPI_Recv call, and thus > you cannot touch the buffer until the function returned. That part makes sense. I'm not allowing the buffer to be read or otherwise used on non-root threads. It makes sense to me that this acts as a MPI_Recv call. The thing that I'm confused by is on the root process, as it seems that the root process' buffer is also written to during the course of the MPI_Bcast; it should act like an MPI_Send. It seems like this is just an implementation detail, and as you pointed out, since the MPI_Bcast buffer is not marked const, anything could happen. >> Background: running on Linux 3.13.0-52 / Ubuntu x86_64. gcc 5.3. MPICH >> 3.2 release version with thread support enabled. I am MPI_Initing with >> thread multiple support. I'm also using thread sanitizer on my >> application (although not on MPICH itself). C++11 standard. >> >> Problem description: I'm running into a SIGABRT at the end of the >> program when MPICH seems to be running its deconstructors. My program >> is running NPROC MPI processes, each with T C++11 threads (so, NPROC * >> T threads total). This simplified application (which I've created to >> exhibit this problem) simply has one root looping through H times, >> doing a MPI_Bcast to other processes each time. Only one thread per >> process participates in the MPI_Bcast. Thread sanitizer is also >> detecting a data race. In fact, thread sanitizer is detecting that >> MPI_Bcast is *writing* to the root's buffer, even though in a >> broadcast, at least semantically, the root should only ever be sending >> data, not receiving data. Taking a quick look at MPI_Bcast (which >> ultimately calls to MPIDI_CH3U_Receive_data_found which invokes the >> memcpy), it does seem that depending on the message size (in my case, >> it's just 10 MPI_INTs per message), scatter and allgather can be used >> or a binomial tree algorithm. I haven't dug in to see which one is >> being used in my case, but this indicates that there's at least a >> possibility that the root's buffer can be received into during a >> MPI_Bcast. My program is producing the "right answer" but could just >> be luck. >> >> >> Here's the basic structure of the program: >> >> if root process >> if root thread >> A: signal using condition variable to (T-1) non-root threads >> that it can read the buffer (via shared memory) >> B: MPI_Bcast to other procs >> else if not root thread >> wait on condition variable until it's ok to read buffer >> read buffer >> else if not root process >> // AFAIK there are no problems with non-root processes >> MPI_Bcast as a non-root >> >> >> >> Thread sanitizer is detecting that on the root process, MPI_Bcast is >> writing to the buffer that is being broadcast. Simultaneously, >> non-root threads in the root process are reading the buffer being >> broadcast *while the MPI_Bcast is happening*. When I change the order >> of statements A and B above, the data race goes away. So, it seems to >> me that my assumption (that on the root node, the buffer being >> broadcast is being written, or at least it's a possibility that the >> MPI call will write the buffer) is incorrect. (FYI, the "writing" >> that's happening in MPI_Bcast is coming from >> MPIDI_CH3U_Receive_data_found >> src/mpid/ch3/src/ch3u_handle_recv_pkt.c:152.) >> >> Also, at the end of the program, I'm hitting a SIGABRT during the >> deconstruction of something in libmpi (__do_global_dtors_aux). Full >> backtrace below. This issue also goes away when I reverse the order of >> statements A and B. I imagine I'm corrupting some state in MPICH, but >> I'm not sure. >> >> I should point out that I'd prefer to have the ordering of A and B as >> above, so some threads can make progress while the MPI_Bcast is >> happening. >> >> So, my questions are: >> >> 1. what assumptions can be made about buffers in general in MPI during >> MPI operations? Does the standard specify anything about this? No >> reading non-const buffers during MPI operations, or can I do this in >> some situations? >> >> 2. Is there any way for me to achieve what I'm trying to do (above, >> where the reading of the buffer is happening simultaneously with an >> MPI operation that shouldn't need to write the user buffer but does)? >> This would be very helpful to know. >> >> Thank you. >> >> >> >> Program received signal SIGABRT, Aborted. >> [Switching to Thread 0x7ff937bf8700 (LWP 32001)] >> 0x00007ff93fd17c37 in __GI_raise (sig=sig@entry=6) at >> ../nptl/sysdeps/unix/sysv/linux/raise.c:56 >> >> (gdb) bt full >> #0 0x00007ff93fd17c37 in __GI_raise (sig=sig@entry=6) at >> ../nptl/sysdeps/unix/sysv/linux/raise.c:56 >> resultvar = 0 >> pid = 31987 >> selftid = 32001 >> #1 0x00007ff93fd1b028 in __GI_abort () at abort.c:89 >> save_stage = 2 >> act = {__sigaction_handler = {sa_handler = 0xbea6495, >> sa_sigaction = 0xbea6495}, sa_mask = {__val = {140708540140376, >> 140708494113584, 134272, 0, 140708494597155, 134240, >> 140708497979320, 131072, 140708494598025, 140708358448000, >> 140708521085690, 140708539470768, 0, 140708540266640, >> 140708551504800, 1}}, sa_flags = 934869088, >> sa_restorer = 0x1} >> sigs = {__val = {32, 0 <repeats 15 times>}} >> #2 0x00007ff93fd62dfa in malloc_printerr (ptr=<optimized out>, >> str=<optimized out>, action=<optimized out>) at malloc.c:5000 >> No locals. >> #3 free_check (mem=<optimized out>, caller=<optimized out>) at >> hooks.c:298 >> p = <optimized out> >> #4 0x00007ff93fd1d53a in __cxa_finalize (d=0x7ff941207708) at >> cxa_finalize.c:56 >> check = 97 >> cxafn = <optimized out> >> cxaarg = <optimized out> >> f = 0x7ff9400a1230 <initial+944> >> funcs = 0x7ff9400a0e80 <initial> >> #5 0x00007ff940ff7123 in __do_global_dtors_aux () from >> mpich/debug/lib/libmpicxx.so.12 >> No symbol table info available. >> #6 0x00007ff937b8e410 in ?? () >> No symbol table info available. >> #7 0x00007ff9426e270a in _dl_fini () at dl-fini.c:252 >> array = 0x7ff941205238 >> i = 0 >> nmaps = 32761 >> nloaded = <optimized out> >> i = 4 >> l = 0x7ff9428d7a00 >> ns = 140708494300474 >> maps = 0x7ff937b8e330 >> maps_size = 140708497985152 >> do_audit = 1116568064 >> __PRETTY_FUNCTION__ = "_dl_fini" >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Reading buffers during MPI call in multithreaded application
by Jeff Hammond 16 Aug '16

16 Aug '16

On Tue, Aug 16, 2016 at 7:18 AM, Mark Davis <markdavisinboston(a)gmail.com> wrote: > Hello, I'm hitting a data race and a deconstruction error when using > MPI_Bcast in an MPI application that also uses multiple threads per > process. I'll go into detail below, but the fundamental question I > have is: what can I assume, if anything, about the state of an MPI > buffer during an MPI call? (I do realize that the buffers are not > declared const in MPI_Bcast, whereas in other situations, such as > MPI_Send, they are.) > Nothing, unless the buffer pointer has the const attribute on it. At the non-root processes, MPI_Bcast behaves like a MPI_Recv call, and thus you cannot touch the buffer until the function returned. Multiple threads cannot use the same buffer to multiple MPI_Bcast calls. Jeff > > Background: running on Linux 3.13.0-52 / Ubuntu x86_64. gcc 5.3. MPICH > 3.2 release version with thread support enabled. I am MPI_Initing with > thread multiple support. I'm also using thread sanitizer on my > application (although not on MPICH itself). C++11 standard. > > Problem description: I'm running into a SIGABRT at the end of the > program when MPICH seems to be running its deconstructors. My program > is running NPROC MPI processes, each with T C++11 threads (so, NPROC * > T threads total). This simplified application (which I've created to > exhibit this problem) simply has one root looping through H times, > doing a MPI_Bcast to other processes each time. Only one thread per > process participates in the MPI_Bcast. Thread sanitizer is also > detecting a data race. In fact, thread sanitizer is detecting that > MPI_Bcast is *writing* to the root's buffer, even though in a > broadcast, at least semantically, the root should only ever be sending > data, not receiving data. Taking a quick look at MPI_Bcast (which > ultimately calls to MPIDI_CH3U_Receive_data_found which invokes the > memcpy), it does seem that depending on the message size (in my case, > it's just 10 MPI_INTs per message), scatter and allgather can be used > or a binomial tree algorithm. I haven't dug in to see which one is > being used in my case, but this indicates that there's at least a > possibility that the root's buffer can be received into during a > MPI_Bcast. My program is producing the "right answer" but could just > be luck. > > > Here's the basic structure of the program: > > if root process > if root thread > A: signal using condition variable to (T-1) non-root threads > that it can read the buffer (via shared memory) > B: MPI_Bcast to other procs > else if not root thread > wait on condition variable until it's ok to read buffer > read buffer > else if not root process > // AFAIK there are no problems with non-root processes > MPI_Bcast as a non-root > > > > Thread sanitizer is detecting that on the root process, MPI_Bcast is > writing to the buffer that is being broadcast. Simultaneously, > non-root threads in the root process are reading the buffer being > broadcast *while the MPI_Bcast is happening*. When I change the order > of statements A and B above, the data race goes away. So, it seems to > me that my assumption (that on the root node, the buffer being > broadcast is being written, or at least it's a possibility that the > MPI call will write the buffer) is incorrect. (FYI, the "writing" > that's happening in MPI_Bcast is coming from > MPIDI_CH3U_Receive_data_found > src/mpid/ch3/src/ch3u_handle_recv_pkt.c:152.) > > Also, at the end of the program, I'm hitting a SIGABRT during the > deconstruction of something in libmpi (__do_global_dtors_aux). Full > backtrace below. This issue also goes away when I reverse the order of > statements A and B. I imagine I'm corrupting some state in MPICH, but > I'm not sure. > > I should point out that I'd prefer to have the ordering of A and B as > above, so some threads can make progress while the MPI_Bcast is > happening. > > So, my questions are: > > 1. what assumptions can be made about buffers in general in MPI during > MPI operations? Does the standard specify anything about this? No > reading non-const buffers during MPI operations, or can I do this in > some situations? > > 2. Is there any way for me to achieve what I'm trying to do (above, > where the reading of the buffer is happening simultaneously with an > MPI operation that shouldn't need to write the user buffer but does)? > This would be very helpful to know. > > Thank you. > > > > Program received signal SIGABRT, Aborted. > [Switching to Thread 0x7ff937bf8700 (LWP 32001)] > 0x00007ff93fd17c37 in __GI_raise (sig=sig@entry=6) at > ../nptl/sysdeps/unix/sysv/linux/raise.c:56 > > (gdb) bt full > #0 0x00007ff93fd17c37 in __GI_raise (sig=sig@entry=6) at > ../nptl/sysdeps/unix/sysv/linux/raise.c:56 > resultvar = 0 > pid = 31987 > selftid = 32001 > #1 0x00007ff93fd1b028 in __GI_abort () at abort.c:89 > save_stage = 2 > act = {__sigaction_handler = {sa_handler = 0xbea6495, > sa_sigaction = 0xbea6495}, sa_mask = {__val = {140708540140376, > 140708494113584, 134272, 0, 140708494597155, 134240, > 140708497979320, 131072, 140708494598025, 140708358448000, > 140708521085690, 140708539470768, 0, 140708540266640, > 140708551504800, 1}}, sa_flags = 934869088, > sa_restorer = 0x1} > sigs = {__val = {32, 0 <repeats 15 times>}} > #2 0x00007ff93fd62dfa in malloc_printerr (ptr=<optimized out>, > str=<optimized out>, action=<optimized out>) at malloc.c:5000 > No locals. > #3 free_check (mem=<optimized out>, caller=<optimized out>) at hooks.c:298 > p = <optimized out> > #4 0x00007ff93fd1d53a in __cxa_finalize (d=0x7ff941207708) at > cxa_finalize.c:56 > check = 97 > cxafn = <optimized out> > cxaarg = <optimized out> > f = 0x7ff9400a1230 <initial+944> > funcs = 0x7ff9400a0e80 <initial> > #5 0x00007ff940ff7123 in __do_global_dtors_aux () from > mpich/debug/lib/libmpicxx.so.12 > No symbol table info available. > #6 0x00007ff937b8e410 in ?? () > No symbol table info available. > #7 0x00007ff9426e270a in _dl_fini () at dl-fini.c:252 > array = 0x7ff941205238 > i = 0 > nmaps = 32761 > nloaded = <optimized out> > i = 4 > l = 0x7ff9428d7a00 > ns = 140708494300474 > maps = 0x7ff937b8e330 > maps_size = 140708497985152 > do_audit = 1116568064 > __PRETTY_FUNCTION__ = "_dl_fini" > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Reading buffers during MPI call in multithreaded application
by Mark Davis 16 Aug '16

16 Aug '16

Hello, I'm hitting a data race and a deconstruction error when using MPI_Bcast in an MPI application that also uses multiple threads per process. I'll go into detail below, but the fundamental question I have is: what can I assume, if anything, about the state of an MPI buffer during an MPI call? (I do realize that the buffers are not declared const in MPI_Bcast, whereas in other situations, such as MPI_Send, they are.) Background: running on Linux 3.13.0-52 / Ubuntu x86_64. gcc 5.3. MPICH 3.2 release version with thread support enabled. I am MPI_Initing with thread multiple support. I'm also using thread sanitizer on my application (although not on MPICH itself). C++11 standard. Problem description: I'm running into a SIGABRT at the end of the program when MPICH seems to be running its deconstructors. My program is running NPROC MPI processes, each with T C++11 threads (so, NPROC * T threads total). This simplified application (which I've created to exhibit this problem) simply has one root looping through H times, doing a MPI_Bcast to other processes each time. Only one thread per process participates in the MPI_Bcast. Thread sanitizer is also detecting a data race. In fact, thread sanitizer is detecting that MPI_Bcast is *writing* to the root's buffer, even though in a broadcast, at least semantically, the root should only ever be sending data, not receiving data. Taking a quick look at MPI_Bcast (which ultimately calls to MPIDI_CH3U_Receive_data_found which invokes the memcpy), it does seem that depending on the message size (in my case, it's just 10 MPI_INTs per message), scatter and allgather can be used or a binomial tree algorithm. I haven't dug in to see which one is being used in my case, but this indicates that there's at least a possibility that the root's buffer can be received into during a MPI_Bcast. My program is producing the "right answer" but could just be luck. Here's the basic structure of the program: if root process if root thread A: signal using condition variable to (T-1) non-root threads that it can read the buffer (via shared memory) B: MPI_Bcast to other procs else if not root thread wait on condition variable until it's ok to read buffer read buffer else if not root process // AFAIK there are no problems with non-root processes MPI_Bcast as a non-root Thread sanitizer is detecting that on the root process, MPI_Bcast is writing to the buffer that is being broadcast. Simultaneously, non-root threads in the root process are reading the buffer being broadcast *while the MPI_Bcast is happening*. When I change the order of statements A and B above, the data race goes away. So, it seems to me that my assumption (that on the root node, the buffer being broadcast is being written, or at least it's a possibility that the MPI call will write the buffer) is incorrect. (FYI, the "writing" that's happening in MPI_Bcast is coming from MPIDI_CH3U_Receive_data_found src/mpid/ch3/src/ch3u_handle_recv_pkt.c:152.) Also, at the end of the program, I'm hitting a SIGABRT during the deconstruction of something in libmpi (__do_global_dtors_aux). Full backtrace below. This issue also goes away when I reverse the order of statements A and B. I imagine I'm corrupting some state in MPICH, but I'm not sure. I should point out that I'd prefer to have the ordering of A and B as above, so some threads can make progress while the MPI_Bcast is happening. So, my questions are: 1. what assumptions can be made about buffers in general in MPI during MPI operations? Does the standard specify anything about this? No reading non-const buffers during MPI operations, or can I do this in some situations? 2. Is there any way for me to achieve what I'm trying to do (above, where the reading of the buffer is happening simultaneously with an MPI operation that shouldn't need to write the user buffer but does)? This would be very helpful to know. Thank you. Program received signal SIGABRT, Aborted. [Switching to Thread 0x7ff937bf8700 (LWP 32001)] 0x00007ff93fd17c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 (gdb) bt full #0 0x00007ff93fd17c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 resultvar = 0 pid = 31987 selftid = 32001 #1 0x00007ff93fd1b028 in __GI_abort () at abort.c:89 save_stage = 2 act = {__sigaction_handler = {sa_handler = 0xbea6495, sa_sigaction = 0xbea6495}, sa_mask = {__val = {140708540140376, 140708494113584, 134272, 0, 140708494597155, 134240, 140708497979320, 131072, 140708494598025, 140708358448000, 140708521085690, 140708539470768, 0, 140708540266640, 140708551504800, 1}}, sa_flags = 934869088, sa_restorer = 0x1} sigs = {__val = {32, 0 <repeats 15 times>}} #2 0x00007ff93fd62dfa in malloc_printerr (ptr=<optimized out>, str=<optimized out>, action=<optimized out>) at malloc.c:5000 No locals. #3 free_check (mem=<optimized out>, caller=<optimized out>) at hooks.c:298 p = <optimized out> #4 0x00007ff93fd1d53a in __cxa_finalize (d=0x7ff941207708) at cxa_finalize.c:56 check = 97 cxafn = <optimized out> cxaarg = <optimized out> f = 0x7ff9400a1230 <initial+944> funcs = 0x7ff9400a0e80 <initial> #5 0x00007ff940ff7123 in __do_global_dtors_aux () from mpich/debug/lib/libmpicxx.so.12 No symbol table info available. #6 0x00007ff937b8e410 in ?? () No symbol table info available. #7 0x00007ff9426e270a in _dl_fini () at dl-fini.c:252 array = 0x7ff941205238 i = 0 nmaps = 32761 nloaded = <optimized out> i = 4 l = 0x7ff9428d7a00 ns = 140708494300474 maps = 0x7ff937b8e330 maps_size = 140708497985152 do_audit = 1116568064 __PRETTY_FUNCTION__ = "_dl_fini" _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH-3.2: SIGSEGV in MPID_Request_create () at src/mpid/ch3/src/ch3u_request.c:101
by Halim Amer 15 Aug '16

15 Aug '16

Good catch! the `most` option implies `mem`, but the root configure failed to forward the `mem` option to the MPL software layer. We will push a fix, but meanwhile you can specify `--enable-g=most,mem` to get the desired behavior. --Halim www.mcs.anl.gov/~aamer On 8/12/16 9:37 PM, Mark Davis wrote: > I've tried both git HEAD (3ea7589) as well as the August 1 master > snapshot and am having trouble building it; I'm getting the same error > in both cases. I've configured with --enable-g=most but otherwise it's > all default. I'm running on OSX (Darwin - 15.6.0) and clang 3.8.1. > > It's erroring on compiling src/mpi/attr/lib_libpmpi_la-attr_delete.lo > due to an issue with the macro MPL_free > > Has anyone seen this before? I'm including the full error trace below: > > > > > > Making all in . > CC src/mpi/attr/lib_libpmpi_la-attr_delete.lo > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:217: > ./src/include/mpir_request.h:281:13: warning: multi-character > character constant [-Wmultichar] > MPL_free(req->u.ureq.greq_fns); > ^ > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: > note: expanded from macro 'MPL_free' > #define MPL_free(a) free((void *)(a)) > ^ > ./src/include/mpir_mem.h:92:27: note: expanded from macro 'free' > #define free(a) 'Error use MPL_free' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:217: > ./src/include/mpir_request.h:281:13: warning: character constant too > long for its type > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: > note: expanded from macro 'MPL_free' > #define MPL_free(a) free((void *)(a)) > ^ > ./src/include/mpir_mem.h:92:27: note: expanded from macro 'free' > #define free(a) 'Error use MPL_free' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:217: > ./src/include/mpir_request.h:281:13: error: expected ';' after expression > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: > note: expanded from macro 'MPL_free' > #define MPL_free(a) free((void *)(a)) > ^ > ./src/include/mpir_mem.h:92:50: note: expanded from macro 'free' > #define free(a) 'Error use MPL_free' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:217: > ./src/include/mpir_request.h:281:13: error: expected expression > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: > note: expanded from macro 'MPL_free' > #define MPL_free(a) free((void *)(a)) > ^ > ./src/include/mpir_mem.h:92:51: note: expanded from macro 'free' > #define free(a) 'Error use MPL_free' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:225: > In file included from ./src/include/mpir_cvars.h:17: > In file included from ./src/include/mpitimpl.h:18: > ./src/include/mpir_utarray.h:238:43: warning: multi-character > character constant [-Wmultichar] > *_dst = (*_src == NULL) ? NULL : (char*)utarray_strdup_(*_src); > ^ > ./src/include/mpir_utarray.h:56:33: note: expanded from macro 'utarray_strdup_' > #define utarray_strdup_(x_) MPL_strdup(x_) > ^ > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:17:20: > note: expanded from macro 'MPL_strdup' > #define MPL_strdup strdup > ^ > ./src/include/mpir_mem.h:100:27: note: expanded from macro 'strdup' > #define strdup(a) 'Error use MPL_strdup' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:225: > In file included from ./src/include/mpir_cvars.h:17: > In file included from ./src/include/mpitimpl.h:18: > ./src/include/mpir_utarray.h:238:43: warning: character constant too > long for its type > ./src/include/mpir_utarray.h:56:33: note: expanded from macro 'utarray_strdup_' > #define utarray_strdup_(x_) MPL_strdup(x_) > ^ > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:17:20: > note: expanded from macro 'MPL_strdup' > #define MPL_strdup strdup > ^ > ./src/include/mpir_mem.h:100:27: note: expanded from macro 'strdup' > #define strdup(a) 'Error use MPL_strdup' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:225: > In file included from ./src/include/mpir_cvars.h:17: > In file included from ./src/include/mpitimpl.h:18: > ./src/include/mpir_utarray.h:238:43: error: expected ';' after expression > ./src/include/mpir_utarray.h:56:33: note: expanded from macro 'utarray_strdup_' > #define utarray_strdup_(x_) MPL_strdup(x_) > ^ > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:17:20: > note: expanded from macro 'MPL_strdup' > #define MPL_strdup strdup > ^ > ./src/include/mpir_mem.h:100:50: note: expanded from macro 'strdup' > #define strdup(a) 'Error use MPL_strdup' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:225: > In file included from ./src/include/mpir_cvars.h:17: > In file included from ./src/include/mpitimpl.h:18: > ./src/include/mpir_utarray.h:238:43: error: expected expression > ./src/include/mpir_utarray.h:56:33: note: expanded from macro 'utarray_strdup_' > #define utarray_strdup_(x_) MPL_strdup(x_) > ^ > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:17:20: > note: expanded from macro 'MPL_strdup' > #define MPL_strdup strdup > ^ > ./src/include/mpir_mem.h:100:51: note: expanded from macro 'strdup' > #define strdup(a) 'Error use MPL_strdup' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:225: > In file included from ./src/include/mpir_cvars.h:17: > In file included from ./src/include/mpitimpl.h:18: > ./src/include/mpir_utarray.h:242:14: warning: multi-character > character constant [-Wmultichar] > if (*eltc) utarray_free_(*eltc); > ^ > ./src/include/mpir_utarray.h:54:33: note: expanded from macro 'utarray_free_' > #define utarray_free_(x_) MPL_free(x_) > ^ > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: > note: expanded from macro 'MPL_free' > #define MPL_free(a) free((void *)(a)) > ^ > ./src/include/mpir_mem.h:92:27: note: expanded from macro 'free' > #define free(a) 'Error use MPL_free' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:225: > In file included from ./src/include/mpir_cvars.h:17: > In file included from ./src/include/mpitimpl.h:18: > ./src/include/mpir_utarray.h:242:14: warning: character constant too > long for its type > ./src/include/mpir_utarray.h:54:33: note: expanded from macro 'utarray_free_' > #define utarray_free_(x_) MPL_free(x_) > ^ > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: > note: expanded from macro 'MPL_free' > #define MPL_free(a) free((void *)(a)) > ^ > ./src/include/mpir_mem.h:92:27: note: expanded from macro 'free' > #define free(a) 'Error use MPL_free' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:225: > In file included from ./src/include/mpir_cvars.h:17: > In file included from ./src/include/mpitimpl.h:18: > ./src/include/mpir_utarray.h:242:14: error: expected ';' after expression > ./src/include/mpir_utarray.h:54:33: note: expanded from macro 'utarray_free_' > #define utarray_free_(x_) MPL_free(x_) > ^ > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: > note: expanded from macro 'MPL_free' > #define MPL_free(a) free((void *)(a)) > ^ > ./src/include/mpir_mem.h:92:50: note: expanded from macro 'free' > #define free(a) 'Error use MPL_free' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:225: > In file included from ./src/include/mpir_cvars.h:17: > In file included from ./src/include/mpitimpl.h:18: > ./src/include/mpir_utarray.h:242:14: error: expected expression > ./src/include/mpir_utarray.h:54:33: note: expanded from macro 'utarray_free_' > #define utarray_free_(x_) MPL_free(x_) > ^ > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: > note: expanded from macro 'MPL_free' > #define MPL_free(a) free((void *)(a)) > ^ > ./src/include/mpir_mem.h:92:51: note: expanded from macro 'free' > #define free(a) 'Error use MPL_free' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:61:29: warning: multi-character > character constant [-Wmultichar] > nIndirect = (int *) MPL_calloc(objmem->indirect_size, sizeof(int)); > ^ > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: > note: expanded from macro 'MPL_calloc' > #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) > ^ > ./src/include/mpir_mem.h:91:27: note: expanded from macro 'calloc' > #define calloc(a,b) 'Error use MPL_calloc' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:61:29: warning: character constant too > long for its type > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: > note: expanded from macro 'MPL_calloc' > #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) > ^ > ./src/include/mpir_mem.h:91:27: note: expanded from macro 'calloc' > #define calloc(a,b) 'Error use MPL_calloc' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:61:29: error: expected ';' after expression > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: > note: expanded from macro 'MPL_calloc' > #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) > ^ > ./src/include/mpir_mem.h:91:50: note: expanded from macro 'calloc' > #define calloc(a,b) 'Error use MPL_calloc' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:61:29: error: expected expression > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: > note: expanded from macro 'MPL_calloc' > #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) > ^ > ./src/include/mpir_mem.h:91:51: note: expanded from macro 'calloc' > #define calloc(a,b) 'Error use MPL_calloc' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:117:9: warning: multi-character > character constant [-Wmultichar] > MPL_free(nIndirect); > ^ > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: > note: expanded from macro 'MPL_free' > #define MPL_free(a) free((void *)(a)) > ^ > ./src/include/mpir_mem.h:92:27: note: expanded from macro 'free' > #define free(a) 'Error use MPL_free' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:117:9: warning: character constant too > long for its type > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: > note: expanded from macro 'MPL_free' > #define MPL_free(a) free((void *)(a)) > ^ > ./src/include/mpir_mem.h:92:27: note: expanded from macro 'free' > #define free(a) 'Error use MPL_free' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:117:9: error: expected ';' after expression > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: > note: expanded from macro 'MPL_free' > #define MPL_free(a) free((void *)(a)) > ^ > ./src/include/mpir_mem.h:92:50: note: expanded from macro 'free' > #define free(a) 'Error use MPL_free' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:117:9: error: expected expression > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: > note: expanded from macro 'MPL_free' > #define MPL_free(a) free((void *)(a)) > ^ > ./src/include/mpir_mem.h:92:51: note: expanded from macro 'free' > #define free(a) 'Error use MPL_free' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:179:9: warning: multi-character > character constant [-Wmultichar] > MPL_free((*indirect)[i]); > ^ > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: > note: expanded from macro 'MPL_free' > #define MPL_free(a) free((void *)(a)) > ^ > ./src/include/mpir_mem.h:92:27: note: expanded from macro 'free' > #define free(a) 'Error use MPL_free' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:179:9: warning: character constant too > long for its type > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: > note: expanded from macro 'MPL_free' > #define MPL_free(a) free((void *)(a)) > ^ > ./src/include/mpir_mem.h:92:27: note: expanded from macro 'free' > #define free(a) 'Error use MPL_free' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:179:9: error: expected ';' after expression > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: > note: expanded from macro 'MPL_free' > #define MPL_free(a) free((void *)(a)) > ^ > ./src/include/mpir_mem.h:92:50: note: expanded from macro 'free' > #define free(a) 'Error use MPL_free' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:179:9: error: expected expression > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: > note: expanded from macro 'MPL_free' > #define MPL_free(a) free((void *)(a)) > ^ > ./src/include/mpir_mem.h:92:51: note: expanded from macro 'free' > #define free(a) 'Error use MPL_free' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:182:9: warning: multi-character > character constant [-Wmultichar] > MPL_free(indirect); > ^ > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: > note: expanded from macro 'MPL_free' > #define MPL_free(a) free((void *)(a)) > ^ > ./src/include/mpir_mem.h:92:27: note: expanded from macro 'free' > #define free(a) 'Error use MPL_free' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:182:9: warning: character constant too > long for its type > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: > note: expanded from macro 'MPL_free' > #define MPL_free(a) free((void *)(a)) > ^ > ./src/include/mpir_mem.h:92:27: note: expanded from macro 'free' > #define free(a) 'Error use MPL_free' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:182:9: error: expected ';' after expression > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: > note: expanded from macro 'MPL_free' > #define MPL_free(a) free((void *)(a)) > ^ > ./src/include/mpir_mem.h:92:50: note: expanded from macro 'free' > #define free(a) 'Error use MPL_free' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:182:9: error: expected expression > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: > note: expanded from macro 'MPL_free' > #define MPL_free(a) free((void *)(a)) > ^ > ./src/include/mpir_mem.h:92:51: note: expanded from macro 'free' > #define free(a) 'Error use MPL_free' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:249:30: warning: multi-character > character constant [-Wmultichar] > *indirect = (void *) MPL_calloc(indirect_num_blocks, sizeof(void *)); > ^ > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: > note: expanded from macro 'MPL_calloc' > #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) > ^ > ./src/include/mpir_mem.h:91:27: note: expanded from macro 'calloc' > #define calloc(a,b) 'Error use MPL_calloc' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:249:30: warning: character constant too > long for its type > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: > note: expanded from macro 'MPL_calloc' > #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) > ^ > ./src/include/mpir_mem.h:91:27: note: expanded from macro 'calloc' > #define calloc(a,b) 'Error use MPL_calloc' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:249:30: error: expected ';' after expression > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: > note: expanded from macro 'MPL_calloc' > #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) > ^ > ./src/include/mpir_mem.h:91:50: note: expanded from macro 'calloc' > #define calloc(a,b) 'Error use MPL_calloc' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:249:30: error: expected expression > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: > note: expanded from macro 'MPL_calloc' > #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) > ^ > ./src/include/mpir_mem.h:91:51: note: expanded from macro 'calloc' > #define calloc(a,b) 'Error use MPL_calloc' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:264:26: warning: multi-character > character constant [-Wmultichar] > block_ptr = (void *) MPL_calloc(indirect_num_indices, obj_size); > ^ > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: > note: expanded from macro 'MPL_calloc' > #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) > ^ > ./src/include/mpir_mem.h:91:27: note: expanded from macro 'calloc' > #define calloc(a,b) 'Error use MPL_calloc' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:264:26: warning: character constant too > long for its type > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: > note: expanded from macro 'MPL_calloc' > #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) > ^ > ./src/include/mpir_mem.h:91:27: note: expanded from macro 'calloc' > #define calloc(a,b) 'Error use MPL_calloc' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:264:26: error: expected ';' after expression > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: > note: expanded from macro 'MPL_calloc' > #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) > ^ > ./src/include/mpir_mem.h:91:50: note: expanded from macro 'calloc' > #define calloc(a,b) 'Error use MPL_calloc' ::: > ^ > In file included from src/mpi/attr/attr_delete.c:8: > In file included from ./src/include/mpiimpl.h:228: > ./src/include/mpir_handlemem.h:264:26: error: expected expression > /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: > note: expanded from macro 'MPL_calloc' > #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) > ^ > ./src/include/mpir_mem.h:91:51: note: expanded from macro 'calloc' > #define calloc(a,b) 'Error use MPL_calloc' ::: > ^ > 18 warnings and 18 errors generated. > make[2]: *** [src/mpi/attr/lib_libpmpi_la-attr_delete.lo] Error 1 > make[1]: *** [all-recursive] Error 1 > Makefile:10270: recipe for target 'all' failed > gmake: *** [all] Error 2 > > On Thu, Aug 11, 2016 at 5:21 PM, Halim Amer <aamer(a)anl.gov> wrote: >> This should be related to the alignment problem reported before >> (http://lists.mpich.org/pipermail/discuss/2016-May/004764.html) >> >> We plan to include a fix in the 3.2.x bug fix release series. Meanwhile, >> please try the repo version (git.mpich.org/mpich.git) which should not >> suffer from this problem. >> >> --Halim >> www.mcs.anl.gov/~aamer >> >> >> On 8/11/16 8:48 AM, Mark Davis wrote: >>> >>> Hello, I'm running into a segfault when I run some relatively simple >>> MPI programs. In this particular case, I'm running a small program in >>> a loop that does MPI_Bcast, once per loop, within MPI_COMM_WORLD. The >>> buffer consists of just 7 doubles. I'm running with 6 procs on a >>> machine with 8 cores on OSX (Darwin - 15.6.0 Darwin Kernel Version >>> 15.6.0: Thu Jun 23 18:25:34 PDT 2016; >>> root:xnu-3248.60.10~1/RELEASE_X86_64 x86_64). When I run the same >>> program with a smaller number of procs, the error usually doesn't show >>> up. My compiler (both for compiling the MPICH source as well as my >>> application) is clang 3.8.1. >>> >>> When I run the same program on linux, also with MPICH-3.2 (I believe >>> the same exact source), compiled with gcc 5.3, I do not get this >>> error. This seems to be something I get only with >>> >>> gdb shows the following stack trace. I have a feeling that this has >>> something to do with my toolchain and/or libraries on my system given >>> that I never get this error on my other system (linux). However, it's >>> possible that there's an application bug as well. >>> >>> I'm running the MPICH-3.2 stable release; I haven't tried anything >>> from the repository yet. >>> >>> Does anyone have any ideas about what's going on here? I'm happy to >>> provide more details. >>> >>> Thank you, >>> Mark >>> >>> >>> Program received signal SIGSEGV, Segmentation fault. >>> MPID_Request_create () at src/mpid/ch3/src/ch3u_request.c:101 >>> 101 req->dev.ext_hdr_ptr = NULL; >>> (gdb) bt full >>> #0 MPID_Request_create () at src/mpid/ch3/src/ch3u_request.c:101 >>> No locals. >>> #1 0x00000001003ac4c9 in MPIDI_CH3U_Recvq_FDP_or_AEU >>> (match=<optimized out>, foundp=0x7fff5fbfe2bc) at >>> src/mpid/ch3/src/ch3u_recvq.c:830 >>> proc_failure_bit_masked = <error reading variable >>> proc_failure_bit_masked (Cannot access memory at address 0x1)> >>> error_bit_masked = <error reading variable error_bit_masked >>> (Cannot access memory at address 0x1)> >>> prev_rreq = <optimized out> >>> channel_matched = <optimized out> >>> rreq = <optimized out> >>> #2 0x00000001003d1ffe in MPIDI_CH3_PktHandler_EagerSend >>> (vc=<optimized out>, pkt=0x1004b3fd8 <MPIU_DBG_MaxLevel>, >>> buflen=0x7fff5fbfe440, rreqp=0x7fff5fbfe438) at >>> src/mpid/ch3/src/ch3u_eager.c:629 >>> mpi_errno = <error reading variable mpi_errno (Cannot access >>> memory at address 0x0)> >>> found = <error reading variable found (Cannot access memory at >>> address 0xefefefefefefefef)> >>> rreq = <optimized out> >>> data_len = <optimized out> >>> complete = <optimized out> >>> #3 0x00000001003f6045 in MPID_nem_handle_pkt (vc=<optimized out>, >>> buf=0x102ad07e0 "", buflen=<optimized out>) at >>> src/mpid/ch3/channels/nemesis/src/ch3_progress.c:760 >>> len = 140734799800192 >>> mpi_errno = <optimized out> >>> complete = <error reading variable complete (Cannot access >>> memory at address 0x1)> >>> rreq = <optimized out> >>> #4 0x00000001003f4e41 in MPIDI_CH3I_Progress >>> (progress_state=0x7fff5fbfe750, is_blocking=1) at >>> src/mpid/ch3/channels/nemesis/src/ch3_progress.c:570 >>> payload_len = 4299898840 >>> cell_buf = <optimized out> >>> rreq = <optimized out> >>> vc = 0x102ad07e8 >>> made_progress = <error reading variable made_progress (Cannot >>> access memory at address 0x0)> >>> mpi_errno = <optimized out> >>> #5 0x000000010035386d in MPIC_Wait (request_ptr=<optimized out>, >>> errflag=<optimized out>) at src/mpi/coll/helper_fns.c:225 >>> progress_state = {ch = {completion_count = -1409286143}} >>> mpi_errno = <error reading variable mpi_errno (Cannot access >>> memory at address 0x0)> >>> #6 0x0000000100353b10 in MPIC_Send (buf=0x100917c30, >>> count=4299945096, datatype=-1581855963, dest=<optimized out>, >>> tag=4975608, comm_ptr=0x1004b3fd8 <MPIU_DBG_MaxLevel>, >>> errflag=<optimized out>) at src/mpi/coll/helper_fns.c:302 >>> mpi_errno = <optimized out> >>> request_ptr = 0x1004bf7e0 <MPID_Request_direct+1760> >>> #7 0x0000000100246031 in MPIR_Bcast_binomial (buffer=<optimized out>, >>> count=<optimized out>, datatype=<optimized out>, root=<optimized out>, >>> comm_ptr=<optimized out>, errflag=<optimized out>) at >>> src/mpi/coll/bcast.c:280 >>> nbytes = <optimized out> >>> mpi_errno_ret = <optimized out> >>> mpi_errno = 0 >>> comm_size = <optimized out> >>> rank = 2 >>> type_size = <optimized out> >>> tmp_buf = 0x0 >>> position = <optimized out> >>> relative_rank = <optimized out> >>> mask = <optimized out> >>> src = <optimized out> >>> status = <optimized out> >>> recvd_size = <optimized out> >>> dst = <optimized out> >>> #8 0x00000001002455a3 in MPIR_SMP_Bcast (buffer=<optimized out>, >>> count=<optimized out>, datatype=<optimized out>, root=<optimized out>, >>> comm_ptr=<optimized out>, errflag=<optimized out>) at >>> src/mpi/coll/bcast.c:1087 >>> mpi_errno_ = <error reading variable mpi_errno_ (Cannot access >>> memory at address 0x0)> >>> mpi_errno = <optimized out> >>> mpi_errno_ret = <optimized out> >>> nbytes = <optimized out> >>> type_size = <optimized out> >>> status = <optimized out> >>> recvd_size = <optimized out> >>> #9 MPIR_Bcast_intra (buffer=0x100917c30, count=<optimized out>, >>> datatype=<optimized out>, root=1, comm_ptr=<optimized out>, >>> errflag=<optimized out>) at src/mpi/coll/bcast.c:1245 >>> nbytes = <optimized out> >>> mpi_errno_ret = <error reading variable mpi_errno_ret (Cannot >>> access memory at address 0x0)> >>> mpi_errno = <optimized out> >>> type_size = <optimized out> >>> comm_size = <optimized out> >>> #10 0x000000010024751e in MPIR_Bcast (buffer=<optimized out>, >>> count=<optimized out>, datatype=<optimized out>, root=<optimized out>, >>> comm_ptr=0x0, errflag=<optimized out>) at src/mpi/coll/bcast.c:1475 >>> mpi_errno = <optimized out> >>> #11 MPIR_Bcast_impl (buffer=0x1004bf7e0 <MPID_Request_direct+1760>, >>> count=-269488145, datatype=-16, root=0, comm_ptr=0x0, >>> errflag=0x1004bf100 <MPID_Request_direct>) at >>> src/mpi/coll/bcast.c:1451 >>> mpi_errno = <optimized out> >>> #12 0x00000001000f3c24 in MPI_Bcast (buffer=<optimized out>, count=7, >>> datatype=1275069445, root=1, comm=<optimized out>) at >>> src/mpi/coll/bcast.c:1585 >>> errflag = 2885681152 >>> mpi_errno = <optimized out> >>> comm_ptr = <optimized out> >>> #13 0x0000000100001df7 in run_test<int> (my_rank=2, >>> num_ranks=<optimized out>, count=<optimized out>, root_rank=1, >>> datatype=@0x7fff5fbfeaec: 1275069445, iterations=<optimized out>) at >>> bcast_test.cpp:83 >>> No locals. >>> #14 0x00000001000019cd in main (argc=<optimized out>, argv=<optimized >>> out>) at bcast_test.cpp:137 >>> root_rank = <optimized out> >>> count = <optimized out> >>> iterations = <optimized out> >>> my_rank = 4978656 >>> num_errors = <optimized out> >>> runtime_ns = <optimized out> >>> stats = {<std::__1::__basic_string_common<true>> = {<No data >>> fields>}, __r_ = >>> {<std::__1::__libcpp_compressed_pair_imp<std::__1::basic_string<char, >>> std::__1::char_traits<char>, std::__1::allocator<char> >::__rep, >>> std::__1::allocator<char>, 2>> = {<std::__1::allocator<char>> = {<No >>> data fields>}, __first_ = {{__l = {__cap_ = 17289301308300324847, >>> __size_ = 17289301308300324847, __data_ = 0xefefefefefefefef <error: >>> Cannot access memory at address 0xefefefefefefefef>} >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH-3.2: SIGSEGV in MPID_Request_create () at src/mpid/ch3/src/ch3u_request.c:101
by Mark Davis 12 Aug '16

12 Aug '16

I've tried both git HEAD (3ea7589) as well as the August 1 master snapshot and am having trouble building it; I'm getting the same error in both cases. I've configured with --enable-g=most but otherwise it's all default. I'm running on OSX (Darwin - 15.6.0) and clang 3.8.1. It's erroring on compiling src/mpi/attr/lib_libpmpi_la-attr_delete.lo due to an issue with the macro MPL_free Has anyone seen this before? I'm including the full error trace below: Making all in . CC src/mpi/attr/lib_libpmpi_la-attr_delete.lo In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:217: ./src/include/mpir_request.h:281:13: warning: multi-character character constant [-Wmultichar] MPL_free(req->u.ureq.greq_fns); ^ /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: note: expanded from macro 'MPL_free' #define MPL_free(a) free((void *)(a)) ^ ./src/include/mpir_mem.h:92:27: note: expanded from macro 'free' #define free(a) 'Error use MPL_free' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:217: ./src/include/mpir_request.h:281:13: warning: character constant too long for its type /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: note: expanded from macro 'MPL_free' #define MPL_free(a) free((void *)(a)) ^ ./src/include/mpir_mem.h:92:27: note: expanded from macro 'free' #define free(a) 'Error use MPL_free' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:217: ./src/include/mpir_request.h:281:13: error: expected ';' after expression /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: note: expanded from macro 'MPL_free' #define MPL_free(a) free((void *)(a)) ^ ./src/include/mpir_mem.h:92:50: note: expanded from macro 'free' #define free(a) 'Error use MPL_free' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:217: ./src/include/mpir_request.h:281:13: error: expected expression /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: note: expanded from macro 'MPL_free' #define MPL_free(a) free((void *)(a)) ^ ./src/include/mpir_mem.h:92:51: note: expanded from macro 'free' #define free(a) 'Error use MPL_free' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:225: In file included from ./src/include/mpir_cvars.h:17: In file included from ./src/include/mpitimpl.h:18: ./src/include/mpir_utarray.h:238:43: warning: multi-character character constant [-Wmultichar] *_dst = (*_src == NULL) ? NULL : (char*)utarray_strdup_(*_src); ^ ./src/include/mpir_utarray.h:56:33: note: expanded from macro 'utarray_strdup_' #define utarray_strdup_(x_) MPL_strdup(x_) ^ /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:17:20: note: expanded from macro 'MPL_strdup' #define MPL_strdup strdup ^ ./src/include/mpir_mem.h:100:27: note: expanded from macro 'strdup' #define strdup(a) 'Error use MPL_strdup' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:225: In file included from ./src/include/mpir_cvars.h:17: In file included from ./src/include/mpitimpl.h:18: ./src/include/mpir_utarray.h:238:43: warning: character constant too long for its type ./src/include/mpir_utarray.h:56:33: note: expanded from macro 'utarray_strdup_' #define utarray_strdup_(x_) MPL_strdup(x_) ^ /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:17:20: note: expanded from macro 'MPL_strdup' #define MPL_strdup strdup ^ ./src/include/mpir_mem.h:100:27: note: expanded from macro 'strdup' #define strdup(a) 'Error use MPL_strdup' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:225: In file included from ./src/include/mpir_cvars.h:17: In file included from ./src/include/mpitimpl.h:18: ./src/include/mpir_utarray.h:238:43: error: expected ';' after expression ./src/include/mpir_utarray.h:56:33: note: expanded from macro 'utarray_strdup_' #define utarray_strdup_(x_) MPL_strdup(x_) ^ /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:17:20: note: expanded from macro 'MPL_strdup' #define MPL_strdup strdup ^ ./src/include/mpir_mem.h:100:50: note: expanded from macro 'strdup' #define strdup(a) 'Error use MPL_strdup' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:225: In file included from ./src/include/mpir_cvars.h:17: In file included from ./src/include/mpitimpl.h:18: ./src/include/mpir_utarray.h:238:43: error: expected expression ./src/include/mpir_utarray.h:56:33: note: expanded from macro 'utarray_strdup_' #define utarray_strdup_(x_) MPL_strdup(x_) ^ /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:17:20: note: expanded from macro 'MPL_strdup' #define MPL_strdup strdup ^ ./src/include/mpir_mem.h:100:51: note: expanded from macro 'strdup' #define strdup(a) 'Error use MPL_strdup' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:225: In file included from ./src/include/mpir_cvars.h:17: In file included from ./src/include/mpitimpl.h:18: ./src/include/mpir_utarray.h:242:14: warning: multi-character character constant [-Wmultichar] if (*eltc) utarray_free_(*eltc); ^ ./src/include/mpir_utarray.h:54:33: note: expanded from macro 'utarray_free_' #define utarray_free_(x_) MPL_free(x_) ^ /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: note: expanded from macro 'MPL_free' #define MPL_free(a) free((void *)(a)) ^ ./src/include/mpir_mem.h:92:27: note: expanded from macro 'free' #define free(a) 'Error use MPL_free' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:225: In file included from ./src/include/mpir_cvars.h:17: In file included from ./src/include/mpitimpl.h:18: ./src/include/mpir_utarray.h:242:14: warning: character constant too long for its type ./src/include/mpir_utarray.h:54:33: note: expanded from macro 'utarray_free_' #define utarray_free_(x_) MPL_free(x_) ^ /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: note: expanded from macro 'MPL_free' #define MPL_free(a) free((void *)(a)) ^ ./src/include/mpir_mem.h:92:27: note: expanded from macro 'free' #define free(a) 'Error use MPL_free' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:225: In file included from ./src/include/mpir_cvars.h:17: In file included from ./src/include/mpitimpl.h:18: ./src/include/mpir_utarray.h:242:14: error: expected ';' after expression ./src/include/mpir_utarray.h:54:33: note: expanded from macro 'utarray_free_' #define utarray_free_(x_) MPL_free(x_) ^ /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: note: expanded from macro 'MPL_free' #define MPL_free(a) free((void *)(a)) ^ ./src/include/mpir_mem.h:92:50: note: expanded from macro 'free' #define free(a) 'Error use MPL_free' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:225: In file included from ./src/include/mpir_cvars.h:17: In file included from ./src/include/mpitimpl.h:18: ./src/include/mpir_utarray.h:242:14: error: expected expression ./src/include/mpir_utarray.h:54:33: note: expanded from macro 'utarray_free_' #define utarray_free_(x_) MPL_free(x_) ^ /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: note: expanded from macro 'MPL_free' #define MPL_free(a) free((void *)(a)) ^ ./src/include/mpir_mem.h:92:51: note: expanded from macro 'free' #define free(a) 'Error use MPL_free' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:61:29: warning: multi-character character constant [-Wmultichar] nIndirect = (int *) MPL_calloc(objmem->indirect_size, sizeof(int)); ^ /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: note: expanded from macro 'MPL_calloc' #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) ^ ./src/include/mpir_mem.h:91:27: note: expanded from macro 'calloc' #define calloc(a,b) 'Error use MPL_calloc' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:61:29: warning: character constant too long for its type /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: note: expanded from macro 'MPL_calloc' #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) ^ ./src/include/mpir_mem.h:91:27: note: expanded from macro 'calloc' #define calloc(a,b) 'Error use MPL_calloc' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:61:29: error: expected ';' after expression /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: note: expanded from macro 'MPL_calloc' #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) ^ ./src/include/mpir_mem.h:91:50: note: expanded from macro 'calloc' #define calloc(a,b) 'Error use MPL_calloc' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:61:29: error: expected expression /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: note: expanded from macro 'MPL_calloc' #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) ^ ./src/include/mpir_mem.h:91:51: note: expanded from macro 'calloc' #define calloc(a,b) 'Error use MPL_calloc' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:117:9: warning: multi-character character constant [-Wmultichar] MPL_free(nIndirect); ^ /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: note: expanded from macro 'MPL_free' #define MPL_free(a) free((void *)(a)) ^ ./src/include/mpir_mem.h:92:27: note: expanded from macro 'free' #define free(a) 'Error use MPL_free' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:117:9: warning: character constant too long for its type /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: note: expanded from macro 'MPL_free' #define MPL_free(a) free((void *)(a)) ^ ./src/include/mpir_mem.h:92:27: note: expanded from macro 'free' #define free(a) 'Error use MPL_free' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:117:9: error: expected ';' after expression /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: note: expanded from macro 'MPL_free' #define MPL_free(a) free((void *)(a)) ^ ./src/include/mpir_mem.h:92:50: note: expanded from macro 'free' #define free(a) 'Error use MPL_free' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:117:9: error: expected expression /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: note: expanded from macro 'MPL_free' #define MPL_free(a) free((void *)(a)) ^ ./src/include/mpir_mem.h:92:51: note: expanded from macro 'free' #define free(a) 'Error use MPL_free' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:179:9: warning: multi-character character constant [-Wmultichar] MPL_free((*indirect)[i]); ^ /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: note: expanded from macro 'MPL_free' #define MPL_free(a) free((void *)(a)) ^ ./src/include/mpir_mem.h:92:27: note: expanded from macro 'free' #define free(a) 'Error use MPL_free' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:179:9: warning: character constant too long for its type /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: note: expanded from macro 'MPL_free' #define MPL_free(a) free((void *)(a)) ^ ./src/include/mpir_mem.h:92:27: note: expanded from macro 'free' #define free(a) 'Error use MPL_free' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:179:9: error: expected ';' after expression /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: note: expanded from macro 'MPL_free' #define MPL_free(a) free((void *)(a)) ^ ./src/include/mpir_mem.h:92:50: note: expanded from macro 'free' #define free(a) 'Error use MPL_free' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:179:9: error: expected expression /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: note: expanded from macro 'MPL_free' #define MPL_free(a) free((void *)(a)) ^ ./src/include/mpir_mem.h:92:51: note: expanded from macro 'free' #define free(a) 'Error use MPL_free' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:182:9: warning: multi-character character constant [-Wmultichar] MPL_free(indirect); ^ /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: note: expanded from macro 'MPL_free' #define MPL_free(a) free((void *)(a)) ^ ./src/include/mpir_mem.h:92:27: note: expanded from macro 'free' #define free(a) 'Error use MPL_free' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:182:9: warning: character constant too long for its type /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: note: expanded from macro 'MPL_free' #define MPL_free(a) free((void *)(a)) ^ ./src/include/mpir_mem.h:92:27: note: expanded from macro 'free' #define free(a) 'Error use MPL_free' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:182:9: error: expected ';' after expression /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: note: expanded from macro 'MPL_free' #define MPL_free(a) free((void *)(a)) ^ ./src/include/mpir_mem.h:92:50: note: expanded from macro 'free' #define free(a) 'Error use MPL_free' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:182:9: error: expected expression /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:110:26: note: expanded from macro 'MPL_free' #define MPL_free(a) free((void *)(a)) ^ ./src/include/mpir_mem.h:92:51: note: expanded from macro 'free' #define free(a) 'Error use MPL_free' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:249:30: warning: multi-character character constant [-Wmultichar] *indirect = (void *) MPL_calloc(indirect_num_blocks, sizeof(void *)); ^ /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: note: expanded from macro 'MPL_calloc' #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) ^ ./src/include/mpir_mem.h:91:27: note: expanded from macro 'calloc' #define calloc(a,b) 'Error use MPL_calloc' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:249:30: warning: character constant too long for its type /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: note: expanded from macro 'MPL_calloc' #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) ^ ./src/include/mpir_mem.h:91:27: note: expanded from macro 'calloc' #define calloc(a,b) 'Error use MPL_calloc' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:249:30: error: expected ';' after expression /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: note: expanded from macro 'MPL_calloc' #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) ^ ./src/include/mpir_mem.h:91:50: note: expanded from macro 'calloc' #define calloc(a,b) 'Error use MPL_calloc' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:249:30: error: expected expression /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: note: expanded from macro 'MPL_calloc' #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) ^ ./src/include/mpir_mem.h:91:51: note: expanded from macro 'calloc' #define calloc(a,b) 'Error use MPL_calloc' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:264:26: warning: multi-character character constant [-Wmultichar] block_ptr = (void *) MPL_calloc(indirect_num_indices, obj_size); ^ /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: note: expanded from macro 'MPL_calloc' #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) ^ ./src/include/mpir_mem.h:91:27: note: expanded from macro 'calloc' #define calloc(a,b) 'Error use MPL_calloc' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:264:26: warning: character constant too long for its type /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: note: expanded from macro 'MPL_calloc' #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) ^ ./src/include/mpir_mem.h:91:27: note: expanded from macro 'calloc' #define calloc(a,b) 'Error use MPL_calloc' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:264:26: error: expected ';' after expression /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: note: expanded from macro 'MPL_calloc' #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) ^ ./src/include/mpir_mem.h:91:50: note: expanded from macro 'calloc' #define calloc(a,b) 'Error use MPL_calloc' ::: ^ In file included from src/mpi/attr/attr_delete.c:8: In file included from ./src/include/mpiimpl.h:228: ./src/include/mpir_handlemem.h:264:26: error: expected expression /Users/m/local/src/mpich-master-v3.2-370-g0d6412303488/src/mpl/include/mpl_trmem.h:109:26: note: expanded from macro 'MPL_calloc' #define MPL_calloc(a,b) calloc((size_t)(a),(size_t)(b)) ^ ./src/include/mpir_mem.h:91:51: note: expanded from macro 'calloc' #define calloc(a,b) 'Error use MPL_calloc' ::: ^ 18 warnings and 18 errors generated. make[2]: *** [src/mpi/attr/lib_libpmpi_la-attr_delete.lo] Error 1 make[1]: *** [all-recursive] Error 1 Makefile:10270: recipe for target 'all' failed gmake: *** [all] Error 2 On Thu, Aug 11, 2016 at 5:21 PM, Halim Amer <aamer(a)anl.gov> wrote: > This should be related to the alignment problem reported before > (http://lists.mpich.org/pipermail/discuss/2016-May/004764.html) > > We plan to include a fix in the 3.2.x bug fix release series. Meanwhile, > please try the repo version (git.mpich.org/mpich.git) which should not > suffer from this problem. > > --Halim > www.mcs.anl.gov/~aamer > > > On 8/11/16 8:48 AM, Mark Davis wrote: >> >> Hello, I'm running into a segfault when I run some relatively simple >> MPI programs. In this particular case, I'm running a small program in >> a loop that does MPI_Bcast, once per loop, within MPI_COMM_WORLD. The >> buffer consists of just 7 doubles. I'm running with 6 procs on a >> machine with 8 cores on OSX (Darwin - 15.6.0 Darwin Kernel Version >> 15.6.0: Thu Jun 23 18:25:34 PDT 2016; >> root:xnu-3248.60.10~1/RELEASE_X86_64 x86_64). When I run the same >> program with a smaller number of procs, the error usually doesn't show >> up. My compiler (both for compiling the MPICH source as well as my >> application) is clang 3.8.1. >> >> When I run the same program on linux, also with MPICH-3.2 (I believe >> the same exact source), compiled with gcc 5.3, I do not get this >> error. This seems to be something I get only with >> >> gdb shows the following stack trace. I have a feeling that this has >> something to do with my toolchain and/or libraries on my system given >> that I never get this error on my other system (linux). However, it's >> possible that there's an application bug as well. >> >> I'm running the MPICH-3.2 stable release; I haven't tried anything >> from the repository yet. >> >> Does anyone have any ideas about what's going on here? I'm happy to >> provide more details. >> >> Thank you, >> Mark >> >> >> Program received signal SIGSEGV, Segmentation fault. >> MPID_Request_create () at src/mpid/ch3/src/ch3u_request.c:101 >> 101 req->dev.ext_hdr_ptr = NULL; >> (gdb) bt full >> #0 MPID_Request_create () at src/mpid/ch3/src/ch3u_request.c:101 >> No locals. >> #1 0x00000001003ac4c9 in MPIDI_CH3U_Recvq_FDP_or_AEU >> (match=<optimized out>, foundp=0x7fff5fbfe2bc) at >> src/mpid/ch3/src/ch3u_recvq.c:830 >> proc_failure_bit_masked = <error reading variable >> proc_failure_bit_masked (Cannot access memory at address 0x1)> >> error_bit_masked = <error reading variable error_bit_masked >> (Cannot access memory at address 0x1)> >> prev_rreq = <optimized out> >> channel_matched = <optimized out> >> rreq = <optimized out> >> #2 0x00000001003d1ffe in MPIDI_CH3_PktHandler_EagerSend >> (vc=<optimized out>, pkt=0x1004b3fd8 <MPIU_DBG_MaxLevel>, >> buflen=0x7fff5fbfe440, rreqp=0x7fff5fbfe438) at >> src/mpid/ch3/src/ch3u_eager.c:629 >> mpi_errno = <error reading variable mpi_errno (Cannot access >> memory at address 0x0)> >> found = <error reading variable found (Cannot access memory at >> address 0xefefefefefefefef)> >> rreq = <optimized out> >> data_len = <optimized out> >> complete = <optimized out> >> #3 0x00000001003f6045 in MPID_nem_handle_pkt (vc=<optimized out>, >> buf=0x102ad07e0 "", buflen=<optimized out>) at >> src/mpid/ch3/channels/nemesis/src/ch3_progress.c:760 >> len = 140734799800192 >> mpi_errno = <optimized out> >> complete = <error reading variable complete (Cannot access >> memory at address 0x1)> >> rreq = <optimized out> >> #4 0x00000001003f4e41 in MPIDI_CH3I_Progress >> (progress_state=0x7fff5fbfe750, is_blocking=1) at >> src/mpid/ch3/channels/nemesis/src/ch3_progress.c:570 >> payload_len = 4299898840 >> cell_buf = <optimized out> >> rreq = <optimized out> >> vc = 0x102ad07e8 >> made_progress = <error reading variable made_progress (Cannot >> access memory at address 0x0)> >> mpi_errno = <optimized out> >> #5 0x000000010035386d in MPIC_Wait (request_ptr=<optimized out>, >> errflag=<optimized out>) at src/mpi/coll/helper_fns.c:225 >> progress_state = {ch = {completion_count = -1409286143}} >> mpi_errno = <error reading variable mpi_errno (Cannot access >> memory at address 0x0)> >> #6 0x0000000100353b10 in MPIC_Send (buf=0x100917c30, >> count=4299945096, datatype=-1581855963, dest=<optimized out>, >> tag=4975608, comm_ptr=0x1004b3fd8 <MPIU_DBG_MaxLevel>, >> errflag=<optimized out>) at src/mpi/coll/helper_fns.c:302 >> mpi_errno = <optimized out> >> request_ptr = 0x1004bf7e0 <MPID_Request_direct+1760> >> #7 0x0000000100246031 in MPIR_Bcast_binomial (buffer=<optimized out>, >> count=<optimized out>, datatype=<optimized out>, root=<optimized out>, >> comm_ptr=<optimized out>, errflag=<optimized out>) at >> src/mpi/coll/bcast.c:280 >> nbytes = <optimized out> >> mpi_errno_ret = <optimized out> >> mpi_errno = 0 >> comm_size = <optimized out> >> rank = 2 >> type_size = <optimized out> >> tmp_buf = 0x0 >> position = <optimized out> >> relative_rank = <optimized out> >> mask = <optimized out> >> src = <optimized out> >> status = <optimized out> >> recvd_size = <optimized out> >> dst = <optimized out> >> #8 0x00000001002455a3 in MPIR_SMP_Bcast (buffer=<optimized out>, >> count=<optimized out>, datatype=<optimized out>, root=<optimized out>, >> comm_ptr=<optimized out>, errflag=<optimized out>) at >> src/mpi/coll/bcast.c:1087 >> mpi_errno_ = <error reading variable mpi_errno_ (Cannot access >> memory at address 0x0)> >> mpi_errno = <optimized out> >> mpi_errno_ret = <optimized out> >> nbytes = <optimized out> >> type_size = <optimized out> >> status = <optimized out> >> recvd_size = <optimized out> >> #9 MPIR_Bcast_intra (buffer=0x100917c30, count=<optimized out>, >> datatype=<optimized out>, root=1, comm_ptr=<optimized out>, >> errflag=<optimized out>) at src/mpi/coll/bcast.c:1245 >> nbytes = <optimized out> >> mpi_errno_ret = <error reading variable mpi_errno_ret (Cannot >> access memory at address 0x0)> >> mpi_errno = <optimized out> >> type_size = <optimized out> >> comm_size = <optimized out> >> #10 0x000000010024751e in MPIR_Bcast (buffer=<optimized out>, >> count=<optimized out>, datatype=<optimized out>, root=<optimized out>, >> comm_ptr=0x0, errflag=<optimized out>) at src/mpi/coll/bcast.c:1475 >> mpi_errno = <optimized out> >> #11 MPIR_Bcast_impl (buffer=0x1004bf7e0 <MPID_Request_direct+1760>, >> count=-269488145, datatype=-16, root=0, comm_ptr=0x0, >> errflag=0x1004bf100 <MPID_Request_direct>) at >> src/mpi/coll/bcast.c:1451 >> mpi_errno = <optimized out> >> #12 0x00000001000f3c24 in MPI_Bcast (buffer=<optimized out>, count=7, >> datatype=1275069445, root=1, comm=<optimized out>) at >> src/mpi/coll/bcast.c:1585 >> errflag = 2885681152 >> mpi_errno = <optimized out> >> comm_ptr = <optimized out> >> #13 0x0000000100001df7 in run_test<int> (my_rank=2, >> num_ranks=<optimized out>, count=<optimized out>, root_rank=1, >> datatype=@0x7fff5fbfeaec: 1275069445, iterations=<optimized out>) at >> bcast_test.cpp:83 >> No locals. >> #14 0x00000001000019cd in main (argc=<optimized out>, argv=<optimized >> out>) at bcast_test.cpp:137 >> root_rank = <optimized out> >> count = <optimized out> >> iterations = <optimized out> >> my_rank = 4978656 >> num_errors = <optimized out> >> runtime_ns = <optimized out> >> stats = {<std::__1::__basic_string_common<true>> = {<No data >> fields>}, __r_ = >> {<std::__1::__libcpp_compressed_pair_imp<std::__1::basic_string<char, >> std::__1::char_traits<char>, std::__1::allocator<char> >::__rep, >> std::__1::allocator<char>, 2>> = {<std::__1::allocator<char>> = {<No >> data fields>}, __first_ = {{__l = {__cap_ = 17289301308300324847, >> __size_ = 17289301308300324847, __data_ = 0xefefefefefefefef <error: >> Cannot access memory at address 0xefefefefefefefef>} >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Managing process distribution
by Kenneth Raffenetti 12 Aug '16

12 Aug '16

You can specify the number of processes per host like so: mpiexec -hosts host1:4,host2:2 -n 6 ./mytask On 08/12/2016 10:43 AM, Doha Ehab wrote: > Hello, > > I am running mpi code on host1(quad core) and host2(dual core) > > mpiexec -hosts host1,host2 -n 6 ./mytask > > I want to assign 4 processes for host1 and 2 for host2. I tried --map-by > core but I found that the processes are distributed 3 for each. > > Regards > Doha > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Managing process distribution
by Doha Ehab 12 Aug '16

12 Aug '16

Hello, I am running mpi code on host1(quad core) and host2(dual core) mpiexec -hosts host1,host2 -n 6 ./mytask I want to assign 4 processes for host1 and 2 for host2. I tried --map-by core but I found that the processes are distributed 3 for each. Regards Doha _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0