May 2016 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Bug reporting for some mpi/rma tests
by Kenneth Raffenetti 31 May '16

31 May '16

Hi, On 05/31/2016 09:31 AM, Clément Foyer wrote: > Hi everyone, > > As I was using your tests I was surprised by some of them. > > The first one is test/mpi/rma/get_acc_local.c, as you accumulate NITER > times acc_val into out_val, initialized at 0. At each step of the loop > you check whether the out_val is equal to i times acc_val. As i start > from 0, you expect the result to be 0 instead of acc_val. I suppose you > should have had i start from 1, or change the test line 36 to "if > (out_val != acc_val * (i + 1)) {", with an adaptation of the debug > message as well. This test looks correct to me. Consider the values of the first few interations. Iteration 0 before Get_accumulate: i = 0, counter = 0, acc_val = 3, out_val = N/A; Iteration 0 after Get_accumulate: i = 0, counter = 3, acc_val = 3, out_val = 0 == (acc_val * i); Iteration 1 before Get_accumulate: i = 1, counter = 3, acc_val = 3, out_val = 0; Iteration 1 after Get_accumulate: i = 1, counter = 6, acc_val = 3, out_val = 3 == (acc_val * i); And so on... > > The second test in which I found errors is mpi/rma/selfrma.c. The > parameters given to the printf function at lines 66, 80 and 95 are in > the wrong order. It should be "j, winbuf[j], sbuf[j]" instead of > "winbuf[j], j, sbuf[j]". And at the line 95, the header to the debug > line is "AccVec", as it seems it should have been "GetVec". Yes, this looks like a bug. We will commit a fix soon. Thanks for reporting. Ken _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Bug reporting for some mpi/rma tests
by Clément Foyer 31 May '16

31 May '16

Hi everyone, As I was using your tests I was surprised by some of them. The first one is test/mpi/rma/get_acc_local.c, as you accumulate NITER times acc_val into out_val, initialized at 0. At each step of the loop you check whether the out_val is equal to i times acc_val. As i start from 0, you expect the result to be 0 instead of acc_val. I suppose you should have had i start from 1, or change the test line 36 to "if (out_val != acc_val * (i + 1)) {", with an adaptation of the debug message as well. The second test in which I found errors is mpi/rma/selfrma.c. The parameters given to the printf function at lines 66, 80 and 95 are in the wrong order. It should be "j, winbuf[j], sbuf[j]" instead of "winbuf[j], j, sbuf[j]". And at the line 95, the header to the debug line is "AccVec", as it seems it should have been "GetVec". I hope that was useful for you. If that was was unnecessary, please tell me so I won't bother you with this kind of message again. Sincerely yours, C. FOYER _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Barrier segmentation fault
by Eric A. Borisch 27 May '16

27 May '16

I believe that this line: (src/include/mpiimpl.h:1531) } MPID_Request ATTRIBUTE((__aligned__(32))); needs to be this: } ATTRIBUTE((__aligned__(32))) MPID_Request; From the GCC docs, the attribute needs to be just after the closing brace. As it is, it aligns single MPID_Request objects to be on a 32-byte boundary, but it does make the sizeof(MPID_Request) a multiple of 32. This, in combination with the compiler optimizing for the advertised-to-be-aligned singleton object with a movaps instruction causes the crash. Changing the above line and re-compiling mpich fixes the issue. There may be other instances of this issue in the codebase; I did not search... - Eric On Fri, May 27, 2016 at 12:42 AM, Ben Whale <ben(a)benwhale.com> wrote: > Ha! Sorry didn’t spot your previous email about XCode. Thanks for the hard work > > I’ve had issues with XCode updates in the past. In the mean time while this is fixed I’ll try building with an old version. > > Ben _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Barrier segmentation fault
by Eric A. Borisch 27 May '16

27 May '16

Well, this wasn't as trivial as I had hoped. With the latest version of Xcode -- 7.3.1 (7D1014) -- it appears there is some bad code being generated; specifically a movaps instruction with a pointer that isn't 16-byte aligned. This is different from what the buildbot [1] -- with Xcode 7.2 (7C68) -- generated. _MPID_Request_create: [....] 000000000012bcaf movq $0x0, 0x1c8(%rax) 000000000012bcba movq $0x0, 0x1c0(%rax) 000000000012bcc5 movq $0x0, 0x1b8(%rax) 000000000012bcd0 xorps %xmm0, %xmm0 000000000012bcd3 movaps %xmm0, 0x230(%rax) <<< CRASH 000000000012bcda movq $0x0, 0x240(%rax) 000000000012bce5 movl $0x2c000000, 0x210(%rax) ## imm = 0x2C000000 000000000012bcef popq %rbp At crash, %rax == 0x000000010af39048 + 230 = 0x10AF39258 (not 16-byte aligned). Fall down go boom. %rax is a pointer to a new handle object returned from MPIU_Handle_obj_alloc() if I'm reading things correctly. vs: _MPID_Request_create: [...] 000000000011d3db movq $0x0, 0x1c0(%rax) 000000000011d3e6 movq $0x0, 0x1b8(%rax) 000000000011d3f1 movq $0x0, 0x240(%rax) 000000000011d3fc movq $0x0, 0x238(%rax) 000000000011d407 movq $0x0, 0x230(%rax) 000000000011d412 movl $0x2c000000, 0x210(%rax) ## imm = 0x2C000000 000000000011d41c popq %rbp I'm not ready to dig into this further tonight, but wanted to post it up here for Ben and awareness. I don't know if the actual error is a "compiler error" in the emitted movaps instruction, or if a pointer that was advertised (in the code) to be 16-byte aligned wasn't for some reason... (If I use the builedbot package, the test example works; if I use the local build, I get a crash just like Ben.) If someone on the MPICH team has the latest XCode and could verify or refute, that would be helpful. We're at n=2/2 so far... - Eric [1] http://packages.macports.org/mpich-default/mpich-default-3.2_0+gcc5.darwin_… On Thu, May 26, 2016 at 8:53 AM, Eric A. Borisch <eborisch(a)gmail.com> wrote: > On Thu, May 26, 2016 at 7:21 AM, Ben Whale <ben(a)benwhale.com> wrote: >> Since I really expected this code to work I suspect that I’ve installed >> mpich incorrectly. >> >> I’m on OSX 10.11.4 and installed mpich using the command sudo port install >> mpich > > Ben, > > Output of "which mpicc", "which mpiexec", and "otool -L > minimal_working_example"? I'm hopeful there will be something > interesting from one of those; we can likely take this off-list unless > others would like to chime in. > > - Eric (I maintain the MacPorts mpich ports.) _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Barrier segmentation fault
by Eric A. Borisch 26 May '16

26 May '16

On Thu, May 26, 2016 at 7:21 AM, Ben Whale <ben(a)benwhale.com> wrote: > Since I really expected this code to work I suspect that I’ve installed > mpich incorrectly. > > I’m on OSX 10.11.4 and installed mpich using the command sudo port install > mpich Ben, Output of "which mpicc", "which mpiexec", and "otool -L minimal_working_example"? I'm hopeful there will be something interesting from one of those; we can likely take this off-list unless others would like to chime in. - Eric (I maintain the MacPorts mpich ports.) _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPI_Barrier segmentation fault
by Ben Whale 26 May '16

26 May '16

Hi I think I’ve done something stupid when installing mpich, so thanks in advance for your patience. The following is a minimal working example of the bug: #include <stdio.h> #include <mpi.h> int main(int argc, char** argv) { MPI_Init(NULL, NULL); printf("Before barrier\n"); MPI_Barrier(MPI_COMM_WORLD); printf("After barrier\n"); MPI_Finalize(); } I compile it with the command mpicc -o minimal_working_example minimal_working_example.c When I run it with the commandmpiexec -n 1 ./minimal_working_example I get the output Before barrier After barrier which is what I expect. When I run it with the command mpiexec -n 2 ./minimal_working_example I get the output Before barrier Before barrier =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 45072 RUNNING AT localhost = EXIT CODE: 11 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault: 11 (signal 11) This typically refers to a problem with your application. Please see the FAQ page for debugging suggestions which is not what I expect. For the record the same occurs when running the `comare_bcast` program found at, https://github.com/wesleykendall/mpitutorial/tree/gh-pages/tutorials/mpi-br… <https://github.com/wesleykendall/mpitutorial/tree/gh-pages/tutorials/mpi-br…>. Since I really expected this code to work I suspect that I’ve installed mpich incorrectly. I’m on OSX 10.11.4 and installed mpich using the command sudo port install mpich Any advice welcome. Thank you very much for your help. Ben _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] RMA calls with MPI Datatypes using structs
by Min Si 25 May '16

25 May '16

Hi Bruce, Thanks for this report and your test code. I believe this is a bug. We will fix it. You can track the status from this ticket. https://trac.mpich.org/projects/mpich/ticket/2335 Min On 5/25/16 11:20 AM, Palmer, Bruce J wrote: > Attached is my test code. I've been trying to emulate the gather-scatter semantics in Global Arrays using MPI RMA calls. The top of the file has a bunch of parameters that can be used to configure different protocols for implementing GA-style puts and gets. The failure I'm seeing seems to be independent of the choice of protocol. The parameter USE_STRUCTS can be used to switch between structs and subarrays. Let me know if you have more questions. > > Bruce > > > > -----Original Message----- > From: Rob Latham [mailto:[email protected]] > Sent: Wednesday, May 25, 2016 9:07 AM > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] RMA calls with MPI Datatypes using structs > > > > On 05/25/2016 10:40 AM, Palmer, Bruce J wrote: >> Hi, >> >> I've been trying to implement code using MPI_Get and MPI_Put that move >> around data using MPI_Datatypes. If I create the data type using >> MPI_Type_create_subarray, everything works fine. If I create the data >> type using MPI_Type_create_struct then I get a failure if the amount >> of data I'm moving around becomes large. At the moment the crossover >> point is moving a total of a million doubles on 8 cores (125000 >> doubles per core), which doesn't seem to be excessively large. The >> error I'm seeing on a Linux cluster using and Infiniband interconnect >> is >> >> Assertion failed in file >> ./src/mpid/ch3/channels/nemesis/include/mpid_nem_inline.h at line 406 >> >> : MPID_NEM_MPICH_DATA_LEN - buf_offset >= ext_hdr_sz >> >> internal ABORT - process 1 >> >> Note that the same code runs fine if the data type is created using >> the subarray call. It looks like some internal limit is being >> exceeded, but if so, I think the threshold may be set a bit on the low >> side. Does anyone know more about this? > Doesn't look like something we've seen before. Can you send us the > working subarray and non-working struct datatype? > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] RMA calls with MPI Datatypes using structs
by Palmer, Bruce J 25 May '16

25 May '16

Attached is my test code. I've been trying to emulate the gather-scatter semantics in Global Arrays using MPI RMA calls. The top of the file has a bunch of parameters that can be used to configure different protocols for implementing GA-style puts and gets. The failure I'm seeing seems to be independent of the choice of protocol. The parameter USE_STRUCTS can be used to switch between structs and subarrays. Let me know if you have more questions. Bruce -----Original Message----- From: Rob Latham [mailto:[email protected]] Sent: Wednesday, May 25, 2016 9:07 AM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] RMA calls with MPI Datatypes using structs On 05/25/2016 10:40 AM, Palmer, Bruce J wrote: > Hi, > > I've been trying to implement code using MPI_Get and MPI_Put that move > around data using MPI_Datatypes. If I create the data type using > MPI_Type_create_subarray, everything works fine. If I create the data > type using MPI_Type_create_struct then I get a failure if the amount > of data I'm moving around becomes large. At the moment the crossover > point is moving a total of a million doubles on 8 cores (125000 > doubles per core), which doesn't seem to be excessively large. The > error I'm seeing on a Linux cluster using and Infiniband interconnect > is > > Assertion failed in file > ./src/mpid/ch3/channels/nemesis/include/mpid_nem_inline.h at line 406 > > : MPID_NEM_MPICH_DATA_LEN - buf_offset >= ext_hdr_sz > > internal ABORT - process 1 > > Note that the same code runs fine if the data type is created using > the subarray call. It looks like some internal limit is being > exceeded, but if so, I think the threshold may be set a bit on the low > side. Does anyone know more about this? Doesn't look like something we've seen before. Can you send us the working subarray and non-working struct datatype? _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] RMA calls with MPI Datatypes using structs
by Rob Latham 25 May '16

25 May '16

On 05/25/2016 10:40 AM, Palmer, Bruce J wrote: > Hi, > > I’ve been trying to implement code using MPI_Get and MPI_Put that move > around data using MPI_Datatypes. If I create the data type using > MPI_Type_create_subarray, everything works fine. If I create the data > type using MPI_Type_create_struct then I get a failure if the amount of > data I’m moving around becomes large. At the moment the crossover point > is moving a total of a million doubles on 8 cores (125000 doubles per > core), which doesn’t seem to be excessively large. The error I’m seeing > on a Linux cluster using and Infiniband interconnect is > > Assertion failed in file > ./src/mpid/ch3/channels/nemesis/include/mpid_nem_inline.h at line 406 > > : MPID_NEM_MPICH_DATA_LEN - buf_offset >= ext_hdr_sz > > internal ABORT - process 1 > > Note that the same code runs fine if the data type is created using the > subarray call. It looks like some internal limit is being exceeded, but > if so, I think the threshold may be set a bit on the low side. Does > anyone know more about this? Doesn't look like something we've seen before. Can you send us the working subarray and non-working struct datatype? _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

RMA calls with MPI Datatypes using structs
by Palmer, Bruce J 25 May '16

25 May '16

Hi, I've been trying to implement code using MPI_Get and MPI_Put that move around data using MPI_Datatypes. If I create the data type using MPI_Type_create_subarray, everything works fine. If I create the data type using MPI_Type_create_struct then I get a failure if the amount of data I'm moving around becomes large. At the moment the crossover point is moving a total of a million doubles on 8 cores (125000 doubles per core), which doesn't seem to be excessively large. The error I'm seeing on a Linux cluster using and Infiniband interconnect is Assertion failed in file ./src/mpid/ch3/channels/nemesis/include/mpid_nem_inline.h at line 406 : MPID_NEM_MPICH_DATA_LEN - buf_offset >= ext_hdr_sz internal ABORT - process 1 Note that the same code runs fine if the data type is created using the subarray call. It looks like some internal limit is being exceeded, but if so, I think the threshold may be set a bit on the low side. Does anyone know more about this? Bruce Palmer _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0