- devel - mailman3.cels.anl.gov

Re: [mpich-devel] Win questions
by Jim Dinan 09 Jun '13

09 Jun '13

Hi Bob, Active target separates exposure and access epochs. It is erroneous for a process to engage in more than one exposure epoch or more than one access epoch per window. So it's ok for many processes to expose (post/wait) to me concurrently, but for a given window I can access only one (start/complete) at a time. Your example code looks ok to me. Are you getting an error message? ~Jim. On Jun 7, 2013 9:11 AM, "Bob Cernohous" <bobc(a)us.ibm.com> wrote: > I haven't worked on rma before but was working on a problem and ran into > this **comment in MPI_Win_post: > > "Starts an RMA exposure epoch for the local window associated with win. > **Only the processes belonging to group should access the window with RMA > calls on win during this epoch. Each process in group must issue a > matching call to MPI_Win_start. MPI_Win_post does not block." > > Would overlapping epochs be violating the ** line? I decided I probably > need to support this but I wondered if it's bending or breaking the 'rules'? > > The problem (code at the bottom of this email) is using a cartesian > communicator and alternating "left/right' accumulates with 'up/down' > accumulates on a single win. So: > > - Ranks 0,1,2,3 are doing a left/right accumulate. > - Ranks 4,5,6,7 are doing a left/right accumulate. > - ... > > and then sometimes... > > - Ranks 0,1,2,3 complete and enter the 'up/down' accumulate epoch > -- Rank 0 does MPI_Win_post to ranks 4,12 > -- Rank 1 doesn MPI_Win_post to ranks 5,13 > ... > > So is Rank 0 posting to Rank 4 while 4 is still in the epoch with 5/6/7 a > violation of "Only the processes belonging to group should access the > window with RMA calls on win during this epoch"? From Rank 4's point of > view, rank 0 isn't in the group for the current win/epoch. > > Putting a barrier (or something) in between or using two different win's > fixes it. I like using two win's since it separates the epochs and clearly > doesn't use the wrong group/rank on the win. > > /* RMA transfers in left-right direction */ > MPI_Win_post(grp_lr, 0, win); > MPI_Win_start(grp_lr, 0, win); > MPI_Accumulate(&i, 1, MPI_INT, ranks_lr[LEFT] , 0, 1, MPI_INT, > MPI_SUM, win); > MPI_Accumulate(&i, 1, MPI_INT, ranks_lr[RIGHT], 0, 1, MPI_INT, > MPI_SUM, win); > MPI_Win_complete(win); > MPI_Win_wait(win); > > /* RMA transfers in up-down direction */ > MPI_Win_post(grp_ud, 0, win); > MPI_Win_start(grp_ud, 0, win); > MPI_Accumulate(&i, 1, MPI_INT, ranks_ud[UP] , 0, 1, MPI_INT, MPI_SUM, > win); > MPI_Accumulate(&i, 1, MPI_INT, ranks_ud[DOWN], 0, 1, MPI_INT, MPI_SUM, > win); > MPI_Win_complete(win); > MPI_Win_wait(win); > >

1 0

Win questions
by Bob Cernohous 07 Jun '13

07 Jun '13

I haven't worked on rma before but was working on a problem and ran into this **comment in MPI_Win_post: "Starts an RMA exposure epoch for the local window associated with win. **Only the processes belonging to group should access the window with RMA calls on win during this epoch. Each process in group must issue a matching call to MPI_Win_start. MPI_Win_post does not block." Would overlapping epochs be violating the ** line? I decided I probably need to support this but I wondered if it's bending or breaking the 'rules'? The problem (code at the bottom of this email) is using a cartesian communicator and alternating "left/right' accumulates with 'up/down' accumulates on a single win. So: - Ranks 0,1,2,3 are doing a left/right accumulate. - Ranks 4,5,6,7 are doing a left/right accumulate. - ... and then sometimes... - Ranks 0,1,2,3 complete and enter the 'up/down' accumulate epoch -- Rank 0 does MPI_Win_post to ranks 4,12 -- Rank 1 doesn MPI_Win_post to ranks 5,13 ... So is Rank 0 posting to Rank 4 while 4 is still in the epoch with 5/6/7 a violation of "Only the processes belonging to group should access the window with RMA calls on win during this epoch"? From Rank 4's point of view, rank 0 isn't in the group for the current win/epoch. Putting a barrier (or something) in between or using two different win's fixes it. I like using two win's since it separates the epochs and clearly doesn't use the wrong group/rank on the win. /* RMA transfers in left-right direction */ MPI_Win_post(grp_lr, 0, win); MPI_Win_start(grp_lr, 0, win); MPI_Accumulate(&i, 1, MPI_INT, ranks_lr[LEFT] , 0, 1, MPI_INT, MPI_SUM, win); MPI_Accumulate(&i, 1, MPI_INT, ranks_lr[RIGHT], 0, 1, MPI_INT, MPI_SUM, win); MPI_Win_complete(win); MPI_Win_wait(win); /* RMA transfers in up-down direction */ MPI_Win_post(grp_ud, 0, win); MPI_Win_start(grp_ud, 0, win); MPI_Accumulate(&i, 1, MPI_INT, ranks_ud[UP] , 0, 1, MPI_INT, MPI_SUM, win); MPI_Accumulate(&i, 1, MPI_INT, ranks_ud[DOWN], 0, 1, MPI_INT, MPI_SUM, win); MPI_Win_complete(win); MPI_Win_wait(win);

1 0

Re: [mpich-devel] MPIR_ERRTEST_ARGNULL and count=0 (or type_size=0)
by Jeff Hammond 03 Jun '13

03 Jun '13

Hi Bill, When I went looking for examples of ERRTEST_ARGNULL, I recall at least most of them being for internally allocated buffers, i.e. used in the MPICH-ized version of "void * b = malloc(); assert(b!=NULL);" I'll look into making both of the changes you describe. It is obviously going to be much more useful to have ERRTEST_USERBUFFER return a message like ERRTEST_ARGNULL. Best, Jeff On Mon, Jun 3, 2013 at 8:25 AM, William Gropp <wgropp(a)illinois.edu> wrote: > The reason that you are not seeing the MPIR_ERRTEST_USERBUFFER is that the > original intent of the design is that easy error tests are carried out in > the error test block and anything else must be handled elsewhere. For > example, some of the group operations have consistency checks in the body of > the code, rather than in the error test block. Because MPI_BOTTOM with an > MPI Datatype containing non-zero offsets is always valid, the simple checks > for NULL were originally not performed in any of the routines. The > ERRTEST_USERBUFFER check will catch many (though not all) null buffers at > relatively low cost, though since other parts of the code must also check > for null, it should not be considered necessary or sufficient. > > Its probably worth adding a comment that it is incorrect to use > ERRTEST_ARGNULL for communication buffers in every routine that has one, > since this issue comes up every few years. > > Also, the ERRTEST_USERBUFFER should include the parameter name and add an > error message that includes the parameter name, since many routines have > multiple communication buffers to which this can be applied. > > Bill > > William Gropp > Director, Parallel Computing Institute > Deputy Director for Research > Institute for Advanced Computing Applications and Technologies > Thomas M. Siebel Chair in Computer Science > University of Illinois Urbana-Champaign > > > > > On May 30, 2013, at 6:40 AM, Jeff Hammond wrote: > > The check for NO_OP in *get_accumulate was most certainly present. > That was the first thing I checked when > MPI_Get_accumulate(MPI_REPLACE) failed to behave like MPI_Get w.r.t. > ARMCI-MPI test codes. > > I'm going to supplement the RMA test suite so that this issue and ones > like it would have been caught by MPICH rather than waiting until the > very last ARMCI-MPI test to be revealed. > > Jeff > > On Thu, May 30, 2013 at 4:53 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > > Ugh, deep shame upon me and whoever reviewed my patch! Lisandro is > > correct. :) > > > For accumulate ops, the op can also cause a buffer to be ignored (e.g. > > GACC/FOP with NO_OP). I think there is code to handle this, but keep an > > eye out for it. > > > ~Jim. > > > On 05/29/2013 03:08 PM, Lisandro Dalcin wrote: > > On 29 May 2013 22:49, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: > > It seems that I didn't understand MPI_BOTTOM completely, so the error > > checks needed to be different than described below, but the high-level > > issues are the same. > > > Interested parties may consult > > http://trac.mpich.org/projects/mpich/ticket/1863 for details and > > further discussion. > > > > Please take a look at the macro MPIR_ERRTEST_USERBUFFER . > > > > > -- > > Lisandro Dalcin > > --------------- > > CIMEC (INTEC/CONICET-UNL) > > Predio CONICET-Santa Fe > > Colectora RN 168 Km 472, Paraje El Pozo > > 3000 Santa Fe, Argentina > > Tel: +54-342-4511594 (ext 1011) > > Tel/Fax: +54-342-4511169 > > > > > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > ALCF docs: http://www.alcf.anl.gov/user-guides > > -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [mpich-devel] MPIR_ERRTEST_ARGNULL and count=0 (or type_size=0)
by William Gropp 03 Jun '13

03 Jun '13

The reason that you are not seeing the MPIR_ERRTEST_USERBUFFER is that the original intent of the design is that easy error tests are carried out in the error test block and anything else must be handled elsewhere. For example, some of the group operations have consistency checks in the body of the code, rather than in the error test block. Because MPI_BOTTOM with an MPI Datatype containing non-zero offsets is always valid, the simple checks for NULL were originally not performed in any of the routines. The ERRTEST_USERBUFFER check will catch many (though not all) null buffers at relatively low cost, though since other parts of the code must also check for null, it should not be considered necessary or sufficient. Its probably worth adding a comment that it is incorrect to use ERRTEST_ARGNULL for communication buffers in every routine that has one, since this issue comes up every few years. Also, the ERRTEST_USERBUFFER should include the parameter name and add an error message that includes the parameter name, since many routines have multiple communication buffers to which this can be applied. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Thomas M. Siebel Chair in Computer Science University of Illinois Urbana-Champaign On May 30, 2013, at 6:40 AM, Jeff Hammond wrote: > The check for NO_OP in *get_accumulate was most certainly present. > That was the first thing I checked when > MPI_Get_accumulate(MPI_REPLACE) failed to behave like MPI_Get w.r.t. > ARMCI-MPI test codes. > > I'm going to supplement the RMA test suite so that this issue and ones > like it would have been caught by MPICH rather than waiting until the > very last ARMCI-MPI test to be revealed. > > Jeff > > On Thu, May 30, 2013 at 4:53 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: >> Ugh, deep shame upon me and whoever reviewed my patch! Lisandro is >> correct. :) >> >> For accumulate ops, the op can also cause a buffer to be ignored (e.g. >> GACC/FOP with NO_OP). I think there is code to handle this, but keep an >> eye out for it. >> >> ~Jim. >> >> On 05/29/2013 03:08 PM, Lisandro Dalcin wrote: >>> On 29 May 2013 22:49, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: >>>> It seems that I didn't understand MPI_BOTTOM completely, so the error >>>> checks needed to be different than described below, but the high-level >>>> issues are the same. >>>> >>>> Interested parties may consult >>>> http://trac.mpich.org/projects/mpich/ticket/1863 for details and >>>> further discussion. >>>> >>> >>> Please take a look at the macro MPIR_ERRTEST_USERBUFFER . >>> >>> >>> >>> -- >>> Lisandro Dalcin >>> --------------- >>> CIMEC (INTEC/CONICET-UNL) >>> Predio CONICET-Santa Fe >>> Colectora RN 168 Km 472, Paraje El Pozo >>> 3000 Santa Fe, Argentina >>> Tel: +54-342-4511594 (ext 1011) >>> Tel/Fax: +54-342-4511169 >>> >> > > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [mpich-devel] "create new ticket" text
by Ken Raffenetti 30 May '13

30 May '13

Mike, I have updated the text on this page to reflect the new names/locations. Ken ----- Original Message ----- > From: "Michael Blocksome" <blocksom(a)us.ibm.com> > To: devel(a)mpich.org > Sent: Wednesday, May 29, 2013 2:01:38 PM > Subject: [mpich-devel] "create new ticket" text > > FYI, the text on the trac page ... > > https://trac.mpich.org/projects/mpich/newticket > > ... talks about "MPICH2" and "mpich-discuss(a)mcs.anl.org". I think > these terms have migrated away by now. > > Michael Blocksome > Blue Gene Messaging > blocksom(a)us.ibm.com >

1 0

Re: [mpich-devel] MPIR_ERRTEST_ARGNULL and count=0 (or type_size=0)
by Jeff Hammond 30 May '13

30 May '13

The check for NO_OP in *get_accumulate was most certainly present. That was the first thing I checked when MPI_Get_accumulate(MPI_REPLACE) failed to behave like MPI_Get w.r.t. ARMCI-MPI test codes. I'm going to supplement the RMA test suite so that this issue and ones like it would have been caught by MPICH rather than waiting until the very last ARMCI-MPI test to be revealed. Jeff On Thu, May 30, 2013 at 4:53 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > Ugh, deep shame upon me and whoever reviewed my patch! Lisandro is > correct. :) > > For accumulate ops, the op can also cause a buffer to be ignored (e.g. > GACC/FOP with NO_OP). I think there is code to handle this, but keep an > eye out for it. > > ~Jim. > > On 05/29/2013 03:08 PM, Lisandro Dalcin wrote: >> On 29 May 2013 22:49, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: >>> It seems that I didn't understand MPI_BOTTOM completely, so the error >>> checks needed to be different than described below, but the high-level >>> issues are the same. >>> >>> Interested parties may consult >>> http://trac.mpich.org/projects/mpich/ticket/1863 for details and >>> further discussion. >>> >> >> Please take a look at the macro MPIR_ERRTEST_USERBUFFER . >> >> >> >> -- >> Lisandro Dalcin >> --------------- >> CIMEC (INTEC/CONICET-UNL) >> Predio CONICET-Santa Fe >> Colectora RN 168 Km 472, Paraje El Pozo >> 3000 Santa Fe, Argentina >> Tel: +54-342-4511594 (ext 1011) >> Tel/Fax: +54-342-4511169 >> > -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [mpich-devel] MPIR_ERRTEST_ARGNULL and count=0 (or type_size=0)
by Jim Dinan 30 May '13

30 May '13

Ugh, deep shame upon me and whoever reviewed my patch! Lisandro is correct. :) For accumulate ops, the op can also cause a buffer to be ignored (e.g. GACC/FOP with NO_OP). I think there is code to handle this, but keep an eye out for it. ~Jim. On 05/29/2013 03:08 PM, Lisandro Dalcin wrote: > On 29 May 2013 22:49, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: >> It seems that I didn't understand MPI_BOTTOM completely, so the error >> checks needed to be different than described below, but the high-level >> issues are the same. >> >> Interested parties may consult >> http://trac.mpich.org/projects/mpich/ticket/1863 for details and >> further discussion. >> > > Please take a look at the macro MPIR_ERRTEST_USERBUFFER . > > > > -- > Lisandro Dalcin > --------------- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169 >

1 0

Re: [mpich-devel] MPIR_ERRTEST_ARGNULL and count=0 (or type_size=0)
by Jeff Hammond 29 May '13

29 May '13

I don't know if you read http://trac.mpich.org/projects/mpich/ticket/1863, but the RMA code in the repo is using MPIR_ERRTEST_ARGNULL in a few places, which caused me problems in ARMCI-MPI. Most of the RMA functions, in particular the ones that existed in MPI-2, were missing any buffer checks. The buffer checks that were present were added to the RMW routines. My patch is relative to remotes/origin/HEAD. "I don't always build MPI from source, but when I do, I use the latest development version of MPICH" </dos equis mem> Best, Jeff On Wed, May 29, 2013 at 4:14 PM, Lisandro Dalcin <dalcinl(a)gmail.com> wrote: > On 29 May 2013 23:55, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: >> Well that's no fun. I was hoping to make it look like I did some >> actual work here :-) >> > > The actual work is figuring out the issue!!! I've spent enough time > with these bugs in other MPI implementations. As I know by testing > myself that MPICH does the right thing for MPI 2.x calls, I guesses > you were missing a more easy way to do it. > > BTW, check this: > > $ grep -r MPIR_ERRTEST_USERBUFFER src/mpi/rma > > I get no output, I think other RMA calls also need better error > checking of buffer addresses... > > > -- > Lisandro Dalcin > --------------- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169 -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [mpich-devel] MPIR_ERRTEST_ARGNULL and count=0 (or type_size=0)
by Lisandro Dalcin 29 May '13

29 May '13

On 29 May 2013 23:55, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: > Well that's no fun. I was hoping to make it look like I did some > actual work here :-) > The actual work is figuring out the issue!!! I've spent enough time with these bugs in other MPI implementations. As I know by testing myself that MPICH does the right thing for MPI 2.x calls, I guesses you were missing a more easy way to do it. BTW, check this: $ grep -r MPIR_ERRTEST_USERBUFFER src/mpi/rma I get no output, I think other RMA calls also need better error checking of buffer addresses... -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

1 0

Re: [mpich-devel] MPIR_ERRTEST_ARGNULL and count=0 (or type_size=0)
by Jeff Hammond 29 May '13

29 May '13

Well that's no fun. I was hoping to make it look like I did some actual work here :-) Thanks, Jeff On Wed, May 29, 2013 at 3:08 PM, Lisandro Dalcin <dalcinl(a)gmail.com> wrote: > On 29 May 2013 22:49, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: >> It seems that I didn't understand MPI_BOTTOM completely, so the error >> checks needed to be different than described below, but the high-level >> issues are the same. >> >> Interested parties may consult >> http://trac.mpich.org/projects/mpich/ticket/1863 for details and >> further discussion. >> > > Please take a look at the macro MPIR_ERRTEST_USERBUFFER . > > > > -- > Lisandro Dalcin > --------------- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169 -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0