March 2017 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Dataloop error message
by Palmer, Bruce J 09 Mar '17

09 Mar '17

For this test, I believe the underlying put/get/accumulate are using simple contiguous data types. Puts and gets use MPI_CHAR, the accumulates use a data type that is dependent on what type of data is being accumulated. In this case it should be MPI_DOUBLE. -----Original Message----- From: Latham, Robert J. [mailto:[email protected]] Sent: Thursday, March 09, 2017 9:23 AM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] Dataloop error message On Wed, 2017-03-08 at 23:41 +0000, Palmer, Bruce J wrote: > Rob, > > Attached are the valgrind logs for a failed run. I've checked out the > code on our side and I don't see anything obviously bogus (not that > that means much). Do these suggest anything to you? I'm still trying > to create a short reproducer, but as you can imagine, all my efforts > so far work just fine. Yeah, these valgrind logs look ok. The warnings about uninitialized bytes are common especially when MPICH not configured with extra debugging to explicitly zero out all those buffers. The dataloop code is getting some kind of garbage datatype. What types are you trying to use? Simple contiguous types? A deeply nested complex user-defined type? ==rob > > Bruce > > -----Original Message----- > From: Latham, Robert J. [mailto:[email protected]] > Sent: Wednesday, March 08, 2017 7:26 AM > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] Dataloop error message > > On Tue, 2017-03-07 at 19:31 +0000, Palmer, Bruce J wrote: > > Hi, > > > > I’m trying to track down a possible race condition in a test program > > that is using MPI RMA from MPICH 3.2. The program repeats a series > > of put/get/accumulate operations to different processors. When I’m > > running on 1 node 4 processors everything is fine but when I move to > > 2 nodes 4 processors I start getting failures. The error messages > > I’m seeing are > > > > Assertion failed in file > > src/mpid/common/datatype/dataloop/dataloop.c > > at line 265: 0 > > that's a strange one! that came from the "Dataloop_update" routine. > It updates pointers after a copy operation. That particular assertion > came from the "handle different types" switch > > switch(dataloop->kind & DLOOP_KIND_MASK) > > which means somehow this code got a datatype that was not one of > CONTIG, VECTOR, BLOCKINDEXED, INDEXED, or STRUCT (in dataloop terms. > MPI type "HINDEXED" for example maps to INDEXED directly, so not all > MPI types are explicitly handled). > > > > Assertion failed in file > > src/mpid/common/datatype/dataloop/dataloop.c > > at line 157: dataloop->loop_params.cm_t.dataloop > > Also inside "Dataloop_update". This assertion > > DLOOP_Assert(dataloop->loop_params.cm_t.dataloop) > > basically suggests garbage was passed to the Dataloop_update routine. > > > Does anyone have a handle on what these routines do and what kind of > > behavior is generating these errors? The test program is allocating > > memory and using it to create a window, followed immediately by a > > call to MPI_Win_lock_all to create a passive synchronization epoch. > > I’ve been using request based RMA calls (Rput, Rget, Raccumulate) > > followed by an immediate call to MPI_Wait for the individual RMA > > operations. Any suggestions about what these errors are telling me? > > If I start putting in print statements to narrow down the location > > of the error, the code runs to completion. > > The two assertions plus your observation that "printf debugging makes > it go away" sure sounds a lot like some kind of memory corruption. > Any chance you can collect some valgrind logs? > > ==rob > _______________________________________________ > discuss mailing list discuss(a)mpich.org To manage subscription > options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org To manage subscription > options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Dataloop error message
by Latham, Robert J. 09 Mar '17

09 Mar '17

On Wed, 2017-03-08 at 23:41 +0000, Palmer, Bruce J wrote: > Rob, > > Attached are the valgrind logs for a failed run. I've checked out the > code on our side and I don't see anything obviously bogus (not that > that means much). Do these suggest anything to you? I'm still trying > to create a short reproducer, but as you can imagine, all my efforts > so far work just fine. Yeah, these valgrind logs look ok. The warnings about uninitialized bytes are common especially when MPICH not configured with extra debugging to explicitly zero out all those buffers. The dataloop code is getting some kind of garbage datatype. What types are you trying to use? Simple contiguous types? A deeply nested complex user-defined type? ==rob > > Bruce > > -----Original Message----- > From: Latham, Robert J. [mailto:[email protected]] > Sent: Wednesday, March 08, 2017 7:26 AM > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] Dataloop error message > > On Tue, 2017-03-07 at 19:31 +0000, Palmer, Bruce J wrote: > > Hi, > > > > I’m trying to track down a possible race condition in a test > > program > > that is using MPI RMA from MPICH 3.2. The program repeats a series > > of > > put/get/accumulate operations to different processors. When I’m > > running on 1 node 4 processors everything is fine but when I move > > to > > 2 nodes 4 processors I start getting failures. The error messages > > I’m > > seeing are > > > > Assertion failed in file > > src/mpid/common/datatype/dataloop/dataloop.c > > at line 265: 0 > > that's a strange one! that came from the "Dataloop_update" routine. > It updates pointers after a copy operation. That particular > assertion came from the "handle different types" switch > > switch(dataloop->kind & DLOOP_KIND_MASK) > > which means somehow this code got a datatype that was not one of > CONTIG, VECTOR, BLOCKINDEXED, INDEXED, or STRUCT (in dataloop > terms. > MPI type "HINDEXED" for example maps to INDEXED directly, so not all > MPI types are explicitly handled). > > > > Assertion failed in file > > src/mpid/common/datatype/dataloop/dataloop.c > > at line 157: dataloop->loop_params.cm_t.dataloop > > Also inside "Dataloop_update". This assertion > > DLOOP_Assert(dataloop->loop_params.cm_t.dataloop) > > basically suggests garbage was passed to the Dataloop_update routine. > > > Does anyone have a handle on what these routines do and what kind > > of > > behavior is generating these errors? The test program is > > allocating > > memory and using it to create a window, followed immediately by a > > call > > to MPI_Win_lock_all to create a passive synchronization epoch. > > I’ve been using request based RMA calls (Rput, Rget, Raccumulate) > > followed by an immediate call to MPI_Wait for the individual RMA > > operations. Any suggestions about what these errors are telling me? > > If I start putting in print statements to narrow down the location > > of > > the error, the code runs to completion. > > The two assertions plus your observation that "printf debugging makes > it go away" sure sounds a lot like some kind of memory > corruption. Any chance you can collect some valgrind logs? > > ==rob > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Dataloop error message
by William Gropp 08 Mar '17

08 Mar '17

I’d still like to see MPICH adopt the Dataloop code that Tarun wrote and that should be much faster and particularly more appropriate for use with RMA. I think that would be more productive in the long term than continuing to maintain the current code. Bill William Gropp Acting Director and Chief Scientist, NCSA Director, Parallel Computing Institute Thomas M. Siebel Chair in Computer Science University of Illinois Urbana-Champaign > On Mar 8, 2017, at 5:41 PM, Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov <mailto:[email protected]>> wrote: > > Rob, > > Attached are the valgrind logs for a failed run. I've checked out the code on our side and I don't see anything obviously bogus (not that that means much). Do these suggest anything to you? I'm still trying to create a short reproducer, but as you can imagine, all my efforts so far work just fine. > > Bruce > > -----Original Message----- > From: Latham, Robert J. [mailto:[email protected] <mailto:[email protected]>] > Sent: Wednesday, March 08, 2017 7:26 AM > To: discuss(a)mpich.org <mailto:[email protected]> > Subject: Re: [mpich-discuss] Dataloop error message > > On Tue, 2017-03-07 at 19:31 +0000, Palmer, Bruce J wrote: >> Hi, >> >> I’m trying to track down a possible race condition in a test program >> that is using MPI RMA from MPICH 3.2. The program repeats a series of >> put/get/accumulate operations to different processors. When I’m >> running on 1 node 4 processors everything is fine but when I move to >> 2 nodes 4 processors I start getting failures. The error messages I’m >> seeing are >> >> Assertion failed in file src/mpid/common/datatype/dataloop/dataloop.c >> at line 265: 0 > > that's a strange one! that came from the "Dataloop_update" routine. > It updates pointers after a copy operation. That particular assertion came from the "handle different types" switch > > switch(dataloop->kind & DLOOP_KIND_MASK) > > which means somehow this code got a datatype that was not one of CONTIG, VECTOR, BLOCKINDEXED, INDEXED, or STRUCT (in dataloop terms. > MPI type "HINDEXED" for example maps to INDEXED directly, so not all MPI types are explicitly handled). > > >> Assertion failed in file src/mpid/common/datatype/dataloop/dataloop.c >> at line 157: dataloop->loop_params.cm_t.dataloop > > Also inside "Dataloop_update". This assertion > > DLOOP_Assert(dataloop->loop_params.cm_t.dataloop) > > basically suggests garbage was passed to the Dataloop_update routine. > >> Does anyone have a handle on what these routines do and what kind of >> behavior is generating these errors? The test program is allocating >> memory and using it to create a window, followed immediately by a call >> to MPI_Win_lock_all to create a passive synchronization epoch. >> I’ve been using request based RMA calls (Rput, Rget, Raccumulate) >> followed by an immediate call to MPI_Wait for the individual RMA >> operations. Any suggestions about what these errors are telling me? >> If I start putting in print statements to narrow down the location of >> the error, the code runs to completion. > > The two assertions plus your observation that "printf debugging makes it go away" sure sounds a lot like some kind of memory corruption. Any chance you can collect some valgrind logs? > > ==rob > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss <https://lists.mpich.org/mailman/listinfo/discuss> > <log.1673><log.1674><log.1729><log.1730>_______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Dataloop error message
by William Gropp 08 Mar '17

08 Mar '17

I’d still like to see MPICH adopt the Dataloop code that Tarun wrote and that should be much faster and particularly more appropriate for use with RMA. I think that would be more productive in the long term than continuing to maintain the current code. Bill William Gropp Acting Director and Chief Scientist, NCSA Director, Parallel Computing Institute Thomas M. Siebel Chair in Computer Science University of Illinois Urbana-Champaign > On Mar 8, 2017, at 5:41 PM, Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> wrote: > > Rob, > > Attached are the valgrind logs for a failed run. I've checked out the code on our side and I don't see anything obviously bogus (not that that means much). Do these suggest anything to you? I'm still trying to create a short reproducer, but as you can imagine, all my efforts so far work just fine. > > Bruce > > -----Original Message----- > From: Latham, Robert J. [mailto:[email protected]] > Sent: Wednesday, March 08, 2017 7:26 AM > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] Dataloop error message > > On Tue, 2017-03-07 at 19:31 +0000, Palmer, Bruce J wrote: >> Hi, >> >> I’m trying to track down a possible race condition in a test program >> that is using MPI RMA from MPICH 3.2. The program repeats a series of >> put/get/accumulate operations to different processors. When I’m >> running on 1 node 4 processors everything is fine but when I move to >> 2 nodes 4 processors I start getting failures. The error messages I’m >> seeing are >> >> Assertion failed in file src/mpid/common/datatype/dataloop/dataloop.c >> at line 265: 0 > > that's a strange one! that came from the "Dataloop_update" routine. > It updates pointers after a copy operation. That particular assertion came from the "handle different types" switch > > switch(dataloop->kind & DLOOP_KIND_MASK) > > which means somehow this code got a datatype that was not one of CONTIG, VECTOR, BLOCKINDEXED, INDEXED, or STRUCT (in dataloop terms. > MPI type "HINDEXED" for example maps to INDEXED directly, so not all MPI types are explicitly handled). > > >> Assertion failed in file src/mpid/common/datatype/dataloop/dataloop.c >> at line 157: dataloop->loop_params.cm_t.dataloop > > Also inside "Dataloop_update". This assertion > > DLOOP_Assert(dataloop->loop_params.cm_t.dataloop) > > basically suggests garbage was passed to the Dataloop_update routine. > >> Does anyone have a handle on what these routines do and what kind of >> behavior is generating these errors? The test program is allocating >> memory and using it to create a window, followed immediately by a call >> to MPI_Win_lock_all to create a passive synchronization epoch. >> I’ve been using request based RMA calls (Rput, Rget, Raccumulate) >> followed by an immediate call to MPI_Wait for the individual RMA >> operations. Any suggestions about what these errors are telling me? >> If I start putting in print statements to narrow down the location of >> the error, the code runs to completion. > > The two assertions plus your observation that "printf debugging makes it go away" sure sounds a lot like some kind of memory corruption. Any chance you can collect some valgrind logs? > > ==rob > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > <log.1673><log.1674><log.1729><log.1730>_______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Dataloop error message
by Palmer, Bruce J 08 Mar '17

08 Mar '17

Rob, Attached are the valgrind logs for a failed run. I've checked out the code on our side and I don't see anything obviously bogus (not that that means much). Do these suggest anything to you? I'm still trying to create a short reproducer, but as you can imagine, all my efforts so far work just fine. Bruce -----Original Message----- From: Latham, Robert J. [mailto:[email protected]] Sent: Wednesday, March 08, 2017 7:26 AM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] Dataloop error message On Tue, 2017-03-07 at 19:31 +0000, Palmer, Bruce J wrote: > Hi, > > I’m trying to track down a possible race condition in a test program > that is using MPI RMA from MPICH 3.2. The program repeats a series of > put/get/accumulate operations to different processors. When I’m > running on 1 node 4 processors everything is fine but when I move to > 2 nodes 4 processors I start getting failures. The error messages I’m > seeing are > > Assertion failed in file src/mpid/common/datatype/dataloop/dataloop.c > at line 265: 0 that's a strange one! that came from the "Dataloop_update" routine. It updates pointers after a copy operation. That particular assertion came from the "handle different types" switch switch(dataloop->kind & DLOOP_KIND_MASK) which means somehow this code got a datatype that was not one of CONTIG, VECTOR, BLOCKINDEXED, INDEXED, or STRUCT (in dataloop terms. MPI type "HINDEXED" for example maps to INDEXED directly, so not all MPI types are explicitly handled). > Assertion failed in file src/mpid/common/datatype/dataloop/dataloop.c > at line 157: dataloop->loop_params.cm_t.dataloop Also inside "Dataloop_update". This assertion DLOOP_Assert(dataloop->loop_params.cm_t.dataloop) basically suggests garbage was passed to the Dataloop_update routine. > Does anyone have a handle on what these routines do and what kind of > behavior is generating these errors? The test program is allocating > memory and using it to create a window, followed immediately by a call > to MPI_Win_lock_all to create a passive synchronization epoch. > I’ve been using request based RMA calls (Rput, Rget, Raccumulate) > followed by an immediate call to MPI_Wait for the individual RMA > operations. Any suggestions about what these errors are telling me? > If I start putting in print statements to narrow down the location of > the error, the code runs to completion. The two assertions plus your observation that "printf debugging makes it go away" sure sounds a lot like some kind of memory corruption. Any chance you can collect some valgrind logs? ==rob _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MCS lock and MPI RMA problem
by Halim Amer 08 Mar '17

08 Mar '17

The bug Ask was referring to is regarding the branch "if (lmem[nextRank] == -1)", as I explained previously. Halim www.mcs.anl.gov/~aamer On 3/8/17 12:09 PM, Jeff Hammond wrote: > It's not a bug in the example. There is a bug in the MPI standard that > tolerates implementations that do not provide asynchronous progress > (https://github.com/mpi-forum/mpi-forum-historic/issues/359) and thus > many of them do not do so by default (because doing so often negatively > impacts the performance of other features). Fortunately, as noted > already, Casper or environment variables fix this issue. > > Jeff, who is an asynchronous RMA zealot :-) > > On Wed, Mar 8, 2017 at 1:06 AM, Ask Jakobsen <afj(a)qeye-labs.com > <mailto:[email protected]>> wrote: > > You are absolutely correct, Halim. Removing the test lmem[nextRank] > == -1 in release fixes the problem. Great work. Now I will try to > understand why you are right. I hope the authors of the book will > credit you for discovering the bug. > > So in conclusion you need to remove the above mentioned test AND > enable asynchronous progression using the environment variable > MPIR_CVAR_ASYNC_PROGRESS=1 in MPICH (BTW I still can't get the code > to work in openmpi). > > > On Tue, Mar 7, 2017 at 5:37 PM, Halim Amer <aamer(a)anl.gov > <mailto:[email protected]>> wrote: > > > detect that another process is being or already enqueued in the MCS queue. > > Actually the problem occurs only when the waiting process > already enqueued itself, i.e., the accumulate operation on the > nextRank field succeeded. > > Halim > www.mcs.anl.gov/~aamer <http://www.mcs.anl.gov/%7Eaamer> > > > On 3/7/17 10:29 AM, Halim Amer wrote: > > In the Release protocol, try removing this test: > > if (lmem[nextRank] == -1) { > If-Block; > } > > but keep the If-Block. > > The hang occurs because the process releasing the MCS lock > fails to > detect that another process is being or already enqueued in > the MCS queue. > > Halim > www.mcs.anl.gov/~aamer <http://www.mcs.anl.gov/%7Eaamer> > > On 3/7/17 6:43 AM, Ask Jakobsen wrote: > > Thanks, Halim. I have now enabled asynchronous progress > in MPICH (can't > find something similar in openmpi) and now all ranks > acquire the lock and > the program finish as expected. However if I put a > while(1) loop > around the > acquire-release code in main.c it will fail again at > random and go > into an > infinite loop. The simple unfair lock does not have this > problem. > > On Tue, Mar 7, 2017 at 12:44 AM, Halim Amer > <aamer(a)anl.gov <mailto:[email protected]>> wrote: > > My understanding is that this code assumes > asynchronous progress. > An example of when the processes hang is as follows: > > 1) P0 Finishes MCSLockAcquire() > 2) P1 is busy waiting in MCSLockAcquire() at > do { > MPI_Win_sync(win); > } while (lmem[blocked] == 1); > 3) P0 executes MCSLockRelease() > 4) P0 waits on MPI_Win_lock_all() inside MCSLockRlease() > > Hang! > > For P1 to get out of the loop, P0 has to get out of > MPI_Win_lock_all() and > executes its Compare_and_swap(). > > For P0 to get out MPI_Win_lock_all(), it needs an > ACK from P1 that it > got > the lock. > > P1 does not make communication progress because > MPI_Win_sync is not > required to do so. It only synchronizes private and > public copies. > > For this hang to disappear, one can either trigger > progress manually by > using heavy-duty synchronization calls instead of > Win_sync (e.g., > Win_unlock_all + Win_lock_all), or enable > asynchronous progress. > > To enable asynchronous progress in MPICH, set the > MPIR_CVAR_ASYNC_PROGRESS > env var to 1. > > Halim > www.mcs.anl.gov/~aamer > <http://www.mcs.anl.gov/%7Eaamer> > <http://www.mcs.anl.gov/%7Eaamer > <http://www.mcs.anl.gov/%7Eaamer>> > > > On 3/6/17 1:11 PM, Ask Jakobsen wrote: > > I am testing on x86_64 platform. > > I have tried to built both the mpich and the mcs > lock code with -O0 to > avoid agressive optimization. After your > suggestion I have also > tried to > make volatile int *pblocked pointing to > lmem[blocked] in the > MCSLockAcquire > function and volatile int *pnextrank pointing to > lmem[nextRank] in > MCSLockRelease, but it does not appear to make a > difference. > > On suggestion from Richard Warren I have also > tried building the code > using > openmpi-2.0.2 without any luck (however it > appears to acquire the > lock a > couple of extra times before failing) which I > find troubling. > > I think I will give up using local load/stores > and will see if I can > figure > out if rewrite using MPI calls like > MPI_Fetch_and_op as you suggest. > Thanks for your help. > > On Mon, Mar 6, 2017 at 7:20 PM, Jeff Hammond > <jeff.science(a)gmail.com > <mailto:[email protected]>> > wrote: > > What processor architecture are you testing? > > > Maybe set lmem to volatile or read it with > MPI_Fetch_and_op rather > than a > load. MPI_Win_sync cannot prevent the > compiler from caching *lmem > in a > register. > > Jeff > > On Sat, Mar 4, 2017 at 12:30 AM, Ask > Jakobsen <afj(a)qeye-labs.com > <mailto:[email protected]>> > wrote: > > Hi, > > > I have downloaded the source code for > the MCS lock from the excellent > book "Using Advanced MPI" from > http://www.mcs.anl.gov/researc > h/projects/mpi/usingmpi/examples-advmpi/rma2/mcs-lock.c > > I have made a very simple piece of test > code for testing the MCS lock > but > it works at random and often never > escapes the busy loops in the > acquire > and release functions (see attached > source code). The code appears > semantically correct to my eyes. > > #include <stdio.h> > #include <mpi.h> > #include "mcs-lock.h" > > int main(int argc, char *argv[]) > { > MPI_Win win; > MPI_Init( &argc, &argv ); > > MCSLockInit(MPI_COMM_WORLD, &win); > > int rank, size; > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > MPI_Comm_size(MPI_COMM_WORLD, &size); > > printf("rank: %d, size: %d\n", rank, > size); > > > MCSLockAcquire(win); > printf("rank %d aquired lock\n", > rank); fflush(stdout); > MCSLockRelease(win); > > > MPI_Win_free(&win); > MPI_Finalize(); > return 0; > } > > > I have tested on several hardware > platforms and mpich-3.2 and > mpich-3.3a2 > but with no luck. > > It appears that the MPI_Win_Sync are not > "refreshing" the local > data or > I > have a bug I can't spot. > > A simple unfair lock like > http://www.mcs.anl.gov/researc > h/projects/mpi/usingmpi/examples-advmpi/rma2/ga_mutex1.c > works > perfectly. > > Best regards, Ask Jakobsen > > > _______________________________________________ > discuss mailing list > discuss(a)mpich.org > <mailto:[email protected]> > To manage subscription options or > unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > <https://lists.mpich.org/mailman/listinfo/discuss> > > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > <mailto:[email protected]> > http://jeffhammond.github.io/ > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > <https://lists.mpich.org/mailman/listinfo/discuss> > > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > <https://lists.mpich.org/mailman/listinfo/discuss> > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > <https://lists.mpich.org/mailman/listinfo/discuss> > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > <https://lists.mpich.org/mailman/listinfo/discuss> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > <https://lists.mpich.org/mailman/listinfo/discuss> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > <https://lists.mpich.org/mailman/listinfo/discuss> > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > <https://lists.mpich.org/mailman/listinfo/discuss> > > > > > -- > Jeff Hammond > jeff.science(a)gmail.com <mailto:[email protected]> > http://jeffhammond.github.io/ > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MCS lock and MPI RMA problem
by Jeff Hammond 08 Mar '17

08 Mar '17

It's not a bug in the example. There is a bug in the MPI standard that tolerates implementations that do not provide asynchronous progress ( https://github.com/mpi-forum/mpi-forum-historic/issues/359) and thus many of them do not do so by default (because doing so often negatively impacts the performance of other features). Fortunately, as noted already, Casper or environment variables fix this issue. Jeff, who is an asynchronous RMA zealot :-) On Wed, Mar 8, 2017 at 1:06 AM, Ask Jakobsen <afj(a)qeye-labs.com> wrote: > You are absolutely correct, Halim. Removing the test lmem[nextRank] == -1 > in release fixes the problem. Great work. Now I will try to understand why > you are right. I hope the authors of the book will credit you for > discovering the bug. > > So in conclusion you need to remove the above mentioned test AND enable > asynchronous progression using the environment variable > MPIR_CVAR_ASYNC_PROGRESS=1 in MPICH (BTW I still can't get the code to work > in openmpi). > > > On Tue, Mar 7, 2017 at 5:37 PM, Halim Amer <aamer(a)anl.gov> wrote: > >> > detect that another process is being or already enqueued in the MCS >> queue. >> >> Actually the problem occurs only when the waiting process already >> enqueued itself, i.e., the accumulate operation on the nextRank field >> succeeded. >> >> Halim >> www.mcs.anl.gov/~aamer <http://www.mcs.anl.gov/%7Eaamer> >> >> >> On 3/7/17 10:29 AM, Halim Amer wrote: >> >>> In the Release protocol, try removing this test: >>> >>> if (lmem[nextRank] == -1) { >>> If-Block; >>> } >>> >>> but keep the If-Block. >>> >>> The hang occurs because the process releasing the MCS lock fails to >>> detect that another process is being or already enqueued in the MCS >>> queue. >>> >>> Halim >>> www.mcs.anl.gov/~aamer <http://www.mcs.anl.gov/%7Eaamer> >>> >>> On 3/7/17 6:43 AM, Ask Jakobsen wrote: >>> >>>> Thanks, Halim. I have now enabled asynchronous progress in MPICH (can't >>>> find something similar in openmpi) and now all ranks acquire the lock >>>> and >>>> the program finish as expected. However if I put a while(1) loop >>>> around the >>>> acquire-release code in main.c it will fail again at random and go >>>> into an >>>> infinite loop. The simple unfair lock does not have this problem. >>>> >>>> On Tue, Mar 7, 2017 at 12:44 AM, Halim Amer <aamer(a)anl.gov> wrote: >>>> >>>> My understanding is that this code assumes asynchronous progress. >>>>> An example of when the processes hang is as follows: >>>>> >>>>> 1) P0 Finishes MCSLockAcquire() >>>>> 2) P1 is busy waiting in MCSLockAcquire() at >>>>> do { >>>>> MPI_Win_sync(win); >>>>> } while (lmem[blocked] == 1); >>>>> 3) P0 executes MCSLockRelease() >>>>> 4) P0 waits on MPI_Win_lock_all() inside MCSLockRlease() >>>>> >>>>> Hang! >>>>> >>>>> For P1 to get out of the loop, P0 has to get out of >>>>> MPI_Win_lock_all() and >>>>> executes its Compare_and_swap(). >>>>> >>>>> For P0 to get out MPI_Win_lock_all(), it needs an ACK from P1 that it >>>>> got >>>>> the lock. >>>>> >>>>> P1 does not make communication progress because MPI_Win_sync is not >>>>> required to do so. It only synchronizes private and public copies. >>>>> >>>>> For this hang to disappear, one can either trigger progress manually by >>>>> using heavy-duty synchronization calls instead of Win_sync (e.g., >>>>> Win_unlock_all + Win_lock_all), or enable asynchronous progress. >>>>> >>>>> To enable asynchronous progress in MPICH, set the >>>>> MPIR_CVAR_ASYNC_PROGRESS >>>>> env var to 1. >>>>> >>>>> Halim >>>>> www.mcs.anl.gov/~aamer <http://www.mcs.anl.gov/%7Eaamer> < >>>>> http://www.mcs.anl.gov/%7Eaamer> >>>>> >>>>> >>>>> On 3/6/17 1:11 PM, Ask Jakobsen wrote: >>>>> >>>>> I am testing on x86_64 platform. >>>>>> >>>>>> I have tried to built both the mpich and the mcs lock code with -O0 to >>>>>> avoid agressive optimization. After your suggestion I have also >>>>>> tried to >>>>>> make volatile int *pblocked pointing to lmem[blocked] in the >>>>>> MCSLockAcquire >>>>>> function and volatile int *pnextrank pointing to lmem[nextRank] in >>>>>> MCSLockRelease, but it does not appear to make a difference. >>>>>> >>>>>> On suggestion from Richard Warren I have also tried building the code >>>>>> using >>>>>> openmpi-2.0.2 without any luck (however it appears to acquire the >>>>>> lock a >>>>>> couple of extra times before failing) which I find troubling. >>>>>> >>>>>> I think I will give up using local load/stores and will see if I can >>>>>> figure >>>>>> out if rewrite using MPI calls like MPI_Fetch_and_op as you suggest. >>>>>> Thanks for your help. >>>>>> >>>>>> On Mon, Mar 6, 2017 at 7:20 PM, Jeff Hammond <jeff.science(a)gmail.com> >>>>>> wrote: >>>>>> >>>>>> What processor architecture are you testing? >>>>>> >>>>>>> >>>>>>> Maybe set lmem to volatile or read it with MPI_Fetch_and_op rather >>>>>>> than a >>>>>>> load. MPI_Win_sync cannot prevent the compiler from caching *lmem >>>>>>> in a >>>>>>> register. >>>>>>> >>>>>>> Jeff >>>>>>> >>>>>>> On Sat, Mar 4, 2017 at 12:30 AM, Ask Jakobsen <afj(a)qeye-labs.com> >>>>>>> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>>> >>>>>>>> I have downloaded the source code for the MCS lock from the >>>>>>>> excellent >>>>>>>> book "Using Advanced MPI" from http://www.mcs.anl.gov/researc >>>>>>>> h/projects/mpi/usingmpi/examples-advmpi/rma2/mcs-lock.c >>>>>>>> >>>>>>>> I have made a very simple piece of test code for testing the MCS >>>>>>>> lock >>>>>>>> but >>>>>>>> it works at random and often never escapes the busy loops in the >>>>>>>> acquire >>>>>>>> and release functions (see attached source code). The code appears >>>>>>>> semantically correct to my eyes. >>>>>>>> >>>>>>>> #include <stdio.h> >>>>>>>> #include <mpi.h> >>>>>>>> #include "mcs-lock.h" >>>>>>>> >>>>>>>> int main(int argc, char *argv[]) >>>>>>>> { >>>>>>>> MPI_Win win; >>>>>>>> MPI_Init( &argc, &argv ); >>>>>>>> >>>>>>>> MCSLockInit(MPI_COMM_WORLD, &win); >>>>>>>> >>>>>>>> int rank, size; >>>>>>>> MPI_Comm_rank(MPI_COMM_WORLD, &rank); >>>>>>>> MPI_Comm_size(MPI_COMM_WORLD, &size); >>>>>>>> >>>>>>>> printf("rank: %d, size: %d\n", rank, size); >>>>>>>> >>>>>>>> >>>>>>>> MCSLockAcquire(win); >>>>>>>> printf("rank %d aquired lock\n", rank); fflush(stdout); >>>>>>>> MCSLockRelease(win); >>>>>>>> >>>>>>>> >>>>>>>> MPI_Win_free(&win); >>>>>>>> MPI_Finalize(); >>>>>>>> return 0; >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> I have tested on several hardware platforms and mpich-3.2 and >>>>>>>> mpich-3.3a2 >>>>>>>> but with no luck. >>>>>>>> >>>>>>>> It appears that the MPI_Win_Sync are not "refreshing" the local >>>>>>>> data or >>>>>>>> I >>>>>>>> have a bug I can't spot. >>>>>>>> >>>>>>>> A simple unfair lock like http://www.mcs.anl.gov/researc >>>>>>>> h/projects/mpi/usingmpi/examples-advmpi/rma2/ga_mutex1.c works >>>>>>>> perfectly. >>>>>>>> >>>>>>>> Best regards, Ask Jakobsen >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> discuss mailing list discuss(a)mpich.org >>>>>>>> To manage subscription options or unsubscribe: >>>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Jeff Hammond >>>>>>> jeff.science(a)gmail.com >>>>>>> http://jeffhammond.github.io/ >>>>>>> >>>>>>> _______________________________________________ >>>>>>> discuss mailing list discuss(a)mpich.org >>>>>>> To manage subscription options or unsubscribe: >>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> discuss mailing list discuss(a)mpich.org >>>>>> To manage subscription options or unsubscribe: >>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>> >>>>>> _______________________________________________ >>>>>> >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> >>>>> >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Dataloop error message
by Latham, Robert J. 08 Mar '17

08 Mar '17

On Tue, 2017-03-07 at 19:31 +0000, Palmer, Bruce J wrote: > Hi, > > I’m trying to track down a possible race condition in a test program > that is using MPI RMA from MPICH 3.2. The program repeats a series of > put/get/accumulate operations to different processors. When I’m > running on 1 node 4 processors everything is fine but when I move to > 2 nodes 4 processors I start getting failures. The error messages > I’m seeing are > > Assertion failed in file src/mpid/common/datatype/dataloop/dataloop.c > at line 265: 0 that's a strange one! that came from the "Dataloop_update" routine. It updates pointers after a copy operation. That particular assertion came from the "handle different types" switch switch(dataloop->kind & DLOOP_KIND_MASK) which means somehow this code got a datatype that was not one of CONTIG, VECTOR, BLOCKINDEXED, INDEXED, or STRUCT (in dataloop terms. MPI type "HINDEXED" for example maps to INDEXED directly, so not all MPI types are explicitly handled). > Assertion failed in file src/mpid/common/datatype/dataloop/dataloop.c > at line 157: dataloop->loop_params.cm_t.dataloop Also inside "Dataloop_update". This assertion DLOOP_Assert(dataloop->loop_params.cm_t.dataloop) basically suggests garbage was passed to the Dataloop_update routine. > Does anyone have a handle on what these routines do and what kind of > behavior is generating these errors? The test program is allocating > memory and using it to create a window, followed immediately by a > call to MPI_Win_lock_all to create a passive synchronization epoch. > I’ve been using request based RMA calls (Rput, Rget, Raccumulate) > followed by an immediate call to MPI_Wait for the individual RMA > operations. Any suggestions about what these errors are telling me? > If I start putting in print statements to narrow down the location of > the error, the code runs to completion. The two assertions plus your observation that "printf debugging makes it go away" sure sounds a lot like some kind of memory corruption. Any chance you can collect some valgrind logs? ==rob _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MCS lock and MPI RMA problem
by Halim Amer 08 Mar '17

08 Mar '17

I cannot claim that I thoroughly verified the correctness of that code, so take it with a grain of salt. Please keep in mind that it is a test code from a tutorial book; those codes are meant for learning purposes not for deployment. If your goal is to have a high performance RMA lock, I suggest you to look into the recent HPDC'16 paper: "High-Performance Distributed RMA Locks". Halim www.mcs.anl.gov/~aamer On 3/8/17 3:06 AM, Ask Jakobsen wrote: > You are absolutely correct, Halim. Removing the test lmem[nextRank] == -1 > in release fixes the problem. Great work. Now I will try to understand why > you are right. I hope the authors of the book will credit you for > discovering the bug. > > So in conclusion you need to remove the above mentioned test AND enable > asynchronous progression using the environment variable > MPIR_CVAR_ASYNC_PROGRESS=1 in MPICH (BTW I still can't get the code to work > in openmpi). > > On Tue, Mar 7, 2017 at 5:37 PM, Halim Amer <aamer(a)anl.gov> wrote: > >>> detect that another process is being or already enqueued in the MCS >> queue. >> >> Actually the problem occurs only when the waiting process already enqueued >> itself, i.e., the accumulate operation on the nextRank field succeeded. >> >> Halim >> www.mcs.anl.gov/~aamer <http://www.mcs.anl.gov/%7Eaamer> >> >> >> On 3/7/17 10:29 AM, Halim Amer wrote: >> >>> In the Release protocol, try removing this test: >>> >>> if (lmem[nextRank] == -1) { >>> If-Block; >>> } >>> >>> but keep the If-Block. >>> >>> The hang occurs because the process releasing the MCS lock fails to >>> detect that another process is being or already enqueued in the MCS queue. >>> >>> Halim >>> www.mcs.anl.gov/~aamer <http://www.mcs.anl.gov/%7Eaamer> >>> >>> On 3/7/17 6:43 AM, Ask Jakobsen wrote: >>> >>>> Thanks, Halim. I have now enabled asynchronous progress in MPICH (can't >>>> find something similar in openmpi) and now all ranks acquire the lock and >>>> the program finish as expected. However if I put a while(1) loop >>>> around the >>>> acquire-release code in main.c it will fail again at random and go >>>> into an >>>> infinite loop. The simple unfair lock does not have this problem. >>>> >>>> On Tue, Mar 7, 2017 at 12:44 AM, Halim Amer <aamer(a)anl.gov> wrote: >>>> >>>> My understanding is that this code assumes asynchronous progress. >>>>> An example of when the processes hang is as follows: >>>>> >>>>> 1) P0 Finishes MCSLockAcquire() >>>>> 2) P1 is busy waiting in MCSLockAcquire() at >>>>> do { >>>>> MPI_Win_sync(win); >>>>> } while (lmem[blocked] == 1); >>>>> 3) P0 executes MCSLockRelease() >>>>> 4) P0 waits on MPI_Win_lock_all() inside MCSLockRlease() >>>>> >>>>> Hang! >>>>> >>>>> For P1 to get out of the loop, P0 has to get out of >>>>> MPI_Win_lock_all() and >>>>> executes its Compare_and_swap(). >>>>> >>>>> For P0 to get out MPI_Win_lock_all(), it needs an ACK from P1 that it >>>>> got >>>>> the lock. >>>>> >>>>> P1 does not make communication progress because MPI_Win_sync is not >>>>> required to do so. It only synchronizes private and public copies. >>>>> >>>>> For this hang to disappear, one can either trigger progress manually by >>>>> using heavy-duty synchronization calls instead of Win_sync (e.g., >>>>> Win_unlock_all + Win_lock_all), or enable asynchronous progress. >>>>> >>>>> To enable asynchronous progress in MPICH, set the >>>>> MPIR_CVAR_ASYNC_PROGRESS >>>>> env var to 1. >>>>> >>>>> Halim >>>>> www.mcs.anl.gov/~aamer <http://www.mcs.anl.gov/%7Eaamer> < >>>>> http://www.mcs.anl.gov/%7Eaamer> >>>>> >>>>> >>>>> On 3/6/17 1:11 PM, Ask Jakobsen wrote: >>>>> >>>>> I am testing on x86_64 platform. >>>>>> >>>>>> I have tried to built both the mpich and the mcs lock code with -O0 to >>>>>> avoid agressive optimization. After your suggestion I have also >>>>>> tried to >>>>>> make volatile int *pblocked pointing to lmem[blocked] in the >>>>>> MCSLockAcquire >>>>>> function and volatile int *pnextrank pointing to lmem[nextRank] in >>>>>> MCSLockRelease, but it does not appear to make a difference. >>>>>> >>>>>> On suggestion from Richard Warren I have also tried building the code >>>>>> using >>>>>> openmpi-2.0.2 without any luck (however it appears to acquire the >>>>>> lock a >>>>>> couple of extra times before failing) which I find troubling. >>>>>> >>>>>> I think I will give up using local load/stores and will see if I can >>>>>> figure >>>>>> out if rewrite using MPI calls like MPI_Fetch_and_op as you suggest. >>>>>> Thanks for your help. >>>>>> >>>>>> On Mon, Mar 6, 2017 at 7:20 PM, Jeff Hammond <jeff.science(a)gmail.com> >>>>>> wrote: >>>>>> >>>>>> What processor architecture are you testing? >>>>>> >>>>>>> >>>>>>> Maybe set lmem to volatile or read it with MPI_Fetch_and_op rather >>>>>>> than a >>>>>>> load. MPI_Win_sync cannot prevent the compiler from caching *lmem >>>>>>> in a >>>>>>> register. >>>>>>> >>>>>>> Jeff >>>>>>> >>>>>>> On Sat, Mar 4, 2017 at 12:30 AM, Ask Jakobsen <afj(a)qeye-labs.com> >>>>>>> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>>> >>>>>>>> I have downloaded the source code for the MCS lock from the excellent >>>>>>>> book "Using Advanced MPI" from http://www.mcs.anl.gov/researc >>>>>>>> h/projects/mpi/usingmpi/examples-advmpi/rma2/mcs-lock.c >>>>>>>> >>>>>>>> I have made a very simple piece of test code for testing the MCS lock >>>>>>>> but >>>>>>>> it works at random and often never escapes the busy loops in the >>>>>>>> acquire >>>>>>>> and release functions (see attached source code). The code appears >>>>>>>> semantically correct to my eyes. >>>>>>>> >>>>>>>> #include <stdio.h> >>>>>>>> #include <mpi.h> >>>>>>>> #include "mcs-lock.h" >>>>>>>> >>>>>>>> int main(int argc, char *argv[]) >>>>>>>> { >>>>>>>> MPI_Win win; >>>>>>>> MPI_Init( &argc, &argv ); >>>>>>>> >>>>>>>> MCSLockInit(MPI_COMM_WORLD, &win); >>>>>>>> >>>>>>>> int rank, size; >>>>>>>> MPI_Comm_rank(MPI_COMM_WORLD, &rank); >>>>>>>> MPI_Comm_size(MPI_COMM_WORLD, &size); >>>>>>>> >>>>>>>> printf("rank: %d, size: %d\n", rank, size); >>>>>>>> >>>>>>>> >>>>>>>> MCSLockAcquire(win); >>>>>>>> printf("rank %d aquired lock\n", rank); fflush(stdout); >>>>>>>> MCSLockRelease(win); >>>>>>>> >>>>>>>> >>>>>>>> MPI_Win_free(&win); >>>>>>>> MPI_Finalize(); >>>>>>>> return 0; >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> I have tested on several hardware platforms and mpich-3.2 and >>>>>>>> mpich-3.3a2 >>>>>>>> but with no luck. >>>>>>>> >>>>>>>> It appears that the MPI_Win_Sync are not "refreshing" the local >>>>>>>> data or >>>>>>>> I >>>>>>>> have a bug I can't spot. >>>>>>>> >>>>>>>> A simple unfair lock like http://www.mcs.anl.gov/researc >>>>>>>> h/projects/mpi/usingmpi/examples-advmpi/rma2/ga_mutex1.c works >>>>>>>> perfectly. >>>>>>>> >>>>>>>> Best regards, Ask Jakobsen >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> discuss mailing list discuss(a)mpich.org >>>>>>>> To manage subscription options or unsubscribe: >>>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Jeff Hammond >>>>>>> jeff.science(a)gmail.com >>>>>>> http://jeffhammond.github.io/ >>>>>>> >>>>>>> _______________________________________________ >>>>>>> discuss mailing list discuss(a)mpich.org >>>>>>> To manage subscription options or unsubscribe: >>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> discuss mailing list discuss(a)mpich.org >>>>>> To manage subscription options or unsubscribe: >>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>> >>>>>> _______________________________________________ >>>>>> >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> >>>>> >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MCS lock and MPI RMA problem
by Ask Jakobsen 08 Mar '17

08 Mar '17

You are absolutely correct, Halim. Removing the test lmem[nextRank] == -1 in release fixes the problem. Great work. Now I will try to understand why you are right. I hope the authors of the book will credit you for discovering the bug. So in conclusion you need to remove the above mentioned test AND enable asynchronous progression using the environment variable MPIR_CVAR_ASYNC_PROGRESS=1 in MPICH (BTW I still can't get the code to work in openmpi). On Tue, Mar 7, 2017 at 5:37 PM, Halim Amer <aamer(a)anl.gov> wrote: > > detect that another process is being or already enqueued in the MCS > queue. > > Actually the problem occurs only when the waiting process already enqueued > itself, i.e., the accumulate operation on the nextRank field succeeded. > > Halim > www.mcs.anl.gov/~aamer <http://www.mcs.anl.gov/%7Eaamer> > > > On 3/7/17 10:29 AM, Halim Amer wrote: > >> In the Release protocol, try removing this test: >> >> if (lmem[nextRank] == -1) { >> If-Block; >> } >> >> but keep the If-Block. >> >> The hang occurs because the process releasing the MCS lock fails to >> detect that another process is being or already enqueued in the MCS queue. >> >> Halim >> www.mcs.anl.gov/~aamer <http://www.mcs.anl.gov/%7Eaamer> >> >> On 3/7/17 6:43 AM, Ask Jakobsen wrote: >> >>> Thanks, Halim. I have now enabled asynchronous progress in MPICH (can't >>> find something similar in openmpi) and now all ranks acquire the lock and >>> the program finish as expected. However if I put a while(1) loop >>> around the >>> acquire-release code in main.c it will fail again at random and go >>> into an >>> infinite loop. The simple unfair lock does not have this problem. >>> >>> On Tue, Mar 7, 2017 at 12:44 AM, Halim Amer <aamer(a)anl.gov> wrote: >>> >>> My understanding is that this code assumes asynchronous progress. >>>> An example of when the processes hang is as follows: >>>> >>>> 1) P0 Finishes MCSLockAcquire() >>>> 2) P1 is busy waiting in MCSLockAcquire() at >>>> do { >>>> MPI_Win_sync(win); >>>> } while (lmem[blocked] == 1); >>>> 3) P0 executes MCSLockRelease() >>>> 4) P0 waits on MPI_Win_lock_all() inside MCSLockRlease() >>>> >>>> Hang! >>>> >>>> For P1 to get out of the loop, P0 has to get out of >>>> MPI_Win_lock_all() and >>>> executes its Compare_and_swap(). >>>> >>>> For P0 to get out MPI_Win_lock_all(), it needs an ACK from P1 that it >>>> got >>>> the lock. >>>> >>>> P1 does not make communication progress because MPI_Win_sync is not >>>> required to do so. It only synchronizes private and public copies. >>>> >>>> For this hang to disappear, one can either trigger progress manually by >>>> using heavy-duty synchronization calls instead of Win_sync (e.g., >>>> Win_unlock_all + Win_lock_all), or enable asynchronous progress. >>>> >>>> To enable asynchronous progress in MPICH, set the >>>> MPIR_CVAR_ASYNC_PROGRESS >>>> env var to 1. >>>> >>>> Halim >>>> www.mcs.anl.gov/~aamer <http://www.mcs.anl.gov/%7Eaamer> < >>>> http://www.mcs.anl.gov/%7Eaamer> >>>> >>>> >>>> On 3/6/17 1:11 PM, Ask Jakobsen wrote: >>>> >>>> I am testing on x86_64 platform. >>>>> >>>>> I have tried to built both the mpich and the mcs lock code with -O0 to >>>>> avoid agressive optimization. After your suggestion I have also >>>>> tried to >>>>> make volatile int *pblocked pointing to lmem[blocked] in the >>>>> MCSLockAcquire >>>>> function and volatile int *pnextrank pointing to lmem[nextRank] in >>>>> MCSLockRelease, but it does not appear to make a difference. >>>>> >>>>> On suggestion from Richard Warren I have also tried building the code >>>>> using >>>>> openmpi-2.0.2 without any luck (however it appears to acquire the >>>>> lock a >>>>> couple of extra times before failing) which I find troubling. >>>>> >>>>> I think I will give up using local load/stores and will see if I can >>>>> figure >>>>> out if rewrite using MPI calls like MPI_Fetch_and_op as you suggest. >>>>> Thanks for your help. >>>>> >>>>> On Mon, Mar 6, 2017 at 7:20 PM, Jeff Hammond <jeff.science(a)gmail.com> >>>>> wrote: >>>>> >>>>> What processor architecture are you testing? >>>>> >>>>>> >>>>>> Maybe set lmem to volatile or read it with MPI_Fetch_and_op rather >>>>>> than a >>>>>> load. MPI_Win_sync cannot prevent the compiler from caching *lmem >>>>>> in a >>>>>> register. >>>>>> >>>>>> Jeff >>>>>> >>>>>> On Sat, Mar 4, 2017 at 12:30 AM, Ask Jakobsen <afj(a)qeye-labs.com> >>>>>> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>>> >>>>>>> I have downloaded the source code for the MCS lock from the excellent >>>>>>> book "Using Advanced MPI" from http://www.mcs.anl.gov/researc >>>>>>> h/projects/mpi/usingmpi/examples-advmpi/rma2/mcs-lock.c >>>>>>> >>>>>>> I have made a very simple piece of test code for testing the MCS lock >>>>>>> but >>>>>>> it works at random and often never escapes the busy loops in the >>>>>>> acquire >>>>>>> and release functions (see attached source code). The code appears >>>>>>> semantically correct to my eyes. >>>>>>> >>>>>>> #include <stdio.h> >>>>>>> #include <mpi.h> >>>>>>> #include "mcs-lock.h" >>>>>>> >>>>>>> int main(int argc, char *argv[]) >>>>>>> { >>>>>>> MPI_Win win; >>>>>>> MPI_Init( &argc, &argv ); >>>>>>> >>>>>>> MCSLockInit(MPI_COMM_WORLD, &win); >>>>>>> >>>>>>> int rank, size; >>>>>>> MPI_Comm_rank(MPI_COMM_WORLD, &rank); >>>>>>> MPI_Comm_size(MPI_COMM_WORLD, &size); >>>>>>> >>>>>>> printf("rank: %d, size: %d\n", rank, size); >>>>>>> >>>>>>> >>>>>>> MCSLockAcquire(win); >>>>>>> printf("rank %d aquired lock\n", rank); fflush(stdout); >>>>>>> MCSLockRelease(win); >>>>>>> >>>>>>> >>>>>>> MPI_Win_free(&win); >>>>>>> MPI_Finalize(); >>>>>>> return 0; >>>>>>> } >>>>>>> >>>>>>> >>>>>>> I have tested on several hardware platforms and mpich-3.2 and >>>>>>> mpich-3.3a2 >>>>>>> but with no luck. >>>>>>> >>>>>>> It appears that the MPI_Win_Sync are not "refreshing" the local >>>>>>> data or >>>>>>> I >>>>>>> have a bug I can't spot. >>>>>>> >>>>>>> A simple unfair lock like http://www.mcs.anl.gov/researc >>>>>>> h/projects/mpi/usingmpi/examples-advmpi/rma2/ga_mutex1.c works >>>>>>> perfectly. >>>>>>> >>>>>>> Best regards, Ask Jakobsen >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> discuss mailing list discuss(a)mpich.org >>>>>>> To manage subscription options or unsubscribe: >>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Jeff Hammond >>>>>> jeff.science(a)gmail.com >>>>>> http://jeffhammond.github.io/ >>>>>> >>>>>> _______________________________________________ >>>>>> discuss mailing list discuss(a)mpich.org >>>>>> To manage subscription options or unsubscribe: >>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> >>>>> _______________________________________________ >>>>> >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0