February 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Amazon ec2 Windows machine
by Nicholas Sgro 08 Feb '13

08 Feb '13

Hi, This is the command I'm using: mpiexec.exe -machinefile machines.txt -env MPICH2_CHANNEL sock -n 2 cpi.exe I have tried using both machine file and hosts in the command line, but I get the same results. The program runs on a single instance with any number of processors. I tried running mpiexec on one instance and using the other as a single host and that also works. -Nicholas On Fri, Feb 8, 2013 at 12:04 PM, Jayesh Krishna <jayesh(a)mcs.anl.gov> wrote: > Hi, > How are you running your job (mpiexec command)? Did you try using a > machine file to specify the hostnames when running the job? > Does the program (cpi) execute correctly on a single ec2 instance? > > Regards, > Jayesh > > ----- Original Message ----- > From: "Nicholas Sgro" <nsgro060(a)gmail.com> > To: "Jayesh Krishna" <jayesh(a)mcs.anl.gov> > Sent: Thursday, February 7, 2013 9:57:55 PM > Subject: Re: [mpich-discuss] Amazon ec2 Windows machine > > I'm using version 1.4.1p1. I tried the sock channel. It doesn't seem to > work either. With sock, I get to the point where I enter the number of > intervals, but then it does nothing. > > Do you know any reason it wouldn't work with ec2 instances? > > > > On Thu, Feb 7, 2013 at 4:29 PM, Jayesh Krishna < jayesh(a)mcs.anl.gov > > wrote: > > > Hi, > Which version of MPICH2 are you using? Did you try the "sock" channel (See > if it works)? > > (PS: We haven't tested MPICH2 on Windows with ec2 instances.) > Regards, > Jayesh > > > ----- Original Message ----- > From: "Nicholas Sgro" < nsgro060(a)gmail.com > > To: discuss(a)mpich.org > Sent: Thursday, February 7, 2013 11:29:57 AM > Subject: [mpich-discuss] Amazon ec2 Windows machine > > > Hi all, > > I am trying to run the example cpi.exe across 2 amazon ec2 instances > running windows. I have different problems depending on the channel I > choose. If I try nemesis, I get the following error: > > Fatal error in MPI_Init: Other MPI error, error stack: > MPIR_Init_thread(392).................: > MPID_Init(139)........................: channel initialization failed > MPIDI_CH3_Init(38)....................: > MPID_nem_init(196)....................: > MPIDI_CH3I_Seg_commit(366)............: > MPIU_SHMW_Hnd_deserialize(324)........: > MPIU_SHMW_Seg_open(863)...............: > MPIU_SHMW_Seg_create_attach_templ(763): unable to allocate shared memory - > OpenFileMapping The system cannot find the file specified. > > If I try to use shm, cpi.exe uses 100% of the processors on both machines, > but makes no progress and I have to cancel the job. > > I am attaching logs from smpd from both machines from the runs with > nemesis and shm. > > I don't have any experience with mpich, so I have no idea what the problem > is. Any guidance would be appreciated. > > Thanks > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > >

1 0

Re: [mpich-discuss] MPI RMA
by Nick Radcliffe 08 Feb '13

08 Feb '13

OK, I think I get it now. Thanks for your help. -Nick ________________________________________ From: discuss-bounces(a)mpich.org [discuss-bounces(a)mpich.org] on behalf of Jeff Hammond [jhammond(a)alcf.anl.gov] Sent: Friday, February 08, 2013 1:02 PM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] MPI RMA I completely agree that the following situation sucks for error checking and that it cries for out fence_start and fence_finish (and possible fence_sync i.e. start+finish). That is what the assertions purport to do, but they don't do it sufficiently. There is an implicit assumption that RMA operations just enqueue until the next synchronization call, particularly for the fence case but that's terrible for performance if one has a decent network (as Cray does). I can't remember what assertions/info there are for windows, but one practical solution would be to accept that error checking is hard to impossible in the general case but much easier if the user indicates that a window will only be used in conjunction with one synchronization mechanism. Jeff On Fri, Feb 8, 2013 at 12:45 PM, Nick Radcliffe <nradclif(a)cray.com> wrote: > Thanks for the quick response. Just to clarify, what if I did something like this: > > MPI_Win_fence(0, win) > if (rank == src) MPI_Put(dest) > MPI_Win_fence(0, win) > > if (rank == src) { > MPI_Win_lock(dest, win) > MPI_Put > MPI_Win_unlock(dest ,win) > } > > if (rank == src) MPI_Put(dest) > MPI_Win_fence(0, win) > > The second call to MPI_Win_fence closes-and-reopens an exposure epoch for the dest rank, because the second call to MPI_Win_fence is followed by another call to MPI_Win_fence, and the dest rank is the target of an RMA operation. > > The problem is that there is no way for the call to MPI_Win_lock to know if the previous call to MPI_Win_fence simply ended an exposure epoch, or if it ended-and-reopened an exposure epoch. What I'm trying to understand is how the call to MPI_Win_lock could do error checking to verify that it is not locking a rank in an exposure epoch, since whether MPI_Win_fence closes or closes-and-reopens an exposure epoch seems to depend on whether there are any future calls to MPI_Put/MPI_Win_fence. > > -Nick > ________________________________________ > From: discuss-bounces(a)mpich.org [discuss-bounces(a)mpich.org] on behalf of Jeff Hammond [jhammond(a)alcf.anl.gov] > Sent: Friday, February 08, 2013 11:58 AM > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] MPI RMA > > From MPI-3 11.5.1: > > - The MPI call MPI_WIN_FENCE(assert, win) synchronizes RMA calls on win. > - The call completes an RMA access epoch if it was preceded by another > fence call and the local process issued RMA communication calls on win > between these two calls. > - The call completes an RMA exposure epoch if it was preceded by > another fence call and the local window was the target of RMA accesses > between these two calls. > - The call starts an RMA access epoch if it is followed by another > fence call and by RMA communication calls issued between these two > fence calls. > - The call starts an exposure epoch if it is followed by another fence > call and the local window is the target of RMA accesses between these > two fence calls. > > MPI_Win_fence may both start or complete an RMA epoch but it may also > complete-one-and-start-another RMA epoch depending on the context and > the assertions. All of the tests below are valid in my opinion. > > What the assertions below in the first example are saying is that the > first call to MPI_Win_fence need not complete any RMA calls, which is > true because it is the first sync call. The last MPI_Win_fence > asserts the same thing in reverse. The middle one asserts nothing > because it is both completing and starting an epoch. > > I think that MPI_Win_fence is poorly designed but that ship has sailed > and I believe that the usage is well-defined in the standard despite > the confusing properties of this function. > > Jeff > > On Fri, Feb 8, 2013 at 11:36 AM, Nick Radcliffe <nradclif(a)cray.com> wrote: >> Hi, >> >> I have a question about MPI RMA, and the ANL regression tests for RMA in particular. The tests mixedsync.c and epochtest.c seem to have contradictory views of fence synchronization. >> >> epochtest.c seems to suggest that access/exposure epochs opened by a call to MPI_Win_fence are not closed until a call to MPI_Win_fence with assert==MPI_MODE_NOSUCCEED. The test looks roughly like this: >> >> MPI_Win_fence(MPI_MODE_NOPRECEDE, win) >> if (rank == src) MPI_Put >> MPI_Win_fence(0, win) >> if (rank == dest) MPI_Put >> etc... >> MPI_Win_fence(MPI_MODE_NOSUCCEED, win) >> >> Since there is a call to MPI_Put after the second call to MPI_Win_fence, it would seem that the second call could not have ended the access epoch for dest, or the exposure epoch for src (which is the target of the second Put). >> >> On the other hand, the test mixedsync.c looks roughly like this: >> >> if (rank == src) { >> MPI_Win_lock(...,win) >> MPI_Put >> MPI_Win_unlock(...,win) >> } >> >> MPI_Win_fence(0, win) >> if (rank == src) MPI_Put >> MPI_Win_fence(0, win) >> >> if (rank == src) { >> MPI_Win_lock(...,win) >> MPI_Put >> MPI_Win_unlock(...,win) >> } >> >> The problem is that it is erroneous to call MPI_Win_lock on a window while that window is exposed due to a call to MPI_Win_fence. If mixedsync.c is not erroneous, then the second call to MPI_Win_fence must end the exposure epoch on win, contradicting what's implied about fence synchronization by epochtest.c. >> >> Sorry for the long post, but if anyone can shed some light on this for me, I would greatly appreciate it. >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI RMA
by Jed Brown 08 Feb '13

08 Feb '13

On Fri, Feb 8, 2013 at 1:08 PM, Jim Dinan <dinan(a)mcs.anl.gov> wrote: > However, it will work fine in MPICH. In MPICH 3.0, I redesigned the > synchronization error detection; as of 3.0, this is the only invalid RMA > synchronization where we don't flag an error. This is not so much because > we choose to support this as an extension to RMA, but because the flush > state needs to change collectively, making this difficult (not O(1)) to > detect. If this is really the only instance, it may not be worth it, but in general, it's helpful to have a run-time (or even compile-time, if necessary) option that does the strictest checking you're willing to implement, even when it implies a performance hit. Certainly useful for people trying to determine whether they have interpreted the standard correctly, or when given someone else's code that is failing mysteriously.

1 0

Re: [mpich-discuss] MPI RMA
by Jim Dinan 08 Feb '13

08 Feb '13

Hi Nick, That is technically an invalid MPI program. However, it will work fine in MPICH. In MPICH 3.0, I redesigned the synchronization error detection; as of 3.0, this is the only invalid RMA synchronization where we don't flag an error. This is not so much because we choose to support this as an extension to RMA, but because the flush state needs to change collectively, making this difficult (not O(1)) to detect. ~Jim. On 2/8/13 12:45 PM, Nick Radcliffe wrote: > Thanks for the quick response. Just to clarify, what if I did something like this: > > MPI_Win_fence(0, win) > if (rank == src) MPI_Put(dest) > MPI_Win_fence(0, win) > > if (rank == src) { > MPI_Win_lock(dest, win) > MPI_Put > MPI_Win_unlock(dest ,win) > } > > if (rank == src) MPI_Put(dest) > MPI_Win_fence(0, win) > > The second call to MPI_Win_fence closes-and-reopens an exposure epoch for the dest rank, because the second call to MPI_Win_fence is followed by another call to MPI_Win_fence, and the dest rank is the target of an RMA operation. > > The problem is that there is no way for the call to MPI_Win_lock to know if the previous call to MPI_Win_fence simply ended an exposure epoch, or if it ended-and-reopened an exposure epoch. What I'm trying to understand is how the call to MPI_Win_lock could do error checking to verify that it is not locking a rank in an exposure epoch, since whether MPI_Win_fence closes or closes-and-reopens an exposure epoch seems to depend on whether there are any future calls to MPI_Put/MPI_Win_fence. > > -Nick > ________________________________________ > From: discuss-bounces(a)mpich.org [discuss-bounces(a)mpich.org] on behalf of Jeff Hammond [jhammond(a)alcf.anl.gov] > Sent: Friday, February 08, 2013 11:58 AM > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] MPI RMA > > From MPI-3 11.5.1: > > - The MPI call MPI_WIN_FENCE(assert, win) synchronizes RMA calls on win. > - The call completes an RMA access epoch if it was preceded by another > fence call and the local process issued RMA communication calls on win > between these two calls. > - The call completes an RMA exposure epoch if it was preceded by > another fence call and the local window was the target of RMA accesses > between these two calls. > - The call starts an RMA access epoch if it is followed by another > fence call and by RMA communication calls issued between these two > fence calls. > - The call starts an exposure epoch if it is followed by another fence > call and the local window is the target of RMA accesses between these > two fence calls. > > MPI_Win_fence may both start or complete an RMA epoch but it may also > complete-one-and-start-another RMA epoch depending on the context and > the assertions. All of the tests below are valid in my opinion. > > What the assertions below in the first example are saying is that the > first call to MPI_Win_fence need not complete any RMA calls, which is > true because it is the first sync call. The last MPI_Win_fence > asserts the same thing in reverse. The middle one asserts nothing > because it is both completing and starting an epoch. > > I think that MPI_Win_fence is poorly designed but that ship has sailed > and I believe that the usage is well-defined in the standard despite > the confusing properties of this function. > > Jeff > > On Fri, Feb 8, 2013 at 11:36 AM, Nick Radcliffe <nradclif(a)cray.com> wrote: >> Hi, >> >> I have a question about MPI RMA, and the ANL regression tests for RMA in particular. The tests mixedsync.c and epochtest.c seem to have contradictory views of fence synchronization. >> >> epochtest.c seems to suggest that access/exposure epochs opened by a call to MPI_Win_fence are not closed until a call to MPI_Win_fence with assert==MPI_MODE_NOSUCCEED. The test looks roughly like this: >> >> MPI_Win_fence(MPI_MODE_NOPRECEDE, win) >> if (rank == src) MPI_Put >> MPI_Win_fence(0, win) >> if (rank == dest) MPI_Put >> etc... >> MPI_Win_fence(MPI_MODE_NOSUCCEED, win) >> >> Since there is a call to MPI_Put after the second call to MPI_Win_fence, it would seem that the second call could not have ended the access epoch for dest, or the exposure epoch for src (which is the target of the second Put). >> >> On the other hand, the test mixedsync.c looks roughly like this: >> >> if (rank == src) { >> MPI_Win_lock(...,win) >> MPI_Put >> MPI_Win_unlock(...,win) >> } >> >> MPI_Win_fence(0, win) >> if (rank == src) MPI_Put >> MPI_Win_fence(0, win) >> >> if (rank == src) { >> MPI_Win_lock(...,win) >> MPI_Put >> MPI_Win_unlock(...,win) >> } >> >> The problem is that it is erroneous to call MPI_Win_lock on a window while that window is exposed due to a call to MPI_Win_fence. If mixedsync.c is not erroneous, then the second call to MPI_Win_fence must end the exposure epoch on win, contradicting what's implied about fence synchronization by epochtest.c. >> >> Sorry for the long post, but if anyone can shed some light on this for me, I would greatly appreciate it. >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] MPI RMA
by Jeff Hammond 08 Feb '13

08 Feb '13

I completely agree that the following situation sucks for error checking and that it cries for out fence_start and fence_finish (and possible fence_sync i.e. start+finish). That is what the assertions purport to do, but they don't do it sufficiently. There is an implicit assumption that RMA operations just enqueue until the next synchronization call, particularly for the fence case but that's terrible for performance if one has a decent network (as Cray does). I can't remember what assertions/info there are for windows, but one practical solution would be to accept that error checking is hard to impossible in the general case but much easier if the user indicates that a window will only be used in conjunction with one synchronization mechanism. Jeff On Fri, Feb 8, 2013 at 12:45 PM, Nick Radcliffe <nradclif(a)cray.com> wrote: > Thanks for the quick response. Just to clarify, what if I did something like this: > > MPI_Win_fence(0, win) > if (rank == src) MPI_Put(dest) > MPI_Win_fence(0, win) > > if (rank == src) { > MPI_Win_lock(dest, win) > MPI_Put > MPI_Win_unlock(dest ,win) > } > > if (rank == src) MPI_Put(dest) > MPI_Win_fence(0, win) > > The second call to MPI_Win_fence closes-and-reopens an exposure epoch for the dest rank, because the second call to MPI_Win_fence is followed by another call to MPI_Win_fence, and the dest rank is the target of an RMA operation. > > The problem is that there is no way for the call to MPI_Win_lock to know if the previous call to MPI_Win_fence simply ended an exposure epoch, or if it ended-and-reopened an exposure epoch. What I'm trying to understand is how the call to MPI_Win_lock could do error checking to verify that it is not locking a rank in an exposure epoch, since whether MPI_Win_fence closes or closes-and-reopens an exposure epoch seems to depend on whether there are any future calls to MPI_Put/MPI_Win_fence. > > -Nick > ________________________________________ > From: discuss-bounces(a)mpich.org [discuss-bounces(a)mpich.org] on behalf of Jeff Hammond [jhammond(a)alcf.anl.gov] > Sent: Friday, February 08, 2013 11:58 AM > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] MPI RMA > > From MPI-3 11.5.1: > > - The MPI call MPI_WIN_FENCE(assert, win) synchronizes RMA calls on win. > - The call completes an RMA access epoch if it was preceded by another > fence call and the local process issued RMA communication calls on win > between these two calls. > - The call completes an RMA exposure epoch if it was preceded by > another fence call and the local window was the target of RMA accesses > between these two calls. > - The call starts an RMA access epoch if it is followed by another > fence call and by RMA communication calls issued between these two > fence calls. > - The call starts an exposure epoch if it is followed by another fence > call and the local window is the target of RMA accesses between these > two fence calls. > > MPI_Win_fence may both start or complete an RMA epoch but it may also > complete-one-and-start-another RMA epoch depending on the context and > the assertions. All of the tests below are valid in my opinion. > > What the assertions below in the first example are saying is that the > first call to MPI_Win_fence need not complete any RMA calls, which is > true because it is the first sync call. The last MPI_Win_fence > asserts the same thing in reverse. The middle one asserts nothing > because it is both completing and starting an epoch. > > I think that MPI_Win_fence is poorly designed but that ship has sailed > and I believe that the usage is well-defined in the standard despite > the confusing properties of this function. > > Jeff > > On Fri, Feb 8, 2013 at 11:36 AM, Nick Radcliffe <nradclif(a)cray.com> wrote: >> Hi, >> >> I have a question about MPI RMA, and the ANL regression tests for RMA in particular. The tests mixedsync.c and epochtest.c seem to have contradictory views of fence synchronization. >> >> epochtest.c seems to suggest that access/exposure epochs opened by a call to MPI_Win_fence are not closed until a call to MPI_Win_fence with assert==MPI_MODE_NOSUCCEED. The test looks roughly like this: >> >> MPI_Win_fence(MPI_MODE_NOPRECEDE, win) >> if (rank == src) MPI_Put >> MPI_Win_fence(0, win) >> if (rank == dest) MPI_Put >> etc... >> MPI_Win_fence(MPI_MODE_NOSUCCEED, win) >> >> Since there is a call to MPI_Put after the second call to MPI_Win_fence, it would seem that the second call could not have ended the access epoch for dest, or the exposure epoch for src (which is the target of the second Put). >> >> On the other hand, the test mixedsync.c looks roughly like this: >> >> if (rank == src) { >> MPI_Win_lock(...,win) >> MPI_Put >> MPI_Win_unlock(...,win) >> } >> >> MPI_Win_fence(0, win) >> if (rank == src) MPI_Put >> MPI_Win_fence(0, win) >> >> if (rank == src) { >> MPI_Win_lock(...,win) >> MPI_Put >> MPI_Win_unlock(...,win) >> } >> >> The problem is that it is erroneous to call MPI_Win_lock on a window while that window is exposed due to a call to MPI_Win_fence. If mixedsync.c is not erroneous, then the second call to MPI_Win_fence must end the exposure epoch on win, contradicting what's implied about fence synchronization by epochtest.c. >> >> Sorry for the long post, but if anyone can shed some light on this for me, I would greatly appreciate it. >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-discuss] MPI RMA
by Nick Radcliffe 08 Feb '13

08 Feb '13

Thanks for the quick response. Just to clarify, what if I did something like this: MPI_Win_fence(0, win) if (rank == src) MPI_Put(dest) MPI_Win_fence(0, win) if (rank == src) { MPI_Win_lock(dest, win) MPI_Put MPI_Win_unlock(dest ,win) } if (rank == src) MPI_Put(dest) MPI_Win_fence(0, win) The second call to MPI_Win_fence closes-and-reopens an exposure epoch for the dest rank, because the second call to MPI_Win_fence is followed by another call to MPI_Win_fence, and the dest rank is the target of an RMA operation. The problem is that there is no way for the call to MPI_Win_lock to know if the previous call to MPI_Win_fence simply ended an exposure epoch, or if it ended-and-reopened an exposure epoch. What I'm trying to understand is how the call to MPI_Win_lock could do error checking to verify that it is not locking a rank in an exposure epoch, since whether MPI_Win_fence closes or closes-and-reopens an exposure epoch seems to depend on whether there are any future calls to MPI_Put/MPI_Win_fence. -Nick ________________________________________ From: discuss-bounces(a)mpich.org [discuss-bounces(a)mpich.org] on behalf of Jeff Hammond [jhammond(a)alcf.anl.gov] Sent: Friday, February 08, 2013 11:58 AM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] MPI RMA >From MPI-3 11.5.1: - The MPI call MPI_WIN_FENCE(assert, win) synchronizes RMA calls on win. - The call completes an RMA access epoch if it was preceded by another fence call and the local process issued RMA communication calls on win between these two calls. - The call completes an RMA exposure epoch if it was preceded by another fence call and the local window was the target of RMA accesses between these two calls. - The call starts an RMA access epoch if it is followed by another fence call and by RMA communication calls issued between these two fence calls. - The call starts an exposure epoch if it is followed by another fence call and the local window is the target of RMA accesses between these two fence calls. MPI_Win_fence may both start or complete an RMA epoch but it may also complete-one-and-start-another RMA epoch depending on the context and the assertions. All of the tests below are valid in my opinion. What the assertions below in the first example are saying is that the first call to MPI_Win_fence need not complete any RMA calls, which is true because it is the first sync call. The last MPI_Win_fence asserts the same thing in reverse. The middle one asserts nothing because it is both completing and starting an epoch. I think that MPI_Win_fence is poorly designed but that ship has sailed and I believe that the usage is well-defined in the standard despite the confusing properties of this function. Jeff On Fri, Feb 8, 2013 at 11:36 AM, Nick Radcliffe <nradclif(a)cray.com> wrote: > Hi, > > I have a question about MPI RMA, and the ANL regression tests for RMA in particular. The tests mixedsync.c and epochtest.c seem to have contradictory views of fence synchronization. > > epochtest.c seems to suggest that access/exposure epochs opened by a call to MPI_Win_fence are not closed until a call to MPI_Win_fence with assert==MPI_MODE_NOSUCCEED. The test looks roughly like this: > > MPI_Win_fence(MPI_MODE_NOPRECEDE, win) > if (rank == src) MPI_Put > MPI_Win_fence(0, win) > if (rank == dest) MPI_Put > etc... > MPI_Win_fence(MPI_MODE_NOSUCCEED, win) > > Since there is a call to MPI_Put after the second call to MPI_Win_fence, it would seem that the second call could not have ended the access epoch for dest, or the exposure epoch for src (which is the target of the second Put). > > On the other hand, the test mixedsync.c looks roughly like this: > > if (rank == src) { > MPI_Win_lock(...,win) > MPI_Put > MPI_Win_unlock(...,win) > } > > MPI_Win_fence(0, win) > if (rank == src) MPI_Put > MPI_Win_fence(0, win) > > if (rank == src) { > MPI_Win_lock(...,win) > MPI_Put > MPI_Win_unlock(...,win) > } > > The problem is that it is erroneous to call MPI_Win_lock on a window while that window is exposed due to a call to MPI_Win_fence. If mixedsync.c is not erroneous, then the second call to MPI_Win_fence must end the exposure epoch on win, contradicting what's implied about fence synchronization by epochtest.c. > > Sorry for the long post, but if anyone can shed some light on this for me, I would greatly appreciate it. > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI RMA
by Jim Dinan 08 Feb '13

08 Feb '13

+1 It's a bit confusing, but both of those examples that Nick included are correct usages of fence. ~Jim. On 2/8/13 11:58 AM, Jeff Hammond wrote: > From MPI-3 11.5.1: > > - The MPI call MPI_WIN_FENCE(assert, win) synchronizes RMA calls on win. > - The call completes an RMA access epoch if it was preceded by another > fence call and the local process issued RMA communication calls on win > between these two calls. > - The call completes an RMA exposure epoch if it was preceded by > another fence call and the local window was the target of RMA accesses > between these two calls. > - The call starts an RMA access epoch if it is followed by another > fence call and by RMA communication calls issued between these two > fence calls. > - The call starts an exposure epoch if it is followed by another fence > call and the local window is the target of RMA accesses between these > two fence calls. > > MPI_Win_fence may both start or complete an RMA epoch but it may also > complete-one-and-start-another RMA epoch depending on the context and > the assertions. All of the tests below are valid in my opinion. > > What the assertions below in the first example are saying is that the > first call to MPI_Win_fence need not complete any RMA calls, which is > true because it is the first sync call. The last MPI_Win_fence > asserts the same thing in reverse. The middle one asserts nothing > because it is both completing and starting an epoch. > > I think that MPI_Win_fence is poorly designed but that ship has sailed > and I believe that the usage is well-defined in the standard despite > the confusing properties of this function. > > Jeff > > On Fri, Feb 8, 2013 at 11:36 AM, Nick Radcliffe <nradclif(a)cray.com> wrote: >> Hi, >> >> I have a question about MPI RMA, and the ANL regression tests for RMA in particular. The tests mixedsync.c and epochtest.c seem to have contradictory views of fence synchronization. >> >> epochtest.c seems to suggest that access/exposure epochs opened by a call to MPI_Win_fence are not closed until a call to MPI_Win_fence with assert==MPI_MODE_NOSUCCEED. The test looks roughly like this: >> >> MPI_Win_fence(MPI_MODE_NOPRECEDE, win) >> if (rank == src) MPI_Put >> MPI_Win_fence(0, win) >> if (rank == dest) MPI_Put >> etc... >> MPI_Win_fence(MPI_MODE_NOSUCCEED, win) >> >> Since there is a call to MPI_Put after the second call to MPI_Win_fence, it would seem that the second call could not have ended the access epoch for dest, or the exposure epoch for src (which is the target of the second Put). >> >> On the other hand, the test mixedsync.c looks roughly like this: >> >> if (rank == src) { >> MPI_Win_lock(...,win) >> MPI_Put >> MPI_Win_unlock(...,win) >> } >> >> MPI_Win_fence(0, win) >> if (rank == src) MPI_Put >> MPI_Win_fence(0, win) >> >> if (rank == src) { >> MPI_Win_lock(...,win) >> MPI_Put >> MPI_Win_unlock(...,win) >> } >> >> The problem is that it is erroneous to call MPI_Win_lock on a window while that window is exposed due to a call to MPI_Win_fence. If mixedsync.c is not erroneous, then the second call to MPI_Win_fence must end the exposure epoch on win, contradicting what's implied about fence synchronization by epochtest.c. >> >> Sorry for the long post, but if anyone can shed some light on this for me, I would greatly appreciate it. >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > >

1 0

Re: [mpich-discuss] MPI RMA
by Jeff Hammond 08 Feb '13

08 Feb '13

>From MPI-3 11.5.1: - The MPI call MPI_WIN_FENCE(assert, win) synchronizes RMA calls on win. - The call completes an RMA access epoch if it was preceded by another fence call and the local process issued RMA communication calls on win between these two calls. - The call completes an RMA exposure epoch if it was preceded by another fence call and the local window was the target of RMA accesses between these two calls. - The call starts an RMA access epoch if it is followed by another fence call and by RMA communication calls issued between these two fence calls. - The call starts an exposure epoch if it is followed by another fence call and the local window is the target of RMA accesses between these two fence calls. MPI_Win_fence may both start or complete an RMA epoch but it may also complete-one-and-start-another RMA epoch depending on the context and the assertions. All of the tests below are valid in my opinion. What the assertions below in the first example are saying is that the first call to MPI_Win_fence need not complete any RMA calls, which is true because it is the first sync call. The last MPI_Win_fence asserts the same thing in reverse. The middle one asserts nothing because it is both completing and starting an epoch. I think that MPI_Win_fence is poorly designed but that ship has sailed and I believe that the usage is well-defined in the standard despite the confusing properties of this function. Jeff On Fri, Feb 8, 2013 at 11:36 AM, Nick Radcliffe <nradclif(a)cray.com> wrote: > Hi, > > I have a question about MPI RMA, and the ANL regression tests for RMA in particular. The tests mixedsync.c and epochtest.c seem to have contradictory views of fence synchronization. > > epochtest.c seems to suggest that access/exposure epochs opened by a call to MPI_Win_fence are not closed until a call to MPI_Win_fence with assert==MPI_MODE_NOSUCCEED. The test looks roughly like this: > > MPI_Win_fence(MPI_MODE_NOPRECEDE, win) > if (rank == src) MPI_Put > MPI_Win_fence(0, win) > if (rank == dest) MPI_Put > etc... > MPI_Win_fence(MPI_MODE_NOSUCCEED, win) > > Since there is a call to MPI_Put after the second call to MPI_Win_fence, it would seem that the second call could not have ended the access epoch for dest, or the exposure epoch for src (which is the target of the second Put). > > On the other hand, the test mixedsync.c looks roughly like this: > > if (rank == src) { > MPI_Win_lock(...,win) > MPI_Put > MPI_Win_unlock(...,win) > } > > MPI_Win_fence(0, win) > if (rank == src) MPI_Put > MPI_Win_fence(0, win) > > if (rank == src) { > MPI_Win_lock(...,win) > MPI_Put > MPI_Win_unlock(...,win) > } > > The problem is that it is erroneous to call MPI_Win_lock on a window while that window is exposed due to a call to MPI_Win_fence. If mixedsync.c is not erroneous, then the second call to MPI_Win_fence must end the exposure epoch on win, contradicting what's implied about fence synchronization by epochtest.c. > > Sorry for the long post, but if anyone can shed some light on this for me, I would greatly appreciate it. > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

MPI RMA
by Nick Radcliffe 08 Feb '13

08 Feb '13

Hi, I have a question about MPI RMA, and the ANL regression tests for RMA in particular. The tests mixedsync.c and epochtest.c seem to have contradictory views of fence synchronization. epochtest.c seems to suggest that access/exposure epochs opened by a call to MPI_Win_fence are not closed until a call to MPI_Win_fence with assert==MPI_MODE_NOSUCCEED. The test looks roughly like this: MPI_Win_fence(MPI_MODE_NOPRECEDE, win) if (rank == src) MPI_Put MPI_Win_fence(0, win) if (rank == dest) MPI_Put etc... MPI_Win_fence(MPI_MODE_NOSUCCEED, win) Since there is a call to MPI_Put after the second call to MPI_Win_fence, it would seem that the second call could not have ended the access epoch for dest, or the exposure epoch for src (which is the target of the second Put). On the other hand, the test mixedsync.c looks roughly like this: if (rank == src) { MPI_Win_lock(...,win) MPI_Put MPI_Win_unlock(...,win) } MPI_Win_fence(0, win) if (rank == src) MPI_Put MPI_Win_fence(0, win) if (rank == src) { MPI_Win_lock(...,win) MPI_Put MPI_Win_unlock(...,win) } The problem is that it is erroneous to call MPI_Win_lock on a window while that window is exposed due to a call to MPI_Win_fence. If mixedsync.c is not erroneous, then the second call to MPI_Win_fence must end the exposure epoch on win, contradicting what's implied about fence synchronization by epochtest.c. Sorry for the long post, but if anyone can shed some light on this for me, I would greatly appreciate it.

1 0

Re: [mpich-discuss] Amazon ec2 Windows machine
by Jayesh Krishna 08 Feb '13

08 Feb '13

Hi, How are you running your job (mpiexec command)? Did you try using a machine file to specify the hostnames when running the job? Does the program (cpi) execute correctly on a single ec2 instance? Regards, Jayesh ----- Original Message ----- From: "Nicholas Sgro" <nsgro060(a)gmail.com> To: "Jayesh Krishna" <jayesh(a)mcs.anl.gov> Sent: Thursday, February 7, 2013 9:57:55 PM Subject: Re: [mpich-discuss] Amazon ec2 Windows machine I'm using version 1.4.1p1. I tried the sock channel. It doesn't seem to work either. With sock, I get to the point where I enter the number of intervals, but then it does nothing. Do you know any reason it wouldn't work with ec2 instances? On Thu, Feb 7, 2013 at 4:29 PM, Jayesh Krishna < jayesh(a)mcs.anl.gov > wrote: Hi, Which version of MPICH2 are you using? Did you try the "sock" channel (See if it works)? (PS: We haven't tested MPICH2 on Windows with ec2 instances.) Regards, Jayesh ----- Original Message ----- From: "Nicholas Sgro" < nsgro060(a)gmail.com > To: discuss(a)mpich.org Sent: Thursday, February 7, 2013 11:29:57 AM Subject: [mpich-discuss] Amazon ec2 Windows machine Hi all, I am trying to run the example cpi.exe across 2 amazon ec2 instances running windows. I have different problems depending on the channel I choose. If I try nemesis, I get the following error: Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(392).................: MPID_Init(139)........................: channel initialization failed MPIDI_CH3_Init(38)....................: MPID_nem_init(196)....................: MPIDI_CH3I_Seg_commit(366)............: MPIU_SHMW_Hnd_deserialize(324)........: MPIU_SHMW_Seg_open(863)...............: MPIU_SHMW_Seg_create_attach_templ(763): unable to allocate shared memory - OpenFileMapping The system cannot find the file specified. If I try to use shm, cpi.exe uses 100% of the processors on both machines, but makes no progress and I have to cancel the job. I am attaching logs from smpd from both machines from the runs with nemesis and shm. I don't have any experience with mpich, so I have no idea what the problem is. Any guidance would be appreciated. Thanks _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0