January 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Deadlock in MPI_Ibarrier over ch3:sock
by Dave Goodell 23 Jan '13

23 Jan '13

On Jan 23, 2013, at 3:17 PM CST, Jed Brown wrote: > On Wed, Jan 23, 2013 at 2:44 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > >> I'm surprised this isn't caught by the "coll/nonblock3" test, although we may just be making too much progress some other way, or we may not have enough coverage of the test itself in that test: http://git.mpich.org/mpich.git/blob/refs/heads/master:/test/mpi/coll/nonblo… >> >> A modest test case would be very helpful. > > <ibarrier.c> Thanks. I've created a ticket to track this: http://trac.mpich.org/projects/mpich/ticket/1785 I suspect the code fix and adding your basic regression test will be easy. I just need to make sure that the nonblocking3 test is doing what we expect it to, which will take a bit more work. -Dave

1 0

Re: [mpich-discuss] Deadlock in MPI_Ibarrier over ch3:sock
by Jim Dinan 23 Jan '13

23 Jan '13

On 1/23/13 3:01 PM, Jeff Hammond wrote: >> We intend to provide support for a default MPI_COMM_SELF implementation of >> shared memory windows in ch3. Shared memory windows are fully implemented >> in Nemesis, but not ch3, so if you were to try using them right now with >> sock, you would get a not implemented error. If anyone needs this >> (presumably for portability), I'd be happy to bump the priority on ticket >> #1666. > > How is this not a no-op implementation for MPI_COMM_SELF? All you > need to support is load-store on local memory, no? We also need to support all of the RMA communication and synchronization operations. This is not an overly complex or large task, it's just currently lower priority that others. That said, if shared memory window support in ch3:sock is something that would prevent you or anyone else from using MPI 3.0 RMA, please speak up and I'll make it higher priority. ~Jim.

1 0

Re: [mpich-discuss] Deadlock in MPI_Ibarrier over ch3:sock
by Jed Brown 23 Jan '13

23 Jan '13

On Wed, Jan 23, 2013 at 2:44 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > Hmm… it looks like we aren't poking NBC progress properly when a "test" or > "iprobe" routine is called. MPIDI_CH3i_Progress_test is missing a call to > MPIDU_Sched_progress: > http://git.mpich.org/mpich.git/blob/refs/heads/master:/src/mpid/ch3/channel… > The attached code demonstrates this. With ch3:sock, it completes with MPI_Wait, but not with MPI_Test. $ ~/usr/mpich-sock/bin/mpicc ibarrier.c -o ibarrier $ mpirun.hydra -n 2 ./ibarrier [0] MPI_Test: 0 [1] MPI_Test: 0 [0] MPI_Test: 1 [1] MPI_Test: 1 [0] MPI_Test: 2 [1] MPI_Test: 2 ... > > > I'm surprised this isn't caught by the "coll/nonblock3" test, although we > may just be making too much progress some other way, or we may not have > enough coverage of the test itself in that test: > http://git.mpich.org/mpich.git/blob/refs/heads/master:/test/mpi/coll/nonblo… > > A modest test case would be very helpful. >

1 0

Re: [mpich-discuss] Deadlock in MPI_Ibarrier over ch3:sock
by Jeff Hammond 23 Jan '13

23 Jan '13

> Actually, you need to support RMA as well, but it should be equivalent > to MPI_WIN_ALLOCATE. Just return the error code MPICH_PSA_STOP_BEING_A_MORON_AND_USE_LOAD_STORE. Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-discuss] Deadlock in MPI_Ibarrier over ch3:sock
by Pavan Balaji 23 Jan '13

23 Jan '13

On 01/23/2013 03:01 PM US Central Time, Jeff Hammond wrote: >> We intend to provide support for a default MPI_COMM_SELF implementation of >> shared memory windows in ch3. Shared memory windows are fully implemented >> in Nemesis, but not ch3, so if you were to try using them right now with >> sock, you would get a not implemented error. If anyone needs this >> (presumably for portability), I'd be happy to bump the priority on ticket >> #1666. > > How is this not a no-op implementation for MPI_COMM_SELF? All you > need to support is load-store on local memory, no? Actually, you need to support RMA as well, but it should be equivalent to MPI_WIN_ALLOCATE. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Deadlock in MPI_Ibarrier over ch3:sock
by Jeff Hammond 23 Jan '13

23 Jan '13

> We intend to provide support for a default MPI_COMM_SELF implementation of > shared memory windows in ch3. Shared memory windows are fully implemented > in Nemesis, but not ch3, so if you were to try using them right now with > sock, you would get a not implemented error. If anyone needs this > (presumably for portability), I'd be happy to bump the priority on ticket > #1666. How is this not a no-op implementation for MPI_COMM_SELF? All you need to support is load-store on local memory, no? Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-discuss] Deadlock in MPI_Ibarrier over ch3:sock
by Jim Dinan 23 Jan '13

23 Jan '13

On 1/23/13 2:50 PM, Pavan Balaji wrote: > > On 01/23/2013 02:44 PM US Central Time, Dave Goodell wrote: >>> Are you still supporting ch3:sock? >> >> We are, although we have chosen not to support a small subset of the >> MPI-3 functionality in ch3:sock, such as shared memory windows. > > It's a pedantic detail, but MPI-3 says that shared memory is supported > on a communicator returned by Comm_split_type. For ch3:sock, we return > MPI_COMM_SELF, so technically we do support shared memory windows, but > only on that process :-D. We intend to provide support for a default MPI_COMM_SELF implementation of shared memory windows in ch3. Shared memory windows are fully implemented in Nemesis, but not ch3, so if you were to try using them right now with sock, you would get a not implemented error. If anyone needs this (presumably for portability), I'd be happy to bump the priority on ticket #1666. https://trac.mpich.org/projects/mpich/ticket/1666 ~Jim.

1 0

Re: [mpich-discuss] Deadlock in MPI_Ibarrier over ch3:sock
by Jeff Hammond 23 Jan '13

23 Jan '13

On Wed, Jan 23, 2013 at 2:44 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: >> Is there some test we can do to check whether ch3:sock is being used so that we can choose a safe algorithm? > > A run-time test that doesn't require looking at which symbols are compiled into the app? I can't think of one off the top of my head, although I suppose we could add one in the future via either MPI_T or the new MPI_Get_library_version routine. I think that MPI_Get_library_version is the most sensible way to do it. There's some precedent for this to return a lot of information: http://blogs.cisco.com/performance/mpi-3-small-new-function-mpi_get_library…. Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-discuss] Deadlock in MPI_Ibarrier over ch3:sock
by Pavan Balaji 23 Jan '13

23 Jan '13

On 01/23/2013 02:44 PM US Central Time, Dave Goodell wrote: >> Are you still supporting ch3:sock? > > We are, although we have chosen not to support a small subset of the > MPI-3 functionality in ch3:sock, such as shared memory windows. It's a pedantic detail, but MPI-3 says that shared memory is supported on a communicator returned by Comm_split_type. For ch3:sock, we return MPI_COMM_SELF, so technically we do support shared memory windows, but only on that process :-D. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Deadlock in MPI_Ibarrier over ch3:sock
by Dave Goodell 23 Jan '13

23 Jan '13

On Jan 23, 2013, at 2:13 PM CST, Jed Brown wrote: > As we've discussed here several times before, PETSc's --download-mpich option uses ch3:sock by default because this issue > > http://trac.mpich.org/projects/mpich/ticket/79 > > makes nemesis unusable when oversubscribed. Your complaint has been duly noted (again). > We're also checking for MPI_Ibarrier and using Torsten's "nonblocking consensus" algorithm when possible. Unfortunately, MPI_Ibarrier deadlocks when run over ch3:sock. (Both procs have posted the Ibarrier, but the requests never complete.) I can prepare a reduced test case if you'd like. Hmm… it looks like we aren't poking NBC progress properly when a "test" or "iprobe" routine is called. MPIDI_CH3i_Progress_test is missing a call to MPIDU_Sched_progress: http://git.mpich.org/mpich.git/blob/refs/heads/master:/src/mpid/ch3/channel… I'm surprised this isn't caught by the "coll/nonblock3" test, although we may just be making too much progress some other way, or we may not have enough coverage of the test itself in that test: http://git.mpich.org/mpich.git/blob/refs/heads/master:/test/mpi/coll/nonblo… A modest test case would be very helpful. > Are you still supporting ch3:sock? We are, although we have chosen not to support a small subset of the MPI-3 functionality in ch3:sock, such as shared memory windows. We still run sock in the nightly tests: http://www.mpich.org/static/cron/tests/#hydra-ch3:sock > Is there some test we can do to check whether ch3:sock is being used so that we can choose a safe algorithm? A run-time test that doesn't require looking at which symbols are compiled into the app? I can't think of one off the top of my head, although I suppose we could add one in the future via either MPI_T or the new MPI_Get_library_version routine. -Dave

1 0