February 2012 - petsc-dev - mailman3.cels.anl.gov

Shared memory and libraries
by Jed Brown 12 Feb '12

12 Feb '12

I thought this thread on MPI shared memory interfaces with possibly weak consistency models is relevant to our discussions about threading. Boehm's paper is http://www.cs.washington.edu/education/courses/cse590p/05au/HPL-2004-209.pdf ---------- Forwarded message ---------- From: Torsten Hoefler <htor(a)illinois.edu> Date: Sun, Feb 12, 2012 at 10:01 Subject: Re: [Mpi3-rma] [Mpi3-hybridpm] Fwd: MPI shared memory allocation issues To: mpi3-hybridpm(a)lists.mpi-forum.org Cc: mpi3-rma(a)lists.mpi-forum.org Dave, On Mon, Mar 28, 2011 at 01:28:48PM -0500, Dave Goodell wrote: > Forwarding this mail to a broader audience, based on our discussion > here at the March 2011 MPI Forum meeting. There was additional > correspondence on this thread that I can forward as needed, but this > forwarded mail contains my core argument against the original > (allocate+free+fence) proposal. Thanks! This is an important discussion. Let me recap below what we discussed in the RMA group when moving towards the newer integrated allocate_shared. > Begin forwarded message: > > > From: Dave Goodell <goodell(a)mcs.anl.gov> > > Date: February 24, 2011 11:18:20 AM CST > > To: Ron Brightwell <rbbrigh(a)sandia.gov>, Douglas Miller < dougmill(a)us.ibm.com>, "Bronis R. de Supinski" <bronis(a)llnl.gov>, Jim Dinan < dinan(a)mcs.anl.gov>, Pavan Balaji <balaji(a)mcs.anl.gov>, Marc Snir < snir(a)illinois.edu> > > Subject: MPI shared memory allocation issues > > > > I voiced concerns at the last MPI forum meeting about the proposed > > MPI extensions for allocating shared memory. In particular I was > > concerned about "MPI_Shm_fence". Pavan asked me to write up a quick > > email to this group in order to help clarify my view in the > > discussion; this is that email. Please widen the distribution list > > as appropriate, I just mailed the addresses that Pavan indicated to > > me. FYI, I am not currently subscribed to the mpi3-hybrid list. > > > > First, I view multithreaded programming within an OS process and > > multiprocess programming using a shared memory region to be > > essentially the same problem. There are probably alternative > > interpretations of the words "process" and "thread" that could muddy > > the picture here, but for the sake of clarity, let's use the > > conventional meanings for the moment. Also, I am not interested in > > discussing distributed shared memory (DSM) here, I think that > > bringing it up just confuses the discussion further. My primary > > objections to the proposal are valid entirely within a discussion of > > conventional shared memory, processes, and threads. > > > > Given that preface, I believe that many, if not all, of the issues > > raised by Boehm's paper, "Threads Cannot Be Implemented As a > > Library" [1], apply here. In particular, some variation on the > > example from section 4.3 is probably an issue, but the others seem > > to apply as well. The performance example is also relevant here, > > but in an even more dramatic fashion given the dearth of > > synchronization primitives offered by the proposal. > > > > I do not believe that we can specify a way to program the provided > > shared memory in any way that is robust and useful to the user, > > because C and Fortran do not give us enough of a specification in > > order to do so. Without getting into the business of compiler > > writing, MPI has no way to give the user any meaningful guarantees. > > Just as Boehm noted about pthreads, we can probably come up with an > > approach that will work most of the time. But that's a pretty > > flimsy guarantee for a standard like MPI. Yes, as Boehm points out, serial compiler optimizations can have very bad effects on concurrently running code accessing close-by memory. However, as you point out, the situation is equivalent to what we have today in pthreads and the proposal does not claim any more, it says "The consistency of load/store accesses from/to the shared memory as observed by the user program depends on the architecture.". We can extend it to include the compiler and maybe reference Boehm's paper (I would see this as a ticket 0 change). I agree to the general sentiment that it is impossible to implement shared memory semantics in a language that doesn't even have a real memory model. However, at the same time, I want to remind us that the behavior of Fortran was *never* 100% correct in MPI <= 2.2 (and we rely in this TR for MPI-3.0). At the same time, Fortran/MPI programs are ubiquitous :-). But let's discuss Boehm's identified correctness issues here: * 4.1 Concurrent modification This is only an issue if users rely on the consistency of the underlying hardware, if they use Win_flush and friends (as advised), such a reordering would be illegal. One downside is that this will only work in C and Fortran will probably may have all kinds of wacky problems with code movement as usual (however, they should be in a position to fix this with the new bindings and the Fortran TR). * 4.2 Rewriting of Adjacent Data This applies to the unified window as well where we simply specify the byte granularity of updates (an architecture could work with larger chunks (e.g., words) and cause the same trouble). So this issue is not limited to the shared memory window, especially when fast remote memory access hardware is used. Here we face the general trade-off between fast hardware access and safety (securing it through a software layer). We decided that byte-consistency is something we can expect from vendors. Also, the vendor library is always free to return MPI_ERR_RMA_SHARED (like he can always choose to not offer the unified memory model). * 4.3 Register Promotion While this is certainly a problem with threads, it would not be one with MPI windows because they have to dereference the address of the accessed memory and would thus prevent register promotion. Copying the data into a faster memory region would also not harm because the remote side has to query the addresses anyway. Again, restrictions may apply for Fortran codes. > > If you ignore the difficulty in specifying an interface that can > > actually be used correctly, then another issue arises. The only > > proposed synchronization mechanism virtually guarantees that the > > user can at best utilize the allocated shared memory region to share > > data that is written once and otherwise read-only. Any other shared > > memory programming techniques are either going to be non-portable > > (e.g., using pthread mutexes or calls/macros from some atomic > > operations library), or they will be limited to potentially slow > > dark-ages techniques such as Dekker's Algorithm with excessive > > MPI_Shm_fence-ing. So does this proposal really empower the user in > > any meaningful way? I agree. This should be addressed by the merge into the RMA context which offers all required functionality (we avoided memory locks on purpose because they are evil). > > I don't see a compelling advantage to putting this into MPI as > > opposed to providing this as some third-party library on top of MPI. > > Sure, it's easy to implement the allocate/free calls inside of MPI > > because the machinery is typically there. But a third-party library > > would be able to escape some of the extremely generic portability > > constraints of the MPI standard and would therefore be able to > > provide a more robust interface to the user. A discussion of DSM > > might make putting it into MPI more compelling because access to the > > network hardware might be involved, but I'm not particularly > > interested in having that discussion right now. I think that MPI-3 > > RMA would probably be more suitable for that use case. First, I am against DSM. Second, I believe that it may be very valuable to have this kind of functionality in MPI because virtually all large-scale codes have to become hybrid. The main issue are the associated memory saving (on-node communication with MPI is often sufficiently fast). I believe the current practice of mixing OpenMP and MPI to achieve this simple goal may be suboptimal (OpenMP supports only the "shared everything" (threaded) model and enables thus a whole new class of bugs and races). All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736 _______________________________________________ mpi3-rma mailing list mpi3-rma(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [petsc-dev] DMGetMatrix --> DMGetMatrices?
by Dmitry Karpeev 12 Feb '12

12 Feb '12

On Sat, Feb 11, 2012 at 8:36 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Sat, Feb 11, 2012 at 08:32, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: > >> This sounds fine to me, but what's the API for doing this? You want to do >> KSPSetDM() and PCSetDM(), >> so that the user has to explicitly pull out the PC to set its DM? >> > > The PC *always* owns both matrices anyway. > I think there is a rather natural API for handling DM grouping, code reuse and dealing with the issue of single matrix per DM: DM should support DMGetKSPDM(), DMGetPCDM() -- similar to DMDAGetCoordinateDA() -- each of which will compute the correct matrix. These calls can return the original DM (or PETSC_NULL) by default. SNES will pull out the KSP DM out to set it on the KSP, KSP will do the same for the PC, etc. DM already has Vec x, so all of the DMs can share x and compute their respective matrices at the same x and reuse whatever computation is reusable. That's up to the implementation, though. Dmitry.

1 0

Re: [petsc-dev] please make myself and Satish admin on bitbucket.org/petsc
by Satish Balay 11 Feb '12

11 Feb '12

Trying out bitbucket - noticed one more thing. premise: We need multiple repos and multiple admins This looks difficult to implement [without admins sharing 'petsc' passwd - and doing everything as the owner - which sean/barray are trying out now]. Single repo - multiple admins is fine. multiple repos with a single 'owner/admin' is fine. With multiple repos we want reposGroupA - access-control-list-A reposGroupB - access-control-list-B [do this either in 1 account or 2 accounts - doesn't matter] To have a common access control list for a group of repos - 'user group' appears to be the thing to use [as we have to set access-list for each repo separately]. However this is not easy to admin: - user 'petsc' can create a group [petsc:groupA] - admin 'balay' can't modify this group [if he need to add another person] - admin 'balay can't look at the contents of [petsc:groupA] - to see who is in it and who is not. - however amdin 'balay' can create a new group [balay:groupB] - and add this to the access control list So we can set the accesscontrol list to repos petsc/petsc, petsc/buildsystem to be the following - but one will not know [without e-mail commuication] the complete list of folks in these groups. petsc:groupA balay:groupB bsmith:groupC This might work - We can ignore duplicate users across groups and keep adding users in the the personal groups - when in doubt. But deleting users will be combersome. So project admintration [with multiple admins] is not as simple as it appeared before.. Satish On Fri, 10 Feb 2012, Satish Balay wrote: > On Fri, 10 Feb 2012, Satish Balay wrote: > > > On Fri, 10 Feb 2012, Sean Farley wrote: > > With petsc.cs.iit.edu - yes I have to create a project manually and > > assemble the group and admin/s . But with bitbucket - you can create > > your own account [ala project] and assemble your own group - aswell as > > set admins. > > minor modification: > > with petsc.cs.iit.edu I create the project - and set the initial > admin/s. They can login and set the group as needed. > > But usually I'm also the admin as most folks didn't want to deal with > the admin part - it was easier to just tell me what was needed. [its > usually manually editing authorized keys file] > > I'll concede that the gui admin for bitbucket is perhaps > better. [instead of logging in shell - you log-into a website and use > the gui]. > > One good aspect of bitbucket thats not on petsc.cs.iit.edu is: If a > user exists in multiple projects - the keys are duplicated manually > for each project account on petsc.cs.iit.edu. However bitbucket > maintains a single copy and uses it across all projects. > > Since the number of projects was relatively small - the lack of this > functionality wasn't a big deal.. > > > After this creation - on petsc.cs.iit.edu any user in the group can > > create repos without my intervention. > > > > hg clone foobar ssh://[email protected]//hg/petsc/foobar > > > > With bitbucket - only owner can create the new repo [not any member of > > a default group]. Don't know about admins.. > > > > wrt delete - petsc.cs.iit.edu admin for project petsc can ssh over and > > delete repos [via shell access]. Looks like with bitbucket - only owner > > can do the deletion of repo [not even admins]

1 0

Re: [petsc-dev] DMGetMatrix --> DMGetMatrices?
by Jed Brown 11 Feb '12

11 Feb '12

On Sat, Feb 11, 2012 at 08:32, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: > This sounds fine to me, but what's the API for doing this? You want to do > KSPSetDM() and PCSetDM(), > so that the user has to explicitly pull out the PC to set its DM? > The PC *always* owns both matrices anyway.

1 0

Re: [petsc-dev] DMGetMatrix --> DMGetMatrices?
by Dmitry Karpeev 11 Feb '12

11 Feb '12

On Sat, Feb 11, 2012 at 8:24 AM, Matthew Knepley <knepley(a)gmail.com> wrote: > Barry is clearly right that each DM should produce a single Mat. However, > the proposed 2 DM (or > multiple DM) interface is just as clearly wrong. The 2 Mat interface > exists so that a user can specify > the system and preconditioning matrix. However, if the system matrix is > missing, we take it from the > DM. > > I think this should be our policy everywhere. The KSP and PC both have > SetDM, so put different DMs > in each one. As discussed, they have to be "compatible", but they generate > different Mats. This is > how we should treat auxiliary operators as well. > This sounds fine to me, but what's the API for doing this? You want to do KSPSetDM() and PCSetDM(), so that the user has to explicitly pull out the PC to set its DM? That's okay, but it doesn't match KSPSetOperators(), which takes care of getting the PC out to set the matrix on it. I guess it's not a big problem, or there could be a convenience method KSPSetDMs() in addition to KSPSetDM() and PCSetDM(). It's against our general rule of allowing the same thing to happen to different ways. Dmitry. > > This solution, however, does not solve our code optimization problems. > First, we need DMs to be > lightweight. I think Jed is right that they can share topology in the > background for now, until we have > an example where it needs to be explicit. Second, how can multiple DMs > share assembly information. > This does not just apply to KSP/PC matrices, since you can make the same > argument for auxiliary > operators (like the Laplacian on the pressure mesh). It seems like we want > a system can group up the > formation of DM operators when the control flow part of the code knows it > needs them at the same time. > Our abstractions for assembly are still quite weak, so I think we need > examples of what they would share. > My first thought is intermediate element calculations, such as stress from > strain. I still do not see the > nice way to make this happen, so I would keep them split up until we have > a better idea. > > Matt > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener >

1 0

Re: [petsc-dev] DMGetMatrix --> DMGetMatrices?
by Matthew Knepley 11 Feb '12

11 Feb '12

Barry is clearly right that each DM should produce a single Mat. However, the proposed 2 DM (or multiple DM) interface is just as clearly wrong. The 2 Mat interface exists so that a user can specify the system and preconditioning matrix. However, if the system matrix is missing, we take it from the DM. I think this should be our policy everywhere. The KSP and PC both have SetDM, so put different DMs in each one. As discussed, they have to be "compatible", but they generate different Mats. This is how we should treat auxiliary operators as well. This solution, however, does not solve our code optimization problems. First, we need DMs to be lightweight. I think Jed is right that they can share topology in the background for now, until we have an example where it needs to be explicit. Second, how can multiple DMs share assembly information. This does not just apply to KSP/PC matrices, since you can make the same argument for auxiliary operators (like the Laplacian on the pressure mesh). It seems like we want a system can group up the formation of DM operators when the control flow part of the code knows it needs them at the same time. Our abstractions for assembly are still quite weak, so I think we need examples of what they would share. My first thought is intermediate element calculations, such as stress from strain. I still do not see the nice way to make this happen, so I would keep them split up until we have a better idea. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] PETSc GPU capabilities
by Paul Mullowney 10 Feb '12

10 Feb '12

We wrote a custom sparse triangular solve. In some case it performed a little better (at most 15-20%). Sometimes it performed worse by a similar amount. I may revive our custom solve at some point. For now, Petsc only uses CUSPARSE. -Paul > Is this a custom sparse triangular solve or is it from CUSPARSE? If > different, how does the performance compare with the CUSPARSE > implementation?

1 0

Re: [petsc-dev] DMGetMatrix --> DMGetMatrices?
by Jed Brown 10 Feb '12

10 Feb '12

On Fri, Feb 10, 2012 at 20:13, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: > Okay. Then what's the need for DMSetPreallocateOnly()? >> > Oh, I see: preallocate the space, but don't set the exact sparsity pattern. > Exactly.

1 0

Re: [petsc-dev] DMGetMatrix --> DMGetMatrices?
by Dmitry Karpeev 10 Feb '12

10 Feb '12

On Fri, Feb 10, 2012 at 8:11 PM, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: > > > On Fri, Feb 10, 2012 at 7:57 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > >> On Fri, Feb 10, 2012 at 19:51, Dmitry Karpeev <karpeev(a)mcs.anl.gov>wrote: >> >>> How does this explanation square with the following code from KSPSetUp? >>> >>> if (ksp->dmActive && !ksp->setupstage) { >>> /* first time in so build matrix and vector data structures using DM >>> */ >>> if (!ksp->vec_rhs) {ierr = >>> DMCreateGlobalVector(ksp->dm,&ksp->vec_rhs);CHKERRQ(ierr);} >>> if (!ksp->vec_sol) {ierr = >>> DMCreateGlobalVector(ksp->dm,&ksp->vec_sol);CHKERRQ(ierr);} >>> ierr = DMCreateMatrix(ksp->dm,MATAIJ,&A);CHKERRQ(ierr); >>> ierr = KSPSetOperators(ksp,A,A,stflg);CHKERRQ(ierr); >>> ierr = PetscObjectDereference((PetscObject)A);CHKERRQ(ierr); >>> } >>> >> >> Look down the page a little, DMComputeJacobian() is what actually >> assembles. >> >> if (ksp->setupstage != KSP_SETUP_NEWRHS) { >> ierr = KSPGetOperators(ksp,&A,&A,PETSC_NULL);CHKERRQ(ierr); >> ierr = >> DMComputeJacobian(ksp->dm,PETSC_NULL,A,A,&stflg);CHKERRQ(ierr); >> ierr = KSPSetOperators(ksp,A,A,stflg);CHKERRQ(ierr); >> } >> >> Okay. Then what's the need for DMSetPreallocateOnly()? > Oh, I see: preallocate the space, but don't set the exact sparsity pattern. Dmitry. > > Dmitry. > >

1 0

Re: [petsc-dev] DMGetMatrix --> DMGetMatrices?
by Dmitry Karpeev 10 Feb '12

10 Feb '12

On Fri, Feb 10, 2012 at 7:57 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Fri, Feb 10, 2012 at 19:51, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: > >> How does this explanation square with the following code from KSPSetUp? >> >> if (ksp->dmActive && !ksp->setupstage) { >> /* first time in so build matrix and vector data structures using DM >> */ >> if (!ksp->vec_rhs) {ierr = >> DMCreateGlobalVector(ksp->dm,&ksp->vec_rhs);CHKERRQ(ierr);} >> if (!ksp->vec_sol) {ierr = >> DMCreateGlobalVector(ksp->dm,&ksp->vec_sol);CHKERRQ(ierr);} >> ierr = DMCreateMatrix(ksp->dm,MATAIJ,&A);CHKERRQ(ierr); >> ierr = KSPSetOperators(ksp,A,A,stflg);CHKERRQ(ierr); >> ierr = PetscObjectDereference((PetscObject)A);CHKERRQ(ierr); >> } >> > > Look down the page a little, DMComputeJacobian() is what actually > assembles. > > if (ksp->setupstage != KSP_SETUP_NEWRHS) { > ierr = KSPGetOperators(ksp,&A,&A,PETSC_NULL);CHKERRQ(ierr); > ierr = > DMComputeJacobian(ksp->dm,PETSC_NULL,A,A,&stflg);CHKERRQ(ierr); > ierr = KSPSetOperators(ksp,A,A,stflg);CHKERRQ(ierr); > } > > Okay. Then what's the need for DMSetPreallocateOnly()? Dmitry.

1 0