November 2011 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] programming model for PETSc
by Matthew Knepley 24 Nov '11

24 Nov '11

On Thu, Nov 24, 2011 at 6:37 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > On Nov 24, 2011, at 4:41 PM, Matthew Knepley wrote: > > > On Thu, Nov 24, 2011 at 4:09 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > > Jed, > > > > Let's stop arguing about whether MPI is or is not a good base for the > next generation of HPC software but instead start a new conversation on > what API (implemented on top of or not on top of MPI/pthreads etc etc) we > want to build PETSc on to scale PETSc up to millions of cores with large > NUMA nodes and GPU like accelerators. > > > > What do you want in the API? > > > > Let's start with the "lowest" level, or at least the smallest. I think > the only sane way to program for portable performance here > > is using CUDA-type vectorization. This SIMT style is explained well here > http://www.yosefk.com/blog/simd-simt-smt-parallelism-in-nvidia-gpus.html > > I think this is much easier and more portable than the intrinsics for > Intel, and more performant and less error prone than threads. > > I think you can show that it will accomplish anything we want to do. > OpenCL seems to have capitulated on this point. Do we agree > > here? > > What syntax do you suggest for writing the code that is "vectorized"? > What tools exist, could exist, for mapping from that syntax to what is > needed by the various compilers/hardware? > Some history. Brook is the basis for CUDA, but like any good foundation, almost everything the creator thought was important was thrown away to make something usable. Brook is a streaming language, much like Thrust. When problems fit this paradigm, it is fantastic. However, CUDA is not a streaming language. The programmer decides exactly what memory to put where, so what is there? Both CUDA and OpenCL inherit the process grid. Of course, MPI already has this (rank), so what is different? All threads in a vector have access to shared memory. I guess you could get the same thing in MPI if you had an idea of a "neighborhood" which had shared memory. This is exactly how OpenCL handles it. You specify a "vector length" (our neighborhood size) and the compiler tries it ass off to vectorize the code. In fact, you could probably manage everything I want with subcommunicators and a runtime code generator. > For daxpy() the syntax doesn't really matter, anything will do. For > other kernels: maxpy, sparse matrix vector product, triangular solves, > P*A*Pt, mesh operations, sorts, indirect access .... the choice of syntax > likely matters a great deal. We should test the syntax out on a wide range > of kernels. For example look at VecMAXPY_kernel vs VecMAXPY_Seq vs > VecMAXPY_VecCUSPMAXPY4 and the three delegators VecMAXPY_SeqPThread, > VecMAXPY_MPI, and VecMAXPY_SeqCUSP; > For things as easy as BLAS, you can go all the way to the streaming-type kernel (Thrust, PyCUDA, etc). I think this is basically solved. I am more interested in kernels like FD residual (ask Jed), FEM residual, and FEM Jacobian. I promise to finish this paper by mid-December. > How does data layout relate to the vectorization you are going to do on > that data and vis-versa? That is the crux. Vectorization is about execution layout (the CUDA thread grid). Somehow we must also layout memory and match them up. This is all of programming (which is why I initially like the stuff from Victor). Matt > > Barry > > > > > Matt > > > > > > Barry > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] programming model for PETSc
by Dmitry Karpeev 24 Nov '11

24 Nov '11

On Thu, Nov 24, 2011 at 6:56 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > On Thu, Nov 24, 2011 at 6:41 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > >> On Thu, Nov 24, 2011 at 18:30, Dmitry Karpeev <karpeev(a)mcs.anl.gov>wrote: >> >>> In any event, this business of storing explicit ranks/offsets for the >>> ghost indices is sort of antithetical to the current >>> PetscLayout-based approach of indices having parallel semantics from the >>> outset. A lot of conceptual difficulties >>> in Sieve and MOAB (and the need for the extra objects such as Overlap >>> and ParallelComm (sic!)) stem from a lack of >>> a global address space. >>> >> >> Global addresses and (rank, offset) are equivalent modulo MPI_Scan() >> which costs essentially the same as MPI_Reduce(). Not free, but not huge >> when done in setup. I think global indices are usually easier for users to >> play with (if they need anything that has global semantics, local indices >> combined with a local-to-global map is even better when sufficient), but >> they eventually need to be converted to (rank, offset) for communication >> (MPI does not know about a "global index" and I don't think it should.) >> > > I think the idea of a local point of view is fundamental, not just for > parallelism. We use LocalToGlobal mappings in FS so that yu do not have > to know about other fields, just the same as you should not have to know > about other processes. > Something has to know about the global layout. In the example about it's the PetscLayout in the Vec or Mat being split. If the PetscLayout takes on a different form, then everything that relies on range in it has to be redesigned. One way to approach this is to make every PETSc object interact with PetscLayout through an API that hides its details, not but looking at the "private" struct members directly. I think that's the right place to start. Dmitry. > Matt > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener >

1 0

Re: [petsc-dev] programming model for PETSc
by Matthew Knepley 24 Nov '11

24 Nov '11

On Thu, Nov 24, 2011 at 6:41 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Thu, Nov 24, 2011 at 18:30, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: > >> In any event, this business of storing explicit ranks/offsets for the >> ghost indices is sort of antithetical to the current >> PetscLayout-based approach of indices having parallel semantics from the >> outset. A lot of conceptual difficulties >> in Sieve and MOAB (and the need for the extra objects such as Overlap and >> ParallelComm (sic!)) stem from a lack of >> a global address space. >> > > Global addresses and (rank, offset) are equivalent modulo MPI_Scan() which > costs essentially the same as MPI_Reduce(). Not free, but not huge when > done in setup. I think global indices are usually easier for users to play > with (if they need anything that has global semantics, local indices > combined with a local-to-global map is even better when sufficient), but > they eventually need to be converted to (rank, offset) for communication > (MPI does not know about a "global index" and I don't think it should.) > I think the idea of a local point of view is fundamental, not just for parallelism. We use LocalToGlobal mappings in FS so that yu do not have to know about other fields, just the same as you should not have to know about other processes. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] programming model for PETSc
by Dmitry Karpeev 24 Nov '11

24 Nov '11

On Thu, Nov 24, 2011 at 6:35 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > On Nov 24, 2011, at 6:30 PM, Dmitry Karpeev wrote: > > > The MPICH paper anticipates O(1B) cores in an exascale machine. > > The number of cores, however, needn't be the number of nodes (with one > MPI process per node), > > if each node has O(1000) cores that are programmed using threads or > whatever. If we can limit the number > > of nodes to O(1M) (at the expense of "fattening" the nodes to whatever > crazy core counts people envisage in > > the future), then the current PetscLayout may be an acceptable price to > pay. > > Otherwise some sort of hierarchical PetscLayout or something entirely > different is necessary. > > > > In any event, this business of storing explicit ranks/offsets for the > ghost indices is sort of antithetical to the current > > PetscLayout-based approach of indices having parallel semantics from the > outset. A lot of conceptual difficulties > > in Sieve and MOAB (and the need for the extra objects such as Overlap > and ParallelComm (sic!)) stem from a lack of > > a global address space. If we are planning to do away with PetscLayout > in its current form, I think > > the whole PETSc parallelization model requires rethinking. That's > definitely not an incremental change. > > Everything is on the table. I could envision a model/implementation of > PETSc that has no concept of global numbering and always works with > rank,local indexing. > > Being able to make a user friendly system with any model/implementation, > of course, is crucial to whether the proposed system makes any sense. > I would start then by defining the API for PetscLayout (or Overlap, or whatever equivalent object), which defines the local view of the global address space. That's the only way, it seems to me, to insulate the API exposed to PETSc from these fundamental choices made underneath. Dmitry. > > Barry > > > > > Dmitry. > > > > On Thu, Nov 24, 2011 at 6:17 PM, Dmitry Karpeev <karpeev(a)mcs.anl.gov> > wrote: > > So then are you planning to re-engineer PetscLayout? > > Or is it acceptable to have a few PetscLayouts, in their current form, > lying around? > > > > On Thu, Nov 24, 2011 at 6:14 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > > On Thu, Nov 24, 2011 at 18:05, Dmitry Karpeev <karpeev(a)mcs.anl.gov> > wrote: > > Is O(commSize) storage considered unscalable in this model? > > > > I would like to avoid it where possible. If we have million-way > parallelism at the network level (e.g. one million MPI processes), then we > have to justify the memory usage. One 64-bit integer per process is 8MB. If > that is needed once for the entire application, it is probably not an > issue, but if it's needed by every object, then it could add up to a > substantial amount (considering the low-memory estimates). > > > > The MPICH guys are thinking about this: > > > > http://www.mcs.anl.gov/~goodell/pdfs/mpi-mem-final.pdf > > > > > >

1 0

Re: [petsc-dev] programming model for PETSc
by Jed Brown 24 Nov '11

24 Nov '11

On Thu, Nov 24, 2011 at 18:30, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: > In any event, this business of storing explicit ranks/offsets for the > ghost indices is sort of antithetical to the current > PetscLayout-based approach of indices having parallel semantics from the > outset. A lot of conceptual difficulties > in Sieve and MOAB (and the need for the extra objects such as Overlap and > ParallelComm (sic!)) stem from a lack of > a global address space. > Global addresses and (rank, offset) are equivalent modulo MPI_Scan() which costs essentially the same as MPI_Reduce(). Not free, but not huge when done in setup. I think global indices are usually easier for users to play with (if they need anything that has global semantics, local indices combined with a local-to-global map is even better when sufficient), but they eventually need to be converted to (rank, offset) for communication (MPI does not know about a "global index" and I don't think it should.)

1 0

Re: [petsc-dev] programming model for PETSc
by Barry Smith 24 Nov '11

24 Nov '11

On Nov 24, 2011, at 4:41 PM, Matthew Knepley wrote: > On Thu, Nov 24, 2011 at 4:09 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Jed, > > Let's stop arguing about whether MPI is or is not a good base for the next generation of HPC software but instead start a new conversation on what API (implemented on top of or not on top of MPI/pthreads etc etc) we want to build PETSc on to scale PETSc up to millions of cores with large NUMA nodes and GPU like accelerators. > > What do you want in the API? > > Let's start with the "lowest" level, or at least the smallest. I think the only sane way to program for portable performance here > is using CUDA-type vectorization. This SIMT style is explained well here http://www.yosefk.com/blog/simd-simt-smt-parallelism-in-nvidia-gpus.html > I think this is much easier and more portable than the intrinsics for Intel, and more performant and less error prone than threads. > I think you can show that it will accomplish anything we want to do. OpenCL seems to have capitulated on this point. Do we agree > here? What syntax do you suggest for writing the code that is "vectorized"? What tools exist, could exist, for mapping from that syntax to what is needed by the various compilers/hardware? For daxpy() the syntax doesn't really matter, anything will do. For other kernels: maxpy, sparse matrix vector product, triangular solves, P*A*Pt, mesh operations, sorts, indirect access .... the choice of syntax likely matters a great deal. We should test the syntax out on a wide range of kernels. For example look at VecMAXPY_kernel vs VecMAXPY_Seq vs VecMAXPY_VecCUSPMAXPY4 and the three delegators VecMAXPY_SeqPThread, VecMAXPY_MPI, and VecMAXPY_SeqCUSP; How does data layout relate to the vectorization you are going to do on that data and vis-versa? Barry > > Matt > > > Barry > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener

1 0

Re: [petsc-dev] programming model for PETSc
by Barry Smith 24 Nov '11

24 Nov '11

On Nov 24, 2011, at 6:30 PM, Dmitry Karpeev wrote: > The MPICH paper anticipates O(1B) cores in an exascale machine. > The number of cores, however, needn't be the number of nodes (with one MPI process per node), > if each node has O(1000) cores that are programmed using threads or whatever. If we can limit the number > of nodes to O(1M) (at the expense of "fattening" the nodes to whatever crazy core counts people envisage in > the future), then the current PetscLayout may be an acceptable price to pay. > Otherwise some sort of hierarchical PetscLayout or something entirely different is necessary. > > In any event, this business of storing explicit ranks/offsets for the ghost indices is sort of antithetical to the current > PetscLayout-based approach of indices having parallel semantics from the outset. A lot of conceptual difficulties > in Sieve and MOAB (and the need for the extra objects such as Overlap and ParallelComm (sic!)) stem from a lack of > a global address space. If we are planning to do away with PetscLayout in its current form, I think > the whole PETSc parallelization model requires rethinking. That's definitely not an incremental change. Everything is on the table. I could envision a model/implementation of PETSc that has no concept of global numbering and always works with rank,local indexing. Being able to make a user friendly system with any model/implementation, of course, is crucial to whether the proposed system makes any sense. Barry > > Dmitry. > > On Thu, Nov 24, 2011 at 6:17 PM, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: > So then are you planning to re-engineer PetscLayout? > Or is it acceptable to have a few PetscLayouts, in their current form, lying around? > > On Thu, Nov 24, 2011 at 6:14 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Thu, Nov 24, 2011 at 18:05, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: > Is O(commSize) storage considered unscalable in this model? > > I would like to avoid it where possible. If we have million-way parallelism at the network level (e.g. one million MPI processes), then we have to justify the memory usage. One 64-bit integer per process is 8MB. If that is needed once for the entire application, it is probably not an issue, but if it's needed by every object, then it could add up to a substantial amount (considering the low-memory estimates). > > The MPICH guys are thinking about this: > > http://www.mcs.anl.gov/~goodell/pdfs/mpi-mem-final.pdf > >

1 0

Re: [petsc-dev] programming model for PETSc
by Dmitry Karpeev 24 Nov '11

24 Nov '11

The MPICH paper anticipates O(1B) cores in an exascale machine. The number of cores, however, needn't be the number of nodes (with one MPI process per node), if each node has O(1000) cores that are programmed using threads or whatever. If we can limit the number of nodes to O(1M) (at the expense of "fattening" the nodes to whatever crazy core counts people envisage in the future), then the current PetscLayout may be an acceptable price to pay. Otherwise some sort of hierarchical PetscLayout or something entirely different is necessary. In any event, this business of storing explicit ranks/offsets for the ghost indices is sort of antithetical to the current PetscLayout-based approach of indices having parallel semantics from the outset. A lot of conceptual difficulties in Sieve and MOAB (and the need for the extra objects such as Overlap and ParallelComm (sic!)) stem from a lack of a global address space. If we are planning to do away with PetscLayout in its current form, I think the whole PETSc parallelization model requires rethinking. That's definitely not an incremental change. Dmitry. On Thu, Nov 24, 2011 at 6:17 PM, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: > So then are you planning to re-engineer PetscLayout? > Or is it acceptable to have a few PetscLayouts, in their current form, > lying around? > > On Thu, Nov 24, 2011 at 6:14 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > >> On Thu, Nov 24, 2011 at 18:05, Dmitry Karpeev <karpeev(a)mcs.anl.gov>wrote: >> >>> Is O(commSize) storage considered unscalable in this model? >> >> >> I would like to avoid it where possible. If we have million-way >> parallelism at the network level (e.g. one million MPI processes), then we >> have to justify the memory usage. One 64-bit integer per process is 8MB. If >> that is needed once for the entire application, it is probably not an >> issue, but if it's needed by every object, then it could add up to a >> substantial amount (considering the low-memory estimates). >> >> The MPICH guys are thinking about this: >> >> http://www.mcs.anl.gov/~goodell/pdfs/mpi-mem-final.pdf >> > >

1 0

Re: [petsc-dev] programming model for PETSc
by Barry Smith 24 Nov '11

24 Nov '11

On Nov 24, 2011, at 6:17 PM, Dmitry Karpeev wrote: > So then are you planning to re-engineer PetscLayout? > Or is it acceptable to have a few PetscLayouts, in their current form, lying around? Everything is on the table in this discussion. Barry > > On Thu, Nov 24, 2011 at 6:14 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Thu, Nov 24, 2011 at 18:05, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: > Is O(commSize) storage considered unscalable in this model? > > I would like to avoid it where possible. If we have million-way parallelism at the network level (e.g. one million MPI processes), then we have to justify the memory usage. One 64-bit integer per process is 8MB. If that is needed once for the entire application, it is probably not an issue, but if it's needed by every object, then it could add up to a substantial amount (considering the low-memory estimates). > > The MPICH guys are thinking about this: > > http://www.mcs.anl.gov/~goodell/pdfs/mpi-mem-final.pdf >

1 0

Re: [petsc-dev] programming model for PETSc
by Dmitry Karpeev 24 Nov '11

24 Nov '11

So then are you planning to re-engineer PetscLayout? Or is it acceptable to have a few PetscLayouts, in their current form, lying around? On Thu, Nov 24, 2011 at 6:14 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Thu, Nov 24, 2011 at 18:05, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: > >> Is O(commSize) storage considered unscalable in this model? > > > I would like to avoid it where possible. If we have million-way > parallelism at the network level (e.g. one million MPI processes), then we > have to justify the memory usage. One 64-bit integer per process is 8MB. If > that is needed once for the entire application, it is probably not an > issue, but if it's needed by every object, then it could add up to a > substantial amount (considering the low-memory estimates). > > The MPICH guys are thinking about this: > > http://www.mcs.anl.gov/~goodell/pdfs/mpi-mem-final.pdf >

1 0