November 2011 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Generality of VecScatter
by Jed Brown 29 Nov '11

29 Nov '11

You don't need any of this unless your threads need to actually sleep instead of spin. I think we can afford to have them spin, so we don't need any locking. You let the threads continue by doing an ordinary write to a memory location. (You can do more sophisticated things with cmpxchg, which you likely need at synchronization points.) On Nov 29, 2011 8:41 AM, "Dmitry Karpeev" <karpeev(a)mcs.anl.gov> wrote: > Barry, > > You might want to take a look at this: > http://stackoverflow.com/questions/2994216/linux-pthread-suspend > The solution suggested there (Answer 1) is to use sigwait(). Answer 3, > although not the actual solution, > explains why the pthread_suspend/pthread_continue-based solution is racy. > I think this may be a general issue > with the problem, which requires the use of mutexes. > > What example do you use to test this? I'd like to take a look. > > Thanks. > Dmitry. > > On Thu, Nov 24, 2011 at 4:10 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > >> On Thu, Nov 24, 2011 at 16:01, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >> >>> Yes, but they can only get access to that shared variable one at a >>> time: first get's it, then second get's it, then third gets, .... Ok for a >>> couple of cores but not for dozens. >>> >>> Take a look at src/sys/objects/pthread.c for the various ways we have >>> coded for "waking" the threads. Maybe I am missing something but this is >>> the best Kerry and I could figure out. >>> >> >> What exactly should I be looking at? Can't you have all the threads spin >> on a normal shared variable (not a mutex) that is only written by the >> thread that needs to spark them? Or use a fetch-and-add atomic if you want >> to keep track of how many are running or limit the number? The latter could >> use a tree to get logarithmic cost, but if they are stored next to each >> other, you would still have O(P) cache invalidations. >> > >

1 0

Re: [petsc-dev] Generality of VecScatter
by Dmitry Karpeev 29 Nov '11

29 Nov '11

Barry, You might want to take a look at this: http://stackoverflow.com/questions/2994216/linux-pthread-suspend The solution suggested there (Answer 1) is to use sigwait(). Answer 3, although not the actual solution, explains why the pthread_suspend/pthread_continue-based solution is racy. I think this may be a general issue with the problem, which requires the use of mutexes. What example do you use to test this? I'd like to take a look. Thanks. Dmitry. On Thu, Nov 24, 2011 at 4:10 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Thu, Nov 24, 2011 at 16:01, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > >> Yes, but they can only get access to that shared variable one at a >> time: first get's it, then second get's it, then third gets, .... Ok for a >> couple of cores but not for dozens. >> >> Take a look at src/sys/objects/pthread.c for the various ways we have >> coded for "waking" the threads. Maybe I am missing something but this is >> the best Kerry and I could figure out. >> > > What exactly should I be looking at? Can't you have all the threads spin > on a normal shared variable (not a mutex) that is only written by the > thread that needs to spark them? Or use a fetch-and-add atomic if you want > to keep track of how many are running or limit the number? The latter could > use a tree to get logarithmic cost, but if they are stored next to each > other, you would still have O(P) cache invalidations. >

1 0

Re: [petsc-dev] uni
by Barry Smith 27 Nov '11

27 Nov '11

Mark, The mpiuni MPI_Comm_create() didn't increase the reference count for the comm hence freed too soon. Also I removed your use of PetscCommDuplicate(), there is no reason to call that before passing the communicator to create a PETSc object, you can just pass any old MPI_Comm and PETSc does the right thing. Only reason to call PetscCommDuplicate() is if you are going to do MPI message passing on the communicator and it may not be a PETSc communicator. Let me know if there are additional problems, Barry On Nov 23, 2011, at 5:47 PM, Mark F. Adams wrote: > > On Nov 23, 2011, at 6:10 PM, Barry Smith wrote: > >> >> On Nov 23, 2011, at 3:14 PM, Mark F. Adams wrote: >> >>> I fixed my code error with valgrind and now I see the same thing (appended) as before (eg, valgrind does not seem to help). >>> >>> I'm suspecting that MatConvert is messed, but I will try not deleting this matrix and see what happens. >>> >>> >>> ==69836== Conditional jump or move depends on uninitialised value(s) >>> ==69836== at 0x1E8A5DE: ATL_dgemvT_a1_x1_b0_y1 (in /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib) >>> ==69836== >>> ==69836== Conditional jump or move depends on uninitialised value(s) >>> ==69836== at 0x1EBAC7A: ATL_dger1_a1_x1_yX (in /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib) >>> ==69836== >>> [0] >> >> Yes it pisses me off to no end that Apple got suckered into using that ATLAS crap and it is buggy with basic memory errors and Jack doesn't give a shit. >> > > Wow .... > > So I fixed the problem. The errors happened when calling the repartitioning code w/o MPI. THe code works fine on one processor with the MPI version. So I think there is a bug in PETSc having to do with MatConvert or Partitioning code w/o MPI. > > I have pushed fix today, but if you really want to debug it ... I'm sure that any no-MPI test using GAMG would trip this. All that needs to change is on line 191 of gamg.c: > > if( s_avoid_repart || npe==1 ) { > > change it to > > if( s_avoid_repart ) { > > This will run the partitioning code even on one processor. > > Let me know if I can help. > > Thanks, > Mark > > >> >> It you can tell us how to run a code (or push an example and tell use it) that reproduces this we can also debug it to see what the problem is. >> >> Barry >> >>> >>> On Nov 23, 2011, at 2:01 PM, Barry Smith wrote: >>> >>>> >>>> On Nov 23, 2011, at 10:31 AM, Mark F. Adams wrote: >>>> >>>>> Yep, need to keep the extra include with uni, but not the lib anymore. >>>>> >>>>> Now I get this error, any ideas? I need a uni regression test apparently. >>>> >>>> After you do the "make all test" on PETSc do "make alltests" and it should run all the tests that run on one process. >>>> >>>> Something is wrong with your reference counting likely here. Does this work with --with-debugging=0? I would run debug version with valgrind first. Then debug version in debugger. >>>> >>>> Barry >>>> >>>> >>>>> >>>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> [0]PETSC ERROR: Corrupt argument: >>>>> see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind! >>>>> [0]PETSC ERROR: Inner MPI_Comm does not have expected reference to outer comm! >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: Petsc Development HG revision: 3dda30872eeec1a2ea5c130f267935c5e0b2534a HG Date: Mon Nov 21 11:19:39 2011 -0800 >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: ./viscousTensorSolve2d.Darwin.g++-4.gfortran.DEBUG.OPT.PETSC.ex on a arch-maco named madams-macbk-3.local by markadams Wed Nov 23 11:25:30 2011 >>>>> [0]PETSC ERROR: Libraries linked from /Users/markadams/Codes/petsc-dev/arch-macosx-gnu-seq/lib >>>>> [0]PETSC ERROR: Configure run at Tue Nov 22 12:41:46 2011 >>>>> [0]PETSC ERROR: Configure options CXX=/sw/lib/gcc4.4/bin/g++-4 CC=/sw/lib/gcc4.4/bin/gcc-4 FC=/sw/lib/gcc4.4/bin/gfortran --with-x=0 --with-clanguage=c++ --with-debugging=0 --with-mpi=0 PETSC_ARCH=arch-macosx-gnu-seq >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: Petsc_DelComm() line 448 in /Users/markadams/Codes/petsc-dev/src/sys/objects/pinit.c >>>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>>> [0]PETSC ERROR: Corrupt argument: >>>>> see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind! >>>>> [0]PETSC ERROR: MPI_Comm does not have tag/name counter nor does it have inner MPI_Comm! >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: Petsc Development HG revision: 3dda30872eeec1a2ea5c130f267935c5e0b2534a HG Date: Mon Nov 21 11:19:39 2011 -0800 >>>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: ./viscousTensorSolve2d.Darwin.g++-4.gfortran.DEBUG.OPT.PETSC.ex on a arch-maco named madams-macbk-3.local by markadams Wed Nov 23 11:25:30 2011 >>>>> [0]PETSC ERROR: Libraries linked from /Users/markadams/Codes/petsc-dev/arch-macosx-gnu-seq/lib >>>>> [0]PETSC ERROR: Configure run at Tue Nov 22 12:41:46 2011 >>>>> [0]PETSC ERROR: Configure options CXX=/sw/lib/gcc4.4/bin/g++-4 CC=/sw/lib/gcc4.4/bin/gcc-4 FC=/sw/lib/gcc4.4/bin/gfortran --with-x=0 --with-clanguage=c++ --with-debugging=0 --with-mpi=0 PETSC_ARCH=arch-macosx-gnu-seq >>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>> [0]PETSC ERROR: PetscCommDestroy() line 229 in /Users/markadams/Codes/petsc-dev/src/sys/objects/tagm.c >>>>> [0]PETSC ERROR: PetscHeaderDestroy_Private() line 110 in /Users/markadams/Codes/petsc-dev/src/sys/objects/inherit.c >>>>> [0]PETSC ERROR: MatDestroy() line 1045 in /Users/markadams/Codes/petsc-dev/src/mat/interface/matrix.c >>>>> [0]PETSC ERROR: partitionLevel() line 347 in /Users/markadams/Codes/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>>>> [0]PETSC ERROR: PCSetUp_GAMG() line 599 in /Users/markadams/Codes/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>>>> [0]PETSC ERROR: PCSetUp() line 819 in /Users/markadams/Codes/petsc-dev/src/ksp/pc/interface/precon.c >>>>> [0]PETSC ERROR: KSPSetUp() line 261 in /Users/markadams/Codes/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>>> [0]PETSC ERROR: solveprivate() line 191 in "unknowndirectory/"PetscLinearSolverI.H >>>>> >>>>> >>>>> >>>>> >>>>> On Nov 22, 2011, at 3:19 PM, Barry Smith wrote: >>>>> >>>>>> >>>>>> You have som other MPI include file hanging around that is getting intercepted before the MPI Uni one. >>>>>> >>>>>> Barry >>>>>> >>>>>> On Nov 22, 2011, at 12:00 PM, Mark F. Adams wrote: >>>>>> >>>>>>> Scorched earth did not work: >>>>>>> >>>>>>> "VecCreate(ompi_communicator_t*, _p_Vec**)", referenced from: >>>>>>> PetscLinearSolver<LevelData<FArrayBox> >::solveprivate(LevelData<FArrayBox>&, LevelData<FArrayBox> const&) in libamrelliptic2d.Darwin.g++-4.gfortran.DEBUG.OPT.PETSC.a(PetscLinearSolver.o) >>>>>>> "_ompi_mpi_comm_self", referenced from: >>>>>>> _ompi_mpi_comm_self$non_lazy_ptr in libamrelliptic2d.Darwin.g++-4.gfortran.DEBUG.OPT.PETSC.a(PetscLinearSolver.o) >>>>>>> >>>>>>> <configure.log.gz> >>>>>>> >>>>>>> On Nov 22, 2011, at 11:47 AM, Satish Balay wrote: >>>>>>> >>>>>>>> Something is strange. >>>>>>>> >>>>>>>> I'll suggeset >>>>>>>> >>>>>>>> rm -rf externalpackages arch-macosx-gnu-seq >>>>>>>> >>>>>>>> and then doing a fresh build. >>>>>>>> >>>>>>>> And then verifying the build with 'make test' >>>>>>>> >>>>>>>> If 'make test' works - and the application build is not working- I'd >>>>>>>> like to see the complete compile log thats producing this error. >>>>>>>> >>>>>>>> >>>>>>>> wrt libmpiuni.a - all the relavent stuff is added to libpetsc.a and a separate >>>>>>>> libmpiuni.a is not created. >>>>>>>> >>>>>>>> Satish >>>>>>>> >>>>>>>> On Tue, 22 Nov 2011, Mark F. Adams wrote: >>>>>>>> >>>>>>>>> I'm trying to build a non MPI code and get this error: >>>>>>>>> >>>>>>>>> Undefined symbols: >>>>>>>>> "_ompi_mpi_comm_self", referenced from: >>>>>>>>> _ompi_mpi_comm_self$non_lazy_ptr in libamrelliptic2d.Darwin.g++-4.gfortran.DEBUG.OPT.PETSC.a(PetscLinearSolver.o) >>>>>>>>> (maybe you meant: _ompi_mpi_comm_self$non_lazy_ptr) >>>>>>>>> >>>>>>>>> >>>>>>>>> In the past I have linked libmpiuni.a but that did not seem to get created. Did libmpiuni.a go away or did my make fail ... >>>>>>>>> >>>>>>>>> Mark >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> >>> >> >> >

1 0

Re: [petsc-dev] petsc4py/numpy: is there a performance difference between these two pieces of code?
by Aron Ahmadia 27 Nov '11

27 Nov '11

On Sun, Nov 27, 2011 at 11:12 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Sun, Nov 27, 2011 at 01:44, Aron Ahmadia <aron.ahmadia(a)kaust.edu.sa>wrote: > >> qbc is a multi-dimensional, Fortran ordered numpy array >> > > Why is it a separate array instead of the one inside of the local vector? > > Are you asking: "Why is this not a view into the local vector?" I'm guessing because we never thought about it that way :) I ask the question mostly because we are reevaluating a large set of performance-related modifications to the pyclaw code base, and I didn't see a clear justification for the use of reset/place instead of localToGloba/reset... I'll have to think a little more about considering qbc a view of the local array instead of a copy, and probably take a closer look into how it's being used in pyclaw to explain that. -A

1 0

Re: [petsc-dev] petsc4py/numpy: is there a performance difference between these two pieces of code?
by Jed Brown 27 Nov '11

27 Nov '11

On Sun, Nov 27, 2011 at 01:44, Aron Ahmadia <aron.ahmadia(a)kaust.edu.sa>wrote: > qbc is a multi-dimensional, Fortran ordered numpy array > Why is it a separate array instead of the one inside of the local vector? Well, let's just think about what actually happens. > # METHOD 1 > > # with fset(self,q) defined as follows: > def fset(self,q): > self.gqVec.setArray(q.reshape([-1], order = 'F')) > This is memcpy > > self.q = qbc[:,mbc:-mbc,mbc:-mbc] > This will do another copy to turn the view into contiguous storage that can be passed on. Note: I think PyClaw's use of fset for setting elements of an array is confusing since the call site looks like it should have different semantics than it does. > > # METHOD 2 > > self.lqVec.placeArray(qbc.reshape([-1], order = 'F')) > I think/hope this does no copy on the Numpy side. > self.q_da.localToGlobal(self.lqVec,self.gqVec) > This updates the ghost values without an intermediate copy. > self.lqVec.resetArray() > > # METHOD 3 > > self.lqVec.placeArray(qbc) > This is the same as before, right? > self.q_da.localToGlobal(self.lqVec,self.gqVec) > self.lqVec.resetArray() >

1 0

petsc4py/numpy: is there a performance difference between these two pieces of code?
by Aron Ahmadia 27 Nov '11

27 Nov '11

Dear All, Just a quick question on whether there should be a performance difference between different approaches in moving data from a local Vec to a global Vec: qbc is a multi-dimensional, Fortran ordered numpy array # METHOD 1 # with fset(self,q) defined as follows: def fset(self,q): self.gqVec.setArray(q.reshape([-1], order = 'F')) self.q = qbc[:,mbc:-mbc,mbc:-mbc] # METHOD 2 self.lqVec.placeArray(qbc.reshape([-1], order = 'F')) self.q_da.localToGlobal(self.lqVec,self.gqVec) self.lqVec.resetArray() # METHOD 3 self.lqVec.placeArray(qbc) self.q_da.localToGlobal(self.lqVec,self.gqVec) self.lqVec.resetArray() A

1 0

Re: [petsc-dev] programming model for PETSc
by Jed Brown 26 Nov '11

26 Nov '11

On Sat, Nov 26, 2011 at 12:34, Matthew Knepley <knepley(a)gmail.com> wrote: > I do not see DG a fundamentally different. > It's not, but there are two traversals in play: an element-based traversal and a face-based traversal. That means that memory layout for efficient traversal is a multi-objective problem. Of course you can do a few things redundantly, but evaluating all the face fluxes redundantly would be hard. Also, if you do a single traversal where cells evaluate fluxes on some of their faces when they come to them, you lose lock-step between threads. These are the reasons why I think it's a better test case to understand. > My first rule would be, never assume library intelligence. So we make it > possible to give a schedule for kernels, but initially there is > no scheduling code. It is user input, with a dumb default. The flat memory > is the thread shared memory, and everything else is messages. > Fine, but now it looks like we have a two-level interface because we have to know what distributed memory operations are needed before launching kernels. The kernels don't see a global address space in any implementation I'm aware of. > > >> Can we have a way to specify the required memory access before launching >> the kernels? >> > > This is exactly what Victor's thing is about Well sure, except that it doesn't say anything about how that would be done. It's a model of distributed computation that currently lacks a statement about what the interface would look like or even which things the user specifies.

1 0

Re: [petsc-dev] Generality of VecScatter
by Jed Brown 26 Nov '11

26 Nov '11

On Sat, Nov 26, 2011 at 14:25, Mark F. Adams <mark.adams(a)columbia.edu>wrote: > So each MPI_Get initiates a message and you pack up a message with an > array of remote pointers or something? > MPI_Get() has an MPI_Datatype for the remote process. That describes where in the remote buffer the values should be gotten from. This is much like how VecScatter can operate on just part of a Vec. > > It sounds like you are saying that you have an auxiliary arrays of remote > global indices for each processor that you communicate with and your > broadcast code looks like: > > for all 'proc' that I talk to > for all proc that I _need_ an update from, I don't know or care who needs an update from me > i = 0 > for all 'v' on proc list > if v.state == not-done > I didn't think we would bother with communicating only those values that were actually updated. Since this should converge in just a few rounds, I figured that we would just update all ghost points. Note that communicating the metadata to send only those values that have actually changed may make the algorithm less latency-tolerant. > data[i++] = [ &v.state, v.id, STATE ] // vague code here... > endif > endfor > MPI_Get(proc,i,data) > endfor > I thought we would skip any loop over vertices and just one-time setup: remotes = {} for each ghost vertex v: remotes[v.owner_rank] += 1 # add/increment entry for this rank nremotes = remotes.size build MPI_Datatype their_datatype[nremotes] (using the remote index stored for each ghosted vertex) and my_datatype[nremotes] (using the sequential locations of the ghost vertices in my array) for each rank in remotes, implement as one loop over ghost vertices communication in each round: MPI_Win_fence(0, window) for (r,rank) in remotes: MPI_Get(my_ghosted_status_array, 1, my_datatype[r], rank, 0, 1, their_datatype[r], window) MPI_Win_fence(0, window) All the packing and unpacking is done by the implementation and I never traverse vertices in user code.

1 0

Re: [petsc-dev] Generality of VecScatter
by Mark F. Adams 26 Nov '11

26 Nov '11

On Nov 26, 2011, at 11:46 AM, Jed Brown wrote: > On Sat, Nov 26, 2011 at 10:35, Mark F. Adams <mark.adams(a)columbia.edu> wrote: > Thats what I'm asking. I assumed that there is some buffering going on here but that is a later question. > > I assumed that you would either (a) go in discrete rounds where every process does as much as it can or (b) do some predetermined amount of work, initiate the update, do another work unit, complete the last update and initiate a new one, do another work unit, etc. >> Remember that my first primitive was "broadcast", which is essentially global-to-local. The map is as a local-to-global map and the "get" is initiated at the local space. > > > I was confused, and perhaps still am, the broadcast is the one sided pull of ghost data to the local space. So a broadcast is passive for the broadcaster whereas it implied an active process to me. The reduce is a push of ghost data with the same (only) map to the global vertex. > > The way I'm writing the algorithm there is no need for a reduce, I think, but this is a detail, this algorithm broadcasts the selected vertex status (and deleted which I forgot in my psuedo-code) and then it infers the local deleted w/o having to have it sent (reduced). > > Sure, either way works. If the data volume for updates was big enough to be a performance limitation (instead of the bottleneck being latency), then we could convert to a message-based model initiated by the owner. It's easy to extract this when we want it, but it's simpler to skip it. > > > So I'm thinking a good way to implement this MIS with your model is after each vertex loop, you do what you call a "broadcast" of all new status. > > Now I want to see exactly how that is implemented, here is a guess: > > As part of the distributed graph, you have (owner rank, index) for every ghosted point. As local setup, the communication API makes a local (where in my array the ghost points reside) and remote (where in the owner's array the points reside) MPI_Datatype. Store these in an array of length num_neighbor_ranks. > > > MPI_Win_fence > loop over all of your boundary vertices that are not "done" yet, and do an MPI_Get (I assume that these requests are buffered in someway) > > Not over boundary vertices, it loops over neighbor each neighbor rank and calls one MPI_Get() to fetch the status on all vertices owned by that rank that are ghosted on this rank. So each MPI_Get initiates a message and you pack up a message with an array of remote pointers or something? It sounds like you are saying that you have an auxiliary arrays of remote global indices for each processor that you communicate with and your broadcast code looks like: for all 'proc' that I talk to i = 0 for all 'v' on proc list if v.state == not-done data[i++] = [ &v.state, v.id, STATE ] // vague code here... endif endfor MPI_Get(proc,i,data) endfor MPI_Win_fence // now ready for next vertex loop Mark > > MPI_Win_fence. > > The communication need only be finished once you need the result, so you can do more work before calling the second MPI_Win_fence().

1 0

Re: [petsc-dev] programming model for PETSc
by Matthew Knepley 26 Nov '11

26 Nov '11

On Sat, Nov 26, 2011 at 12:07 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Fri, Nov 25, 2011 at 16:48, Matthew Knepley <knepley(a)gmail.com> wrote: > >> Synopsis of what I said before to elicit comment: >> >> 1) I think the only thing we can learn from Brook, CUDA, OpenCL is that >> you identify threads by a grid ID. >> >> 2) Things like BLAS are so easy that you can move up to the streaming >> model, but this does not work for >> >> - FD and FEM residual evaluation (Jed has an FD example with Aron, SNES >> ex52 is my FEM example) >> >> - FD and FEM Jacobian evaluation >> > > I think these are also probably too simple. Discontinuous Galerkin with > overlapped flux computations and interior integration would be a somewhat > better model problem. Nonlinear Gauss-Seidel in a multigrid context would > be another. > I do not see DG a fundamentally different. > > >> >> 3) If you look at ex52 I do a "thread transposition" meaning threads >> start working on different areas of >> memory which looks like a transpose on a 2D grid. I can do this using >> shared memory for the vector group. >> >> The API is very simple. Give grid indices to the thread, and its done in >> CUDA and OpenCL essentially the >> same way. >> > > As is, this seems to assume a flat memory model and the memory access only > appears in how the kernel uses threadIdx to determine what memory to > operate on. If we could say something about this up-front, then the library > could schedule tasks relative to memory and perhaps handle some updates for > distributed memory. > My first rule would be, never assume library intelligence. So we make it possible to give a schedule for kernels, but initially there is no scheduling code. It is user input, with a dumb default. The flat memory is the thread shared memory, and everything else is messages. > Can we have a way to specify the required memory access before launching > the kernels? > This is exactly what Victor's thing is about Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0