On Nov 24, 2011, at 4:41 PM, Matthew Knepley wrote:

On Thu, Nov 24, 2011 at 4:09 PM, Barry Smith <[email protected]> wrote:

Jed,

Let's stop arguing about whether MPI is or is not a good base for the next generation of HPC software but instead start a new conversation on what API (implemented on top of or not on top of MPI/pthreads etc etc) we want to build PETSc on to scale PETSc up to millions of cores with large NUMA nodes and GPU like accelerators.

What do you want in the API?

Let's start with the "lowest" level, or at least the smallest. I think the only sane way to program for portable performance here is using CUDA-type vectorization. This SIMT style is explained well here http://www.yosefk.com/blog/simd-simt-smt-parallelism-in-nvidia-gpus.html I think this is much easier and more portable than the intrinsics for Intel, and more performant and less error prone than threads. I think you can show that it will accomplish anything we want to do. OpenCL seems to have capitulated on this point. Do we agree here?

What syntax do you suggest for writing the code that is "vectorized"? What tools exist, could exist, for mapping from that syntax to what is needed by the various compilers/hardware? For daxpy() the syntax doesn't really matter, anything will do. For other kernels: maxpy, sparse matrix vector product, triangular solves, P*A*Pt, mesh operations, sorts, indirect access .... the choice of syntax likely matters a great deal. We should test the syntax out on a wide range of kernels. For example look at VecMAXPY_kernel vs VecMAXPY_Seq vs VecMAXPY_VecCUSPMAXPY4 and the three delegators VecMAXPY_SeqPThread, VecMAXPY_MPI, and VecMAXPY_SeqCUSP; How does data layout relate to the vectorization you are going to do on that data and vis-versa? Barry

Matt

Barry

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener