Maybe a general class of opportunities for PETSc could be wrapped up under "good abstractions for memory space awareness". That is, all these topics have been discussed recently: 1. Re Jeff's excellent suggestion about OpenMP, it would be nice if more were done to promote the alternative, MPI-based shared memory capability. It's quite clear that this is a good way to go, but this doesn't mean that people (who often equate shared memory parallelism with threads) will use it, so the more that can be done to make the internals of the library use this approach and the more that the classes (DM in particular) can make it easy to do things "the right way", the better. 2. As we saw from Richard's talk on KNL, that device will feature (when used a certain way) one NUMA domain which can provide much greater memory bandwidth. As in the discussions here before on the topic, it's a real challenge to figure out how to to make something like this usable via PETSc, and an even greater one to pick defaults in a way that won't sometimes produce bewildering performance results. Nevertheless, there's a good chance this is the kind of hardware that people will be running PETSc on, and given that this is something which attacks memory bandwidth bottlenecks, something that could speed up a lot of PETSc code. 3. People will likely also be running on machines with coprocessors/accelerators/etc for a while, and there needs to be plumbing to deal with the fact that each device may provide an extra memory space which needs to be managed properly in the flat MPI environment. This is of course related to point 1, as good use of MPI-3 shared memory seems like a reasonable way forward. 4. Related topics re MPI-4/endpoints and how PETSc really could be used properly from an existing multi-threaded environment. Karl has talked about the challenges of doing this the right way a lot recently. With all of these, introducing some clever abstractions to allow these sorts of things to be as transparent/automatic/encapsulated as possible might be very valuable and well-used additions to the library. On Sat, Jul 2, 2016 at 1:13 AM, Jeff Hammond <[email protected]> wrote:

Obviously, holistic support for OpenMP is critical to the future of PETSc :-D

On a more serious note, Matt and I have discussed the use of PETSc for sparse multidimensional array computations for dimensions greater than 2, also known as tensor computations. The associated paper describing previous work with dense arrays is http://www.eecs.berkeley.edu/Pubs/TechRpts/2012/EECS-2012-210.html. There was even an unsuccessful SciDAC application proposal that described how PETSc could be used for that domain when sparsity is important. To start, all we'd need is sparse matrix x sparse matrix multiplication, which I hear the multigrid folks also need. Sparse times dense is also important. Sparse tensor factorization would also help, but I get that there are enough open math questions there that it might be impractical to try to implement something in PETSc in the near future.

Maybe I am just biased because I spend all of my time reading www.nextplatform.com, but I hear machine learning is becoming an important HPC workload. While the most hyped efforts related to running inaccurate - the technical term is half-precision - dense matrix multiplication as fast as possible, I suspect that more elegant approaches will prevail. Presumably there is something that PETSc can do to enable machine learning algorithms. As most of the existing approaches use silly programming models based on MapReduce, it can't be too hard for PETSc to do better.

Jeff

On Fri, Jul 1, 2016 at 2:32 PM, Barry Smith <[email protected]> wrote:

...

The DOE SciDAC institutes have supported PETSc linear solver research/code development for the past fifteen years.

This email is to solicit ideas for linear solver research/code development work for the next round of SciDAC institutes (which will be a 4 year period) in PETSc. Please send me any ideas, no matter how crazy, on things you feel are missing, broken, or incomplete in PETSc with regard to linear solvers that we should propose to work on. In particular, issues coming from particular classes of applications would be good. Generic "multi physics" coupling types of things are too general (and old :-)) while work for extreme large scale is also out since that is covered under another call (ECP). But particular types of optimizations etc for existing or new codes could be in, just not for the very large scale.

Rough ideas and pointers to publications are all useful. There is an extremely short fuse so the sooner the better,

Thanks

Barry

-- Jeff Hammond [email protected] http://jeffhammond.github.io/