On Mon, Feb 27, 2012 at 16:31, Gerard Gorman <[email protected]>wrote:

I had a quick go at trying to get some sensible benchmarks for this but I was getting too much system noise. I am particularly interested in seeing if the overhead goes to zero if num_threads(1) is used.

What timing method did you use? I did not see overhead going to zero when num_threads goes to 1 when using GCC compilers, but Intel seems to do fairly well.

I'm surprised by this. I not aware of any compiler that doesn't have OpenMP support - and then you do not actually enable OpenMP compilers generally just ignore the pragma. Do you know of any compiler that does not have OpenMP support which will complain?

Sean points out that omp.h might not be available, but that misses the point. As far as I know, recent mainstream compilers have enough sense to at least ignore these directives, but I'm sure there are still cases where it would be an issue. More importantly, #pragma was a misfeature that should never be used now that _Pragma() exists. The latter is better not just because it can be turned off, but because it can be manipulated using macros and can be explicitly compiled out.

This may not be flexible enough. You frequently want to have a parallel region, and then have multiple omp for's within that one region.

PetscPragmaOMPObject(obj, parallel) { PetscPragmaOMP(whetever you normally write for this loop) for (....) { } ... and so on }

I think what you describe is close to Fig 3 of this paper written by your neighbours: http://greg.bronevetsky.com/papers/2008IWOMP.pdf However, before making the implementation more complex, it would be good to benchmark the current approach and use a tool like likwid to measure the NUMA traffic so we can get a good handle on the costs.

Sure.

Well this is where the implementation details get richer and there are many options - they also become less portable. For example, what does all this mean for the sparc64 processors which are UMA.

Delay to runtime, use an ignorant partition for UMA. (Blue Gene/Q is also essentially uniform.) But note that even with uniform memory, cache still makes it somewhat hierarchical.

Not to mention Intel MIC which also supports OpenMP. I guess I am cautious about getting too bogged down with very invasive optimisations until we have benchmarked the basic approach which in a wide range of use cases will achieve good thread/page locality as illustrated previously.

I guess I'm just interested in exposing enough semantic information to be able to schedule a few different ways using run-time (or, if absolutely necessary, configure-time) options. I don't want to have to revisit individual loops.