For example, on an 8-core node, I was hoping to get async progress on 7 processes by pinning 7 comm threads to the 8th core.
Did this work at all?
What is your definition of work?
Does it make better async progress than the default?
There's absolutely no reason why it should. I am not trying to maximize progress; I am trying to maximize computational performance. Pinning the 7 comm threads to one core is going to be terrible for them, but I am assuming that I don't need that much progress, whereas I do need to the computation to run at top speed. DGEMM on 7 cores and 1 core of MPI should be much better than DGEMM on 8 cores where each core is time-shared with a comm thread.
As far as I know, there is no API in MPICH for controlling thread affinity. The right way to improve this could would be to move it inside of Nemesis and then add support for hwloc for comm threads. I assume you can move the parent processes around such that your 7 comm-intensive procs are closer to the NIC though. You should look at hwloc though.
I'm thinking about how to interact with other threads _because_ I'm writing hwloc-based affinity management now. The behavior that I think people will want is to use MPI to set process affinity and then tell me to use some affinity policy to _restrict_ the process affinity to each thread. I prefer using MPI to set process affinity because I think it's more messy for software at a higher level than MPI to "figure out" what other processes are running on the same node and agree among themselves how to divvy up resources. If there was an important configuration that could not be supported this way, however, then I suppose I could do it.
You should run Intel MPI, MKL, OpenMP and TBB and only compile your code with Intel compilers. That's your best chance to have everything work together. I really don't see how Hydra is supposed to know what GOMP is doing and try to deal with it. Maybe I am reading too much into your statement, but it sounds a lot like "give me a low calorie stout that tastes fantastic." :-)
My diff w.r.t. the SVN trunk (I did all of this before the SVN->Git conversion) is below. It is clearly a hack and I don't care. It only works on Linux or other systems that support CPU_SET. It does not work on my Mac, for example.
I have not done very much experimenting with this code other than to verify that it works (as in "does not crash and gives the same result for cpi"). Eventually, I am going to see how it works with ARMCI-MPI.
Cool, I'll be curious.
What testing I have done of MPICH_NEMESIS_ASYNC_PROGRESS=1 on Cray XC30 indicates that NWChem is better off without comm threads since it communicates and computes at fine enough granularity such that the lack of progress is less than the overhead of internal locking (because ASYNC_PROGRESS implies MPI_THREAD_MULTIPLE) and competition for execution resources (even though XC30 has Intel SNB with HT enabled). Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute [email protected] / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond