Why are you building MPICH from source on a Cray machine? Cray MPI is based on MPICH and highly optimized for that system. Jeff On Thu, Dec 8, 2016 at 2:38 PM, <[email protected]> wrote:
HI Halim,
Thanks for getting back to me.
On 6 Dec 2016, at 3:48 am, Halim Amer <[email protected]> wrote:
2) The latency changes by > 10x over 100 iterations. Is that normal?
What is the baseline you are comparing against? Do you mean memory latency? If yes, how do you measure it and from where do you fetch the data?
I mean the (max - min)/min latency for a call to mpi_gather() as reported by the osu_gather [1] benchmark for a 16MB message size with 12 ranks running on the same node:
mpirun -n 12 ./osu_gather -m 33554432 -f -M 1073741842 # Size Avg Latency(us) Min Latency(us) Max Latency(us) Iterations ... 16777216 19515.15 3522.29 49930.88 100
I haven’t checked the osu_gather benchmark. I imagine it initialises some arrays and just runs mpi_gather.
I don’t know how to find out whether mpich is using shared memory, or going over some network interface. In either case, the jitter in latency (x13) is much larger on this mpich setup than I’ve seen on another machine (in which it’s a few percent).
Moreover, that latency should be much smaller if it is using shared memory. If I understand it right, the average throughput = message_size/latency = 6.5 Gbits/sec, which resembles a network throughput to me. I’m sure the memory bus on this machine can sustain much more than that.
vmstat doesn’t report any swapping, but I’m wondering whether there’s some problem with how the shared memory is working? Some virtual memory setup problem?
What is your hardware?
Cray XC30 Single node: Memory: 64 GB CPU model name : Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz Architecture : x86_64 cpu MHz : 1999.873 cache size : 20480 KB (Last Level)
Total Number of Sockets : 2 Total Number of Cores : 16 (8 per socket) Hyperthreading : ON Total Number of Physical Processors : 16 Total Number of Logical Processors : 32 (2 per Phys Processor)
MPICH configure: --prefix=/group/astronomy856/
ban115/mpich/build-ingest-debug --enable-error-messages=all --enable-timing=all --enable-g=most
You are trying to understand if there is a performance anomaly, yet you
build MPICH in debugging mode. I suggest building with *--enable-fast=O3,ndebug* and remove the other flags you supplied.
I’ve compiled as you say, and got essentailly the same results. :-(
[1] http://mvapich.cse.ohio-state.edu/benchmarks/ -- KEITH BANNISTER CSIRO Astronomy and Space Science T +61 2 9372 4295 E [email protected]
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
-- Jeff Hammond [email protected] http://jeffhammond.github.io/ _______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss