December 2016 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] mpi_gather slow on a single node
by Keith.Bannister＠csiro.au 08 Dec '16

08 Dec '16

On 9 Dec 2016, at 9:38 am, Keith Bannister <ban115(a)csiro.au> wrote: > Moreover, that latency should be much smaller if it is using shared memory. If I understand it right, the average throughput = message_size/latency = 6.5 Gbits/sec, which resembles a network throughput to me. I’m sure the memory bus on this machine can sustain much more than that. Ah, my mistake on this point. Looking at the osu_gather.c code, the correct calculation should be throughput = message_size * nranks / latency = 78 Gbits/sec. So the throughput is OK, but the latency variability is still pretty wild. K _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpi_gather slow on a single node
by Jeff Hammond 08 Dec '16

08 Dec '16

Why are you building MPICH from source on a Cray machine? Cray MPI is based on MPICH and highly optimized for that system. Jeff On Thu, Dec 8, 2016 at 2:38 PM, <Keith.Bannister(a)csiro.au> wrote: > HI Halim, > > Thanks for getting back to me. > > On 6 Dec 2016, at 3:48 am, Halim Amer <aamer(a)anl.gov> wrote: > > > 2) The latency changes by > 10x over 100 iterations. Is that normal? > > > > What is the baseline you are comparing against? Do you mean memory > latency? If yes, how do you measure it and from where do you fetch the data? > > I mean the (max - min)/min latency for a call to mpi_gather() as reported > by the osu_gather [1] benchmark for a 16MB message size with 12 ranks > running on the same node: > > mpirun -n 12 ./osu_gather -m 33554432 -f -M 1073741842 > # Size Avg Latency(us) Min Latency(us) Max Latency(us) > Iterations > ... > 16777216 19515.15 3522.29 49930.88 > 100 > > I haven’t checked the osu_gather benchmark. I imagine it initialises some > arrays and just runs mpi_gather. > > I don’t know how to find out whether mpich is using shared memory, or > going over some network interface. In either case, the jitter in latency > (x13) is much larger on this mpich setup than I’ve seen on another machine > (in which it’s a few percent). > > Moreover, that latency should be much smaller if it is using shared > memory. If I understand it right, the average throughput = > message_size/latency = 6.5 Gbits/sec, which resembles a network throughput > to me. I’m sure the memory bus on this machine can sustain much more than > that. > > vmstat doesn’t report any swapping, but I’m wondering whether there’s some > problem with how the shared memory is working? Some virtual memory setup > problem? > > > What is your hardware? > > Cray XC30 > Single node: > Memory: 64 GB > CPU > model name : Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz > Architecture : x86_64 > cpu MHz : 1999.873 > cache size : 20480 KB (Last Level) > > Total Number of Sockets : 2 > Total Number of Cores : 16 (8 per socket) > Hyperthreading : ON > Total Number of Physical Processors : 16 > Total Number of Logical Processors : 32 (2 per Phys Processor) > > > > > > MPICH configure: --prefix=/group/astronomy856/ > ban115/mpich/build-ingest-debug --enable-error-messages=all > --enable-timing=all --enable-g=most > > > > You are trying to understand if there is a performance anomaly, yet you > build MPICH in debugging mode. I suggest building with > *--enable-fast=O3,ndebug* and remove the other flags you supplied. > > I’ve compiled as you say, and got essentailly the same results. :-( > > > > [1] http://mvapich.cse.ohio-state.edu/benchmarks/ > -- > KEITH BANNISTER > CSIRO Astronomy and Space Science > T +61 2 9372 4295 > E keith.bannister(a)csiro.au > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpi_gather slow on a single node
by Keith.Bannister＠csiro.au 08 Dec '16

08 Dec '16

HI Halim, Thanks for getting back to me. On 6 Dec 2016, at 3:48 am, Halim Amer <aamer(a)anl.gov> wrote: > > 2) The latency changes by > 10x over 100 iterations. Is that normal? > > What is the baseline you are comparing against? Do you mean memory latency? If yes, how do you measure it and from where do you fetch the data? I mean the (max - min)/min latency for a call to mpi_gather() as reported by the osu_gather [1] benchmark for a 16MB message size with 12 ranks running on the same node: mpirun -n 12 ./osu_gather -m 33554432 -f -M 1073741842 # Size Avg Latency(us) Min Latency(us) Max Latency(us) Iterations ... 16777216 19515.15 3522.29 49930.88 100 I haven’t checked the osu_gather benchmark. I imagine it initialises some arrays and just runs mpi_gather. I don’t know how to find out whether mpich is using shared memory, or going over some network interface. In either case, the jitter in latency (x13) is much larger on this mpich setup than I’ve seen on another machine (in which it’s a few percent). Moreover, that latency should be much smaller if it is using shared memory. If I understand it right, the average throughput = message_size/latency = 6.5 Gbits/sec, which resembles a network throughput to me. I’m sure the memory bus on this machine can sustain much more than that. vmstat doesn’t report any swapping, but I’m wondering whether there’s some problem with how the shared memory is working? Some virtual memory setup problem? > What is your hardware? Cray XC30 Single node: Memory: 64 GB CPU model name : Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz Architecture : x86_64 cpu MHz : 1999.873 cache size : 20480 KB (Last Level) Total Number of Sockets : 2 Total Number of Cores : 16 (8 per socket) Hyperthreading : ON Total Number of Physical Processors : 16 Total Number of Logical Processors : 32 (2 per Phys Processor) > > > MPICH configure: --prefix=/group/astronomy856/ban115/mpich/build-ingest-debug --enable-error-messages=all --enable-timing=all --enable-g=most > > You are trying to understand if there is a performance anomaly, yet you build MPICH in debugging mode. I suggest building with *--enable-fast=O3,ndebug* and remove the other flags you supplied. I’ve compiled as you say, and got essentailly the same results. :-( [1] http://mvapich.cse.ohio-state.edu/benchmarks/ -- KEITH BANNISTER CSIRO Astronomy and Space Science T +61 2 9372 4295 E keith.bannister(a)csiro.au _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] osx sierra troubles mpich 3.2
by John DelSignore 08 Dec '16

08 Dec '16

I'm not sure if this is your problem or not, but we found that the Mac OS X update from 10.12.0 to 10.12.1 broke TotalView. The root cause of the problem was security related. In Mac OS X 10.12.1, when a process calls execve() (as is done when a debugger launches a target application), Darwin changes the Mach task and thread ports, and the debugger is now required to use the new ports after it receives the execve() exception event from the operating system. The latest release of TotalView (TotalView 2016.07) support Sierra and its various updates, and it's a parallel debugger designed to with MPICH, Open MPI, and other MPIs. See: http://www.roguewave.com/products-services/totalview/whats-new Hope this helps. Cheers, John D. On 12/07/16 12:39, Burlen Loring wrote: > after upgrading to osx sierra and latest xcode some of my regression tests crash in MPI calls, that's as far as I've got as MPI init seems to fail when run in the debugger. See below. Can any one advise how to run the program in the debugger on osx sierrra? > > > (lldb) target create "./bin/test/test_table_reader_distribute" > Current executable set to './bin/test/test_table_reader_distribute' (x86_64). > (lldb) settings set -- target.run-args "/Users/bloring/TECA/../TECA_data/test_tc_candidates_20.bin" "/Users/bloring/TECA/../TECA_data/test_table_reader_distribute_20.bin" "step" "0" "-1" "1" > (lldb) r > Process 75610 launched: './bin/test/test_table_reader_distribute' (x86_64) > [cli_1]: write_line error; fd=8 buf=:cmd=init pmi_version=1 pmi_subversion=1 > : > system msg for write_line failure : Bad file descriptor > [cli_1]: Unable to write to PMI_fd > [cli_1]: write_line error; fd=8 buf=:cmd=get_appnum > : > system msg for write_line failure : Bad file descriptor > Fatal error in MPI_Init_thread: Other MPI error, error stack: > MPIR_Init_thread(474): > MPID_Init(152).......: channel initialization failed > MPID_Init(426).......: PMI_Get_appnum returned -1 > [cli_1]: write_line error; fd=8 buf=:cmd=abort exitcode=1094159 > : > system msg for write_line failure : Bad file descriptor > Process 75610 exited with status = 15 (0x0000000f) > (lldb) > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] osx sierra troubles mpich 3.2
by Seo, Sangmin 08 Dec '16

08 Dec '16

Hi Burlen, You’re right. The same issue has been sitting there for a long time. We added your error case to the github issue (https://github.com/pmodels/mpich/issues/2063, we migrated all trac tickets to github), and we are investigating the issue. You can subscribe on the issue page if you’d like to get notifications about the issue. Thanks, Sangmin On Dec 7, 2016, at 5:38 PM, Burlen Loring <burlen.loring(a)gmail.com<mailto:[email protected]>> wrote: unfortunately how to start the program in a debugger is not the issue here. multiple pieces of our stack no longer even compile with gcc on OSX, and GDB does not work with clang. So clang/lldb seems to be the only viable option right now on OSX. I've seen related posts to this list and located 3 year old open bug report on mpich.org<http://mpich.org>(link below) with identical errors. This seems to be an ongoing issue with mpich on OSX. In the mean time I've switched to OpenMPI and found that our regression suite runs cleanly, not sure what's going on with MPICH but will not be able to do much else without lldb. bug report: https://trac.mpich.org/projects/mpich/ticket/2063 On 12/07/2016 01:32 PM, Seo, Sangmin wrote: You can find how to debug your MPI application at the links below: https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Debugging… https://wiki.mpich.org/mpich/index.php/Building_MPICH_For_Development#Use_g… Regards, Sangmin On Dec 7, 2016, at 11:39 AM, Burlen Loring <burlen.loring(a)gmail.com<mailto:[email protected]>> wrote: after upgrading to osx sierra and latest xcode some of my regression tests crash in MPI calls, that's as far as I've got as MPI init seems to fail when run in the debugger. See below. Can any one advise how to run the program in the debugger on osx sierrra? (lldb) target create "./bin/test/test_table_reader_distribute" Current executable set to './bin/test/test_table_reader_distribute' (x86_64). (lldb) settings set -- target.run-args "/Users/bloring/TECA/../TECA_data/test_tc_candidates_20.bin" "/Users/bloring/TECA/../TECA_data/test_table_reader_distribute_20.bin" "step" "0" "-1" "1" (lldb) r Process 75610 launched: './bin/test/test_table_reader_distribute' (x86_64) [cli_1]: write_line error; fd=8 buf=:cmd=init pmi_version=1 pmi_subversion=1 : system msg for write_line failure : Bad file descriptor [cli_1]: Unable to write to PMI_fd [cli_1]: write_line error; fd=8 buf=:cmd=get_appnum : system msg for write_line failure : Bad file descriptor Fatal error in MPI_Init_thread: Other MPI error, error stack: MPIR_Init_thread(474): MPID_Init(152).......: channel initialization failed MPID_Init(426).......: PMI_Get_appnum returned -1 [cli_1]: write_line error; fd=8 buf=:cmd=abort exitcode=1094159 : system msg for write_line failure : Bad file descriptor Process 75610 exited with status = 15 (0x0000000f) (lldb) _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] osx sierra troubles mpich 3.2
by Burlen Loring 07 Dec '16

07 Dec '16

unfortunately how to start the program in a debugger is not the issue here. multiple pieces of our stack no longer even compile with gcc on OSX, and GDB does not work with clang. So clang/lldb seems to be the only viable option right now on OSX. I've seen related posts to this list and located 3 year old open bug report on mpich.org(link below) with identical errors. This seems to be an ongoing issue with mpich on OSX. In the mean time I've switched to OpenMPI and found that our regression suite runs cleanly, not sure what's going on with MPICH but will not be able to do much else without lldb. bug report: https://trac.mpich.org/projects/mpich/ticket/2063 On 12/07/2016 01:32 PM, Seo, Sangmin wrote: > You can find how to debug your MPI application at the links below: > > https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Debugging… > https://wiki.mpich.org/mpich/index.php/Building_MPICH_For_Development#Use_g… > > Regards, > Sangmin > >> On Dec 7, 2016, at 11:39 AM, Burlen Loring <burlen.loring(a)gmail.com >> <mailto:[email protected]>> wrote: >> >> after upgrading to osx sierra and latest xcode some of my regression >> tests crash in MPI calls, that's as far as I've got as MPI init seems >> to fail when run in the debugger. See below. Can any one advise how >> to run the program in the debugger on osx sierrra? >> >> >> (lldb) target create "./bin/test/test_table_reader_distribute" >> Current executable set to './bin/test/test_table_reader_distribute' >> (x86_64). >> (lldb) settings set -- target.run-args >> "/Users/bloring/TECA/../TECA_data/test_tc_candidates_20.bin" >> "/Users/bloring/TECA/../TECA_data/test_table_reader_distribute_20.bin" >> "step" "0" "-1" "1" >> (lldb) r >> Process 75610 launched: './bin/test/test_table_reader_distribute' >> (x86_64) >> [cli_1]: write_line error; fd=8 buf=:cmd=init pmi_version=1 >> pmi_subversion=1 >> : >> system msg for write_line failure : Bad file descriptor >> [cli_1]: Unable to write to PMI_fd >> [cli_1]: write_line error; fd=8 buf=:cmd=get_appnum >> : >> system msg for write_line failure : Bad file descriptor >> Fatal error in MPI_Init_thread: Other MPI error, error stack: >> MPIR_Init_thread(474): >> MPID_Init(152).......: channel initialization failed >> MPID_Init(426).......: PMI_Get_appnum returned -1 >> [cli_1]: write_line error; fd=8 buf=:cmd=abort exitcode=1094159 >> : >> system msg for write_line failure : Bad file descriptor >> Process 75610 exited with status = 15 (0x0000000f) >> (lldb) >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] osx sierra troubles mpich 3.2
by Seo, Sangmin 07 Dec '16

07 Dec '16

You can find how to debug your MPI application at the links below: https://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Debugging… https://wiki.mpich.org/mpich/index.php/Building_MPICH_For_Development#Use_g… Regards, Sangmin On Dec 7, 2016, at 11:39 AM, Burlen Loring <burlen.loring(a)gmail.com<mailto:[email protected]>> wrote: after upgrading to osx sierra and latest xcode some of my regression tests crash in MPI calls, that's as far as I've got as MPI init seems to fail when run in the debugger. See below. Can any one advise how to run the program in the debugger on osx sierrra? (lldb) target create "./bin/test/test_table_reader_distribute" Current executable set to './bin/test/test_table_reader_distribute' (x86_64). (lldb) settings set -- target.run-args "/Users/bloring/TECA/../TECA_data/test_tc_candidates_20.bin" "/Users/bloring/TECA/../TECA_data/test_table_reader_distribute_20.bin" "step" "0" "-1" "1" (lldb) r Process 75610 launched: './bin/test/test_table_reader_distribute' (x86_64) [cli_1]: write_line error; fd=8 buf=:cmd=init pmi_version=1 pmi_subversion=1 : system msg for write_line failure : Bad file descriptor [cli_1]: Unable to write to PMI_fd [cli_1]: write_line error; fd=8 buf=:cmd=get_appnum : system msg for write_line failure : Bad file descriptor Fatal error in MPI_Init_thread: Other MPI error, error stack: MPIR_Init_thread(474): MPID_Init(152).......: channel initialization failed MPID_Init(426).......: PMI_Get_appnum returned -1 [cli_1]: write_line error; fd=8 buf=:cmd=abort exitcode=1094159 : system msg for write_line failure : Bad file descriptor Process 75610 exited with status = 15 (0x0000000f) (lldb) _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

osx sierra troubles mpich 3.2
by Burlen Loring 07 Dec '16

07 Dec '16

after upgrading to osx sierra and latest xcode some of my regression tests crash in MPI calls, that's as far as I've got as MPI init seems to fail when run in the debugger. See below. Can any one advise how to run the program in the debugger on osx sierrra? (lldb) target create "./bin/test/test_table_reader_distribute" Current executable set to './bin/test/test_table_reader_distribute' (x86_64). (lldb) settings set -- target.run-args "/Users/bloring/TECA/../TECA_data/test_tc_candidates_20.bin" "/Users/bloring/TECA/../TECA_data/test_table_reader_distribute_20.bin" "step" "0" "-1" "1" (lldb) r Process 75610 launched: './bin/test/test_table_reader_distribute' (x86_64) [cli_1]: write_line error; fd=8 buf=:cmd=init pmi_version=1 pmi_subversion=1 : system msg for write_line failure : Bad file descriptor [cli_1]: Unable to write to PMI_fd [cli_1]: write_line error; fd=8 buf=:cmd=get_appnum : system msg for write_line failure : Bad file descriptor Fatal error in MPI_Init_thread: Other MPI error, error stack: MPIR_Init_thread(474): MPID_Init(152).......: channel initialization failed MPID_Init(426).......: PMI_Get_appnum returned -1 [cli_1]: write_line error; fd=8 buf=:cmd=abort exitcode=1094159 : system msg for write_line failure : Bad file descriptor Process 75610 exited with status = 15 (0x0000000f) (lldb) _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpi_gather slow on a single node
by Halim Amer 05 Dec '16

05 Dec '16

Hi Keith, There is too little information to answer your question. > 2) The latency changes by > 10x over 100 iterations. Is that normal? What is the baseline you are comparing against? Do you mean memory latency? If yes, how do you measure it and from where do you fetch the data? What is your hardware? > MPICH configure: --prefix=/group/astronomy856/ban115/mpich/build-ingest-debug --enable-error-messages=all --enable-timing=all --enable-g=most You are trying to understand if there is a performance anomaly, yet you build MPICH in debugging mode. I suggest building with *--enable-fast=O3,ndebug* and remove the other flags you supplied. Halim www.mcs.anl.gov/~aamer On 12/1/16 5:10 PM, Keith.Bannister(a)csiro.au wrote: > Hi, > > I think I have a problem with my supercomputer :-( > > I’m new to MPI and HPC. > > Below are the results of osu_gather running on a single node. I’m interested in the 16 MB message size. > > 1) I expected mpi_gather on a single node to have throughput of roughly the memory bandwidth. Currently it’s averaging my less, i.e. ~ 7 Gbps for the 16MB message size. Adding MPICH_NO_LOCAL=1 gives essentially identical results. > 2) The latency changes by > 10x over 100 iterations. Is that normal? > > >> mpirun -n 12 ./osu_gather -m 33554432 -f -M 1073741842 > > # OSU MPI Gather Latency Test v5.3.2 > > # Size Avg Latency(us) Min Latency(us) Max Latency(us) Iterations > 1 2.49 0.52 7.51 1000 > 2 2.48 0.48 7.50 1000 > 4 2.58 0.52 7.70 1000 > 8 2.53 0.54 7.61 1000 > 16 2.66 0.54 8.08 1000 > 32 2.86 0.64 8.44 1000 > 64 2.92 0.61 8.59 1000 > 128 3.08 0.74 8.95 1000 > 256 3.27 0.84 9.57 1000 > 512 3.62 0.90 10.97 1000 > 1024 6.80 0.70 20.08 1000 > 2048 8.38 0.76 26.14 1000 > 4096 11.27 1.09 36.73 1000 > 8192 16.37 1.69 55.65 1000 > 16384 32.40 3.24 104.44 100 > 32768 57.54 6.66 174.95 100 > 65536 109.20 18.30 315.05 100 > 131072 201.53 32.24 578.51 100 > 262144 226.00 40.95 631.46 100 > 524288 405.27 72.43 1129.47 100 > 1048576 925.78 148.69 2608.16 100 > 2097152 2066.67 342.48 5678.46 100 > 4194304 4287.71 806.01 11652.06 100 > 8388608 8589.63 1754.01 23237.47 100 > 16777216 19515.15 3522.29 49930.88 100 > 33554432 40325.59 12634.90 98571.13 100 > > >> mpichversion > MPICH Version: 3.2 > MPICH Release date: Wed Nov 11 22:06:48 CST 2015 > MPICH Device: ch3:nemesis > MPICH configure: --prefix=/group/astronomy856/ban115/mpich/build-ingest-debug --enable-error-messages=all --enable-timing=all --enable-g=most --enable-fortran=no > MPICH CC: gcc -g -O2 > MPICH CXX: g++ -g -O2 > MPICH F77: gfortran -g > MPICH FC: gfortran -g > -- > KEITH BANNISTER > CSIRO Astronomy and Space Science > T +61 2 9372 4295 > E keith.bannister(a)csiro.au > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

mpi_gather slow on a single node
by Keith.Bannister＠csiro.au 01 Dec '16

01 Dec '16

Hi, I think I have a problem with my supercomputer :-( I’m new to MPI and HPC. Below are the results of osu_gather running on a single node. I’m interested in the 16 MB message size. 1) I expected mpi_gather on a single node to have throughput of roughly the memory bandwidth. Currently it’s averaging my less, i.e. ~ 7 Gbps for the 16MB message size. Adding MPICH_NO_LOCAL=1 gives essentially identical results. 2) The latency changes by > 10x over 100 iterations. Is that normal? > mpirun -n 12 ./osu_gather -m 33554432 -f -M 1073741842 # OSU MPI Gather Latency Test v5.3.2 # Size Avg Latency(us) Min Latency(us) Max Latency(us) Iterations 1 2.49 0.52 7.51 1000 2 2.48 0.48 7.50 1000 4 2.58 0.52 7.70 1000 8 2.53 0.54 7.61 1000 16 2.66 0.54 8.08 1000 32 2.86 0.64 8.44 1000 64 2.92 0.61 8.59 1000 128 3.08 0.74 8.95 1000 256 3.27 0.84 9.57 1000 512 3.62 0.90 10.97 1000 1024 6.80 0.70 20.08 1000 2048 8.38 0.76 26.14 1000 4096 11.27 1.09 36.73 1000 8192 16.37 1.69 55.65 1000 16384 32.40 3.24 104.44 100 32768 57.54 6.66 174.95 100 65536 109.20 18.30 315.05 100 131072 201.53 32.24 578.51 100 262144 226.00 40.95 631.46 100 524288 405.27 72.43 1129.47 100 1048576 925.78 148.69 2608.16 100 2097152 2066.67 342.48 5678.46 100 4194304 4287.71 806.01 11652.06 100 8388608 8589.63 1754.01 23237.47 100 16777216 19515.15 3522.29 49930.88 100 33554432 40325.59 12634.90 98571.13 100 > mpichversion MPICH Version: 3.2 MPICH Release date: Wed Nov 11 22:06:48 CST 2015 MPICH Device: ch3:nemesis MPICH configure: --prefix=/group/astronomy856/ban115/mpich/build-ingest-debug --enable-error-messages=all --enable-timing=all --enable-g=most --enable-fortran=no MPICH CC: gcc -g -O2 MPICH CXX: g++ -g -O2 MPICH F77: gfortran -g MPICH FC: gfortran -g -- KEITH BANNISTER CSIRO Astronomy and Space Science T +61 2 9372 4295 E keith.bannister(a)csiro.au _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0