Hello Min and Jeff, Here is my experiment results. Default number of iterations in osu_latency for 0B – 8KB is 10,000. With that setting I had run the osu_latency 100 times and found standard deviation 33 for 8KB message size. So later I have set the iteration to 50,000 and 100,000 for 1KB – 16KB message size. Then run osu_latency for 100 times for each setting and take the average and standard deviation. Msg Size in Bytes Avg time in us (50K iterations) Avg time in us (100k iterations) Standard deviation (50K iterations) Standard deviation (100K iterations) 1k 85.10 84.9 0.55 0.45 2k 75.79 74.63 5.09 4.44 4k 273.80 274.71 4.18 2.45 8k 258.56 249.83 21.14 28 16k 281.31 281.02 3.22 4.10 The standard deviation of 8K message is so high and that implies it actually not producing any consistent latency time. Looks like that's the reason for 8K is taking less time than 4K. Meanwhile, 2K has standard deviation less than 5 but 1K message latency timing are more densely populated than 2K. So probably this is the explanation for 2K message less latency time. Thank you for your suggestions. Best Regards, Abu Naser ________________________________ From: Abu Naser Sent: Wednesday, June 20, 2018 1:48:53 PM To: [email protected] Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Hello Min, Thanks for the clarification. I will do the experiment. Thanks. Best Regards, Abu Naser ________________________________ From: Min Si <[email protected]> Sent: Wednesday, June 20, 2018 1:39:30 PM To: [email protected] Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Hi Abu, I think Jeff means that you should run your experiment with more iterations in order to get a stable results. - Increase the iteration of for loop in each execution (I think osu benchmark allows you to set it) - Run the experiments 10 or 100 times, and take the average and standard deviation. If you see a very small standard deviation (e.g., <=5%), then the trend is stable and you might not see such gaps. Best regards, Min On 2018/06/20 12:14, Abu Naser wrote: Hello Jeff, Yes, I am using a switch and other machines are also connected with that switch. If I remove other machines and just use my two node with the switch, then will it improve the performance by 200 ~ 400 iterations? Meanwhile I will give a try with a single dedicated cable. Thank you. Best Regards, Abu Naser ________________________________ From: Jeff Hammond <[email protected]><mailto:[email protected]> Sent: Wednesday, June 20, 2018 12:52:06 PM To: MPICH Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Is the ethernet connection a single dedicated cable between the two machines or are you running through a switch that handles other traffic? My best guess is that this is noise and that you may be able to avoid it by running a very long time, e.g. 10000 iterations. Jeff On Wed, Jun 20, 2018 at 6:53 AM, Abu Naser <[email protected]<mailto:[email protected]>> wrote: Good day to all, I had run point to point osu_latency test in two nodes for 200 times. Followings are the average time in microsecond for various size of the messages - 1KB 84.8514 us 2KB 73.52535 us 4KB 272.55275 us 8KB 234.86385 us 16KB 288.88 us 32KB 523.3725 us 64KB 910.4025 us