Hello Min,

After compiling my mpich-3.2.1 with sock, while I was trying to run a= ny program including osu benchmark or examples/cpi in two m= achines, I have received following error -

Process 3 of 4 is on dhcp16194

Process 1 of 4 is on dhcp16194

Process 0 of 4 is on dhcp16198

Process 2 of 4 is on dhcp16198

Fatal error in PMPI_Bcast: Unknown erro= r class, error stack:

PMPI_Bcast(1600).......................= .....: MPI_Bcast(buf=3D0x7ffc1808542c, count=3D1, MPI_INT, root=3D0, MPI_CO= MM_WORLD) failed

MPIR_Bcast_impl(1452)..................= .....:

MPIR_Bcast(1476).......................= .....:

MPIR_Bcast_intra(1249).................= .....:

MPIR_SMP_Bcast(1081)...................= .....:

MPIR_Bcast_binomial(285)...............= .....:

MPIC_Send(303).........................= .....:

MPIC_Wait(226).........................= .....:

MPIDI_CH3i_Progress_wait(242)..........= .....: an error occurred while handling an event returned by MPIDU_Sock_Wai= t()

MPIDI_CH3I_Progress_handle_sock_event(6= 98)..:

MPIDI_CH3_Sockconn_handle_connect_event= (597): [ch3:sock] failed to connnect to remote process

MPIDU_Socki_handle_connect(808)........= .....: connection failure (set=3D0,sock=3D1,errno=3D111:Connection refused)=

MPIR_SMP_Bcast(1088)...................= .....:

MPIR_Bcast_binomial(310)...............= .....: Failure during collective

Fatal error in PMPI_Bcast: Other MPI er= ror, error stack:

PMPI_Bcast(1600)........: MPI_Bcast(buf= =3D0x7ffd9eeebdac, count=3D1, MPI_INT, root=3D0, MPI_COMM_WORLD) failed

MPIR_Bcast_impl(1452)...: <= /i>

MPIR_Bcast(1476)........: <= /i>

MPIR_Bcast_intra(1249)..: <= /i>

MPIR_SMP_Bcast(1088)....: <= /i>

MPIR_Bcast_binomial(310): Failure durin= g collective

I checked the mpich FAQ and also mpic= h discussion list. Based on that I have checked followings a= nd found they are fine in my machines -

- firewall is disabled in both machine

- I can do password less ssh in both machine

- /etc/hosts in both machine configured wi= th ip address and name properly

- I have updated the library path and used= absolute path for mpiexec

- Most importantly when I configured and b= uild mpich with tcp, it works fine.

I think I am missing something but could not figure = out yet. Any help would be appreciated.

Thank you.

Best Regards,

Abu Naser

From: Min Si <msi@anl.go= v>
Sent: Tuesday, June 26, 2018 12:54:29 PM
To: discuss@mpich.org
Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less ti= me than 4KB and 2KB takes less time than 1KB?

Hi Abu,

I think the results are stable enough. Perhaps you could also try the follo= wing tests, and see if similar trend exists:
- MPICH/socket (set `--with-device=3Dch3: sock` at configure)
- A socket-based pingpong test without MPI.

At this point, I could not think of any MPI-specific design for 2k/8k messa= ges. My guess is that it is related to your network connection.

Min

On 2018/06/24 11: 09, Abu Naser wrote:

Hello Min and Jeff,

Here is my experiment results. Default number of iterations in osu_= latency for 0B =96 8KB is 10,000. With that setting I had run the osu_laten= cy 100 times and found standard deviation 33 for 8KB message size.

So later I have set the iteration to 50,000 and 100,000 for 1KB =96 16KB= message size. Then run osu_latency for 100 times for each setting and take= the average and standard deviation.

Msg Size in Bytes

Avg time in us (50K iterations)

Avg time in us (100k iterations)

Standard deviation (50K iterations)

Standard deviation (100K iterations)

1k

85.10

84.9

0.55

0.45

2k

75.79

74.63

5.09

4.44

4k

273.80

274.71

4.18

2.45

8k

258.56

249.83

21.14

28

16k

281.31

281.02

3.22

4.10

The standard deviation of 8K message is so high and that implies it actu= ally not producing any consistent latency time. Looks like that's the = reason for 8K is taking less time than 4K.

Meanwhile, 2K has standard deviation less than 5 but 1K message latency = timing are more densely populated than 2K. So probably this is the explanat= ion for 2K message less latency time.

Thank you for your suggestions.

Best Regards,

Abu Naser

From: Abu Naser
Sent: Wednesday, June 20, 2018 1:48:53 PM
To: discuss@mpich.org
Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less ti= me than 4KB and 2KB takes less time than 1KB?

Hello Min,

Thanks for the clarification. I will do the experiment.

Thanks.

Best Regards,

Abu Naser

From: Min Si <msi@anl.gov>
Sent: Wednesday, June 20, 2018 1:39:30 PM
To: discuss@mpich.org
Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less ti= me than 4KB and 2KB takes less time than 1KB?

Hi Abu,

I think Jeff means that you should run your experiment with more iterations= in order to get a stable results.
- Increase the iteration of for loop in each execution (I think osu benchma= rk allows you to set it)
- Run the experiments 10 or 100 times, and take the average and standard de= viation.

If you see a very small standard deviation (e.g., <=3D5%), then the tren= d is stable and you might not see such gaps.

Best regards,
Min

On 2018/06/20 12: 14, Abu Naser wrote:<= br>

Hello Jeff,

Yes, I am using a switch and other machines are also connected with= that switch.

If I remove other machines and just use my two node with the switch, the= n will it improve the performance by 200 ~ 400 iterations?

Meanwhile I will give a try with a single dedicated cable. =

Thank you.

Best Regards,

Abu Naser

From: Jeff Hammond <jeff.science@gmail.com>
Sent: Wednesday, June 20, 2018 12:52:06 PM
To: MPICH
Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less ti= me than 4KB and 2KB takes less time than 1KB?

Is the ethernet connection a single dedicated cable betwee= n the two machines or are you running through a switch that handles other t= raffic?

My best guess is that this is noise and that you may be able to avoid = it by running a very long time, e.g. 10000 iterations.

Jeff

On Wed, Jun 20, 2018 at 6: 53 AM, Abu Nas= er <an16e@my.fsu.edu>= ; wrote:

Good day to all,

I had run point to point osu_latency test in two nodes for 200 times.&nb= sp; Followings are the average time in microsecond for various size of the = messages -

1KB   84.8514 us
2KB   73.52535 us
4KB   272.55275 us
8KB   234.86385 us
16KB   288.88 us
32KB   523.3725 us
64KB   910.4025 us

From the above looks like, 2KB message has less latency than 1 KB and 8K= B has less latency than 4KB.

I was looking for explanation of this behavior but did not get any= .

MPIR_CVAR_CH3_EAGER_MAX_MSG_SIZE is set to 128K= B. So none of the above message size is using Rendezvous protocol. Is there= any partition inside eager protocol (e.g. 0 - 512 bytes, 1KB - 8KB, 16KB -= 64KB)? If yes then what are the boundaries for them? Can I log them with debug-event-logging?

Setup I am using:

- two nodes has intel core i7, one with 16gb memory another one 8gb

- mpich 3.2.1, configured and build to use nemesis tcp

- 1gb Ethernet connection

- NFS is using for sharing

- osu_latency: uses MPI_Send and MPI_Recv

- MPIR_CVAR_CH3_EAGER_MAX_MSG_SIZE=3D 131072 (128KB)

Can anyone help me on that? Thanks in advance.

Best Regards,

Abu Naser

_______________________________________________
discuss mailing list disc= uss@mpich.org
To manage subscription options or unsubscribe:
https: //lists.mpich.org/mailman/listinfo/discuss<= br>

--

Jeff Hammond
jeff.science@gmail.co= m
http: //jeffhammond.gi= thub.io/

_______________________________________________=0A= discuss mailing list discuss@mpich.org=0A= To manage subscription options or unsubscribe: =0A= https: //lists.mpich.org/mailman/listinfo/discuss=0A=

_______________________________________________=0A= discuss mailing list discuss@mpich.org=0A= To manage subscription options or unsubscribe: =0A= = https: //lists.mpich.org/mailman/listinfo/discuss=0A=

Msg Size in Bytes	Avg time in us (50K iterations)	Avg time in us (100k iterations)	Standard deviation (50K iterations)	Standard deviation (100K iterations)
1k	85.10	84.9	0.55	0.45
2k	75.79	74.63	5.09	4.44
4k	273.80	274.71	4.18	2.45
8k	258.56	249.83	21.14	28
16k	281.31	281.02	3.22	4.10