July 2018 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] error when running mpich hydra with sge
by Reuti 16 Jul '18

16 Jul '18

Hi, Am 16.07.2018 um 03:16 schrieb Shuwei Zhao: > Hi, All, > > I met with an issue that when run mpich-3.2.1 with SGE grid. After reading the wikipage https://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager, I tried to run a 4 process application on SGE but still get the following error: > "error: executing task of job 151347 failed: execution daemon on host "host1" didn't accept task" > "error: executing task of job 151347 failed: execution daemon on host "host1" didn't accept task" > "error: executing task of job 151347 failed: execution daemon on host "host1" didn't accept task" You requested a PE and the definition of the PE includes: control_slaves TRUE -- Reuti > Here is the list of commands I tried to run: > mpiexec -n 4 - f machine_file ./my_executable > mpiexec -rmk sge -n 4 -f machine_file ./my_executable > mpiexec -launcher sge -n 4 -f machine_file ./my_executable > mpiexec -rmk sge -launcher sge -n 4 -f machine_file ./my_executable > > Has anyone can share with me successful experience on running mpich in sge? Or can anyone point me how to fix this error? > > Appreciate your help ahead! > > Thanks, > Shuwei > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

error when running mpich hydra with sge
by Shuwei Zhao 15 Jul '18

15 Jul '18

Hi, All, I met with an issue that when run mpich-3.2.1 with SGE grid. After reading the wikipage https://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager, I tried to run a 4 process application on SGE but still get the following error: "error: executing task of job 151347 failed: execution daemon on host "host1" didn't accept task" "error: executing task of job 151347 failed: execution daemon on host "host1" didn't accept task" "error: executing task of job 151347 failed: execution daemon on host "host1" didn't accept task" Here is the list of commands I tried to run: mpiexec -n 4 - f machine_file ./my_executable mpiexec -rmk sge -n 4 -f machine_file ./my_executable mpiexec -launcher sge -n 4 -f machine_file ./my_executable mpiexec -rmk sge -launcher sge -n 4 -f machine_file ./my_executable Has anyone can share with me successful experience on running mpich in sge? Or can anyone point me how to fix this error? Appreciate your help ahead! Thanks, Shuwei _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB?
by Abu Naser 02 Jul '18

02 Jul '18

Hello Min, Now for some cases it is working and for some cases not. Cases when it worked: - when application binary (e.g cpi, osu_bw, osu_latency) is compiled with other mpicc (generated when configured with tcp), then mpiexec (generated when configured with sock) could run it. Cases not working: - application binary (e.g cpi) is compiled with mpicc (generated when configured with sock), then mpiexec (generated when configured with sock) could not run it and produce the same error message. [lib path was set] Thank you. Best Regards, Abu Naser ________________________________ From: Min Si <msi(a)anl.gov> Sent: Monday, July 2, 2018 2:10:23 PM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Could you please try mpich-3.3b3 ? http://www.mpich.org/static/downloads/3.3b3/mpich-3.3b3.tar.gz Min On 2018/07/02 13:01, Abu Naser wrote: Hello Min, I have downloaded it from http://www.mpich.org/static/downloads/3.2.1/mpich-3.2.1.tar.gz but it did not work. I have received almost same error. Except this time no process information from my remote machine. Previously I have received this - Process 3 of 4 is on dhcp16194 Process 1 of 4 is on dhcp16194 Process 0 of 4 is on dhcp16198 Process 2 of 4 is on dhcp16198 With the new source code - Process 0 of 4 is on dhcp16198 Process 2 of 4 is on dhcp16198 Entire error message is: Process 0 of 4 is on dhcp16198 Process 2 of 4 is on dhcp16198 Fatal error in PMPI_Bcast: Unknown error class, error stack: PMPI_Bcast(1600)............................: MPI_Bcast(buf=0x7ffd1ee145f0, count=1, MPI_INT, root=0, MPI_COMM_WORLD) failed MPIR_Bcast_impl(1452).......................: MPIR_Bcast(1476)............................: MPIR_Bcast_intra(1249)......................: MPIR_SMP_Bcast(1081)........................: MPIR_Bcast_binomial(285)....................: MPIC_Send(303)..............................: MPIC_Wait(226)..............................: MPIDI_CH3i_Progress_wait(242)...............: an error occurred while handling an event returned by MPIDU_Sock_Wait() MPIDI_CH3I_Progress_handle_sock_event(698)..: MPIDI_CH3_Sockconn_handle_connect_event(597): [ch3:sock] failed to connnect to remote process MPIDU_Socki_handle_connect(808).............: connection failure (set=0,sock=1,errno=111:Connection refused) MPIR_SMP_Bcast(1088)........................: MPIR_Bcast_binomial(310)....................: Failure during collective Fatal error in PMPI_Bcast: Other MPI error, error stack: PMPI_Bcast(1600)........: MPI_Bcast(buf=0x7ffe2eeb90f0, count=1, MPI_INT, root=0, MPI_COMM_WORLD) failed MPIR_Bcast_impl(1452)...: MPIR_Bcast(1476)........: MPIR_Bcast_intra(1249)..: MPIR_SMP_Bcast(1088)....: MPIR_Bcast_binomial(310): Failure during collective Again if I configure the new source with tcp, it works fine. Thank You. Best Regards, Abu Naser ________________________________ From: Min Si <msi(a)anl.gov><mailto:[email protected]> Sent: Monday, July 2, 2018 11:56:51 AM To: discuss(a)mpich.org<mailto:[email protected]> Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Hi Abu, Thanks for reporting this. Can you please try the latest release with ch3/sock and see if you still have this error ? Min On 2018/07/01 21:47, Abu Naser wrote: Hello Min, After compiling my mpich-3.2.1 with sock, while I was trying to run any program including osu benchmark or examples/cpi in two machines, I have received following error - Process 3 of 4 is on dhcp16194 Process 1 of 4 is on dhcp16194 Process 0 of 4 is on dhcp16198 Process 2 of 4 is on dhcp16198 Fatal error in PMPI_Bcast: Unknown error class, error stack: PMPI_Bcast(1600)............................: MPI_Bcast(buf=0x7ffc1808542c, count=1, MPI_INT, root=0, MPI_COMM_WORLD) failed MPIR_Bcast_impl(1452).......................: MPIR_Bcast(1476)............................: MPIR_Bcast_intra(1249)......................: MPIR_SMP_Bcast(1081)........................: MPIR_Bcast_binomial(285)....................: MPIC_Send(303)..............................: MPIC_Wait(226)..............................: MPIDI_CH3i_Progress_wait(242)...............: an error occurred while handling an event returned by MPIDU_Sock_Wait() MPIDI_CH3I_Progress_handle_sock_event(698)..: MPIDI_CH3_Sockconn_handle_connect_event(597): [ch3:sock] failed to connnect to remote process MPIDU_Socki_handle_connect(808).............: connection failure (set=0,sock=1,errno=111:Connection refused) MPIR_SMP_Bcast(1088)........................: MPIR_Bcast_binomial(310)....................: Failure during collective Fatal error in PMPI_Bcast: Other MPI error, error stack: PMPI_Bcast(1600)........: MPI_Bcast(buf=0x7ffd9eeebdac, count=1, MPI_INT, root=0, MPI_COMM_WORLD) failed MPIR_Bcast_impl(1452)...: MPIR_Bcast(1476)........: MPIR_Bcast_intra(1249)..: MPIR_SMP_Bcast(1088)....: MPIR_Bcast_binomial(310): Failure during collective I checked the mpich FAQ and also mpich discussion list. Based on that I have checked followings and found they are fine in my machines - - firewall is disabled in both machine - I can do password less ssh in both machine - /etc/hosts in both machine configured with ip address and name properly - I have updated the library path and used absolute path for mpiexec - Most importantly when I configured and build mpich with tcp, it works fine. I think I am missing something but could not figure out yet. Any help would be appreciated. Thank you. Best Regards, Abu Naser ________________________________ From: Min Si <msi(a)anl.gov><mailto:[email protected]> Sent: Tuesday, June 26, 2018 12:54:29 PM To: discuss(a)mpich.org<mailto:[email protected]> Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Hi Abu, I think the results are stable enough. Perhaps you could also try the following tests, and see if similar trend exists: - MPICH/socket (set `--with-device=ch3:sock` at configure) - A socket-based pingpong test without MPI. At this point, I could not think of any MPI-specific design for 2k/8k messages. My guess is that it is related to your network connection. Min On 2018/06/24 11:09, Abu Naser wrote: Hello Min and Jeff, Here is my experiment results. Default number of iterations in osu_latency for 0B – 8KB is 10,000. With that setting I had run the osu_latency 100 times and found standard deviation 33 for 8KB message size. So later I have set the iteration to 50,000 and 100,000 for 1KB – 16KB message size. Then run osu_latency for 100 times for each setting and take the average and standard deviation. Msg Size in Bytes Avg time in us (50K iterations) Avg time in us (100k iterations) Standard deviation (50K iterations) Standard deviation (100K iterations) 1k 85.10 84.9 0.55 0.45 2k 75.79 74.63 5.09 4.44 4k 273.80 274.71 4.18 2.45 8k 258.56 249.83 21.14 28 16k 281.31 281.02 3.22 4.10 The standard deviation of 8K message is so high and that implies it actually not producing any consistent latency time. Looks like that's the reason for 8K is taking less time than 4K. Meanwhile, 2K has standard deviation less than 5 but 1K message latency timing are more densely populated than 2K. So probably this is the explanation for 2K message less latency time. Thank you for your suggestions. Best Regards, Abu Naser ________________________________ From: Abu Naser Sent: Wednesday, June 20, 2018 1:48:53 PM To: discuss(a)mpich.org<mailto:[email protected]> Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Hello Min, Thanks for the clarification. I will do the experiment. Thanks. Best Regards, Abu Naser ________________________________ From: Min Si <msi(a)anl.gov><mailto:[email protected]> Sent: Wednesday, June 20, 2018 1:39:30 PM To: discuss(a)mpich.org<mailto:[email protected]> Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Hi Abu, I think Jeff means that you should run your experiment with more iterations in order to get a stable results. - Increase the iteration of for loop in each execution (I think osu benchmark allows you to set it) - Run the experiments 10 or 100 times, and take the average and standard deviation. If you see a very small standard deviation (e.g., <=5%), then the trend is stable and you might not see such gaps. Best regards, Min On 2018/06/20 12:14, Abu Naser wrote: Hello Jeff, Yes, I am using a switch and other machines are also connected with that switch. If I remove other machines and just use my two node with the switch, then will it improve the performance by 200 ~ 400 iterations? Meanwhile I will give a try with a single dedicated cable. Thank you. Best Regards, Abu Naser ________________________________ From: Jeff Hammond <jeff.science(a)gmail.com><mailto:[email protected]> Sent: Wednesday, June 20, 2018 12:52:06 PM To: MPICH Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Is the ethernet connection a single dedicated cable between the two machines or are you running through a switch that handles other traffic? My best guess is that this is noise and that you may be able to avoid it by running a very long time, e.g. 10000 iterations. Jeff On Wed, Jun 20, 2018 at 6:53 AM, Abu Naser <an16e(a)my.fsu.edu<mailto:[email protected]>> wrote: Good day to all, I had run point to point osu_latency test in two nodes for 200 times. Followings are the average time in microsecond for various size of the messages - 1KB 84.8514 us 2KB 73.52535 us 4KB 272.55275 us 8KB 234.86385 us 16KB 288.88 us 32KB 523.3725 us 64KB 910.4025 us

1 0

Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB?
by Min Si 02 Jul '18

02 Jul '18

Could you please try mpich-3.3b3 ? http://www.mpich.org/static/downloads/3.3b3/mpich-3.3b3.tar.gz Min On 2018/07/02 13:01, Abu Naser wrote: > > Hello Min, > > > I have downloaded it from > http://www.mpich.org/static/downloads/3.2.1/mpich-3.2.1.tar.gz but it > did not work. I have received almost same error. Except this time no > process information from my remote machine. > > *Previously I have received this - * > > /Process 3 of 4 is on dhcp16194/ > /Process 1 of 4 is on dhcp16194/ > /Process 0 of 4 is on dhcp16198/ > /Process 2 of 4 is on dhcp16198/ > > *With the new source code -* > > /Process 0 of 4 is on dhcp16198/ > /Process 2 of 4 is on dhcp16198/ > > > *Entire error message is:* > > /Process 0 of 4 is on dhcp16198/ > /Process 2 of 4 is on dhcp16198/ > /Fatal error in PMPI_Bcast: Unknown error class, error stack:/ > /PMPI_Bcast(1600)............................: > MPI_Bcast(buf=0x7ffd1ee145f0, count=1, MPI_INT, root=0, > MPI_COMM_WORLD) failed/ > /MPIR_Bcast_impl(1452).......................: / > /MPIR_Bcast(1476)............................: / > /MPIR_Bcast_intra(1249)......................: / > /MPIR_SMP_Bcast(1081)........................: / > /MPIR_Bcast_binomial(285)....................: / > /MPIC_Send(303)..............................: / > /MPIC_Wait(226)..............................: / > /MPIDI_CH3i_Progress_wait(242)...............: an error occurred while > handling an event returned by MPIDU_Sock_Wait()/ > /MPIDI_CH3I_Progress_handle_sock_event(698)..: / > /MPIDI_CH3_Sockconn_handle_connect_event(597): [ch3:sock] failed to > connnect to remote process/ > /MPIDU_Socki_handle_connect(808).............: connection failure > (set=0,sock=1,errno=111:Connection refused)/ > /MPIR_SMP_Bcast(1088)........................: / > /MPIR_Bcast_binomial(310)....................: Failure during collective/ > /Fatal error in PMPI_Bcast: Other MPI error, error stack:/ > /PMPI_Bcast(1600)........: MPI_Bcast(buf=0x7ffe2eeb90f0, count=1, > MPI_INT, root=0, MPI_COMM_WORLD) failed/ > /MPIR_Bcast_impl(1452)...: / > /MPIR_Bcast(1476)........: / > /MPIR_Bcast_intra(1249)..: / > /MPIR_SMP_Bcast(1088)....: / > /MPIR_Bcast_binomial(310): Failure during collective/ > > Again if I configure the new source with tcp, it works fine. > > > Thank You. > > > Best Regards, > > Abu Naser > > ------------------------------------------------------------------------ > *From:* Min Si <msi(a)anl.gov> > *Sent:* Monday, July 2, 2018 11:56:51 AM > *To:* discuss(a)mpich.org > *Subject:* Re: [mpich-discuss] osu_latency test: why 8KB takes less > time than 4KB and 2KB takes less time than 1KB? > Hi Abu, > > Thanks for reporting this. Can you please try the latest release with > ch3/sock and see if you still have this error ? > > Min > On 2018/07/01 21:47, Abu Naser wrote: >> >> Hello Min, >> >> >> After compiling my mpich-3.2.1 with sock, while I was trying to runï¿½ >> any program including osuï¿½benchmark orï¿½examples/cpiï¿½ in two machines, >> I have received following error - >> >> >> /Process 3 of 4 is on dhcp16194/ >> /Process 1 of 4 is on dhcp16194/ >> /Process 0 of 4 is on dhcp16198/ >> /Process 2 of 4 is on dhcp16198/ >> /Fatal error in PMPI_Bcast: Unknown error class, error stack:/ >> /PMPI_Bcast(1600)............................: >> MPI_Bcast(buf=0x7ffc1808542c, count=1, MPI_INT, root=0, >> MPI_COMM_WORLD) failed/ >> /MPIR_Bcast_impl(1452).......................: / >> /MPIR_Bcast(1476)............................: / >> /MPIR_Bcast_intra(1249)......................: / >> /MPIR_SMP_Bcast(1081)........................: / >> /MPIR_Bcast_binomial(285)....................: / >> /MPIC_Send(303)..............................: / >> /MPIC_Wait(226)..............................: / >> /MPIDI_CH3i_Progress_wait(242)...............: an error occurred >> while handling an event returned by MPIDU_Sock_Wait()/ >> /MPIDI_CH3I_Progress_handle_sock_event(698)..: / >> /MPIDI_CH3_Sockconn_handle_connect_event(597): [ch3:sock] failed to >> connnect to remote process/ >> /MPIDU_Socki_handle_connect(808).............: connection failure >> (set=0,sock=1,errno=111:Connection refused)/ >> /MPIR_SMP_Bcast(1088)........................: / >> /MPIR_Bcast_binomial(310)....................: Failure during collective/ >> /Fatal error in PMPI_Bcast: Other MPI error, error stack:/ >> /PMPI_Bcast(1600)........: MPI_Bcast(buf=0x7ffd9eeebdac, count=1, >> MPI_INT, root=0, MPI_COMM_WORLD) failed/ >> /MPIR_Bcast_impl(1452)...: / >> /MPIR_Bcast(1476)........: / >> /MPIR_Bcast_intra(1249)..: / >> /MPIR_SMP_Bcast(1088)....: / >> /MPIR_Bcast_binomial(310): Failure during collective/ >> >> I checked the mpichï¿½FAQ and also mpich discussion list. Based on that >> I have checked followingsand foundï¿½ï¿½theyï¿½are fine in my machines - >> >> - firewall is disabled in both machine >> >> - I can do password lessï¿½ssh in both machine >> >> - /etc/hosts in both machine configured with ipï¿½address and name properly >> >> - I have updated the library path and used absolute path for mpiexec >> >> - Most importantly when I configured and build mpich with tcp, it >> works fine. >> >> >> ï¿½I think I am missing something but could not figure out yet. Any >> help would be appreciated. >> >> >> Thank you. >> >> >> >> >> >> >> Best Regards, >> >> Abu Naser >> >> ------------------------------------------------------------------------ >> *From:* Min Si <msi(a)anl.gov> <mailto:[email protected]> >> *Sent:* Tuesday, June 26, 2018 12:54:29 PM >> *To:* discuss(a)mpich.org <mailto:[email protected]> >> *Subject:* Re: [mpich-discuss] osu_latency test: why 8KB takes less >> time than 4KB and 2KB takes less time than 1KB? >> Hi Abu, >> >> I think the results are stable enough. Perhaps you could also try the >> following tests, and see if similar trend exists: >> - MPICH/socket (set `--with-device=ch3:sock` at configure) >> - A socket-based pingpong test without MPI. >> >> At this point, I could not think of any MPI-specific design for 2k/8k >> messages. My guess is that it is related to your network connection. >> >> Min >> >> On 2018/06/24 11:09, Abu Naser wrote: >>> >>> Hello Min and Jeff, >>> >>> >>> Here is my experiment results.ï¿½Default number of iterations in >>> osu_latency for 0B ï¿½ 8KB is 10,000. With that setting I had run the >>> osu_latency 100 times and found standard deviation 33 for 8KB >>> message size. >>> >>> >>> So later I have set the iteration to 50,000 and 100,000 for 1KB ï¿½ >>> 16KB message size. Then run osu_latency for 100 times for each >>> setting and take the average and standard deviation. >>> >>> >>> *Msg Size in Bytes* >>> >>> >>> >>> *Avg time in us (50K iterations)* >>> >>> >>> >>> *Avg time in us (100k iterations)* >>> >>> >>> >>> *Standard deviation (50K iterations)* >>> >>> >>> >>> *Standard deviation (100K iterations)* >>> >>> 1k >>> >>> >>> >>> 85.10 >>> >>> >>> >>> 84.9 >>> >>> >>> >>> 0.55 >>> >>> >>> >>> 0.45 >>> >>> 2k >>> >>> >>> >>> 75.79 >>> >>> >>> >>> 74.63 >>> >>> >>> >>> 5.09 >>> >>> >>> >>> 4.44 >>> >>> 4k >>> >>> >>> >>> 273.80 >>> >>> >>> >>> 274.71 >>> >>> >>> >>> 4.18 >>> >>> >>> >>> 2.45 >>> >>> 8k >>> >>> >>> >>> 258.56 >>> >>> >>> >>> 249.83 >>> >>> >>> >>> 21.14 >>> >>> >>> >>> 28 >>> >>> 16k >>> >>> >>> >>> 281.31 >>> >>> >>> >>> 281.02 >>> >>> >>> >>> 3.22 >>> >>> >>> >>> 4.10 >>> >>> >>> >>> The standard deviation of 8K message is so high and that implies it >>> actually not producing any consistent latency time. Looks >>> likeï¿½that's the reason for 8K is taking less time than 4K. >>> >>> >>> Meanwhile, 2K has standard deviation less than 5 but 1K message >>> latency timing are more densely populated than 2K. So probably this >>> is the explanation for 2K message less latency time. >>> >>> >>> Thank you for your suggestions. >>> >>> >>> >>> >>> Best Regards, >>> >>> Abu Naser >>> >>> ------------------------------------------------------------------------ >>> *From:* Abu Naser >>> *Sent:* Wednesday, June 20, 2018 1:48:53 PM >>> *To:* discuss(a)mpich.org <mailto:[email protected]> >>> *Subject:* Re: [mpich-discuss] osu_latency test: why 8KB takes less >>> time than 4KB and 2KB takes less time than 1KB? >>> >>> Hello Min, >>> >>> >>> Thanks for the clarification.ï¿½ I will do the experiment. >>> >>> >>> Thanks. >>> >>> Best Regards, >>> >>> Abu Naser >>> >>> ------------------------------------------------------------------------ >>> *From:* Min Si <msi(a)anl.gov> <mailto:[email protected]> >>> *Sent:* Wednesday, June 20, 2018 1:39:30 PM >>> *To:* discuss(a)mpich.org <mailto:[email protected]> >>> *Subject:* Re: [mpich-discuss] osu_latency test: why 8KB takes less >>> time than 4KB and 2KB takes less time than 1KB? >>> Hi Abu, >>> >>> I think Jeff means that you should run your experiment with more >>> iterations in order to get a stable results. >>> - Increase the iteration of for loop in each execution (I think osu >>> benchmark allows you to set it) >>> - Run the experiments 10 or 100 times, and take the average and >>> standard deviation. >>> >>> If you see a very small standard deviation (e.g., <=5%), then the >>> trend is stable and you might not see such gaps. >>> >>> Best regards, >>> Min >>> On 2018/06/20 12:14, Abu Naser wrote: >>>> >>>> Hello Jeff, >>>> >>>> >>>> Yes, I am using a switch and other machinesï¿½are also connected with >>>> that switch. >>>> >>>> If I remove other machines and just use my two node with the >>>> switch, then will it improve the performance by 200 ~ 400 iterations? >>>> >>>> Meanwhile I will give a try with a single dedicated cable. >>>> >>>> >>>> Thank you. >>>> >>>> >>>> Best Regards, >>>> >>>> Abu Naser >>>> >>>> ------------------------------------------------------------------------ >>>> *From:* Jeff Hammond <jeff.science(a)gmail.com> >>>> <mailto:[email protected]> >>>> *Sent:* Wednesday, June 20, 2018 12:52:06 PM >>>> *To:* MPICH >>>> *Subject:* Re: [mpich-discuss] osu_latency test: why 8KB takes less >>>> time than 4KB and 2KB takes less time than 1KB? >>>> Is the ethernet connection a single dedicated cable between the two >>>> machines or are you running through a switch that handles other >>>> traffic? >>>> >>>> My best guess is that this is noise and that you may be able to >>>> avoid it by running a very long time, e.g. 10000 iterations. >>>> >>>> Jeff >>>> >>>> On Wed, Jun 20, 2018 at 6:53 AM, Abu Naser <an16e(a)my.fsu.edu >>>> <mailto:[email protected]>> wrote: >>>> >>>> >>>> Good day to all, >>>> >>>> >>>> I had run point to point osu_latency test in two nodes for 200 >>>> times.ï¿½ Followings are the average time in microsecond for >>>> various size of the messages - >>>> >>>> 1KBï¿½ï¿½ ï¿½84.8514 us >>>> 2KBï¿½ï¿½ ï¿½73.52535 us >>>> 4KBï¿½ï¿½ ï¿½272.55275 us >>>> 8KBï¿½ï¿½ ï¿½234.86385 us >>>> 16KBï¿½ï¿½ ï¿½288.88 us >>>> 32KBï¿½ï¿½ ï¿½523.3725 us >>>> 64KBï¿½ï¿½ ï¿½910.4025 us >>>> >>>> >>>> From the above looks like, 2KB message has less latency than 1 >>>> KB and 8KB has less latency than 4KB. >>>> >>>> I was looking for explanation of this behaviorï¿½ but did not get >>>> any. >>>> >>>> >>>> 1. MPIR_CVAR_CH3_EAGER_MAX_MSG_SIZEis set to 128KB. So none of >>>> the above message size is using Rendezvous protocol. Is >>>> there any partition inside eager protocol (e.g. 0 - 512 >>>> bytes, 1KB - 8KB, 16KB - 64KB)? If yes then what are the >>>> boundaries for them? Can I log them with debug-event-logging? >>>> >>>> >>>> Setup I am using: >>>> >>>> - two nodes has intel core i7, one with 16gb memory another one 8gb >>>> >>>> - mpich 3.2.1, configured and build to use nemesis tcp >>>> >>>> - 1gb Ethernet connection >>>> >>>> - NFS is using for sharing >>>> >>>> - osu_latency : uses MPI_Send and MPI_Recv >>>> >>>> - MPIR_CVAR_CH3_EAGER_MAX_MSG_SIZE= 131072 (128KB) >>>> >>>> >>>> Can anyone help me on that? Thanks in advance. >>>> >>>> >>>> >>>> >>>> Best Regards, >>>> >>>> Abu Naser >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> <https://lists.mpich.org/mailman/listinfo/discuss> >>>> >>>> >>>> >>>> >>>> -- >>>> Jeff Hammond >>>> jeff.science(a)gmail.com <mailto:[email protected]> >>>> http://jeffhammond.github.io/ >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing listdiscuss(a)mpich.org <mailto:[email protected]> >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> >>> >>> _______________________________________________ >>> discuss mailing listdiscuss(a)mpich.org <mailto:[email protected]> >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> _______________________________________________ >> discuss mailing listdiscuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB?
by Abu Naser 02 Jul '18

02 Jul '18

Hello Min, I have downloaded it from http://www.mpich.org/static/downloads/3.2.1/mpich-3.2.1.tar.gz but it did not work. I have received almost same error. Except this time no process information from my remote machine. Previously I have received this - Process 3 of 4 is on dhcp16194 Process 1 of 4 is on dhcp16194 Process 0 of 4 is on dhcp16198 Process 2 of 4 is on dhcp16198 With the new source code - Process 0 of 4 is on dhcp16198 Process 2 of 4 is on dhcp16198 Entire error message is: Process 0 of 4 is on dhcp16198 Process 2 of 4 is on dhcp16198 Fatal error in PMPI_Bcast: Unknown error class, error stack: PMPI_Bcast(1600)............................: MPI_Bcast(buf=0x7ffd1ee145f0, count=1, MPI_INT, root=0, MPI_COMM_WORLD) failed MPIR_Bcast_impl(1452).......................: MPIR_Bcast(1476)............................: MPIR_Bcast_intra(1249)......................: MPIR_SMP_Bcast(1081)........................: MPIR_Bcast_binomial(285)....................: MPIC_Send(303)..............................: MPIC_Wait(226)..............................: MPIDI_CH3i_Progress_wait(242)...............: an error occurred while handling an event returned by MPIDU_Sock_Wait() MPIDI_CH3I_Progress_handle_sock_event(698)..: MPIDI_CH3_Sockconn_handle_connect_event(597): [ch3:sock] failed to connnect to remote process MPIDU_Socki_handle_connect(808).............: connection failure (set=0,sock=1,errno=111:Connection refused) MPIR_SMP_Bcast(1088)........................: MPIR_Bcast_binomial(310)....................: Failure during collective Fatal error in PMPI_Bcast: Other MPI error, error stack: PMPI_Bcast(1600)........: MPI_Bcast(buf=0x7ffe2eeb90f0, count=1, MPI_INT, root=0, MPI_COMM_WORLD) failed MPIR_Bcast_impl(1452)...: MPIR_Bcast(1476)........: MPIR_Bcast_intra(1249)..: MPIR_SMP_Bcast(1088)....: MPIR_Bcast_binomial(310): Failure during collective Again if I configure the new source with tcp, it works fine. Thank You. Best Regards, Abu Naser ________________________________ From: Min Si <msi(a)anl.gov> Sent: Monday, July 2, 2018 11:56:51 AM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Hi Abu, Thanks for reporting this. Can you please try the latest release with ch3/sock and see if you still have this error ? Min On 2018/07/01 21:47, Abu Naser wrote: Hello Min, After compiling my mpich-3.2.1 with sock, while I was trying to run any program including osu benchmark or examples/cpi in two machines, I have received following error - Process 3 of 4 is on dhcp16194 Process 1 of 4 is on dhcp16194 Process 0 of 4 is on dhcp16198 Process 2 of 4 is on dhcp16198 Fatal error in PMPI_Bcast: Unknown error class, error stack: PMPI_Bcast(1600)............................: MPI_Bcast(buf=0x7ffc1808542c, count=1, MPI_INT, root=0, MPI_COMM_WORLD) failed MPIR_Bcast_impl(1452).......................: MPIR_Bcast(1476)............................: MPIR_Bcast_intra(1249)......................: MPIR_SMP_Bcast(1081)........................: MPIR_Bcast_binomial(285)....................: MPIC_Send(303)..............................: MPIC_Wait(226)..............................: MPIDI_CH3i_Progress_wait(242)...............: an error occurred while handling an event returned by MPIDU_Sock_Wait() MPIDI_CH3I_Progress_handle_sock_event(698)..: MPIDI_CH3_Sockconn_handle_connect_event(597): [ch3:sock] failed to connnect to remote process MPIDU_Socki_handle_connect(808).............: connection failure (set=0,sock=1,errno=111:Connection refused) MPIR_SMP_Bcast(1088)........................: MPIR_Bcast_binomial(310)....................: Failure during collective Fatal error in PMPI_Bcast: Other MPI error, error stack: PMPI_Bcast(1600)........: MPI_Bcast(buf=0x7ffd9eeebdac, count=1, MPI_INT, root=0, MPI_COMM_WORLD) failed MPIR_Bcast_impl(1452)...: MPIR_Bcast(1476)........: MPIR_Bcast_intra(1249)..: MPIR_SMP_Bcast(1088)....: MPIR_Bcast_binomial(310): Failure during collective I checked the mpich FAQ and also mpich discussion list. Based on that I have checked followings and found they are fine in my machines - - firewall is disabled in both machine - I can do password less ssh in both machine - /etc/hosts in both machine configured with ip address and name properly - I have updated the library path and used absolute path for mpiexec - Most importantly when I configured and build mpich with tcp, it works fine. I think I am missing something but could not figure out yet. Any help would be appreciated. Thank you. Best Regards, Abu Naser ________________________________ From: Min Si <msi(a)anl.gov><mailto:[email protected]> Sent: Tuesday, June 26, 2018 12:54:29 PM To: discuss(a)mpich.org<mailto:[email protected]> Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Hi Abu, I think the results are stable enough. Perhaps you could also try the following tests, and see if similar trend exists: - MPICH/socket (set `--with-device=ch3:sock` at configure) - A socket-based pingpong test without MPI. At this point, I could not think of any MPI-specific design for 2k/8k messages. My guess is that it is related to your network connection. Min On 2018/06/24 11:09, Abu Naser wrote: Hello Min and Jeff, Here is my experiment results. Default number of iterations in osu_latency for 0B – 8KB is 10,000. With that setting I had run the osu_latency 100 times and found standard deviation 33 for 8KB message size. So later I have set the iteration to 50,000 and 100,000 for 1KB – 16KB message size. Then run osu_latency for 100 times for each setting and take the average and standard deviation. Msg Size in Bytes Avg time in us (50K iterations) Avg time in us (100k iterations) Standard deviation (50K iterations) Standard deviation (100K iterations) 1k 85.10 84.9 0.55 0.45 2k 75.79 74.63 5.09 4.44 4k 273.80 274.71 4.18 2.45 8k 258.56 249.83 21.14 28 16k 281.31 281.02 3.22 4.10 The standard deviation of 8K message is so high and that implies it actually not producing any consistent latency time. Looks like that's the reason for 8K is taking less time than 4K. Meanwhile, 2K has standard deviation less than 5 but 1K message latency timing are more densely populated than 2K. So probably this is the explanation for 2K message less latency time. Thank you for your suggestions. Best Regards, Abu Naser ________________________________ From: Abu Naser Sent: Wednesday, June 20, 2018 1:48:53 PM To: discuss(a)mpich.org<mailto:[email protected]> Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Hello Min, Thanks for the clarification. I will do the experiment. Thanks. Best Regards, Abu Naser ________________________________ From: Min Si <msi(a)anl.gov><mailto:[email protected]> Sent: Wednesday, June 20, 2018 1:39:30 PM To: discuss(a)mpich.org<mailto:[email protected]> Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Hi Abu, I think Jeff means that you should run your experiment with more iterations in order to get a stable results. - Increase the iteration of for loop in each execution (I think osu benchmark allows you to set it) - Run the experiments 10 or 100 times, and take the average and standard deviation. If you see a very small standard deviation (e.g., <=5%), then the trend is stable and you might not see such gaps. Best regards, Min On 2018/06/20 12:14, Abu Naser wrote: Hello Jeff, Yes, I am using a switch and other machines are also connected with that switch. If I remove other machines and just use my two node with the switch, then will it improve the performance by 200 ~ 400 iterations? Meanwhile I will give a try with a single dedicated cable. Thank you. Best Regards, Abu Naser ________________________________ From: Jeff Hammond <jeff.science(a)gmail.com><mailto:[email protected]> Sent: Wednesday, June 20, 2018 12:52:06 PM To: MPICH Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Is the ethernet connection a single dedicated cable between the two machines or are you running through a switch that handles other traffic? My best guess is that this is noise and that you may be able to avoid it by running a very long time, e.g. 10000 iterations. Jeff On Wed, Jun 20, 2018 at 6:53 AM, Abu Naser <an16e(a)my.fsu.edu<mailto:[email protected]>> wrote: Good day to all, I had run point to point osu_latency test in two nodes for 200 times. Followings are the average time in microsecond for various size of the messages - 1KB 84.8514 us 2KB 73.52535 us 4KB 272.55275 us 8KB 234.86385 us 16KB 288.88 us 32KB 523.3725 us 64KB 910.4025 us

1 0

Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB?
by Min Si 02 Jul '18

02 Jul '18

Hi Abu, Thanks for reporting this. Can you please try the latest release with ch3/sock and see if you still have this error ? Min On 2018/07/01 21:47, Abu Naser wrote: > > Hello Min, > > > After compiling my mpich-3.2.1 with sock, while I was trying to runï¿½ > any program including osuï¿½benchmark orï¿½examples/cpi in two machines, I > have received following error - > > > /Process 3 of 4 is on dhcp16194/ > /Process 1 of 4 is on dhcp16194/ > /Process 0 of 4 is on dhcp16198/ > /Process 2 of 4 is on dhcp16198/ > /Fatal error in PMPI_Bcast: Unknown error class, error stack:/ > /PMPI_Bcast(1600)............................: > MPI_Bcast(buf=0x7ffc1808542c, count=1, MPI_INT, root=0, > MPI_COMM_WORLD) failed/ > /MPIR_Bcast_impl(1452).......................: / > /MPIR_Bcast(1476)............................: / > /MPIR_Bcast_intra(1249)......................: / > /MPIR_SMP_Bcast(1081)........................: / > /MPIR_Bcast_binomial(285)....................: / > /MPIC_Send(303)..............................: / > /MPIC_Wait(226)..............................: / > /MPIDI_CH3i_Progress_wait(242)...............: an error occurred while > handling an event returned by MPIDU_Sock_Wait()/ > /MPIDI_CH3I_Progress_handle_sock_event(698)..: / > /MPIDI_CH3_Sockconn_handle_connect_event(597): [ch3:sock] failed to > connnect to remote process/ > /MPIDU_Socki_handle_connect(808).............: connection failure > (set=0,sock=1,errno=111:Connection refused)/ > /MPIR_SMP_Bcast(1088)........................: / > /MPIR_Bcast_binomial(310)....................: Failure during collective/ > /Fatal error in PMPI_Bcast: Other MPI error, error stack:/ > /PMPI_Bcast(1600)........: MPI_Bcast(buf=0x7ffd9eeebdac, count=1, > MPI_INT, root=0, MPI_COMM_WORLD) failed/ > /MPIR_Bcast_impl(1452)...: / > /MPIR_Bcast(1476)........: / > /MPIR_Bcast_intra(1249)..: / > /MPIR_SMP_Bcast(1088)....: / > /MPIR_Bcast_binomial(310): Failure during collective/ > > I checked the mpichï¿½FAQ and also mpich discussion list. Based on that > I have checked followingsand foundï¿½ï¿½theyï¿½are fine in my machines - > > - firewall is disabled in both machine > > - I can do password lessï¿½ssh in both machine > > - /etc/hosts in both machine configured with ipï¿½address and name properly > > - I have updated the library path and used absolute path for mpiexec > > - Most importantly when I configured and build mpich with tcp, it > works fine. > > > ï¿½I think I am missing something but could not figure out yet. Any help > would be appreciated. > > > Thank you. > > > > > > > Best Regards, > > Abu Naser > > ------------------------------------------------------------------------ > *From:* Min Si <msi(a)anl.gov> > *Sent:* Tuesday, June 26, 2018 12:54:29 PM > *To:* discuss(a)mpich.org > *Subject:* Re: [mpich-discuss] osu_latency test: why 8KB takes less > time than 4KB and 2KB takes less time than 1KB? > Hi Abu, > > I think the results are stable enough. Perhaps you could also try the > following tests, and see if similar trend exists: > - MPICH/socket (set `--with-device=ch3:sock` at configure) > - A socket-based pingpong test without MPI. > > At this point, I could not think of any MPI-specific design for 2k/8k > messages. My guess is that it is related to your network connection. > > Min > > On 2018/06/24 11:09, Abu Naser wrote: >> >> Hello Min and Jeff, >> >> >> Here is my experiment results.ï¿½Default number of iterations in >> osu_latency for 0B ï¿½ 8KB is 10,000. With that setting I had run the >> osu_latency 100 times and found standard deviation 33 for 8KB message >> size. >> >> >> So later I have set the iteration to 50,000 and 100,000 for 1KB ï¿½ >> 16KB message size. Then run osu_latency for 100 times for each >> setting and take the average and standard deviation. >> >> >> *Msg Size in Bytes* >> >> >> >> *Avg time in us (50K iterations)* >> >> >> >> *Avg time in us (100k iterations)* >> >> >> >> *Standard deviation (50K iterations)* >> >> >> >> *Standard deviation (100K iterations)* >> >> 1k >> >> >> >> 85.10 >> >> >> >> 84.9 >> >> >> >> 0.55 >> >> >> >> 0.45 >> >> 2k >> >> >> >> 75.79 >> >> >> >> 74.63 >> >> >> >> 5.09 >> >> >> >> 4.44 >> >> 4k >> >> >> >> 273.80 >> >> >> >> 274.71 >> >> >> >> 4.18 >> >> >> >> 2.45 >> >> 8k >> >> >> >> 258.56 >> >> >> >> 249.83 >> >> >> >> 21.14 >> >> >> >> 28 >> >> 16k >> >> >> >> 281.31 >> >> >> >> 281.02 >> >> >> >> 3.22 >> >> >> >> 4.10 >> >> >> >> The standard deviation of 8K message is so high and that implies it >> actually not producing any consistent latency time. Looks likeï¿½that's >> the reason for 8K is taking less time than 4K. >> >> >> Meanwhile, 2K has standard deviation less than 5 but 1K message >> latency timing are more densely populated than 2K. So probably this >> is the explanation for 2K message less latency time. >> >> >> Thank you for your suggestions. >> >> >> >> >> Best Regards, >> >> Abu Naser >> >> ------------------------------------------------------------------------ >> *From:* Abu Naser >> *Sent:* Wednesday, June 20, 2018 1:48:53 PM >> *To:* discuss(a)mpich.org <mailto:[email protected]> >> *Subject:* Re: [mpich-discuss] osu_latency test: why 8KB takes less >> time than 4KB and 2KB takes less time than 1KB? >> >> Hello Min, >> >> >> Thanks for the clarification.ï¿½ I will do the experiment. >> >> >> Thanks. >> >> Best Regards, >> >> Abu Naser >> >> ------------------------------------------------------------------------ >> *From:* Min Si <msi(a)anl.gov> <mailto:[email protected]> >> *Sent:* Wednesday, June 20, 2018 1:39:30 PM >> *To:* discuss(a)mpich.org <mailto:[email protected]> >> *Subject:* Re: [mpich-discuss] osu_latency test: why 8KB takes less >> time than 4KB and 2KB takes less time than 1KB? >> Hi Abu, >> >> I think Jeff means that you should run your experiment with more >> iterations in order to get a stable results. >> - Increase the iteration of for loop in each execution (I think osu >> benchmark allows you to set it) >> - Run the experiments 10 or 100 times, and take the average and >> standard deviation. >> >> If you see a very small standard deviation (e.g., <=5%), then the >> trend is stable and you might not see such gaps. >> >> Best regards, >> Min >> On 2018/06/20 12:14, Abu Naser wrote: >>> >>> Hello Jeff, >>> >>> >>> Yes, I am using a switch and other machinesï¿½are also connected with >>> that switch. >>> >>> If I remove other machines and just use my two node with the switch, >>> then will it improve the performance by 200 ~ 400 iterations? >>> >>> Meanwhile I will give a try with a single dedicated cable. >>> >>> >>> Thank you. >>> >>> >>> Best Regards, >>> >>> Abu Naser >>> >>> ------------------------------------------------------------------------ >>> *From:* Jeff Hammond <jeff.science(a)gmail.com> >>> <mailto:[email protected]> >>> *Sent:* Wednesday, June 20, 2018 12:52:06 PM >>> *To:* MPICH >>> *Subject:* Re: [mpich-discuss] osu_latency test: why 8KB takes less >>> time than 4KB and 2KB takes less time than 1KB? >>> Is the ethernet connection a single dedicated cable between the two >>> machines or are you running through a switch that handles other >>> traffic? >>> >>> My best guess is that this is noise and that you may be able to >>> avoid it by running a very long time, e.g. 10000 iterations. >>> >>> Jeff >>> >>> On Wed, Jun 20, 2018 at 6:53 AM, Abu Naser <an16e(a)my.fsu.edu >>> <mailto:[email protected]>> wrote: >>> >>> >>> Good day to all, >>> >>> >>> I had run point to point osu_latency test in two nodes for 200 >>> times.ï¿½ Followings are the average time in microsecond for >>> various size of the messages - >>> >>> 1KBï¿½ï¿½ ï¿½84.8514 us >>> 2KBï¿½ï¿½ ï¿½73.52535 us >>> 4KBï¿½ï¿½ ï¿½272.55275 us >>> 8KBï¿½ï¿½ ï¿½234.86385 us >>> 16KBï¿½ï¿½ ï¿½288.88 us >>> 32KBï¿½ï¿½ ï¿½523.3725 us >>> 64KBï¿½ï¿½ ï¿½910.4025 us >>> >>> >>> From the above looks like, 2KB message has less latency than 1 >>> KB and 8KB has less latency than 4KB. >>> >>> I was looking for explanation of this behaviorï¿½ but did not get any. >>> >>> >>> 1. MPIR_CVAR_CH3_EAGER_MAX_MSG_SIZEis set to 128KB. So none of >>> the above message size is using Rendezvous protocol. Is >>> there any partition inside eager protocol (e.g. 0 - 512 >>> bytes, 1KB - 8KB, 16KB - 64KB)? If yes then what are the >>> boundaries for them? Can I log them with debug-event-logging? >>> >>> >>> Setup I am using: >>> >>> - two nodes has intel core i7, one with 16gb memory another one 8gb >>> >>> - mpich 3.2.1, configured and build to use nemesis tcp >>> >>> - 1gb Ethernet connection >>> >>> - NFS is using for sharing >>> >>> - osu_latency : uses MPI_Send and MPI_Recv >>> >>> - MPIR_CVAR_CH3_EAGER_MAX_MSG_SIZE= 131072 (128KB) >>> >>> >>> Can anyone help me on that? Thanks in advance. >>> >>> >>> >>> >>> Best Regards, >>> >>> Abu Naser >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> <https://lists.mpich.org/mailman/listinfo/discuss> >>> >>> >>> >>> >>> -- >>> Jeff Hammond >>> jeff.science(a)gmail.com <mailto:[email protected]> >>> http://jeffhammond.github.io/ >>> >>> >>> _______________________________________________ >>> discuss mailing listdiscuss(a)mpich.org <mailto:[email protected]> >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> _______________________________________________ >> discuss mailing listdiscuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] what's the right way to run mpiexec with sge?
by Shuwei Zhao 02 Jul '18

02 Jul '18

can anyone provide a simple example of launching mpich distribution application with SGE? and explain the integration of MPICH3 with lsf, sge....? Thanks, Shuwei On Mon, Jun 25, 2018 at 3:15 PM, Shuwei Zhao <shuweizhao1991(a)gmail.com> wrote: > Hi, > > I'm a bit confused by the correct way for interacting mpiexec with sge. > here is my original way of interacting mpich2 with sge successfully but > failed with mpich3. > > 1. using qsub submit 2 subjobs at the same time: > qsub -P bhosts - pe mt 1 ./myscript > qsub -P bhosts - pe mt 1 ./myscript > 2. Using lock file handle to a temporary file to decide master and slave. > 3. Both master and slave echo host name to a hosts file. > 4. run mpiexec -n 2 -f hosts ./mybinary(this one always fail) > 5. I tried to change command in 4th to mpiexec -launcher sge -n 2 -f > hosts ./mybinary but also fails. > > as mpiexec -help says, mpiexec will automatically interact with resource > manager like sge, lsf, I'm a bit confused about the right way to interact > with sge. > do you know what's wrong with my application? > > Thanks, > Shuwei > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB?
by Abu Naser 01 Jul '18

01 Jul '18

Hello Min, After compiling my mpich-3.2.1 with sock, while I was trying to run any program including osu benchmark or examples/cpi in two machines, I have received following error - Process 3 of 4 is on dhcp16194 Process 1 of 4 is on dhcp16194 Process 0 of 4 is on dhcp16198 Process 2 of 4 is on dhcp16198 Fatal error in PMPI_Bcast: Unknown error class, error stack: PMPI_Bcast(1600)............................: MPI_Bcast(buf=0x7ffc1808542c, count=1, MPI_INT, root=0, MPI_COMM_WORLD) failed MPIR_Bcast_impl(1452).......................: MPIR_Bcast(1476)............................: MPIR_Bcast_intra(1249)......................: MPIR_SMP_Bcast(1081)........................: MPIR_Bcast_binomial(285)....................: MPIC_Send(303)..............................: MPIC_Wait(226)..............................: MPIDI_CH3i_Progress_wait(242)...............: an error occurred while handling an event returned by MPIDU_Sock_Wait() MPIDI_CH3I_Progress_handle_sock_event(698)..: MPIDI_CH3_Sockconn_handle_connect_event(597): [ch3:sock] failed to connnect to remote process MPIDU_Socki_handle_connect(808).............: connection failure (set=0,sock=1,errno=111:Connection refused) MPIR_SMP_Bcast(1088)........................: MPIR_Bcast_binomial(310)....................: Failure during collective Fatal error in PMPI_Bcast: Other MPI error, error stack: PMPI_Bcast(1600)........: MPI_Bcast(buf=0x7ffd9eeebdac, count=1, MPI_INT, root=0, MPI_COMM_WORLD) failed MPIR_Bcast_impl(1452)...: MPIR_Bcast(1476)........: MPIR_Bcast_intra(1249)..: MPIR_SMP_Bcast(1088)....: MPIR_Bcast_binomial(310): Failure during collective I checked the mpich FAQ and also mpich discussion list. Based on that I have checked followings and found they are fine in my machines - - firewall is disabled in both machine - I can do password less ssh in both machine - /etc/hosts in both machine configured with ip address and name properly - I have updated the library path and used absolute path for mpiexec - Most importantly when I configured and build mpich with tcp, it works fine. I think I am missing something but could not figure out yet. Any help would be appreciated. Thank you. Best Regards, Abu Naser ________________________________ From: Min Si <msi(a)anl.gov> Sent: Tuesday, June 26, 2018 12:54:29 PM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Hi Abu, I think the results are stable enough. Perhaps you could also try the following tests, and see if similar trend exists: - MPICH/socket (set `--with-device=ch3:sock` at configure) - A socket-based pingpong test without MPI. At this point, I could not think of any MPI-specific design for 2k/8k messages. My guess is that it is related to your network connection. Min On 2018/06/24 11:09, Abu Naser wrote: Hello Min and Jeff, Here is my experiment results. Default number of iterations in osu_latency for 0B – 8KB is 10,000. With that setting I had run the osu_latency 100 times and found standard deviation 33 for 8KB message size. So later I have set the iteration to 50,000 and 100,000 for 1KB – 16KB message size. Then run osu_latency for 100 times for each setting and take the average and standard deviation. Msg Size in Bytes Avg time in us (50K iterations) Avg time in us (100k iterations) Standard deviation (50K iterations) Standard deviation (100K iterations) 1k 85.10 84.9 0.55 0.45 2k 75.79 74.63 5.09 4.44 4k 273.80 274.71 4.18 2.45 8k 258.56 249.83 21.14 28 16k 281.31 281.02 3.22 4.10 The standard deviation of 8K message is so high and that implies it actually not producing any consistent latency time. Looks like that's the reason for 8K is taking less time than 4K. Meanwhile, 2K has standard deviation less than 5 but 1K message latency timing are more densely populated than 2K. So probably this is the explanation for 2K message less latency time. Thank you for your suggestions. Best Regards, Abu Naser ________________________________ From: Abu Naser Sent: Wednesday, June 20, 2018 1:48:53 PM To: discuss(a)mpich.org<mailto:[email protected]> Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Hello Min, Thanks for the clarification. I will do the experiment. Thanks. Best Regards, Abu Naser ________________________________ From: Min Si <msi(a)anl.gov><mailto:[email protected]> Sent: Wednesday, June 20, 2018 1:39:30 PM To: discuss(a)mpich.org<mailto:[email protected]> Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Hi Abu, I think Jeff means that you should run your experiment with more iterations in order to get a stable results. - Increase the iteration of for loop in each execution (I think osu benchmark allows you to set it) - Run the experiments 10 or 100 times, and take the average and standard deviation. If you see a very small standard deviation (e.g., <=5%), then the trend is stable and you might not see such gaps. Best regards, Min On 2018/06/20 12:14, Abu Naser wrote: Hello Jeff, Yes, I am using a switch and other machines are also connected with that switch. If I remove other machines and just use my two node with the switch, then will it improve the performance by 200 ~ 400 iterations? Meanwhile I will give a try with a single dedicated cable. Thank you. Best Regards, Abu Naser ________________________________ From: Jeff Hammond <jeff.science(a)gmail.com><mailto:[email protected]> Sent: Wednesday, June 20, 2018 12:52:06 PM To: MPICH Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Is the ethernet connection a single dedicated cable between the two machines or are you running through a switch that handles other traffic? My best guess is that this is noise and that you may be able to avoid it by running a very long time, e.g. 10000 iterations. Jeff On Wed, Jun 20, 2018 at 6:53 AM, Abu Naser <an16e(a)my.fsu.edu<mailto:[email protected]>> wrote: Good day to all, I had run point to point osu_latency test in two nodes for 200 times. Followings are the average time in microsecond for various size of the messages - 1KB 84.8514 us 2KB 73.52535 us 4KB 272.55275 us 8KB 234.86385 us 16KB 288.88 us 32KB 523.3725 us 64KB 910.4025 us

1 0