June 2018 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB?
by Min Si 26 Jun '18

26 Jun '18

Hi Abu, I think the results are stable enough. Perhaps you could also try the following tests, and see if similar trend exists: - MPICH/socket (set `--with-device=ch3:sock` at configure) - A socket-based pingpong test without MPI. At this point, I could not think of any MPI-specific design for 2k/8k messages. My guess is that it is related to your network connection. Min On 2018/06/24 11:09, Abu Naser wrote: > > Hello Min and Jeff, > > > Here is my experiment results.ï¿½Default number of iterations in > osu_latency for 0B ï¿½ 8KB is 10,000. With that setting I had run the > osu_latency 100 times and found standard deviation 33 for 8KB message > size. > > > So later I have set the iteration to 50,000 and 100,000 for 1KB ï¿½ 16KB > message size. Then run osu_latency for 100 times for each setting and > take the average and standard deviation. > > > *Msg Size in Bytes* > > > > *Avg time in us (50K iterations)* > > > > *Avg time in us (100k iterations)* > > > > *Standard deviation (50K iterations)* > > > > *Standard deviation (100K iterations)* > > 1k > > > > 85.10 > > > > 84.9 > > > > 0.55 > > > > 0.45 > > 2k > > > > 75.79 > > > > 74.63 > > > > 5.09 > > > > 4.44 > > 4k > > > > 273.80 > > > > 274.71 > > > > 4.18 > > > > 2.45 > > 8k > > > > 258.56 > > > > 249.83 > > > > 21.14 > > > > 28 > > 16k > > > > 281.31 > > > > 281.02 > > > > 3.22 > > > > 4.10 > > > > The standard deviation of 8K message is so high and that implies it > actually not producing any consistent latency time. Looks likeï¿½that's > the reason for 8K is taking less time than 4K. > > > Meanwhile, 2K has standard deviation less than 5 but 1K message > latency timing are more densely populated than 2K. So probably this is > the explanation for 2K message less latency time. > > > Thank you for your suggestions. > > > > > Best Regards, > > Abu Naser > > ------------------------------------------------------------------------ > *From:* Abu Naser > *Sent:* Wednesday, June 20, 2018 1:48:53 PM > *To:* discuss(a)mpich.org > *Subject:* Re: [mpich-discuss] osu_latency test: why 8KB takes less > time than 4KB and 2KB takes less time than 1KB? > > Hello Min, > > > Thanks for the clarification.ï¿½ I will do the experiment. > > > Thanks. > > Best Regards, > > Abu Naser > > ------------------------------------------------------------------------ > *From:* Min Si <msi(a)anl.gov> > *Sent:* Wednesday, June 20, 2018 1:39:30 PM > *To:* discuss(a)mpich.org > *Subject:* Re: [mpich-discuss] osu_latency test: why 8KB takes less > time than 4KB and 2KB takes less time than 1KB? > Hi Abu, > > I think Jeff means that you should run your experiment with more > iterations in order to get a stable results. > - Increase the iteration of for loop in each execution (I think osu > benchmark allows you to set it) > - Run the experiments 10 or 100 times, and take the average and > standard deviation. > > If you see a very small standard deviation (e.g., <=5%), then the > trend is stable and you might not see such gaps. > > Best regards, > Min > On 2018/06/20 12:14, Abu Naser wrote: >> >> Hello Jeff, >> >> >> Yes, I am using a switch and other machinesï¿½are also connected with >> that switch. >> >> If I remove other machines and just use my two node with the switch, >> then will it improve the performance by 200 ~ 400 iterations? >> >> Meanwhile I will give a try with a single dedicated cable. >> >> >> Thank you. >> >> >> Best Regards, >> >> Abu Naser >> >> ------------------------------------------------------------------------ >> *From:* Jeff Hammond <jeff.science(a)gmail.com> >> <mailto:[email protected]> >> *Sent:* Wednesday, June 20, 2018 12:52:06 PM >> *To:* MPICH >> *Subject:* Re: [mpich-discuss] osu_latency test: why 8KB takes less >> time than 4KB and 2KB takes less time than 1KB? >> Is the ethernet connection a single dedicated cable between the two >> machines or are you running through a switch that handles other traffic? >> >> My best guess is that this is noise and that you may be able to avoid >> it by running a very long time, e.g. 10000 iterations. >> >> Jeff >> >> On Wed, Jun 20, 2018 at 6:53 AM, Abu Naser <an16e(a)my.fsu.edu >> <mailto:[email protected]>> wrote: >> >> >> Good day to all, >> >> >> I had run point to point osu_latency test in two nodes for 200 >> times.ï¿½ Followings are the average time in microsecond for >> various size of the messages - >> >> 1KBï¿½ï¿½ ï¿½84.8514 us >> 2KBï¿½ï¿½ ï¿½73.52535 us >> 4KBï¿½ï¿½ ï¿½272.55275 us >> 8KBï¿½ï¿½ ï¿½234.86385 us >> 16KBï¿½ï¿½ ï¿½288.88 us >> 32KBï¿½ï¿½ ï¿½523.3725 us >> 64KBï¿½ï¿½ ï¿½910.4025 us >> >> >> From the above looks like, 2KB message has less latency than 1 KB >> and 8KB has less latency than 4KB. >> >> I was looking for explanation of this behaviorï¿½ but did not get any. >> >> >> 1. MPIR_CVAR_CH3_EAGER_MAX_MSG_SIZEis set to 128KB. So none of >> the above message size is using Rendezvous protocol. Is there >> any partition inside eager protocol (e.g. 0 - 512 bytes, 1KB >> - 8KB, 16KB - 64KB)? If yes then what are the boundaries for >> them? Can I log them with debug-event-logging? >> >> >> Setup I am using: >> >> - two nodes has intel core i7, one with 16gb memory another one 8gb >> >> - mpich 3.2.1, configured and build to use nemesis tcp >> >> - 1gb Ethernet connection >> >> - NFS is using for sharing >> >> - osu_latency : uses MPI_Send and MPI_Recv >> >> - MPIR_CVAR_CH3_EAGER_MAX_MSG_SIZE= 131072 (128KB) >> >> >> Can anyone help me on that? Thanks in advance. >> >> >> >> >> Best Regards, >> >> Abu Naser >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> <https://lists.mpich.org/mailman/listinfo/discuss> >> >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com <mailto:[email protected]> >> http://jeffhammond.github.io/ >> >> >> _______________________________________________ >> discuss mailing listdiscuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Right level of thread safety
by Thakur, Rajeev 26 Jun '18

26 Jun '18

You need to follow the MPI specification and use MPI_THREAD_MULTIPLE. It is not a good idea to make assumptions about what an MPI implementation may do within an MPI function. Otherwise your code may not work with some MPI implementation on some platform. MPI_Test is not read only. Internally it may cause progress to occur on other pending communication operations. Rajeev > On Jun 26, 2018, at 4:26 AM, Florian Lindner <mailinglists(a)xgm.de> wrote: > > Hello, > > I have a multi-threaded MPI application. Actually, only two threads. The main thread does all kinds of MPI operations, > the secondary thread, launched by std::thread does only MPI_Test, no other MPI operations. > > What is the required level of thread safety? > > According to https://www.mpich.org/static/docs/v3.1/www3/MPI_Init_thread.html > > MPI_THREAD_SINGLE > Only one thread will execute. > > MPI_THREAD_FUNNELED > The process may be multi-threaded, but only the main thread will make MPI calls (all MPI calls are funneled to the > main thread). > > MPI_THREAD_SERIALIZED > The process may be multi-threaded, and multiple threads may make MPI calls, but only one at a time: MPI calls are > not made concurrently from two distinct threads (all MPI calls are serialized). > > MPI_THREAD_MULTIPLE > Multiple threads may call MPI, with no restrictions. > > > its THREAD_MULTIPLE, since I can not synchronize the main threads MPI calls and MPI_Test from the other thread. > > Is that perception correct? I am asking, because I think Test is a relatively small operations and probably read only > and therefore maybe a lower thread safety level suffices. > > Thanks, > Florian > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Right level of thread safety
by Florian Lindner 26 Jun '18

26 Jun '18

Hello, I have a multi-threaded MPI application. Actually, only two threads. The main thread does all kinds of MPI operations, the secondary thread, launched by std::thread does only MPI_Test, no other MPI operations. What is the required level of thread safety? According to https://www.mpich.org/static/docs/v3.1/www3/MPI_Init_thread.html MPI_THREAD_SINGLE Only one thread will execute. MPI_THREAD_FUNNELED The process may be multi-threaded, but only the main thread will make MPI calls (all MPI calls are funneled to the main thread). MPI_THREAD_SERIALIZED The process may be multi-threaded, and multiple threads may make MPI calls, but only one at a time: MPI calls are not made concurrently from two distinct threads (all MPI calls are serialized). MPI_THREAD_MULTIPLE Multiple threads may call MPI, with no restrictions. its THREAD_MULTIPLE, since I can not synchronize the main threads MPI calls and MPI_Test from the other thread. Is that perception correct? I am asking, because I think Test is a relatively small operations and probably read only and therefore maybe a lower thread safety level suffices. Thanks, Florian _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

what's the right way to run mpiexec with sge?
by Shuwei Zhao 25 Jun '18

25 Jun '18

Hi, I'm a bit confused by the correct way for interacting mpiexec with sge. here is my original way of interacting mpich2 with sge successfully but failed with mpich3. 1. using qsub submit 2 subjobs at the same time: qsub -P bhosts - pe mt 1 ./myscript qsub -P bhosts - pe mt 1 ./myscript 2. Using lock file handle to a temporary file to decide master and slave. 3. Both master and slave echo host name to a hosts file. 4. run mpiexec -n 2 -f hosts ./mybinary(this one always fail) 5. I tried to change command in 4th to mpiexec -launcher sge -n 2 -f hosts ./mybinary but also fails. as mpiexec -help says, mpiexec will automatically interact with resource manager like sge, lsf, I'm a bit confused about the right way to interact with sge. do you know what's wrong with my application? Thanks, Shuwei _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB?
by Abu Naser 24 Jun '18

24 Jun '18

Hello Min and Jeff, Here is my experiment results. Default number of iterations in osu_latency for 0B – 8KB is 10,000. With that setting I had run the osu_latency 100 times and found standard deviation 33 for 8KB message size. So later I have set the iteration to 50,000 and 100,000 for 1KB – 16KB message size. Then run osu_latency for 100 times for each setting and take the average and standard deviation. Msg Size in Bytes Avg time in us (50K iterations) Avg time in us (100k iterations) Standard deviation (50K iterations) Standard deviation (100K iterations) 1k 85.10 84.9 0.55 0.45 2k 75.79 74.63 5.09 4.44 4k 273.80 274.71 4.18 2.45 8k 258.56 249.83 21.14 28 16k 281.31 281.02 3.22 4.10 The standard deviation of 8K message is so high and that implies it actually not producing any consistent latency time. Looks like that's the reason for 8K is taking less time than 4K. Meanwhile, 2K has standard deviation less than 5 but 1K message latency timing are more densely populated than 2K. So probably this is the explanation for 2K message less latency time. Thank you for your suggestions. Best Regards, Abu Naser ________________________________ From: Abu Naser Sent: Wednesday, June 20, 2018 1:48:53 PM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Hello Min, Thanks for the clarification. I will do the experiment. Thanks. Best Regards, Abu Naser ________________________________ From: Min Si <msi(a)anl.gov> Sent: Wednesday, June 20, 2018 1:39:30 PM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Hi Abu, I think Jeff means that you should run your experiment with more iterations in order to get a stable results. - Increase the iteration of for loop in each execution (I think osu benchmark allows you to set it) - Run the experiments 10 or 100 times, and take the average and standard deviation. If you see a very small standard deviation (e.g., <=5%), then the trend is stable and you might not see such gaps. Best regards, Min On 2018/06/20 12:14, Abu Naser wrote: Hello Jeff, Yes, I am using a switch and other machines are also connected with that switch. If I remove other machines and just use my two node with the switch, then will it improve the performance by 200 ~ 400 iterations? Meanwhile I will give a try with a single dedicated cable. Thank you. Best Regards, Abu Naser ________________________________ From: Jeff Hammond <jeff.science(a)gmail.com><mailto:[email protected]> Sent: Wednesday, June 20, 2018 12:52:06 PM To: MPICH Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Is the ethernet connection a single dedicated cable between the two machines or are you running through a switch that handles other traffic? My best guess is that this is noise and that you may be able to avoid it by running a very long time, e.g. 10000 iterations. Jeff On Wed, Jun 20, 2018 at 6:53 AM, Abu Naser <an16e(a)my.fsu.edu<mailto:[email protected]>> wrote: Good day to all, I had run point to point osu_latency test in two nodes for 200 times. Followings are the average time in microsecond for various size of the messages - 1KB 84.8514 us 2KB 73.52535 us 4KB 272.55275 us 8KB 234.86385 us 16KB 288.88 us 32KB 523.3725 us 64KB 910.4025 us

1 0

Re: [mpich-discuss] Help with debugging
by Kochunas, Brendan 21 Jun '18

21 Jun '18

Hi All, I wanted to provide an update as I've been debugging this fairly intensely the past 2 days. First some more information about the communication pattern. The test case uses 9 processors on a 3x3 cartesian grid. Each communications with its neighbors (so there are 12 communicating pairs). There are 768 simultaneous (64 messages per communicating pair) for which Isend and Irecv have been called but no wait operations. The total data over all messages is about 128 MB, this works out to be about 10 MB per communicating pair. Individual messages range from 270 KB down to 13.5 KB. The data being passed are just vanilla Fortran arrays that are REAL(8)'s e.g. doubles. No crazy derived types. I double checked the tags and message sizes for all 768 messages and everything checks out. No duplicate tags, information on both processors in a communicating pair is correct. The machine I'm testing on is RHEL7 workstation with the packages for gcc-4.8.5 and mpich-3.2. I built and tested mpich-3.2, mpich-3.2.1, and mpich-3.3b2 with gcc 4.8.5 also. It has two 48-core Intel Skylake processors. So all messages are transmitted via shared memory. At the suggestion of Jeff Hammond I tried compiling MPI with different different devices in place of nemesis. For mpich-3.2 I tried sockets, and for mpich-3.3b2 I also tried libfabric-1.6.1 for ch4:ofi. The problem also does not occur for OpenMPI or MVAPICH. If I did not use nemesis, the problem did not occur for my test case. If I did use nemesis the problem was encountered for all versions of MPICH I tested. As undeterministic as these types of bugs might be, I do seem able to reproduce it reliably. However, if I add print statements to my code in certain, seemingly random places. So my question to the list is: does this sound like an issue with nemesis? Or is it possible there is still some application error? Since I was able to build MPICH with debug info, when I ran this case under valgrind I received a slightly more meaningful stack trace: ==99610== Invalid read of size 2 ==99610== at 0xB3B70BF: memcpy@@GLIBC_2.14 (vg_replace_strmem.c:1022) ==99610== by 0xE7F8FBB: MPID_nem_mpich_sendv_header (mpid_nem_inline.h:363) ==99610== by 0xE7F8FBB: MPIDI_CH3I_Shm_send_progress (ch3_progress.c:207) ==99610== by 0xE7FBD93: MPIDI_CH3I_Progress (ch3_progress.c:583) ==99610== by 0xE72754E: MPIR_Waitall_impl (waitall.c:217) ==99610== by 0xE727BB5: PMPI_Waitall (waitall.c:378) ==99610== by 0xDFE4DFD: mpi_waitall (waitallf.c:275) ==99610== Address 0x149ea1e4 is 180 bytes inside a block of size 216 free'd ==99610== at 0xB3B3CBD: free (vg_replace_malloc.c:530) ==99610== by 0xE83FB41: MPL_trfree (in /home/bkochuna/sw/lib/libmpi.so.12.1.0) ==99610== by 0xE7837D8: MPIU_trfree (trmem.c:37) ==99610== by 0xE80DC08: MPIU_SHMW_Hnd_free (mpiu_shm_wrappers.h:247) ==99610== by 0xE80DC08: MPIU_SHMW_Hnd_finalize (mpiu_shm_wrappers.h:443) ==99610== by 0xE80DC08: MPID_nem_delete_shm_region (mpid_nem_lmt_shm.c:963) ==99610== by 0xE80DC08: MPID_nem_lmt_shm_start_send (mpid_nem_lmt_shm.c:270) ==99610== by 0xE8097E0: pkt_CTS_handler (mpid_nem_lmt.c:352) ==99610== by 0xE7FBD0C: MPIDI_CH3I_Progress (ch3_progress.c:541) ==99610== by 0xE72754E: MPIR_Waitall_impl (waitall.c:217) ==99610== by 0xE727BB5: PMPI_Waitall (waitall.c:378) ==99610== by 0xDFE4DFD: mpi_waitall (waitallf.c:275) ==99610== Block was alloc'd at ==99610== at 0xB3B2BC3: malloc (vg_replace_malloc.c:299) ==99610== by 0xE83F38F: MPL_trmalloc (in /home/bkochuna/sw/lib/libmpi.so.12.1.0) ==99610== by 0xE783526: MPIU_trmalloc (trmem.c:29) ==99610== by 0xE80CF74: MPIU_SHMW_Ghnd_alloc (mpiu_shm_wrappers.h:188) ==99610== by 0xE80CF74: MPIU_SHMW_Seg_create_attach_templ (mpiu_shm_wrappers.h:622) ==99610== by 0xE80CF74: MPIU_SHMW_Seg_create_and_attach (mpiu_shm_wrappers.h:894) ==99610== by 0xE80CF74: MPID_nem_allocate_shm_region (mpid_nem_lmt_shm.c:885) ==99610== by 0xE80CF74: MPID_nem_lmt_shm_start_recv (mpid_nem_lmt_shm.c:180) ==99610== by 0xE8094AF: do_cts (mpid_nem_lmt.c:560) ==99610== by 0xE809EBE: pkt_RTS_handler (mpid_nem_lmt.c:276) ==99610== by 0xE7FB3C5: MPID_nem_handle_pkt (ch3_progress.c:760) ==99610== by 0xE7FBF8D: MPIDI_CH3I_Progress (ch3_progress.c:570) ==99610== by 0xE72754E: MPIR_Waitall_impl (waitall.c:217) ==99610== by 0xE727BB5: PMPI_Waitall (waitall.c:378) ==99610== by 0xDFE4DFD: mpi_waitall (waitallf.c:275) ==99610== There are basically 8 sets of messages like where the numbers of bytes changes slightly. (e.g. invalid read of 8 bytes instead of 2). There are no other valgrind errors or run time check errors. If I'm able to free up enough time over the next couple days, I hope to write a simple program that demonstrates the problem. Until then, thoughts and opinions are welcome! Thanks, -Brendan On 6/19/2018 10:21 PM, discuss-request(a)mpich.org wrote: > Date: Tue, 19 Jun 2018 20:52:52 -0400 > From: "Kochunas, Brendan" <bkochuna(a)umich.edu> > To: discuss(a)mpich.org > Subject: [mpich-discuss] Help with debugging > Message-ID: <1d6a3e4e-0145-400c-77f8-d98b8121877d(a)umich.edu> > Content-Type: text/plain; charset=utf-8; format=flowed > > We've been dealing with a particularly nasty bug in our code and are > having trouble debugging it. > > We are using RHEL7 with default GNU compiler colleciton (4.8.5) and yum > package for mpich (mpich-3.2-x86_64) > > The rest of this email describes briefly: > > * Our communication pattern > * what we observe > * what we've tested (and think we know) > > Description of communication pattern: > This is point-to-point communication using MPI_Isend, MPI_Irecv, and > MPI_Waitall. Its effectively a structured cartesian grid and we send > multiple messages per face that are probably large (message sizes are a > few KB). > > What we observe: > The problem we run into is that when we get to the MPI_Waitall we > receive something like: > > Fatal error in PMPI_Waitall: Other MPI error, error stack: > PMPI_Waitall(323)...............: MPI_Waitall(count=1152, > req_array=0x13493b90, status_array=0x1) failed > MPIR_Waitall_impl(166)..........: > MPIDI_CH3I_Progress(422)........: > MPID_nem_handle_pkt(642)........: > pkt_CTS_handler(321)............: > MPID_nem_lmt_shm_start_send(270): > MPID_nem_delete_shm_region(923).: > MPIU_SHMW_Seg_detach(707).......: unable to remove shared memory - > unlink No such file or directory > > Our test case is 9 processors (a 3x3 grid) > > We have tested several things that allow us to avoid the problem (but > not reliably) if: > * we send multiple messages (about 50x) that are smaller, instead of one > large message. > * Adding debug print statements near the MPI_Waitall > * Removing compiler optimizations > * Running under valgrind (e.g. mpirun -np <n> valgrind <exe>) > > * We have also observed this error on multiple machines (Cray XK7, > workstations with Intel Xeon's, SGI ICE-X with MVAPICH) > > The output from valgrind is: > > ==210233== Invalid read of size 8 > ==210233==??? at 0xB3B7060: memcpy@@GLIBC_2.14 (vg_replace_strmem.c:1022) > ==210233==??? by 0xE7F8FBB: MPID_nem_mpich_sendv_header > (mpid_nem_inline.h:363) > ==210233==??? by 0xE7F8FBB: MPIDI_CH3I_Shm_send_progress > (ch3_progress.c:207) > ==210233==??? by 0xE7FBD93: MPIDI_CH3I_Progress (ch3_progress.c:583) > ==210233==??? by 0xE7276B4: MPIR_Waitall_impl (waitall.c:164) > ==210233==??? by 0xE727BB5: PMPI_Waitall (waitall.c:378) > ==210233==??? by 0xDFE4DFD: mpi_waitall (waitallf.c:275) > ==210233==??? by 0x49FF27F: __updatebc_MOD_finish_updatebc_base > (UpdateBC.f90:361) > ==210233==? Address 0x142ec268 is 152 bytes inside a block of size 216 > free'd > ==210233==??? at 0xB3B3CBD: free (vg_replace_malloc.c:530) > ==210233==??? by 0xE83FB41: MPL_trfree (in > /home/bkochuna/sw/lib/libmpi.so.12.1.0) > ==210233==??? by 0xE7837D8: MPIU_trfree (trmem.c:37) > ==210233==??? by 0xE80DC08: MPIU_SHMW_Hnd_free (mpiu_shm_wrappers.h:247) > ==210233==??? by 0xE80DC08: MPIU_SHMW_Hnd_finalize (mpiu_shm_wrappers.h:443) > ==210233==??? by 0xE80DC08: MPID_nem_delete_shm_region > (mpid_nem_lmt_shm.c:963) > ==210233==??? by 0xE80DC08: MPID_nem_lmt_shm_start_send > (mpid_nem_lmt_shm.c:270) > ==210233==??? by 0xE8097E0: pkt_CTS_handler (mpid_nem_lmt.c:352) > ==210233==??? by 0xE7FB3C5: MPID_nem_handle_pkt (ch3_progress.c:760) > ==210233==??? by 0xE7FBF8D: MPIDI_CH3I_Progress (ch3_progress.c:570) > ==210233==??? by 0xE7276B4: MPIR_Waitall_impl (waitall.c:164) > ==210233==??? by 0xE727BB5: PMPI_Waitall (waitall.c:378) > ==210233==??? by 0xDFE4DFD: mpi_waitall (waitallf.c:275) > ==210233==??? by 0x49FF27F: __updatebc_MOD_finish_updatebc_base > (UpdateBC.f90:361) > ==210233==? Block was alloc'd at > ==210233==??? at 0xB3B2BC3: malloc (vg_replace_malloc.c:299) > ==210233==??? by 0xE83F38F: MPL_trmalloc (in > /home/bkochuna/sw/lib/libmpi.so.12.1.0) > ==210233==??? by 0xE783526: MPIU_trmalloc (trmem.c:29) > ==210233==??? by 0xE80CF74: MPIU_SHMW_Ghnd_alloc (mpiu_shm_wrappers.h:188) > ==210233==??? by 0xE80CF74: MPIU_SHMW_Seg_create_attach_templ > (mpiu_shm_wrappers.h:622) > ==210233==??? by 0xE80CF74: MPIU_SHMW_Seg_create_and_attach > (mpiu_shm_wrappers.h:894) > ==210233==??? by 0xE80CF74: MPID_nem_allocate_shm_region > (mpid_nem_lmt_shm.c:885) > ==210233==??? by 0xE80CF74: MPID_nem_lmt_shm_start_recv > (mpid_nem_lmt_shm.c:180) > ==210233==??? by 0xE8094AF: do_cts (mpid_nem_lmt.c:560) > ==210233==??? by 0xE809EBE: pkt_RTS_handler (mpid_nem_lmt.c:276) > ==210233==??? by 0xE7FB3C5: MPID_nem_handle_pkt (ch3_progress.c:760) > ==210233==??? by 0xE7FBF8D: MPIDI_CH3I_Progress (ch3_progress.c:570) > ==210233==??? by 0xE7276B4: MPIR_Waitall_impl (waitall.c:164) > ==210233==??? by 0xE727BB5: PMPI_Waitall (waitall.c:378) > ==210233==??? by 0xDFE4DFD: mpi_waitall (waitallf.c:275) > ==210233==??? by 0x49FF27F: __updatebc_MOD_finish_updatebc_base > (UpdateBC.f90:361) > > So my main questions to the list are: > * are there any known issues with mpich-3.2 and the shared memory > communication within nemesis that might affect non-blocking communication? > * Does this look more like an issue in our code or within MPICH? > * If it looks like an issue in our code what is the best way to debug > this... seems like our current efforts of basic print statements change > the behavior and prevent us from identifying it. I suspect there may be > some memory overwrite beyond some array bounds, but compiling in debug > with -fbounds-check does not expose the problem (and due to the nature > of the MPI interfaces, I would not expect this to help). We've stared at > the code for the calls to MPI_Isend and MPI_Irecv and everything looks > correct (e.g. we don't touch the buffers, the sizes match up, etc.) > * Are there limits to the number of simultaneous messages? (we are below > the maximum number of MPI requests) or message sizes (I don't believe > there is a limit here beyond integer overflow) > * We've tried changing from MPI_Waitall to individual MPI_Waits and > MPI_Tests, but the problem still happens with the individual waits. > > Any insights are welcome. > > Thanks, > -Brendan > > > > ------------------------------ > > Message: 4 > Date: Tue, 19 Jun 2018 19:20:35 -0700 > From: Jeff Hammond <jeff.science(a)gmail.com> > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] Help with debugging > Message-ID: <2226E1B6-75F7-43E2-A6C3-101860313B18(a)gmail.com> > Content-Type: text/plain; charset=utf-8 > > You can rule out MPICH shm bugs if the code runs fine with Open-MPI or by disabling shm in MPICH, eg by using ch3:sock. > > The fact that small messages work suggest rendezvous deadlock is the issue, which is often but not always an application error. > > It?s impossible to reason about the precise nature of the bug without source code. > > Jeff > > Sent from my iPhone > >> On Jun 19, 2018, at 5:52 PM, Kochunas, Brendan <bkochuna(a)umich.edu> wrote: >> >> We've been dealing with a particularly nasty bug in our code and are having trouble debugging it. >> >> We are using RHEL7 with default GNU compiler colleciton (4.8.5) and yum package for mpich (mpich-3.2-x86_64) >> >> The rest of this email describes briefly: >> >> * Our communication pattern >> * what we observe >> * what we've tested (and think we know) >> >> Description of communication pattern: >> This is point-to-point communication using MPI_Isend, MPI_Irecv, and MPI_Waitall. Its effectively a structured cartesian grid and we send multiple messages per face that are probably large (message sizes are a few KB). >> >> What we observe: >> The problem we run into is that when we get to the MPI_Waitall we receive something like: >> >> Fatal error in PMPI_Waitall: Other MPI error, error stack: >> PMPI_Waitall(323)...............: MPI_Waitall(count=1152, req_array=0x13493b90, status_array=0x1) failed >> MPIR_Waitall_impl(166)..........: >> MPIDI_CH3I_Progress(422)........: >> MPID_nem_handle_pkt(642)........: >> pkt_CTS_handler(321)............: >> MPID_nem_lmt_shm_start_send(270): >> MPID_nem_delete_shm_region(923).: >> MPIU_SHMW_Seg_detach(707).......: unable to remove shared memory - unlink No such file or directory >> >> Our test case is 9 processors (a 3x3 grid) >> >> We have tested several things that allow us to avoid the problem (but not reliably) if: >> * we send multiple messages (about 50x) that are smaller, instead of one large message. >> * Adding debug print statements near the MPI_Waitall >> * Removing compiler optimizations >> * Running under valgrind (e.g. mpirun -np <n> valgrind <exe>) >> >> * We have also observed this error on multiple machines (Cray XK7, workstations with Intel Xeon's, SGI ICE-X with MVAPICH) >> >> The output from valgrind is: >> >> ==210233== Invalid read of size 8 >> ==210233== at 0xB3B7060: memcpy@@GLIBC_2.14 (vg_replace_strmem.c:1022) >> ==210233== by 0xE7F8FBB: MPID_nem_mpich_sendv_header (mpid_nem_inline.h:363) >> ==210233== by 0xE7F8FBB: MPIDI_CH3I_Shm_send_progress (ch3_progress.c:207) >> ==210233== by 0xE7FBD93: MPIDI_CH3I_Progress (ch3_progress.c:583) >> ==210233== by 0xE7276B4: MPIR_Waitall_impl (waitall.c:164) >> ==210233== by 0xE727BB5: PMPI_Waitall (waitall.c:378) >> ==210233== by 0xDFE4DFD: mpi_waitall (waitallf.c:275) >> ==210233== by 0x49FF27F: __updatebc_MOD_finish_updatebc_base (UpdateBC.f90:361) >> ==210233== Address 0x142ec268 is 152 bytes inside a block of size 216 free'd >> ==210233== at 0xB3B3CBD: free (vg_replace_malloc.c:530) >> ==210233== by 0xE83FB41: MPL_trfree (in /home/bkochuna/sw/lib/libmpi.so.12.1.0) >> ==210233== by 0xE7837D8: MPIU_trfree (trmem.c:37) >> ==210233== by 0xE80DC08: MPIU_SHMW_Hnd_free (mpiu_shm_wrappers.h:247) >> ==210233== by 0xE80DC08: MPIU_SHMW_Hnd_finalize (mpiu_shm_wrappers.h:443) >> ==210233== by 0xE80DC08: MPID_nem_delete_shm_region (mpid_nem_lmt_shm.c:963) >> ==210233== by 0xE80DC08: MPID_nem_lmt_shm_start_send (mpid_nem_lmt_shm.c:270) >> ==210233== by 0xE8097E0: pkt_CTS_handler (mpid_nem_lmt.c:352) >> ==210233== by 0xE7FB3C5: MPID_nem_handle_pkt (ch3_progress.c:760) >> ==210233== by 0xE7FBF8D: MPIDI_CH3I_Progress (ch3_progress.c:570) >> ==210233== by 0xE7276B4: MPIR_Waitall_impl (waitall.c:164) >> ==210233== by 0xE727BB5: PMPI_Waitall (waitall.c:378) >> ==210233== by 0xDFE4DFD: mpi_waitall (waitallf.c:275) >> ==210233== by 0x49FF27F: __updatebc_MOD_finish_updatebc_base (UpdateBC.f90:361) >> ==210233== Block was alloc'd at >> ==210233== at 0xB3B2BC3: malloc (vg_replace_malloc.c:299) >> ==210233== by 0xE83F38F: MPL_trmalloc (in /home/bkochuna/sw/lib/libmpi.so.12.1.0) >> ==210233== by 0xE783526: MPIU_trmalloc (trmem.c:29) >> ==210233== by 0xE80CF74: MPIU_SHMW_Ghnd_alloc (mpiu_shm_wrappers.h:188) >> ==210233== by 0xE80CF74: MPIU_SHMW_Seg_create_attach_templ (mpiu_shm_wrappers.h:622) >> ==210233== by 0xE80CF74: MPIU_SHMW_Seg_create_and_attach (mpiu_shm_wrappers.h:894) >> ==210233== by 0xE80CF74: MPID_nem_allocate_shm_region (mpid_nem_lmt_shm.c:885) >> ==210233== by 0xE80CF74: MPID_nem_lmt_shm_start_recv (mpid_nem_lmt_shm.c:180) >> ==210233== by 0xE8094AF: do_cts (mpid_nem_lmt.c:560) >> ==210233== by 0xE809EBE: pkt_RTS_handler (mpid_nem_lmt.c:276) >> ==210233== by 0xE7FB3C5: MPID_nem_handle_pkt (ch3_progress.c:760) >> ==210233== by 0xE7FBF8D: MPIDI_CH3I_Progress (ch3_progress.c:570) >> ==210233== by 0xE7276B4: MPIR_Waitall_impl (waitall.c:164) >> ==210233== by 0xE727BB5: PMPI_Waitall (waitall.c:378) >> ==210233== by 0xDFE4DFD: mpi_waitall (waitallf.c:275) >> ==210233== by 0x49FF27F: __updatebc_MOD_finish_updatebc_base (UpdateBC.f90:361) >> >> So my main questions to the list are: >> * are there any known issues with mpich-3.2 and the shared memory communication within nemesis that might affect non-blocking communication? >> * Does this look more like an issue in our code or within MPICH? >> * If it looks like an issue in our code what is the best way to debug this... seems like our current efforts of basic print statements change the behavior and prevent us from identifying it. I suspect there may be some memory overwrite beyond some array bounds, but compiling in debug with -fbounds-check does not expose the problem (and due to the nature of the MPI interfaces, I would not expect this to help). We've stared at the code for the calls to MPI_Isend and MPI_Irecv and everything looks correct (e.g. we don't touch the buffers, the sizes match up, etc.) >> * Are there limits to the number of simultaneous messages? (we are below the maximum number of MPI requests) or message sizes (I don't believe there is a limit here beyond integer overflow) >> * We've tried changing from MPI_Waitall to individual MPI_Waits and MPI_Tests, but the problem still happens with the individual waits. >> >> Any insights are welcome. >> >> Thanks, >> -Brendan >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > discuss mailing list > discuss(a)mpich.org > https://lists.mpich.org/mailman/listinfo/discuss > > > ------------------------------ > > End of discuss Digest, Vol 68, Issue 12 > *************************************** _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB?
by Abu Naser 20 Jun '18

20 Jun '18

Hello Min, Thanks for the clarification. I will do the experiment. Thanks. Best Regards, Abu Naser ________________________________ From: Min Si <msi(a)anl.gov> Sent: Wednesday, June 20, 2018 1:39:30 PM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Hi Abu, I think Jeff means that you should run your experiment with more iterations in order to get a stable results. - Increase the iteration of for loop in each execution (I think osu benchmark allows you to set it) - Run the experiments 10 or 100 times, and take the average and standard deviation. If you see a very small standard deviation (e.g., <=5%), then the trend is stable and you might not see such gaps. Best regards, Min On 2018/06/20 12:14, Abu Naser wrote: Hello Jeff, Yes, I am using a switch and other machines are also connected with that switch. If I remove other machines and just use my two node with the switch, then will it improve the performance by 200 ~ 400 iterations? Meanwhile I will give a try with a single dedicated cable. Thank you. Best Regards, Abu Naser ________________________________ From: Jeff Hammond <jeff.science(a)gmail.com><mailto:[email protected]> Sent: Wednesday, June 20, 2018 12:52:06 PM To: MPICH Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Is the ethernet connection a single dedicated cable between the two machines or are you running through a switch that handles other traffic? My best guess is that this is noise and that you may be able to avoid it by running a very long time, e.g. 10000 iterations. Jeff On Wed, Jun 20, 2018 at 6:53 AM, Abu Naser <an16e(a)my.fsu.edu<mailto:[email protected]>> wrote: Good day to all, I had run point to point osu_latency test in two nodes for 200 times. Followings are the average time in microsecond for various size of the messages - 1KB 84.8514 us 2KB 73.52535 us 4KB 272.55275 us 8KB 234.86385 us 16KB 288.88 us 32KB 523.3725 us 64KB 910.4025 us

1 0

Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB?
by Min Si 20 Jun '18

20 Jun '18

Hi Abu, I think Jeff means that you should run your experiment with more iterations in order to get a stable results. - Increase the iteration of for loop in each execution (I think osu benchmark allows you to set it) - Run the experiments 10 or 100 times, and take the average and standard deviation. If you see a very small standard deviation (e.g., <=5%), then the trend is stable and you might not see such gaps. Best regards, Min On 2018/06/20 12:14, Abu Naser wrote: > > Hello Jeff, > > > Yes, I am using a switch and other machinesï¿½are also connected with > that switch. > > If I remove other machines and just use my two node with the switch, > then will it improve the performance by 200 ~ 400 iterations? > > Meanwhile I will give a try with a single dedicated cable. > > > Thank you. > > > Best Regards, > > Abu Naser > > ------------------------------------------------------------------------ > *From:* Jeff Hammond <jeff.science(a)gmail.com> > *Sent:* Wednesday, June 20, 2018 12:52:06 PM > *To:* MPICH > *Subject:* Re: [mpich-discuss] osu_latency test: why 8KB takes less > time than 4KB and 2KB takes less time than 1KB? > Is the ethernet connection a single dedicated cable between the two > machines or are you running through a switch that handles other traffic? > > My best guess is that this is noise and that you may be able to avoid > it by running a very long time, e.g. 10000 iterations. > > Jeff > > On Wed, Jun 20, 2018 at 6:53 AM, Abu Naser <an16e(a)my.fsu.edu > <mailto:[email protected]>> wrote: > > > Good day to all, > > > I had run point to point osu_latency test in two nodes for 200 > times.ï¿½ Followings are the average time in microsecond for various > size of the messages - > > 1KBï¿½ï¿½ ï¿½84.8514 us > 2KBï¿½ï¿½ ï¿½73.52535 us > 4KBï¿½ï¿½ ï¿½272.55275 us > 8KBï¿½ï¿½ ï¿½234.86385 us > 16KBï¿½ï¿½ ï¿½288.88 us > 32KBï¿½ï¿½ ï¿½523.3725 us > 64KBï¿½ï¿½ ï¿½910.4025 us > > > From the above looks like, 2KB message has less latency than 1 KB > and 8KB has less latency than 4KB. > > I was looking for explanation of this behaviorï¿½ but did not get any. > > > 1. MPIR_CVAR_CH3_EAGER_MAX_MSG_SIZEis set to 128KB. So none of > the above message size is using Rendezvous protocol. Is there > any partition inside eager protocol (e.g. 0 - 512 bytes, 1KB - > 8KB, 16KB - 64KB)? If yes then what are the boundaries for > them? Can I log them with debug-event-logging? > > > Setup I am using: > > - two nodes has intel core i7, one with 16gb memory another one 8gb > > - mpich 3.2.1, configured and build to use nemesis tcp > > - 1gb Ethernet connection > > - NFS is using for sharing > > - osu_latency : uses MPI_Send and MPI_Recv > > - MPIR_CVAR_CH3_EAGER_MAX_MSG_SIZE= 131072 (128KB) > > > Can anyone help me on that? Thanks in advance. > > > > > Best Regards, > > Abu Naser > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > <https://lists.mpich.org/mailman/listinfo/discuss> > > > > > -- > Jeff Hammond > jeff.science(a)gmail.com <mailto:[email protected]> > http://jeffhammond.github.io/ > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB?
by Abu Naser 20 Jun '18

20 Jun '18

Hello Jeff, Yes, I am using a switch and other machines are also connected with that switch. If I remove other machines and just use my two node with the switch, then will it improve the performance by 200 ~ 400 iterations? Meanwhile I will give a try with a single dedicated cable. Thank you. Best Regards, Abu Naser ________________________________ From: Jeff Hammond <jeff.science(a)gmail.com> Sent: Wednesday, June 20, 2018 12:52:06 PM To: MPICH Subject: Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB? Is the ethernet connection a single dedicated cable between the two machines or are you running through a switch that handles other traffic? My best guess is that this is noise and that you may be able to avoid it by running a very long time, e.g. 10000 iterations. Jeff On Wed, Jun 20, 2018 at 6:53 AM, Abu Naser <an16e(a)my.fsu.edu<mailto:[email protected]>> wrote: Good day to all, I had run point to point osu_latency test in two nodes for 200 times. Followings are the average time in microsecond for various size of the messages - 1KB 84.8514 us 2KB 73.52535 us 4KB 272.55275 us 8KB 234.86385 us 16KB 288.88 us 32KB 523.3725 us 64KB 910.4025 us

1 0

Re: [mpich-discuss] osu_latency test: why 8KB takes less time than 4KB and 2KB takes less time than 1KB?
by Jeff Hammond 20 Jun '18

20 Jun '18

Is the ethernet connection a single dedicated cable between the two machines or are you running through a switch that handles other traffic? My best guess is that this is noise and that you may be able to avoid it by running a very long time, e.g. 10000 iterations. Jeff On Wed, Jun 20, 2018 at 6:53 AM, Abu Naser <an16e(a)my.fsu.edu> wrote: > > Good day to all, > > > I had run point to point osu_latency test in two nodes for 200 times. > Followings are the average time in microsecond for various size of the > messages - > > 1KB 84.8514 us > 2KB 73.52535 us > 4KB 272.55275 us > 8KB 234.86385 us > 16KB 288.88 us > 32KB 523.3725 us > 64KB 910.4025 us > > > From the above looks like, 2KB message has less latency than 1 KB and 8KB > has less latency than 4KB. > > I was looking for explanation of this behavior but did not get any. > > > > 1. MPIR_CVAR_CH3_EAGER_MAX_MSG_SIZE is set to 128KB. So none of the > above message size is using Rendezvous protocol. Is there any partition > inside eager protocol (e.g. 0 - 512 bytes, 1KB - 8KB, 16KB - 64KB)? If yes > then what are the boundaries for them? Can I log them with > debug-event-logging? > > > Setup I am using: > > - two nodes has intel core i7, one with 16gb memory another one 8gb > > - mpich 3.2.1, configured and build to use nemesis tcp > > - 1gb Ethernet connection > > - NFS is using for sharing > > - osu_latency : uses MPI_Send and MPI_Recv > > - MPIR_CVAR_CH3_EAGER_MAX_MSG_SIZE= 131072 (128KB) > > > Can anyone help me on that? Thanks in advance. > > > > > Best Regards, > > Abu Naser > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0