November 2019 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Overlapping non-blocking collectives leads to deadlock
by Mark Davis 18 Nov '19

18 Nov '19

I realized something else relevant: I mentioned above that this deadlock occurs sometimes but not all of the time; I think I've narrowed down when it happens. Here's the above example with thread IDs annotated in: PROCESS 0 (root for ireduce and ibcast): // T0 is always the thread that calls MPI functions T0: MPI_Ireduce(..., &req) T0: MPI_Wait(&req); <-- blocking here ... T0: MPI_Ibcast(..., &req2); T0: MPI_Wait(&req2); PROCESS 1 (non-root for ireduce and ibcast): // T0 is the root for the reduce T0: MPI_Ireduce(..., &req) T0:MPI_Wait(&req); ... // T1 is the root for the bcast T1: MPI_Ibcast(..., &req2); T1: MPI_Wait(&req2); <-- blocking here Note that the non-root process has two different threads, T0 and T1, and T0 does the Ireduce and T1 does the bcast. I believe the T0 call to MPI_Ireduce is concurrent with the T1 call to MPI_Ibcast (both as non-roots). So, I believe the question is: is it legal in MPI to have two threads in a given MPI process call different non-blocking collectives (e.g., reduce and bcast) concurrently with MPI_THREAD_MULTIPLE enabled? Thank you On Mon, Nov 18, 2019 at 10:05 AM Mark Davis <markdavisinboston(a)gmail.com> wrote: > > Hello, I'm experimenting with non-blocking collectives using MPICH in > a multithreaded C++ program (with MPI_THREAD_MULTIPLE initialization). > > I'm currently doing a non-blocking reduce followed by a non-blocking > broadcast (I realize I can just use an allreduce but for my > experiment, I need to serialize these operations). I was able to > produce this bug with only two MPI processes. I see in gdb that the > root process is stuck trying to execute the MPI_Ireduce in cases where > the non-root process does the MPI_Ireduce and gets to the MPI_Ibcast > quickly. That is, process 0 (root) isn't able to complete the > MPI_Ireduce wait while process 1 is stuck in the MPI_Ibcast wait. > > PROCESS 0 (root for ireduce and ibcast): > MPI_Ireduce(..., &req) > MPI_Wait(&req); <-- blocking here > ... > MPI_Ibcast(..., &req2); > MPI_Wait(&req2); > > PROCESS 1 (non-root for ireduce and ibcast): > MPI_Ireduce(..., &req) > MPI_Wait(&req); > ... > MPI_Ibcast(..., &req2); > MPI_Wait(&req2); <-- blocking here > > Much of the time, the program deadlocks as shown above; sometimes this > works fine, though, perhaps due to subtle timing differences. I > mentioned above that this is a multithreaded program. I'm able to > produce the issue with two threads with two MPI procs. The other > threads are not calling MPI functions -- they are helping with other > computation. I've verified that I don't have any TSAN or ASAN errors > in this program. However, when I only have one thread per process, I > don't have this issue. I think there's a decent chance, though, that > this has to do with timing differences as opposed to changing anything > with the MPI calls. I have verified that only one thread per process > is calling the MPI routines in the multithreaded case. > > When I change the MPI_Ireduce to a blocking MPI_Reduce and I keep the > MPI_Ibcast non-blocking, the program runs fine. Only when BOTH > MPI_Ireduce and MPI_Ibcast happen serially do I see this deadlock > (again, some of the time). > > Unfortunately, this program is part of a very large system and it's > not straightforward to give a fully working example. So, I'm just > looking for any ideas anyone has for what sort of thing may be > happening, any information that may be helpful about how two > coincident non-blocking requests could interact with each other, etc. > > Also, if anyone has tips on how to debug this sort of thing in gdb > that would be helpful. For example, are there ways to introspect the > MPI_Request object, etc.? > > Thanks

1 0

Overlapping non-blocking collectives leads to deadlock
by Mark Davis 18 Nov '19

18 Nov '19

Hello, I'm experimenting with non-blocking collectives using MPICH in a multithreaded C++ program (with MPI_THREAD_MULTIPLE initialization). I'm currently doing a non-blocking reduce followed by a non-blocking broadcast (I realize I can just use an allreduce but for my experiment, I need to serialize these operations). I was able to produce this bug with only two MPI processes. I see in gdb that the root process is stuck trying to execute the MPI_Ireduce in cases where the non-root process does the MPI_Ireduce and gets to the MPI_Ibcast quickly. That is, process 0 (root) isn't able to complete the MPI_Ireduce wait while process 1 is stuck in the MPI_Ibcast wait. PROCESS 0 (root for ireduce and ibcast): MPI_Ireduce(..., &req) MPI_Wait(&req); <-- blocking here ... MPI_Ibcast(..., &req2); MPI_Wait(&req2); PROCESS 1 (non-root for ireduce and ibcast): MPI_Ireduce(..., &req) MPI_Wait(&req); ... MPI_Ibcast(..., &req2); MPI_Wait(&req2); <-- blocking here Much of the time, the program deadlocks as shown above; sometimes this works fine, though, perhaps due to subtle timing differences. I mentioned above that this is a multithreaded program. I'm able to produce the issue with two threads with two MPI procs. The other threads are not calling MPI functions -- they are helping with other computation. I've verified that I don't have any TSAN or ASAN errors in this program. However, when I only have one thread per process, I don't have this issue. I think there's a decent chance, though, that this has to do with timing differences as opposed to changing anything with the MPI calls. I have verified that only one thread per process is calling the MPI routines in the multithreaded case. When I change the MPI_Ireduce to a blocking MPI_Reduce and I keep the MPI_Ibcast non-blocking, the program runs fine. Only when BOTH MPI_Ireduce and MPI_Ibcast happen serially do I see this deadlock (again, some of the time). Unfortunately, this program is part of a very large system and it's not straightforward to give a fully working example. So, I'm just looking for any ideas anyone has for what sort of thing may be happening, any information that may be helpful about how two coincident non-blocking requests could interact with each other, etc. Also, if anyone has tips on how to debug this sort of thing in gdb that would be helpful. For example, are there ways to introspect the MPI_Request object, etc.? Thanks

1 0

Re: [mpich-discuss] Fortran 2008 MPI Interface Bug
by Thomas König 15 Nov '19

15 Nov '19

Hi Junchao, > Could you check if it is a gcc bug in handling type(*), dimension(..) Unfortunately, I don't know the code that well to be able to do it. If you could put together a simple, self-contained test case showing i there is a problem, you could submit that to gcc's bugzilla (or I could do that for you). I could also check a simplified test case against current gfortran trunk, which will not compile mpich 3.3.1 because it now flags argument mismatches as errors. (And yes, argument mismatches are real errors, which cause wrong code; see for example https://bugs.debian.org/944127 .)

1 0

Re: [mpich-discuss] Fortran 2008 MPI Interface Bug
by Zhang, Junchao 14 Nov '19

14 Nov '19

Hi, Thomas, I tried gcc-9-branch and I could build MPICH f08 with it. I could also reproduce the problem Pedro Ricardo Souza reported. Could you check if it is a gcc bug in handling type(*), dimension(..) --Junchao Zhang On Thu, Nov 14, 2019 at 6:27 AM Thomas König via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi, just a general remark on this. If you were using gfortran 9, that has seen a fair number of bug fixes in that field in the recent weeks, which will be included in the next release of gcc 9 and the upcoming release of gcc 10. If you can, you might give the current gcc 9 branch a spin to see if the problem goes away. Any additional bug reports would be wecome 😉 _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs
by Douglas Dommermuth 14 Nov '19

14 Nov '19

Hi Joachim, Thanks for your help and the help of others who contributed to my understanding. I did some tests with just one medium-sized mpi job: mpirun.mpich -n 128 mprog => 371.5s mpirun.mpich -bind-to numa -n 128 myprog => 366.4s mpirun.mpich -bind-to user:0,1,2,3,...,127 -n 128 myprog => 358.4s mpirun.mpich -bind-to rr -n 128 myprog => 355.1s mpirun.mpich -n 128 myprog with hyperthreading disabled => 365.6s mpirun.mpich -bind-to rr -n 128 myprog with hyperthreading disabled => 356.0s I also did one test with two medium-sized mpi jobs running concurrently: mpirun.mpich -bind-to user:0,1,2,3,4,...,127 -n 128 myprog and mpirun.mpich -bind-to user:128,129,130,131,...,255 -n 128 myprog =>690.8s rr binding cannot be used for multiple jobs because it pins it to the same threads. Manually entering the threads yields the best performance for multiple jobs, but it is awkward to enter 128 threads. It would be nice to have an option such as 0-127+1. Running two jobs concurrently is only slightly faster than running two jobs sequentially. I think that manually entering the threads is only useful for very long multiple jobs if all the available threads are used because I don't think it is desirable to bind to single threads over extended periods of time. Unlike manual binding, numa binding spreads the burden over all the threads evenly. Thanks again, Doug. ________________________________________ From: Joachim Protze <protze(a)itc.rwth-aachen.de> Sent: Thursday, November 14, 2019 1:33 AM To: discuss(a)mpich.org Cc: Douglas Dommermuth Subject: Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs Hi Doug, This is somewhat expected behavior. For many simulation codes you will see that they hit the memory bandwidth of a socket using just 30-70% of the cores. Adding more processes/threads doesn't improve the runtime in such case. In your first experiment, when you don't bind the processes to the same numa domain (socket), the 64 processes will spread across both sockets and use the memory bandwidth of both sockets. By binding one job to one socket, this job can use the same memory bandwidth as two jobs spread on both sockets. Based on the numbers you provided, my guess is that ~44 processes of your application are sufficient to fully utilize the memory bandwidth. Adding more processes to the socket will only slightly improve the runtime. So the question for optimization might be: is there a number of processes in 44..64, which allows best decomposition of your problem? - Joachim On 11/13/19 11:03 PM, Douglas Dommermuth via discuss wrote: > I tried the following: > > mpirun.mpich -bind-to user:0,1,2,3,4,5,6,7, .... ,57,58,59,60,61,62,63 -n 64 myprog. > > The timing was 76.03s. It was interesting to follow it on the system monitor. > ________________________________________ > From: Douglas Dommermuth via discuss <discuss(a)mpich.org> > Sent: Wednesday, November 13, 2019 1:37 PM > To: discuss(a)mpich.org > Cc: Douglas Dommermuth > Subject: Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs > > Hi Giuseppe, > > Thanks for your help. For NUMA Node 0, the hardware threads are 0-63 and 128-191. Thread 0 is paired with 128 on the same core, 1 is paired with 129, etc. For NUMA Node 1, the hardware threads are 64-127 and 192-255. How do I run the two jobs on completely separated hardware threads and numas? > > Thank you, Doug. > ________________________________________ > From: Congiu, Giuseppe <gcongiu(a)anl.gov> > Sent: Wednesday, November 13, 2019 12:57 PM > To: discuss(a)mpich.org > Cc: Douglas Dommermuth > Subject: Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs > > You can try with only two jobs running on completely separated hwthreads and numas and see if you get runtime comparable to the case in which you have only one job. Then you can add the other two jobs and see how much you increase the runtime. If it’s twice as long you are running jobs serially (as Joachim noted), otherwise there is some room for sharing among hwthreads. In any case, consider that by default MPICH uses shared memory for intranode communication, thus hwthreads/processes will be involved in copying data and leave little time to other hwthreads that want to run. On top of this memory bandwidth might also be an issue as you have all the cores possibly accessing memory at the same time. > >> On Nov 13, 2019, at 2:06 PM, Douglas Dommermuth via discuss <discuss(a)mpich.org> wrote: >> >> The timings for 4 runs are 163.1s, 159.9s, 160.9s, and 158.3s. >> >> The subdomain size is currently 32^3. I could see how 64^3 and 128^3 scale to boost the work relative to the communication. The solver is incompressible with multigrid, which makes it a bit tricky. Also, AMD recommends specific BIOS settings. However, the machine is currently Top 5 on Geekbench 4 for multicore results. >> ________________________________________ >> From: Congiu, Giuseppe <gcongiu(a)anl.gov> >> Sent: Wednesday, November 13, 2019 11:42 AM >> To: discuss(a)mpich.org >> Cc: Douglas Dommermuth >> Subject: Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs >> >> Is this an average runtime over multiple runs? >> >>> On Nov 13, 2019, at 1:40 PM, Douglas Dommermuth via discuss <discuss(a)mpich.org> wrote: >>> >>> Hi Giuseppe, >>> >>> It took 163.1s for this case: >>> >>> mpirun.mpich -bind-to user:0+1 -n 64 myprog & >>> mpirun.mpich -bind-to user:64+1 -n 64 myprog & >>> mpirun.mpich -bind-to user:128+1 -n 64 myprog & >>> mpirun.mpich -bind-to user:192+1 -n 64 myprog & >>> >>> Thanks, Doug. >>> ________________________________________ >>> From: Congiu, Giuseppe <gcongiu(a)anl.gov> >>> Sent: Wednesday, November 13, 2019 11:10 AM >>> To: discuss(a)mpich.org >>> Cc: Douglas Dommermuth >>> Subject: Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs >>> >>> Try binding all the ranks of a job to the same numa. See if something like this works better: >>> >>> mpirun.mpich -bind-to user:0+1 -n 64 myprog & >>> mpirun.mpich -bind-to user:64+1 -n 64 myprog & >>> mpirun.mpich -bind-to user:128+1 -n 64 myprog & >>> mpirun.mpich -bind-to user:192+1 -n 64 myprog & >>> >>> However this might not solve completely the problem as MPI processes can still move around across different cores in the numa. >>> >>> —Giuseppe >>> >>>> On Nov 13, 2019, at 1:00 PM, Douglas Dommermuth via discuss <discuss(a)mpich.org> wrote: >>>> >>>> Hi Giuseppe and Joachim, >>>> >>>> I will look into turning off hyperthreading and running two jobs with a corresponding change in the sizes of the jobs. Meanwhile, I ran the following case, which took 159.6s: >>>> >>>> mpirun.mpich -bind-to user:0+4 -n 64 myprog & >>>> mpirun.mpich -bind-to user:1+4 -n 64 myprog & >>>> mpirun.mpich -bind-to user:2+4 -n 64 myprog & >>>> mpirun.mpich -bind-to user:3+4 -n 64 myprog & >>>> >>>> Thank you, Doug. >>>> ________________________________________ >>>> From: Joachim Protze <protze(a)itc.rwth-aachen.de> >>>> Sent: Wednesday, November 13, 2019 9:56 AM >>>> To: discuss(a)mpich.org >>>> Cc: Douglas Dommermuth >>>> Subject: Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs >>>> >>>> Hi Doug, >>>> >>>> in general, using hyperthreads only improves execution time, if you do >>>> not utilize the core with a single process/thread. I.e., if you see >>>> ~100% cpu utilization per process in "top" for the single job execution, >>>> doubling the execution time from 2 to 4 mpi jobs sounds reasonable. >>>> If your application is mostly calculating (as an MPI application >>>> hopefully does), the two processes/threads running on the same core >>>> share the execution time of the core and will finally end up with double >>>> execution time. >>>> >>>> Depending on your application, additional processes/threads also might >>>> increase the pressure on the memory bus and therefore slow down the >>>> other application by making it wait for memory accesses. This might also >>>> explain the execution time increase from one to two mpi jobs. >>>> All this depends on the cpu/memory configuration of this machine. >>>> >>>> Best >>>> Joachim >>>> >>>> On 11/13/19 5:39 PM, Douglas Dommermuth via discuss wrote: >>>>> I am running Ubuntu 18.04.3 with MPICH 3.3~a2-4 and GFortran >>>>> 4:7.4.0-1ubuntu2.3 and GCC 4:7.4.0-1ubuntu2.3CC on dual AMD EPYC 7742 >>>>> processors with hyper threading enabled. My codes are written in MPI >>>>> and Fortran. The dual AMD processors have 128 cores and 256 threads. >>>>> I want to optimize the runtime for 4 mpi jobs running concurrently >>>>> with 64 threads each. Some timings are provided here: >>>>> >>>>> 1. One mpi job with mpiexec.hydra -n 64 myprog => 57.32s >>>>> 2. One mpi job with mpiexec.hydra -bind-to numa -n 64 => 50.52s >>>>> 3. Two mpi jobs with mpiexec.hydra -n 64 myprog => 99.77s >>>>> 4. Two mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 72.23s >>>>> 5. Four mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 159.2s >>>>> >>>>> The option "-bind-to numa" helps, but even so, running four mpi >>>>> jobs concurrently with 64 threads each is considerably slower than >>>>> running one mpi job with 64 threads. I can almost run four mpi jobs >>>>> sequentially and match the time for running four mpi jobs concurrently. >>>>> How can I improve on the result for running 4 mpi jobs concurrently? >>>>> Thanks, Doug. >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> >>>> >>>> >>>> -- >>>> Dipl.-Inf. Joachim Protze >>>> >>>> IT Center >>>> Group: High Performance Computing >>>> Division: Computational Science and Engineering >>>> RWTH Aachen University >>>> Seffenter Weg 23 >>>> D 52074 Aachen (Germany) >>>> Tel: +49 241 80- 24765 >>>> Fax: +49 241 80-624765 >>>> protze(a)itc.rwth-aachen.de >>>> www.itc.rwth-aachen.de >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpich-discuss] Fortran 2008 MPI Interface Bug
by Zhang, Junchao 14 Nov '19

14 Nov '19

Hi, Thomas, On my Macbook, I have gfortran-9.2 installed by homebrew. I could also install it from spack. `spack info gcc` shows Preferred version: 9.2.0 https://ftpmirror.gnu.org/gcc/gcc-9.2.0/gcc-9.2.0.tar.xz Safe versions: develop [svn] svn://gcc.gnu.org/svn/gcc/trunk<http://gcc.gnu.org/svn/gcc/trunk> Do I need to select develop to have bug fixes you mentioned? Thanks. --Junchao Zhang On Thu, Nov 14, 2019 at 6:27 AM Thomas König via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi, just a general remark on this. If you were using gfortran 9, that has seen a fair number of bug fixes in that field in the recent weeks, which will be included in the next release of gcc 9 and the upcoming release of gcc 10. If you can, you might give the current gcc 9 branch a spin to see if the problem goes away. Any additional bug reports would be wecome 😉 _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Fortran 2008 MPI Interface Bug
by Thomas König 14 Nov '19

14 Nov '19

Hi, just a general remark on this. If you were using gfortran 9, that has seen a fair number of bug fixes in that field in the recent weeks, which will be included in the next release of gcc 9 and the upcoming release of gcc 10. If you can, you might give the current gcc 9 branch a spin to see if the problem goes away. Any additional bug reports would be wecome 😉

1 0

Re: [mpich-discuss] Fortran 2008 MPI Interface Bug
by Pedro Ricardo Souza 14 Nov '19

14 Nov '19

Ok, I'll give it a try ... But Gcc versions alone doesn't explain the bug... As I said earlier: # The usage of your old Fortran interface "use mpi" does not show the index bug # The OpenMPI new "use mpi_f08" also doesn't have it ... All compiled from scratch with Gcc 9.1.1 It seems too easy to just blame the compiler for this behavior. I also encourage you to test the program I submitted with the versions is Gcc that you find to be stabler, to reproduce the error. It's much more likely that you just missed something in your new interface that OpenMPI got covered. -- Best regards, M.Sc.Eng. Pedro Ricardo C. Souza Em qui, 14 de nov de 2019 10:27, Thomas König via discuss <discuss(a)mpich.org> escreveu: > Hi, > > just a general remark on this. > > If you were using gfortran 9, that has seen a fair number of bug fixes in > that field in the recent weeks, which will be included in the next release > of gcc 9 and the upcoming release of gcc 10. > > If you can, you might give the current gcc 9 branch a spin to see if the > problem goes away. Any additional bug reports would be wecome 😉 > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs
by Joachim Protze 14 Nov '19

14 Nov '19

Hi Doug, This is somewhat expected behavior. For many simulation codes you will see that they hit the memory bandwidth of a socket using just 30-70% of the cores. Adding more processes/threads doesn't improve the runtime in such case. In your first experiment, when you don't bind the processes to the same numa domain (socket), the 64 processes will spread across both sockets and use the memory bandwidth of both sockets. By binding one job to one socket, this job can use the same memory bandwidth as two jobs spread on both sockets. Based on the numbers you provided, my guess is that ~44 processes of your application are sufficient to fully utilize the memory bandwidth. Adding more processes to the socket will only slightly improve the runtime. So the question for optimization might be: is there a number of processes in 44..64, which allows best decomposition of your problem? - Joachim On 11/13/19 11:03 PM, Douglas Dommermuth via discuss wrote: > I tried the following: > > mpirun.mpich -bind-to user:0,1,2,3,4,5,6,7, .... ,57,58,59,60,61,62,63 -n 64 myprog. > > The timing was 76.03s. It was interesting to follow it on the system monitor. > ________________________________________ > From: Douglas Dommermuth via discuss <discuss(a)mpich.org> > Sent: Wednesday, November 13, 2019 1:37 PM > To: discuss(a)mpich.org > Cc: Douglas Dommermuth > Subject: Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs > > Hi Giuseppe, > > Thanks for your help. For NUMA Node 0, the hardware threads are 0-63 and 128-191. Thread 0 is paired with 128 on the same core, 1 is paired with 129, etc. For NUMA Node 1, the hardware threads are 64-127 and 192-255. How do I run the two jobs on completely separated hardware threads and numas? > > Thank you, Doug. > ________________________________________ > From: Congiu, Giuseppe <gcongiu(a)anl.gov> > Sent: Wednesday, November 13, 2019 12:57 PM > To: discuss(a)mpich.org > Cc: Douglas Dommermuth > Subject: Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs > > You can try with only two jobs running on completely separated hwthreads and numas and see if you get runtime comparable to the case in which you have only one job. Then you can add the other two jobs and see how much you increase the runtime. If it’s twice as long you are running jobs serially (as Joachim noted), otherwise there is some room for sharing among hwthreads. In any case, consider that by default MPICH uses shared memory for intranode communication, thus hwthreads/processes will be involved in copying data and leave little time to other hwthreads that want to run. On top of this memory bandwidth might also be an issue as you have all the cores possibly accessing memory at the same time. > >> On Nov 13, 2019, at 2:06 PM, Douglas Dommermuth via discuss <discuss(a)mpich.org> wrote: >> >> The timings for 4 runs are 163.1s, 159.9s, 160.9s, and 158.3s. >> >> The subdomain size is currently 32^3. I could see how 64^3 and 128^3 scale to boost the work relative to the communication. The solver is incompressible with multigrid, which makes it a bit tricky. Also, AMD recommends specific BIOS settings. However, the machine is currently Top 5 on Geekbench 4 for multicore results. >> ________________________________________ >> From: Congiu, Giuseppe <gcongiu(a)anl.gov> >> Sent: Wednesday, November 13, 2019 11:42 AM >> To: discuss(a)mpich.org >> Cc: Douglas Dommermuth >> Subject: Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs >> >> Is this an average runtime over multiple runs? >> >>> On Nov 13, 2019, at 1:40 PM, Douglas Dommermuth via discuss <discuss(a)mpich.org> wrote: >>> >>> Hi Giuseppe, >>> >>> It took 163.1s for this case: >>> >>> mpirun.mpich -bind-to user:0+1 -n 64 myprog & >>> mpirun.mpich -bind-to user:64+1 -n 64 myprog & >>> mpirun.mpich -bind-to user:128+1 -n 64 myprog & >>> mpirun.mpich -bind-to user:192+1 -n 64 myprog & >>> >>> Thanks, Doug. >>> ________________________________________ >>> From: Congiu, Giuseppe <gcongiu(a)anl.gov> >>> Sent: Wednesday, November 13, 2019 11:10 AM >>> To: discuss(a)mpich.org >>> Cc: Douglas Dommermuth >>> Subject: Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs >>> >>> Try binding all the ranks of a job to the same numa. See if something like this works better: >>> >>> mpirun.mpich -bind-to user:0+1 -n 64 myprog & >>> mpirun.mpich -bind-to user:64+1 -n 64 myprog & >>> mpirun.mpich -bind-to user:128+1 -n 64 myprog & >>> mpirun.mpich -bind-to user:192+1 -n 64 myprog & >>> >>> However this might not solve completely the problem as MPI processes can still move around across different cores in the numa. >>> >>> —Giuseppe >>> >>>> On Nov 13, 2019, at 1:00 PM, Douglas Dommermuth via discuss <discuss(a)mpich.org> wrote: >>>> >>>> Hi Giuseppe and Joachim, >>>> >>>> I will look into turning off hyperthreading and running two jobs with a corresponding change in the sizes of the jobs. Meanwhile, I ran the following case, which took 159.6s: >>>> >>>> mpirun.mpich -bind-to user:0+4 -n 64 myprog & >>>> mpirun.mpich -bind-to user:1+4 -n 64 myprog & >>>> mpirun.mpich -bind-to user:2+4 -n 64 myprog & >>>> mpirun.mpich -bind-to user:3+4 -n 64 myprog & >>>> >>>> Thank you, Doug. >>>> ________________________________________ >>>> From: Joachim Protze <protze(a)itc.rwth-aachen.de> >>>> Sent: Wednesday, November 13, 2019 9:56 AM >>>> To: discuss(a)mpich.org >>>> Cc: Douglas Dommermuth >>>> Subject: Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs >>>> >>>> Hi Doug, >>>> >>>> in general, using hyperthreads only improves execution time, if you do >>>> not utilize the core with a single process/thread. I.e., if you see >>>> ~100% cpu utilization per process in "top" for the single job execution, >>>> doubling the execution time from 2 to 4 mpi jobs sounds reasonable. >>>> If your application is mostly calculating (as an MPI application >>>> hopefully does), the two processes/threads running on the same core >>>> share the execution time of the core and will finally end up with double >>>> execution time. >>>> >>>> Depending on your application, additional processes/threads also might >>>> increase the pressure on the memory bus and therefore slow down the >>>> other application by making it wait for memory accesses. This might also >>>> explain the execution time increase from one to two mpi jobs. >>>> All this depends on the cpu/memory configuration of this machine. >>>> >>>> Best >>>> Joachim >>>> >>>> On 11/13/19 5:39 PM, Douglas Dommermuth via discuss wrote: >>>>> I am running Ubuntu 18.04.3 with MPICH 3.3~a2-4 and GFortran >>>>> 4:7.4.0-1ubuntu2.3 and GCC 4:7.4.0-1ubuntu2.3CC on dual AMD EPYC 7742 >>>>> processors with hyper threading enabled. My codes are written in MPI >>>>> and Fortran. The dual AMD processors have 128 cores and 256 threads. >>>>> I want to optimize the runtime for 4 mpi jobs running concurrently >>>>> with 64 threads each. Some timings are provided here: >>>>> >>>>> 1. One mpi job with mpiexec.hydra -n 64 myprog => 57.32s >>>>> 2. One mpi job with mpiexec.hydra -bind-to numa -n 64 => 50.52s >>>>> 3. Two mpi jobs with mpiexec.hydra -n 64 myprog => 99.77s >>>>> 4. Two mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 72.23s >>>>> 5. Four mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 159.2s >>>>> >>>>> The option "-bind-to numa" helps, but even so, running four mpi >>>>> jobs concurrently with 64 threads each is considerably slower than >>>>> running one mpi job with 64 threads. I can almost run four mpi jobs >>>>> sequentially and match the time for running four mpi jobs concurrently. >>>>> How can I improve on the result for running 4 mpi jobs concurrently? >>>>> Thanks, Doug. >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> >>>> >>>> >>>> -- >>>> Dipl.-Inf. Joachim Protze >>>> >>>> IT Center >>>> Group: High Performance Computing >>>> Division: Computational Science and Engineering >>>> RWTH Aachen University >>>> Seffenter Weg 23 >>>> D 52074 Aachen (Germany) >>>> Tel: +49 241 80- 24765 >>>> Fax: +49 241 80-624765 >>>> protze(a)itc.rwth-aachen.de >>>> www.itc.rwth-aachen.de >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpich-discuss] Fortran 2008 MPI Interface Bug
by Zhou, Hui 13 Nov '19

13 Nov '19

Hi Pedro Ricardo, Thanks for reporting the issue. I have opened a GitHub issue, https://github.com/pmodels/mpich/issues/4170, for better tracking. `mpi_f08` tries to use the iso-c-binding as much as it can and it has not been receiving as much test as `use mpi`. One of the reason is the compiler support has been sporadic. May I ask which compiler (include version) were you using? Anyway, we’ll look into it and hopefully get it fixed shortly. — Hui Zhou Principal Software Development Specialist MCS | ANL On Nov 13, 2019, at 1:59 PM, Pedro Ricardo Souza via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi there, My name is Pedro Ricardo and I'm currently a researcher at a Fluid Dynamics Laboratory in Brazil. We have a CFD software with 12 continuous years of development mostly in Fortran, and we've been using Mpich and Openmpi since the very beginning. Recently we started to refactor the code and write it using Fortran2008 standard and this includes the usage of the the newest MPI module for Fortran "mpi_f08". So far no errors were present in Openmpi implementations and everything worked like a charm, but when we started compiling with Mpich lots of errors happened during execution. I was able so isolate and reproduce the error in a simpler code. Whenever "use mpi_f08" is present, the arrays passed to a MPI subroutine return with C-like indexes. If I allocate an standard array in Fortran (that starts with index 1), after the call of some MPI routine this array come back starting with index 0. And you can imagine that all access to this array will be wrong after that. It is a funny behavior because if you put "use mpi" instead of "use mpi_f08", the indexes do not change. The test is simple, allocate and array, print it's bound indexes, calls a MPI_Reduce routine and print the array bounds again. I've also tested other routines like MPI_ISend and the error also happens. program check_bounds !Check Works ! use mpi !Check Fails use mpi_f08 implicit none integer:: mpi_ierr, proc_id, n_proc real, allocatable, dimension(:):: t1, t1_0 call MPI_Init(mpi_ierr) call MPI_Comm_rank(MPI_COMM_WORLD, proc_id, mpi_ierr) call MPI_Comm_size(MPI_COMM_WORLD, n_proc, mpi_ierr) allocate(t1(5), t1_0(5)) t1 = 1.0; t1_0 = 1.0 if (proc_id==0) then write(*,*)'Array Configuration Before:' write(*,'(a6,2(1x,i2))')'Low:',lbound(t1),lbound(t1_0) write(*,'(a6,2(1x,i2))')'Upper:',ubound(t1),ubound(t1_0) end if CALL MPI_REDUCE(t1, t1_0, 5, MPI_REAL, MPI_SUM, 0, MPI_COMM_WORLD, mpi_ierr) if (proc_id==0) then write(*,*)'Array Configuration After:' write(*,'(a6,2(1x,i2))')'Low:',lbound(t1),lbound(t1_0) write(*,'(a6,2(1x,i2))')'Upper:',ubound(t1),ubound(t1_0) end if deallocate(t1, t1_0) call MPI_Finalize(mpi_ierr) end program check_bounds Thanks for the attention,

1 0