- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs
by Douglas Dommermuth 13 Nov '19

13 Nov '19

The timings for 4 runs are 163.1s, 159.9s, 160.9s, and 158.3s. The subdomain size is currently 32^3. I could see how 64^3 and 128^3 scale to boost the work relative to the communication. The solver is incompressible with multigrid, which makes it a bit tricky. Also, AMD recommends specific BIOS settings. However, the machine is currently Top 5 on Geekbench 4 for multicore results. ________________________________________ From: Congiu, Giuseppe <gcongiu(a)anl.gov> Sent: Wednesday, November 13, 2019 11:42 AM To: discuss(a)mpich.org Cc: Douglas Dommermuth Subject: Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs Is this an average runtime over multiple runs? > On Nov 13, 2019, at 1:40 PM, Douglas Dommermuth via discuss <discuss(a)mpich.org> wrote: > > Hi Giuseppe, > > It took 163.1s for this case: > > mpirun.mpich -bind-to user:0+1 -n 64 myprog & > mpirun.mpich -bind-to user:64+1 -n 64 myprog & > mpirun.mpich -bind-to user:128+1 -n 64 myprog & > mpirun.mpich -bind-to user:192+1 -n 64 myprog & > > Thanks, Doug. > ________________________________________ > From: Congiu, Giuseppe <gcongiu(a)anl.gov> > Sent: Wednesday, November 13, 2019 11:10 AM > To: discuss(a)mpich.org > Cc: Douglas Dommermuth > Subject: Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs > > Try binding all the ranks of a job to the same numa. See if something like this works better: > > mpirun.mpich -bind-to user:0+1 -n 64 myprog & > mpirun.mpich -bind-to user:64+1 -n 64 myprog & > mpirun.mpich -bind-to user:128+1 -n 64 myprog & > mpirun.mpich -bind-to user:192+1 -n 64 myprog & > > However this might not solve completely the problem as MPI processes can still move around across different cores in the numa. > > —Giuseppe > >> On Nov 13, 2019, at 1:00 PM, Douglas Dommermuth via discuss <discuss(a)mpich.org> wrote: >> >> Hi Giuseppe and Joachim, >> >> I will look into turning off hyperthreading and running two jobs with a corresponding change in the sizes of the jobs. Meanwhile, I ran the following case, which took 159.6s: >> >> mpirun.mpich -bind-to user:0+4 -n 64 myprog & >> mpirun.mpich -bind-to user:1+4 -n 64 myprog & >> mpirun.mpich -bind-to user:2+4 -n 64 myprog & >> mpirun.mpich -bind-to user:3+4 -n 64 myprog & >> >> Thank you, Doug. >> ________________________________________ >> From: Joachim Protze <protze(a)itc.rwth-aachen.de> >> Sent: Wednesday, November 13, 2019 9:56 AM >> To: discuss(a)mpich.org >> Cc: Douglas Dommermuth >> Subject: Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs >> >> Hi Doug, >> >> in general, using hyperthreads only improves execution time, if you do >> not utilize the core with a single process/thread. I.e., if you see >> ~100% cpu utilization per process in "top" for the single job execution, >> doubling the execution time from 2 to 4 mpi jobs sounds reasonable. >> If your application is mostly calculating (as an MPI application >> hopefully does), the two processes/threads running on the same core >> share the execution time of the core and will finally end up with double >> execution time. >> >> Depending on your application, additional processes/threads also might >> increase the pressure on the memory bus and therefore slow down the >> other application by making it wait for memory accesses. This might also >> explain the execution time increase from one to two mpi jobs. >> All this depends on the cpu/memory configuration of this machine. >> >> Best >> Joachim >> >> On 11/13/19 5:39 PM, Douglas Dommermuth via discuss wrote: >>> I am running Ubuntu 18.04.3 with MPICH 3.3~a2-4 and GFortran >>> 4:7.4.0-1ubuntu2.3 and GCC 4:7.4.0-1ubuntu2.3CC on dual AMD EPYC 7742 >>> processors with hyper threading enabled. My codes are written in MPI >>> and Fortran. The dual AMD processors have 128 cores and 256 threads. >>> I want to optimize the runtime for 4 mpi jobs running concurrently >>> with 64 threads each. Some timings are provided here: >>> >>> 1. One mpi job with mpiexec.hydra -n 64 myprog => 57.32s >>> 2. One mpi job with mpiexec.hydra -bind-to numa -n 64 => 50.52s >>> 3. Two mpi jobs with mpiexec.hydra -n 64 myprog => 99.77s >>> 4. Two mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 72.23s >>> 5. Four mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 159.2s >>> >>> The option "-bind-to numa" helps, but even so, running four mpi >>> jobs concurrently with 64 threads each is considerably slower than >>> running one mpi job with 64 threads. I can almost run four mpi jobs >>> sequentially and match the time for running four mpi jobs concurrently. >>> How can I improve on the result for running 4 mpi jobs concurrently? >>> Thanks, Doug. >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> >> -- >> Dipl.-Inf. Joachim Protze >> >> IT Center >> Group: High Performance Computing >> Division: Computational Science and Engineering >> RWTH Aachen University >> Seffenter Weg 23 >> D 52074 Aachen (Germany) >> Tel: +49 241 80- 24765 >> Fax: +49 241 80-624765 >> protze(a)itc.rwth-aachen.de >> www.itc.rwth-aachen.de >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Fortran 2008 MPI Interface Bug
by Pedro Ricardo Souza 13 Nov '19

13 Nov '19

Hi there, My name is Pedro Ricardo and I'm currently a researcher at a Fluid Dynamics Laboratory in Brazil. We have a CFD software with 12 continuous years of development mostly in Fortran, and we've been using Mpich and Openmpi since the very beginning. Recently we started to refactor the code and write it using Fortran2008 standard and this includes the usage of the the newest MPI module for Fortran "*mpi_f08*". So far no errors were present in Openmpi implementations and everything worked like a charm, but when we started compiling with Mpich lots of errors happened during execution. I was able so isolate and reproduce the error in a simpler code. Whenever "*use mpi_f08*" is present, the arrays passed to a MPI subroutine return with *C*-like indexes. If I allocate an standard array in Fortran (that starts with index *1*), after the call of some MPI routine this array come back starting with index *0*. And you can imagine that all access to this array will be wrong after that. It is a funny behavior because if you put "*use mpi*" instead of "*use mpi_f08*", the indexes do not change. The test is simple, allocate and array, print it's bound indexes, calls a *MPI_Reduce* routine and print the array bounds again. I've also tested other routines like *MPI_ISend* and the error also happens. program check_bounds > !Check Works > ! use mpi > !Check Fails > use mpi_f08 > > implicit none > integer:: mpi_ierr, proc_id, n_proc > real, allocatable, dimension(:):: t1, t1_0 > call MPI_Init(mpi_ierr) > call MPI_Comm_rank(MPI_COMM_WORLD, proc_id, mpi_ierr) > call MPI_Comm_size(MPI_COMM_WORLD, n_proc, mpi_ierr) > allocate(t1(5), t1_0(5)) > t1 = 1.0; t1_0 = 1.0 > if (proc_id==0) then > write(*,*)'Array Configuration Before:' > write(*,'(a6,2(1x,i2))')'Low:',lbound(t1),lbound(t1_0) > write(*,'(a6,2(1x,i2))')'Upper:',ubound(t1),ubound(t1_0) > end if > CALL MPI_REDUCE(t1, t1_0, 5, MPI_REAL, MPI_SUM, 0, MPI_COMM_WORLD, > mpi_ierr) > if (proc_id==0) then > write(*,*)'Array Configuration After:' > write(*,'(a6,2(1x,i2))')'Low:',lbound(t1),lbound(t1_0) > write(*,'(a6,2(1x,i2))')'Upper:',ubound(t1),ubound(t1_0) > end if > deallocate(t1, t1_0) > call MPI_Finalize(mpi_ierr) > end program check_bounds Thanks for the attention, -- Best regards, M.Sc.Eng. Pedro Ricardo C. Souza

1 0

Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs
by Congiu, Giuseppe 13 Nov '19

13 Nov '19

Is this an average runtime over multiple runs? > On Nov 13, 2019, at 1:40 PM, Douglas Dommermuth via discuss <discuss(a)mpich.org> wrote: > > Hi Giuseppe, > > It took 163.1s for this case: > > mpirun.mpich -bind-to user:0+1 -n 64 myprog & > mpirun.mpich -bind-to user:64+1 -n 64 myprog & > mpirun.mpich -bind-to user:128+1 -n 64 myprog & > mpirun.mpich -bind-to user:192+1 -n 64 myprog & > > Thanks, Doug. > ________________________________________ > From: Congiu, Giuseppe <gcongiu(a)anl.gov> > Sent: Wednesday, November 13, 2019 11:10 AM > To: discuss(a)mpich.org > Cc: Douglas Dommermuth > Subject: Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs > > Try binding all the ranks of a job to the same numa. See if something like this works better: > > mpirun.mpich -bind-to user:0+1 -n 64 myprog & > mpirun.mpich -bind-to user:64+1 -n 64 myprog & > mpirun.mpich -bind-to user:128+1 -n 64 myprog & > mpirun.mpich -bind-to user:192+1 -n 64 myprog & > > However this might not solve completely the problem as MPI processes can still move around across different cores in the numa. > > —Giuseppe > >> On Nov 13, 2019, at 1:00 PM, Douglas Dommermuth via discuss <discuss(a)mpich.org> wrote: >> >> Hi Giuseppe and Joachim, >> >> I will look into turning off hyperthreading and running two jobs with a corresponding change in the sizes of the jobs. Meanwhile, I ran the following case, which took 159.6s: >> >> mpirun.mpich -bind-to user:0+4 -n 64 myprog & >> mpirun.mpich -bind-to user:1+4 -n 64 myprog & >> mpirun.mpich -bind-to user:2+4 -n 64 myprog & >> mpirun.mpich -bind-to user:3+4 -n 64 myprog & >> >> Thank you, Doug. >> ________________________________________ >> From: Joachim Protze <protze(a)itc.rwth-aachen.de> >> Sent: Wednesday, November 13, 2019 9:56 AM >> To: discuss(a)mpich.org >> Cc: Douglas Dommermuth >> Subject: Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs >> >> Hi Doug, >> >> in general, using hyperthreads only improves execution time, if you do >> not utilize the core with a single process/thread. I.e., if you see >> ~100% cpu utilization per process in "top" for the single job execution, >> doubling the execution time from 2 to 4 mpi jobs sounds reasonable. >> If your application is mostly calculating (as an MPI application >> hopefully does), the two processes/threads running on the same core >> share the execution time of the core and will finally end up with double >> execution time. >> >> Depending on your application, additional processes/threads also might >> increase the pressure on the memory bus and therefore slow down the >> other application by making it wait for memory accesses. This might also >> explain the execution time increase from one to two mpi jobs. >> All this depends on the cpu/memory configuration of this machine. >> >> Best >> Joachim >> >> On 11/13/19 5:39 PM, Douglas Dommermuth via discuss wrote: >>> I am running Ubuntu 18.04.3 with MPICH 3.3~a2-4 and GFortran >>> 4:7.4.0-1ubuntu2.3 and GCC 4:7.4.0-1ubuntu2.3CC on dual AMD EPYC 7742 >>> processors with hyper threading enabled. My codes are written in MPI >>> and Fortran. The dual AMD processors have 128 cores and 256 threads. >>> I want to optimize the runtime for 4 mpi jobs running concurrently >>> with 64 threads each. Some timings are provided here: >>> >>> 1. One mpi job with mpiexec.hydra -n 64 myprog => 57.32s >>> 2. One mpi job with mpiexec.hydra -bind-to numa -n 64 => 50.52s >>> 3. Two mpi jobs with mpiexec.hydra -n 64 myprog => 99.77s >>> 4. Two mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 72.23s >>> 5. Four mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 159.2s >>> >>> The option "-bind-to numa" helps, but even so, running four mpi >>> jobs concurrently with 64 threads each is considerably slower than >>> running one mpi job with 64 threads. I can almost run four mpi jobs >>> sequentially and match the time for running four mpi jobs concurrently. >>> How can I improve on the result for running 4 mpi jobs concurrently? >>> Thanks, Doug. >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> >> -- >> Dipl.-Inf. Joachim Protze >> >> IT Center >> Group: High Performance Computing >> Division: Computational Science and Engineering >> RWTH Aachen University >> Seffenter Weg 23 >> D 52074 Aachen (Germany) >> Tel: +49 241 80- 24765 >> Fax: +49 241 80-624765 >> protze(a)itc.rwth-aachen.de >> www.itc.rwth-aachen.de >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs
by Douglas Dommermuth 13 Nov '19

13 Nov '19

Hi Giuseppe, It took 163.1s for this case: mpirun.mpich -bind-to user:0+1 -n 64 myprog & mpirun.mpich -bind-to user:64+1 -n 64 myprog & mpirun.mpich -bind-to user:128+1 -n 64 myprog & mpirun.mpich -bind-to user:192+1 -n 64 myprog & Thanks, Doug. ________________________________________ From: Congiu, Giuseppe <gcongiu(a)anl.gov> Sent: Wednesday, November 13, 2019 11:10 AM To: discuss(a)mpich.org Cc: Douglas Dommermuth Subject: Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs Try binding all the ranks of a job to the same numa. See if something like this works better: mpirun.mpich -bind-to user:0+1 -n 64 myprog & mpirun.mpich -bind-to user:64+1 -n 64 myprog & mpirun.mpich -bind-to user:128+1 -n 64 myprog & mpirun.mpich -bind-to user:192+1 -n 64 myprog & However this might not solve completely the problem as MPI processes can still move around across different cores in the numa. —Giuseppe > On Nov 13, 2019, at 1:00 PM, Douglas Dommermuth via discuss <discuss(a)mpich.org> wrote: > > Hi Giuseppe and Joachim, > > I will look into turning off hyperthreading and running two jobs with a corresponding change in the sizes of the jobs. Meanwhile, I ran the following case, which took 159.6s: > > mpirun.mpich -bind-to user:0+4 -n 64 myprog & > mpirun.mpich -bind-to user:1+4 -n 64 myprog & > mpirun.mpich -bind-to user:2+4 -n 64 myprog & > mpirun.mpich -bind-to user:3+4 -n 64 myprog & > > Thank you, Doug. > ________________________________________ > From: Joachim Protze <protze(a)itc.rwth-aachen.de> > Sent: Wednesday, November 13, 2019 9:56 AM > To: discuss(a)mpich.org > Cc: Douglas Dommermuth > Subject: Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs > > Hi Doug, > > in general, using hyperthreads only improves execution time, if you do > not utilize the core with a single process/thread. I.e., if you see > ~100% cpu utilization per process in "top" for the single job execution, > doubling the execution time from 2 to 4 mpi jobs sounds reasonable. > If your application is mostly calculating (as an MPI application > hopefully does), the two processes/threads running on the same core > share the execution time of the core and will finally end up with double > execution time. > > Depending on your application, additional processes/threads also might > increase the pressure on the memory bus and therefore slow down the > other application by making it wait for memory accesses. This might also > explain the execution time increase from one to two mpi jobs. > All this depends on the cpu/memory configuration of this machine. > > Best > Joachim > > On 11/13/19 5:39 PM, Douglas Dommermuth via discuss wrote: >> I am running Ubuntu 18.04.3 with MPICH 3.3~a2-4 and GFortran >> 4:7.4.0-1ubuntu2.3 and GCC 4:7.4.0-1ubuntu2.3CC on dual AMD EPYC 7742 >> processors with hyper threading enabled. My codes are written in MPI >> and Fortran. The dual AMD processors have 128 cores and 256 threads. >> I want to optimize the runtime for 4 mpi jobs running concurrently >> with 64 threads each. Some timings are provided here: >> >> 1. One mpi job with mpiexec.hydra -n 64 myprog => 57.32s >> 2. One mpi job with mpiexec.hydra -bind-to numa -n 64 => 50.52s >> 3. Two mpi jobs with mpiexec.hydra -n 64 myprog => 99.77s >> 4. Two mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 72.23s >> 5. Four mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 159.2s >> >> The option "-bind-to numa" helps, but even so, running four mpi >> jobs concurrently with 64 threads each is considerably slower than >> running one mpi job with 64 threads. I can almost run four mpi jobs >> sequentially and match the time for running four mpi jobs concurrently. >> How can I improve on the result for running 4 mpi jobs concurrently? >> Thanks, Doug. >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > -- > Dipl.-Inf. Joachim Protze > > IT Center > Group: High Performance Computing > Division: Computational Science and Engineering > RWTH Aachen University > Seffenter Weg 23 > D 52074 Aachen (Germany) > Tel: +49 241 80- 24765 > Fax: +49 241 80-624765 > protze(a)itc.rwth-aachen.de > www.itc.rwth-aachen.de > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs
by Congiu, Giuseppe 13 Nov '19

13 Nov '19

Try binding all the ranks of a job to the same numa. See if something like this works better: mpirun.mpich -bind-to user:0+1 -n 64 myprog & mpirun.mpich -bind-to user:64+1 -n 64 myprog & mpirun.mpich -bind-to user:128+1 -n 64 myprog & mpirun.mpich -bind-to user:192+1 -n 64 myprog & However this might not solve completely the problem as MPI processes can still move around across different cores in the numa. —Giuseppe > On Nov 13, 2019, at 1:00 PM, Douglas Dommermuth via discuss <discuss(a)mpich.org> wrote: > > Hi Giuseppe and Joachim, > > I will look into turning off hyperthreading and running two jobs with a corresponding change in the sizes of the jobs. Meanwhile, I ran the following case, which took 159.6s: > > mpirun.mpich -bind-to user:0+4 -n 64 myprog & > mpirun.mpich -bind-to user:1+4 -n 64 myprog & > mpirun.mpich -bind-to user:2+4 -n 64 myprog & > mpirun.mpich -bind-to user:3+4 -n 64 myprog & > > Thank you, Doug. > ________________________________________ > From: Joachim Protze <protze(a)itc.rwth-aachen.de> > Sent: Wednesday, November 13, 2019 9:56 AM > To: discuss(a)mpich.org > Cc: Douglas Dommermuth > Subject: Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs > > Hi Doug, > > in general, using hyperthreads only improves execution time, if you do > not utilize the core with a single process/thread. I.e., if you see > ~100% cpu utilization per process in "top" for the single job execution, > doubling the execution time from 2 to 4 mpi jobs sounds reasonable. > If your application is mostly calculating (as an MPI application > hopefully does), the two processes/threads running on the same core > share the execution time of the core and will finally end up with double > execution time. > > Depending on your application, additional processes/threads also might > increase the pressure on the memory bus and therefore slow down the > other application by making it wait for memory accesses. This might also > explain the execution time increase from one to two mpi jobs. > All this depends on the cpu/memory configuration of this machine. > > Best > Joachim > > On 11/13/19 5:39 PM, Douglas Dommermuth via discuss wrote: >> I am running Ubuntu 18.04.3 with MPICH 3.3~a2-4 and GFortran >> 4:7.4.0-1ubuntu2.3 and GCC 4:7.4.0-1ubuntu2.3CC on dual AMD EPYC 7742 >> processors with hyper threading enabled. My codes are written in MPI >> and Fortran. The dual AMD processors have 128 cores and 256 threads. >> I want to optimize the runtime for 4 mpi jobs running concurrently >> with 64 threads each. Some timings are provided here: >> >> 1. One mpi job with mpiexec.hydra -n 64 myprog => 57.32s >> 2. One mpi job with mpiexec.hydra -bind-to numa -n 64 => 50.52s >> 3. Two mpi jobs with mpiexec.hydra -n 64 myprog => 99.77s >> 4. Two mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 72.23s >> 5. Four mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 159.2s >> >> The option "-bind-to numa" helps, but even so, running four mpi >> jobs concurrently with 64 threads each is considerably slower than >> running one mpi job with 64 threads. I can almost run four mpi jobs >> sequentially and match the time for running four mpi jobs concurrently. >> How can I improve on the result for running 4 mpi jobs concurrently? >> Thanks, Doug. >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > -- > Dipl.-Inf. Joachim Protze > > IT Center > Group: High Performance Computing > Division: Computational Science and Engineering > RWTH Aachen University > Seffenter Weg 23 > D 52074 Aachen (Germany) > Tel: +49 241 80- 24765 > Fax: +49 241 80-624765 > protze(a)itc.rwth-aachen.de > www.itc.rwth-aachen.de > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs
by Douglas Dommermuth 13 Nov '19

13 Nov '19

Hi Giuseppe and Joachim, I will look into turning off hyperthreading and running two jobs with a corresponding change in the sizes of the jobs. Meanwhile, I ran the following case, which took 159.6s: mpirun.mpich -bind-to user:0+4 -n 64 myprog & mpirun.mpich -bind-to user:1+4 -n 64 myprog & mpirun.mpich -bind-to user:2+4 -n 64 myprog & mpirun.mpich -bind-to user:3+4 -n 64 myprog & Thank you, Doug. ________________________________________ From: Joachim Protze <protze(a)itc.rwth-aachen.de> Sent: Wednesday, November 13, 2019 9:56 AM To: discuss(a)mpich.org Cc: Douglas Dommermuth Subject: Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs Hi Doug, in general, using hyperthreads only improves execution time, if you do not utilize the core with a single process/thread. I.e., if you see ~100% cpu utilization per process in "top" for the single job execution, doubling the execution time from 2 to 4 mpi jobs sounds reasonable. If your application is mostly calculating (as an MPI application hopefully does), the two processes/threads running on the same core share the execution time of the core and will finally end up with double execution time. Depending on your application, additional processes/threads also might increase the pressure on the memory bus and therefore slow down the other application by making it wait for memory accesses. This might also explain the execution time increase from one to two mpi jobs. All this depends on the cpu/memory configuration of this machine. Best Joachim On 11/13/19 5:39 PM, Douglas Dommermuth via discuss wrote: > I am running Ubuntu 18.04.3 with MPICH 3.3~a2-4 and GFortran > 4:7.4.0-1ubuntu2.3 and GCC 4:7.4.0-1ubuntu2.3CC on dual AMD EPYC 7742 > processors with hyper threading enabled. My codes are written in MPI > and Fortran. The dual AMD processors have 128 cores and 256 threads. > I want to optimize the runtime for 4 mpi jobs running concurrently > with 64 threads each. Some timings are provided here: > > 1. One mpi job with mpiexec.hydra -n 64 myprog => 57.32s > 2. One mpi job with mpiexec.hydra -bind-to numa -n 64 => 50.52s > 3. Two mpi jobs with mpiexec.hydra -n 64 myprog => 99.77s > 4. Two mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 72.23s > 5. Four mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 159.2s > > The option "-bind-to numa" helps, but even so, running four mpi > jobs concurrently with 64 threads each is considerably slower than > running one mpi job with 64 threads. I can almost run four mpi jobs > sequentially and match the time for running four mpi jobs concurrently. > How can I improve on the result for running 4 mpi jobs concurrently? > Thanks, Doug. > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs
by Joachim Protze 13 Nov '19

13 Nov '19

Hi Doug, in general, using hyperthreads only improves execution time, if you do not utilize the core with a single process/thread. I.e., if you see ~100% cpu utilization per process in "top" for the single job execution, doubling the execution time from 2 to 4 mpi jobs sounds reasonable. If your application is mostly calculating (as an MPI application hopefully does), the two processes/threads running on the same core share the execution time of the core and will finally end up with double execution time. Depending on your application, additional processes/threads also might increase the pressure on the memory bus and therefore slow down the other application by making it wait for memory accesses. This might also explain the execution time increase from one to two mpi jobs. All this depends on the cpu/memory configuration of this machine. Best Joachim On 11/13/19 5:39 PM, Douglas Dommermuth via discuss wrote: > I am running Ubuntu 18.04.3 with MPICH 3.3~a2-4 and GFortran > 4:7.4.0-1ubuntu2.3 and GCC 4:7.4.0-1ubuntu2.3CC on dual AMD EPYC 7742 > processors with hyper threading enabled. My codes are written in MPI > and Fortran. The dual AMD processors have 128 cores and 256 threads. > I want to optimize the runtime for 4 mpi jobs running concurrently > with 64 threads each. Some timings are provided here: > > 1. One mpi job with mpiexec.hydra -n 64 myprog => 57.32s > 2. One mpi job with mpiexec.hydra -bind-to numa -n 64 => 50.52s > 3. Two mpi jobs with mpiexec.hydra -n 64 myprog => 99.77s > 4. Two mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 72.23s > 5. Four mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 159.2s > > The option "-bind-to numa" helps, but even so, running four mpi > jobs concurrently with 64 threads each is considerably slower than > running one mpi job with 64 threads. I can almost run four mpi jobs > sequentially and match the time for running four mpi jobs concurrently. > How can I improve on the result for running 4 mpi jobs concurrently? > Thanks, Doug. > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs
by Douglas Dommermuth 13 Nov '19

13 Nov '19

Hi Hui, Here is the updated list with your request: 1. One mpi job with mpiexec.hydra -n 64 myprog => 57.32s 2. One mpi job with mpiexec.hydra -bind-to numa -n 64 myprog => 50.52s 3. Two mpi jobs with mpiexec.hydra -n 64 myprog => 99.77s 4. Two mpi jobs with mpiexec.hydra -bind-to numa -n 64 myprog => 72.23s 5. Four mpi jobs with mpiexec.hydra -bind-to numa -n 64 myprog => 159.2s 6. Four mpi jobs with mpiexec.hydra -n 64 myprog => 159.9s Case 6 is the same speed as Case 5. The code is a CFD code based on a subdomain approach. There is communication between neighboring subdomains. The machine is a supermicro server with 512GB of memory. These small test cases use about 3% of the available memory. I want to run 4 larger MPI jobs concurrently. ________________________________ From: Zhou, Hui <zhouh(a)anl.gov> Sent: Wednesday, November 13, 2019 8:55 AM To: discuss(a)mpich.org Cc: Douglas Dommermuth Subject: Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs Hi Doug, What is your number running 4 mpi jobs without -bind-to numa? - Hui Zhou On Nov 13, 2019, at 10:39 AM, Douglas Dommermuth via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: I am running Ubuntu 18.04.3 with MPICH 3.3~a2-4 and GFortran 4:7.4.0-1ubuntu2.3 and GCC 4:7.4.0-1ubuntu2.3CC on dual AMD EPYC 7742 processors with hyper threading enabled. My codes are written in MPI and Fortran. The dual AMD processors have 128 cores and 256 threads. I want to optimize the runtime for 4 mpi jobs running concurrently with 64 threads each. Some timings are provided here: 1. One mpi job with mpiexec.hydra -n 64 myprog => 57.32s 2. One mpi job with mpiexec.hydra -bind-to numa -n 64 => 50.52s 3. Two mpi jobs with mpiexec.hydra -n 64 myprog => 99.77s 4. Two mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 72.23s 5. Four mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 159.2s The option "-bind-to numa" helps, but even so, running four mpi jobs concurrently with 64 threads each is considerably slower than running one mpi job with 64 threads. I can almost run four mpi jobs sequentially and match the time for running four mpi jobs concurrently. How can I improve on the result for running 4 mpi jobs concurrently? Thanks, Doug. _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs
by Congiu, Giuseppe 13 Nov '19

13 Nov '19

Hi Doug, When you do -map-by numa, processes should be bound to available numas using a round robin mapping. Thus if you are running only one MPI job with -bind-to numa and you have two numas, you are binding rank 0 to numa0:core0:hwthread0, rank 1 to numa1:core64:hwthread128, rank 2 to numa0:core0:hwthread1, and so on. Which is what you were expecting. However, if you run multiple MPI jobs with the same bind policy, these are independent from each other and will all map to the same core and hwthreads. Thus if you have 256 MPI processes running over 4 separate jobs you are still using half of the cores and a forth of the hwthreads. If you want to map to different hwthreads you can use a custom policy different for every job: mpiexec -bind-to --help -bind-to: Process-core binding type to use Binding type options: Default: none -- no binding (default) Architecture unaware options: rr -- round-robin as OS assigned processor IDs user:0+2,1+4,3,2 -- user specified binding The user policy allows you to do exactly that. There you define what hwthreads you want to bind your ranks to (+N) means your MPI process can move across hwthreads skipping every N hwthreads. So if you have 32 hwthreads and one process and request -bind-to user:0+2 you get your process moving around across hwthreads: 0, 2, 4, 6, 8, … If you want to see what the binding is for your processes you can export HYDRA_TOPO_DEBUG=1 and use /bin/true as simple example code to inspect the binding you get for different options. If you don’t do any binding MPI processes will be re-scheduled over the course of the runtime on different cores, hwthreads (losing cached data) and possibly even different numas. In Linux memory is allocated on first touch policy by default. Which means that physical memory is allocated on the numa your process is running when it accessed it first. If your process gets moved to another numa later on during the runtime it will be accessing its memory across numa domains which has a bigger impact on latency. So if your application is latency sensitive you may get a big hit on performance. Hope this helps. Best, Giuseppe On Nov 13, 2019, at 10:39 AM, Douglas Dommermuth via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: 256

1 0

Re: [mpich-discuss] Optimizing runtime for 4 mpiexec.hydra jobs
by Zhou, Hui 13 Nov '19

13 Nov '19

Hi Doug, What is your number running 4 mpi jobs without -bind-to numa? — Hui Zhou On Nov 13, 2019, at 10:39 AM, Douglas Dommermuth via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: I am running Ubuntu 18.04.3 with MPICH 3.3~a2-4 and GFortran 4:7.4.0-1ubuntu2.3 and GCC 4:7.4.0-1ubuntu2.3CC on dual AMD EPYC 7742 processors with hyper threading enabled. My codes are written in MPI and Fortran. The dual AMD processors have 128 cores and 256 threads. I want to optimize the runtime for 4 mpi jobs running concurrently with 64 threads each. Some timings are provided here: 1. One mpi job with mpiexec.hydra -n 64 myprog => 57.32s 2. One mpi job with mpiexec.hydra -bind-to numa -n 64 => 50.52s 3. Two mpi jobs with mpiexec.hydra -n 64 myprog => 99.77s 4. Two mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 72.23s 5. Four mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 159.2s The option "-bind-to numa" helps, but even so, running four mpi jobs concurrently with 64 threads each is considerably slower than running one mpi job with 64 threads. I can almost run four mpi jobs sequentially and match the time for running four mpi jobs concurrently. How can I improve on the result for running 4 mpi jobs concurrently? Thanks, Doug. _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0