May 2020 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Better alternatives of MPI_Allreduce()
by Joachim Protze 05 May '20

05 May '20

Hello, it is important to understand, that most of the time you see is not the cost of the allreduce, but the cost of synchronization (caused by load imbalance). You can do an easy experiment and add a barrier before the allreduce. Then you will see the actual cost of the allreduce, while the cost of synchronization will go into the barrier. Now, think about dependencies in your algorithm: do you need the output value immediately? Is this the same time, where you have the input value ready? -> otherwise use non-blocking communication and perform independent work in between In any case: fix your load imbalance (the root cause of synchronization cost). Best Joachim Am 05.05.20 um 07:38 schrieb hritikesh semwal via discuss: > Hello all, > > I am working on the development of a parallel CFD solver and I am using > MPI_Allreduce for the global summation of the local errors calculated on > all processes of a group and the summation is to be used by all the > processes. My concern is that MPI_Allreduce is taking almost 27-30% of > the total time used, which is a significant amount. So, I want to ask if > anyone can suggest me better alternative/s to replace MPI_Allreduce > which can reduce the time consumption. > > Thank you. > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpich-discuss] Better alternatives of MPI_Allreduce()
by Benson Muite 05 May '20

05 May '20

On Tue, May 5, 2020, at 9:57 AM, Benson Muite wrote: > > > > On Tue, May 5, 2020, at 8:38 AM, hritikesh semwal via discuss wrote: >> Hello all, >> >> I am working on the development of a parallel CFD solver and I am using MPI_Allreduce for the global summation of the local errors calculated on all processes of a group and the summation is to be used by all the processes. My concern is that MPI_Allreduce is taking almost 27-30% of the total time used, which is a significant amount. So, I want to ask if anyone can suggest me better alternative/s to replace MPI_Allreduce which can reduce the time consumption. >> >> Thank you. >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > Hi Hitesh, > > What hardware are you running on and what is the interconnect? > Have you tried changing any of the MPI settings? > Can the reduction be done asynchronously? > > Regards, > Benson Also, is your work load balanced? One way to check this might be to place a barrier just before the all-reduce call. If the barrier ends up taking most of your time, then it is likely you will need to determine a better way to distribute the computational work.

1 0

Re: [mpich-discuss] Better alternatives of MPI_Allreduce()
by Benson Muite 05 May '20

05 May '20

On Tue, May 5, 2020, at 8:38 AM, hritikesh semwal via discuss wrote: > Hello all, > > I am working on the development of a parallel CFD solver and I am using MPI_Allreduce for the global summation of the local errors calculated on all processes of a group and the summation is to be used by all the processes. My concern is that MPI_Allreduce is taking almost 27-30% of the total time used, which is a significant amount. So, I want to ask if anyone can suggest me better alternative/s to replace MPI_Allreduce which can reduce the time consumption. > > Thank you. > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > Hi Hitesh, What hardware are you running on and what is the interconnect? Have you tried changing any of the MPI settings? Can the reduction be done asynchronously? Regards, Benson

1 0

Better alternatives of MPI_Allreduce()
by hritikesh semwal 05 May '20

05 May '20

Hello all, I am working on the development of a parallel CFD solver and I am using MPI_Allreduce for the global summation of the local errors calculated on all processes of a group and the summation is to be used by all the processes. My concern is that MPI_Allreduce is taking almost 27-30% of the total time used, which is a significant amount. So, I want to ask if anyone can suggest me better alternative/s to replace MPI_Allreduce which can reduce the time consumption. Thank you.

1 0

Re: [mpich-discuss] MPICH 3.3.2 and ch4:ucx launcher
by Tony Curtis 01 May '20

01 May '20

> On May 1, 2020, at 4:04 PM, Sajid Ali <sajidsyed2021(a)u.northwestern.edu> wrote: > > Hi Tony, > > Yes, and be sure to set the correct mpi launcher at runtime via the --mpi argument to srun (for more info check this page https://slurm.schedmd.com/mpi_guide.html <https://slurm.schedmd.com/mpi_guide.html> ). > Thanks - thought so. I’m just a j. Random user so I can’t do much about this. Tony

1 0

Re: [mpich-discuss] MPICH 3.3.2 and ch4:ucx launcher
by Sajid Ali 01 May '20

01 May '20

Hi Tony, Yes, and be sure to set the correct mpi launcher at runtime via the --mpi argument to srun (for more info check this page https://slurm.schedmd.com/mpi_guide.html ). -- Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io

1 0

Re: [mpich-discuss] MPICH 3.3.2 and ch4:ucx launcher
by Tony Curtis 01 May '20

01 May '20

> On May 1, 2020, at 3:30 PM, Sajid Ali <sajidsyed2021(a)u.northwestern.edu> wrote: > > Hi Tony, > > Since you have a `--with-slurm` in your configure options, you should be able to launch mpi jobs via srun. > Thanks for the quick reply. I suspected this, but wanted to ask “cleanly” without too much noise. Does this mean that to use ch4 + pmix, I would need the SLURM pmix plugin to make srun work? Tony

1 0

Re: [mpich-discuss] MPICH 3.3.2 and ch4:ucx launcher
by Sajid Ali 01 May '20

01 May '20

Hi Tony, Since you have a `--with-slurm` in your configure options, you should be able to launch mpi jobs via srun. -- Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io

1 0

MPICH 3.3.2 and ch4:ucx launcher
by Tony Curtis 01 May '20

01 May '20

Hi all, I am trying out MPICH with ch4/UCX. Our local cluster has SLURM. I configure with e.g. --with-pmix=/path/to/pmix/github-head \ --with-device=ch4:ucx \ --with-ucx=/path/to/ucx/1.7.0 \ --with-pm=no \ --with-slurm Is this correct? I don’t get an mpi{run, exec} when I make install, assume that’s a consequence of —with-pm=no ? If so, how do I get a suitable launcher for ch4:ucx ? thanks Tony

1 0