- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Better alternatives of MPI_Allreduce()
by hritikesh semwal 05 May '20

05 May '20

On Tue, May 5, 2020 at 4:51 PM Benson Muite <benson_muite(a)emailplus.org> wrote: > > > > > Hi Hitesh, > > > > What hardware are you running on and what is the interconnect? > > > Right now I am using a cluster. > > > What is the interconnect? > I don't know about this. Is it relevant? > > > > Have you tried changing any of the MPI settings? > > > What do you mean by MPI settings? > > Given your comment on the barrier, this is probably not so useful at the > moment. > > > > > Can the reduction be done asynchronously? > > > I did not get your question. > > > For example using a non blocking all reduce: > https://www.mpi-forum.org/docs/mpi-3.1/mpi31-report/node135.htm > > I tried using a non-blocking call but after this code is not working correctly. > > > > > > Regards, > > Benson > > Also, is your work load balanced? One way to check this might be to place > a barrier just before the all-reduce call. If the barrier ends up taking > most of your time, then it is likely you will need to determine a better > way to distribute the computational work. > > > Thanks for your response. > > Yes, you are right. I have put barrier just before Allreduce and out of > the total time consumed by Allreduce, 79% time is consumed by the barrier. > But my computational work is balanced. Right now, I have distributed 97336 > cells among 24 processors and maximum and minimum cell distribution among > all processors is 4057 and 4055 respectively which is not too bad. Is there > any solution to get rid of this? > > Please help me in this regard. > > > >

1 0

Re: [mpich-discuss] Better alternatives of MPI_Allreduce()
by Benson Muite 05 May '20

05 May '20

On Tue, May 5, 2020, at 2:16 PM, hritikesh semwal via discuss wrote: > I want to add two more questions about my solver, > 1. I am using MPI_Neighbor_alltoallw() for exchanging the data by generating a distributed graph topology communicator. My concern is that most of the time my code is working fine but sometimes I guess it is going into deadlock (as it is not showing any output). But MPI_Neighbor_alltoallw uses MPI_Waitall inside it so I am not getting why exactly this is happening. May want to check sending and receiving correct data. Perhaps also try MPI_Neighbor_alltoallw > 2. Is it possible that every time I run the code the processors times for completion of the task may vary? For example, for one run it all processors takes around 100 seconds and for another run, all processors take 110 seconds. There is usually some variability. Do you solve the same system each time? What is the method of solution? If your code is available it can sometimes be easier to give suggestions. > > Please help in above two matters. > > On Tue, May 5, 2020 at 4:28 PM hritikesh semwal <hritikesh.semwal(a)gmail.com> wrote: >> Thanks for your response. >> >> Yes, you are right. I have put barrier just before Allreduce and out of the total time consumed by Allreduce, 79% time is consumed by the barrier. But my computational work is balanced. Right now, I have distributed 97336 cells among 24 processors and maximum and minimum cell distribution among all processors is 4057 and 4055 respectively which is not too bad. Is there any solution to get rid of this. Try profiling your code not just looking at cell distribution. Are any profling tools already installed on your cluster? >> On Tue, May 5, 2020 at 12:30 PM Joachim Protze <protze(a)itc.rwth-aachen.de> wrote: >>> Hello, >>> >>> it is important to understand, that most of the time you see is not the >>> cost of the allreduce, but the cost of synchronization (caused by load >>> imbalance). >>> >>> You can do an easy experiment and add a barrier before the allreduce. >>> Then you will see the actual cost of the allreduce, while the cost of >>> synchronization will go into the barrier. >>> >>> Now, think about dependencies in your algorithm: do you need the output >>> value immediately? Is this the same time, where you have the input value >>> ready? >>> -> otherwise use non-blocking communication and perform independent work >>> in between >>> >>> In any case: fix your load imbalance (the root cause of synchronization >>> cost). >>> >>> Best >>> Joachim >>> >>> Am 05.05.20 um 07:38 schrieb hritikesh semwal via discuss: >>> > Hello all, >>> > >>> > I am working on the development of a parallel CFD solver and I am using >>> > MPI_Allreduce for the global summation of the local errors calculated on >>> > all processes of a group and the summation is to be used by all the >>> > processes. My concern is that MPI_Allreduce is taking almost 27-30% of >>> > the total time used, which is a significant amount. So, I want to ask if >>> > anyone can suggest me better alternative/s to replace MPI_Allreduce >>> > which can reduce the time consumption. >>> > >>> > Thank you. >>> > >>> > _______________________________________________ >>> > discuss mailing list discuss(a)mpich.org >>> > To manage subscription options or unsubscribe: >>> > https://lists.mpich.org/mailman/listinfo/discuss >>> > >>> >>> >>> -- >>> Dipl.-Inf. Joachim Protze >>> >>> IT Center >>> Group: High Performance Computing >>> Division: Computational Science and Engineering >>> RWTH Aachen University >>> Seffenter Weg 23 >>> D 52074 Aachen (Germany) >>> Tel: +49 241 80- 24765 >>> Fax: +49 241 80-624765 >>> protze(a)itc.rwth-aachen.de >>> www.itc.rwth-aachen.de >>> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Better alternatives of MPI_Allreduce()
by Benson Muite 05 May '20

05 May '20

>> > >> > Hi Hitesh, >> > >> > What hardware are you running on and what is the interconnect? > > Right now I am using a cluster. What is the interconnect? > >> > Have you tried changing any of the MPI settings? > > What do you mean by MPI settings? Given your comment on the barrier, this is probably not so useful at the moment. > >> > Can the reduction be done asynchronously? > > I did not get your question. For example using a non blocking all reduce: https://www.mpi-forum.org/docs/mpi-3.1/mpi31-report/node135.htm > >> > >> > Regards, >> > Benson >> >> Also, is your work load balanced? One way to check this might be to place a barrier just before the all-reduce call. If the barrier ends up taking most of your time, then it is likely you will need to determine a better way to distribute the computational work. > > Thanks for your response. > > Yes, you are right. I have put barrier just before Allreduce and out of the total time consumed by Allreduce, 79% time is consumed by the barrier. But my computational work is balanced. Right now, I have distributed 97336 cells among 24 processors and maximum and minimum cell distribution among all processors is 4057 and 4055 respectively which is not too bad. Is there any solution to get rid of this.

1 0

Re: [mpich-discuss] Better alternatives of MPI_Allreduce()
by hritikesh semwal 05 May '20

05 May '20

I want to add two more questions about my solver, 1. I am using MPI_Neighbor_alltoallw() for exchanging the data by generating a distributed graph topology communicator. My concern is that most of the time my code is working fine but sometimes I guess it is going into deadlock (as it is not showing any output). But MPI_Neighbor_alltoallw uses MPI_Waitall inside it so I am not getting why exactly this is happening. 2. Is it possible that every time I run the code the processors times for completion of the task may vary? For example, for one run it all processors takes around 100 seconds and for another run, all processors take 110 seconds. Please help in above two matters. On Tue, May 5, 2020 at 4:28 PM hritikesh semwal <hritikesh.semwal(a)gmail.com> wrote: > Thanks for your response. > > Yes, you are right. I have put barrier just before Allreduce and out of > the total time consumed by Allreduce, 79% time is consumed by the barrier. > But my computational work is balanced. Right now, I have distributed 97336 > cells among 24 processors and maximum and minimum cell distribution among > all processors is 4057 and 4055 respectively which is not too bad. Is there > any solution to get rid of this. > > On Tue, May 5, 2020 at 12:30 PM Joachim Protze <protze(a)itc.rwth-aachen.de> > wrote: > >> Hello, >> >> it is important to understand, that most of the time you see is not the >> cost of the allreduce, but the cost of synchronization (caused by load >> imbalance). >> >> You can do an easy experiment and add a barrier before the allreduce. >> Then you will see the actual cost of the allreduce, while the cost of >> synchronization will go into the barrier. >> >> Now, think about dependencies in your algorithm: do you need the output >> value immediately? Is this the same time, where you have the input value >> ready? >> -> otherwise use non-blocking communication and perform independent work >> in between >> >> In any case: fix your load imbalance (the root cause of synchronization >> cost). >> >> Best >> Joachim >> >> Am 05.05.20 um 07:38 schrieb hritikesh semwal via discuss: >> > Hello all, >> > >> > I am working on the development of a parallel CFD solver and I am using >> > MPI_Allreduce for the global summation of the local errors calculated >> on >> > all processes of a group and the summation is to be used by all the >> > processes. My concern is that MPI_Allreduce is taking almost 27-30% of >> > the total time used, which is a significant amount. So, I want to ask >> if >> > anyone can suggest me better alternative/s to replace MPI_Allreduce >> > which can reduce the time consumption. >> > >> > Thank you. >> > >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> > >> >> >> -- >> Dipl.-Inf. Joachim Protze >> >> IT Center >> Group: High Performance Computing >> Division: Computational Science and Engineering >> RWTH Aachen University >> Seffenter Weg 23 >> D 52074 Aachen (Germany) >> Tel: +49 241 80- 24765 >> Fax: +49 241 80-624765 >> protze(a)itc.rwth-aachen.de >> www.itc.rwth-aachen.de >> >>

1 0

Re: [mpich-discuss] discuss Digest, Vol 91, Issue 3
by hritikesh semwal 05 May '20

05 May '20

On Tue, May 5, 2020 at 12:30 PM <discuss-request(a)mpich.org> wrote: > Send discuss mailing list submissions to > discuss(a)mpich.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.mpich.org/mailman/listinfo/discuss > or, via email, send a message with subject or body 'help' to > discuss-request(a)mpich.org > > You can reach the person managing the list at > discuss-owner(a)mpich.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of discuss digest..." > > > Today's Topics: > > 1. Better alternatives of MPI_Allreduce() (hritikesh semwal) > 2. Re: Better alternatives of MPI_Allreduce() (Benson Muite) > 3. Re: Better alternatives of MPI_Allreduce() (Benson Muite) > 4. Re: Better alternatives of MPI_Allreduce() (Joachim Protze) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 5 May 2020 11:08:20 +0530 > From: hritikesh semwal <hritikesh.semwal(a)gmail.com> > To: discuss(a)mpich.org > Subject: [mpich-discuss] Better alternatives of MPI_Allreduce() > Message-ID: > <CAA+35d1zgxVjcfH= > jpio5rLm1FkjxMC3Bis-qB9D43qjfEogRw(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hello all, > > I am working on the development of a parallel CFD solver and I am using > MPI_Allreduce for the global summation of the local errors calculated on > all processes of a group and the summation is to be used by all the > processes. My concern is that MPI_Allreduce is taking almost 27-30% of the > total time used, which is a significant amount. So, I want to ask if anyone > can suggest me better alternative/s to replace MPI_Allreduce which can > reduce the time consumption. > > Thank you. >

1 0

Re: [mpich-discuss] Better alternatives of MPI_Allreduce()
by hritikesh semwal 05 May '20

05 May '20

Thanks for your response. Yes, you are right. I have put barrier just before Allreduce and out of the total time consumed by Allreduce, 79% time is consumed by the barrier. But my computational work is balanced. Right now, I have distributed 97336 cells among 24 processors and maximum and minimum cell distribution among all processors is 4057 and 4055 respectively which is not too bad. Is there any solution to get rid of this. On Tue, May 5, 2020 at 12:30 PM Joachim Protze <protze(a)itc.rwth-aachen.de> wrote: > Hello, > > it is important to understand, that most of the time you see is not the > cost of the allreduce, but the cost of synchronization (caused by load > imbalance). > > You can do an easy experiment and add a barrier before the allreduce. > Then you will see the actual cost of the allreduce, while the cost of > synchronization will go into the barrier. > > Now, think about dependencies in your algorithm: do you need the output > value immediately? Is this the same time, where you have the input value > ready? > -> otherwise use non-blocking communication and perform independent work > in between > > In any case: fix your load imbalance (the root cause of synchronization > cost). > > Best > Joachim > > Am 05.05.20 um 07:38 schrieb hritikesh semwal via discuss: > > Hello all, > > > > I am working on the development of a parallel CFD solver and I am using > > MPI_Allreduce for the global summation of the local errors calculated on > > all processes of a group and the summation is to be used by all the > > processes. My concern is that MPI_Allreduce is taking almost 27-30% of > > the total time used, which is a significant amount. So, I want to ask if > > anyone can suggest me better alternative/s to replace MPI_Allreduce > > which can reduce the time consumption. > > > > Thank you. > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > -- > Dipl.-Inf. Joachim Protze > > IT Center > Group: High Performance Computing > Division: Computational Science and Engineering > RWTH Aachen University > Seffenter Weg 23 > D 52074 Aachen (Germany) > Tel: +49 241 80- 24765 > Fax: +49 241 80-624765 > protze(a)itc.rwth-aachen.de > www.itc.rwth-aachen.de > >

1 0

Re: [mpich-discuss] Better alternatives of MPI_Allreduce()
by Joachim Protze 05 May '20

05 May '20

Hello, it is important to understand, that most of the time you see is not the cost of the allreduce, but the cost of synchronization (caused by load imbalance). You can do an easy experiment and add a barrier before the allreduce. Then you will see the actual cost of the allreduce, while the cost of synchronization will go into the barrier. Now, think about dependencies in your algorithm: do you need the output value immediately? Is this the same time, where you have the input value ready? -> otherwise use non-blocking communication and perform independent work in between In any case: fix your load imbalance (the root cause of synchronization cost). Best Joachim Am 05.05.20 um 07:38 schrieb hritikesh semwal via discuss: > Hello all, > > I am working on the development of a parallel CFD solver and I am using > MPI_Allreduce for the global summation of the local errors calculated on > all processes of a group and the summation is to be used by all the > processes. My concern is that MPI_Allreduce is taking almost 27-30% of > the total time used, which is a significant amount. So, I want to ask if > anyone can suggest me better alternative/s to replace MPI_Allreduce > which can reduce the time consumption. > > Thank you. > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpich-discuss] Better alternatives of MPI_Allreduce()
by Benson Muite 05 May '20

05 May '20

On Tue, May 5, 2020, at 9:57 AM, Benson Muite wrote: > > > > On Tue, May 5, 2020, at 8:38 AM, hritikesh semwal via discuss wrote: >> Hello all, >> >> I am working on the development of a parallel CFD solver and I am using MPI_Allreduce for the global summation of the local errors calculated on all processes of a group and the summation is to be used by all the processes. My concern is that MPI_Allreduce is taking almost 27-30% of the total time used, which is a significant amount. So, I want to ask if anyone can suggest me better alternative/s to replace MPI_Allreduce which can reduce the time consumption. >> >> Thank you. >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > Hi Hitesh, > > What hardware are you running on and what is the interconnect? > Have you tried changing any of the MPI settings? > Can the reduction be done asynchronously? > > Regards, > Benson Also, is your work load balanced? One way to check this might be to place a barrier just before the all-reduce call. If the barrier ends up taking most of your time, then it is likely you will need to determine a better way to distribute the computational work.

1 0

Re: [mpich-discuss] Better alternatives of MPI_Allreduce()
by Benson Muite 05 May '20

05 May '20

On Tue, May 5, 2020, at 8:38 AM, hritikesh semwal via discuss wrote: > Hello all, > > I am working on the development of a parallel CFD solver and I am using MPI_Allreduce for the global summation of the local errors calculated on all processes of a group and the summation is to be used by all the processes. My concern is that MPI_Allreduce is taking almost 27-30% of the total time used, which is a significant amount. So, I want to ask if anyone can suggest me better alternative/s to replace MPI_Allreduce which can reduce the time consumption. > > Thank you. > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > Hi Hitesh, What hardware are you running on and what is the interconnect? Have you tried changing any of the MPI settings? Can the reduction be done asynchronously? Regards, Benson

1 0

Better alternatives of MPI_Allreduce()
by hritikesh semwal 05 May '20

05 May '20

Hello all, I am working on the development of a parallel CFD solver and I am using MPI_Allreduce for the global summation of the local errors calculated on all processes of a group and the summation is to be used by all the processes. My concern is that MPI_Allreduce is taking almost 27-30% of the total time used, which is a significant amount. So, I want to ask if anyone can suggest me better alternative/s to replace MPI_Allreduce which can reduce the time consumption. Thank you.

1 0