January 2021 - devel - mailman3.cels.anl.gov

Re: [mpich-devel] mpich3 error
by Brent Morgan 17 Jan '21

17 Jan '21

Hi Mpich dev community, Based on what we're seeing, I can guess it concurrently broadcasts for 23 processes, then serially goes to the next set of processes (4 processes in this case). This remains more true as I increase the # of processes- more and more processes are serially broadcasted after the first set of processes. Some processes are finished before some enter- so there does seem to be a limitation on MPI_Bcast() on how many concurrent processes are being broadcasted at a time. Does this explain why there's a timing drop as the # of processes grows? If so, possible fixes: multiple communicators simultaneously (I think), or MPI_Reduce()- but MPI_Reduce() will eventually run into a limitation as well? Am I correct? Thank you, Best, Brent On Sat, Jan 16, 2021 at 7:05 PM Brent Morgan <brent.taylormorgan(a)gmail.com> wrote: > Hi Rajeev, > > For the 27 processes, it consists of 27 physical cores, 7 devices. > > I understand Gather may overall be slower, but that should result in a > smaller slope of a linear speedup... The observed results leads to a > timing 2x of what is expected for 27 processes. > [image: image.png] > > One thing I notice is that, in line 126 of my sent code, ranks 20-24 is > processed *after* ranks 1-19, 25-27. Printing things out above line 126, > ranks 20-24 is printed always after 1-19, 25-27, and a pause; it's as if > the program broadcasts 23 (instead of 27) ranks at a time. There is > nothing faulty about my devices/cores, as this behavior is consistent when > I switch nodes in the hostname file. > > Best, > Brent > > On Sat, Jan 16, 2021 at 6:44 PM Thakur, Rajeev <thakur(a)anl.gov> wrote: > >> There is no limitation. Gather concatenates data at the root so all the >> data needs to be sent. Reduce calculates intermediate sums in a binomial >> tree fashion, so if you are summing a single number, only a single number >> is communicated. There are many papers on collective communication >> algorithms. >> >> >> >> When you run 27 processes, are there 27 physical cores for them to run >> concurrently? If not, the performance measurements will not show speedups. >> >> >> >> *From: *Brent Morgan <brent.taylormorgan(a)gmail.com> >> *Date: *Saturday, January 16, 2021 at 7:37 PM >> *To: *"Thakur, Rajeev" <thakur(a)anl.gov> >> *Cc: *Robert Katona <robert.katona(a)hotmail.com>, "Zhou, Hui" < >> zhouh(a)anl.gov>, "devel(a)mpich.org" <devel(a)mpich.org> >> *Subject: *Re: [mpich-devel] mpich3 error >> >> >> >> Why isn’t the sum all computed after doing a gather to the root- is there >> a limitation of 27 processes (7 devices) for gather? We thought a >> collector could handle much more, >> >> >> >> Best, >> >> Brent >> >> >> >> On Sat, Jan 16, 2021 at 2:18 PM Thakur, Rajeev <thakur(a)anl.gov> wrote: >> >> It uses a different algorithm and communicates much less data. The sum is >> not all computed after doing a gather to the root. >> >> >> >> *From: *Brent Morgan <brent.taylormorgan(a)gmail.com> >> *Date: *Saturday, January 16, 2021 at 2:58 PM >> *To: *"Thakur, Rajeev" <thakur(a)anl.gov> >> *Cc: *Robert Katona <robert.katona(a)hotmail.com>, "Zhou, Hui" < >> zhouh(a)anl.gov>, "devel(a)mpich.org" <devel(a)mpich.org> >> *Subject: *Re: [mpich-devel] mpich3 error >> >> >> >> We will try MPI_Reduce which will improve our code, but it will not solve >> the underlying problem. >> >> >> >> Best, >> >> Brent >> >> >> >> On Sat, Jan 16, 2021 at 1:31 PM Thakur, Rajeev <thakur(a)anl.gov> wrote: >> >> Your mail all the way below says “We are using MPI_Gather collector for >> merely calculating the sum of the result of N processes”. Why don’t you use >> MPI_Reduce instead then? >> >> >> >> Rajeev >> >> >> >> >> >> *From: *Brent Morgan via devel <devel(a)mpich.org> >> *Reply-To: *"devel(a)mpich.org" <devel(a)mpich.org> >> *Date: *Saturday, January 16, 2021 at 1:38 PM >> *To: *"Zhou, Hui" <zhouh(a)anl.gov> >> *Cc: *Brent Morgan <brent.taylormorgan(a)gmail.com>, "devel(a)mpich.org" < >> devel(a)mpich.org>, Robert Katona <robert.katona(a)hotmail.com> >> *Subject: *Re: [mpich-devel] mpich3 error >> >> >> >> Hi Hui, Mpich community, >> >> >> >> Thanks for the response. You're right, I'll provide a toy program that >> replicates the code structure (and results). The toy program is >> calculating a sum value from each process- the value isn't too important >> for this toy program. The timing, however, is the only thing important in >> our demonstration. It exactly replicates what we are observing for our >> actual program. This directly relates to the MPI functionality- we can't >> find out what the issue is. >> >> >> >> I have attached the code. Is something wrong with our implementation? >> It starts with the main() function. Thank you very much for any help, >> >> >> >> Best, >> >> Brent >> >> PS My subscription to discuss(a)mpich.org is pending currently. >> >> >> >> On Sat, Jan 16, 2021 at 12:24 PM Brent Morgan < >> brent.taylormorgan(a)gmail.com> wrote: >> >> Hi Hui, Mpich community, >> >> >> >> Thanks for the response. You're right, I'll provide a toy program that >> replicates the code structure (and results). The toy program is >> calculating a sum value from each process- the value isn't too important >> for this toy program. The timing, however, is the only thing important in >> our demonstration. It exactly replicates what we are observing for our >> actual program. This directly relates to the MPI functionality- we can't >> find out what the issue is. >> >> >> >> I have attached the code. Is something wrong with our implementation? >> It starts with the main() function. Thank you very much for any help, >> >> >> >> Best, >> >> Brent >> >> >> >> On Fri, Jan 15, 2021 at 10:43 PM Zhou, Hui <zhouh(a)anl.gov> wrote: >> >> Your description only mentions MPI_Gather. If there is indeed problem >> with MPI_Gather, then you should be able to reproduce the issue with a >> sample program. Share with us and we can better assist you. If you can’t >> reproduce the issue with a simple example, then I suspect there are other >> problems that you are not able to fully describe. We really can’t help much >> without able to see the code. >> >> >> >> That said, I am not even sure what is the issue you are describing. 100 >> process MPI_Gather will be slower than 50 process MPI_Gather. And since it >> is a collective, if one of your process is delayed due to some computations >> or else, the whole collective will take longer to finish just due to >> waiting for the late process. You really need tell us what your program is >> doing in order for us to even offer an intelligent guess. >> >> >> >> -- >> Hui Zhou >> >> >> >> >> >> *From: *Brent Morgan <brent.taylormorgan(a)gmail.com> >> *Date: *Friday, January 15, 2021 at 10:42 PM >> *To: *Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> >> *Cc: *Robert Katona <robert.katona(a)hotmail.com> >> *Subject: *Re: [mpich-devel] mpich3 error >> >> Hi MPICH community, >> >> >> >> My team has downloaded mpich 3.3.2 (using ch3 as default) and implemented >> MPI, and for small # of processes (<50), everything worked fine for our MPI >> implementation. For >=50 processes, there was a ch3 error and crashed the >> program after a random amount of seconds (sometimes 10seconds, sometimes >> 100seconds). So we compiled mpich 3.3.2 with ch4 (instead of default ch3) >> using '--with-device=ch4:ofi` flag and this got rid of the error- but for >> >12 processes, the speed would slow down to 2x slower suddenly. >> >> >> >> Upon Hui's suggestion, we upgraded to mpich 3.4 and compiled with >> '--with-device=ch4:ofi` flag (where ch4 is default for mpich 3.4). >> Everything worked fine until we hit 20 processes; after >=20 processes, the >> 2x slowdown is happening again. >> >> >> >> We have tried 1 communicator and multiple communicators in an attempt to >> make the MPI implementation faster, but there's no significant difference >> in observations. We are using MPI_Gather collector for merely calculating >> the sum of the result of N processes, but we can't seem to maintain >> stability within MPI as we increase N processes. Is there something we are >> missing that is ultimately causing this error? We are at a loss here, >> thank you. >> >> >> >> Best, >> >> Brent >> >> >> >> >> >>

1 0

Re: [mpich-devel] mpich3 error
by Brent Morgan 16 Jan '21

16 Jan '21

Hi Rajeev, For the 27 processes, it consists of 27 physical cores, 7 devices. I understand Gather may overall be slower, but that should result in a smaller slope of a linear speedup... The observed results leads to a timing 2x of what is expected for 27 processes. [image: image.png] One thing I notice is that, in line 126 of my sent code, ranks 20-24 is processed *after* ranks 1-19, 25-27. Printing things out above line 126, ranks 20-24 is printed always after 1-19, 25-27, and a pause; it's as if the program broadcasts 23 (instead of 27) ranks at a time. There is nothing faulty about my devices/cores, as this behavior is consistent when I switch nodes in the hostname file. Best, Brent On Sat, Jan 16, 2021 at 6:44 PM Thakur, Rajeev <thakur(a)anl.gov> wrote: > There is no limitation. Gather concatenates data at the root so all the > data needs to be sent. Reduce calculates intermediate sums in a binomial > tree fashion, so if you are summing a single number, only a single number > is communicated. There are many papers on collective communication > algorithms. > > > > When you run 27 processes, are there 27 physical cores for them to run > concurrently? If not, the performance measurements will not show speedups. > > > > *From: *Brent Morgan <brent.taylormorgan(a)gmail.com> > *Date: *Saturday, January 16, 2021 at 7:37 PM > *To: *"Thakur, Rajeev" <thakur(a)anl.gov> > *Cc: *Robert Katona <robert.katona(a)hotmail.com>, "Zhou, Hui" < > zhouh(a)anl.gov>, "devel(a)mpich.org" <devel(a)mpich.org> > *Subject: *Re: [mpich-devel] mpich3 error > > > > Why isn’t the sum all computed after doing a gather to the root- is there > a limitation of 27 processes (7 devices) for gather? We thought a > collector could handle much more, > > > > Best, > > Brent > > > > On Sat, Jan 16, 2021 at 2:18 PM Thakur, Rajeev <thakur(a)anl.gov> wrote: > > It uses a different algorithm and communicates much less data. The sum is > not all computed after doing a gather to the root. > > > > *From: *Brent Morgan <brent.taylormorgan(a)gmail.com> > *Date: *Saturday, January 16, 2021 at 2:58 PM > *To: *"Thakur, Rajeev" <thakur(a)anl.gov> > *Cc: *Robert Katona <robert.katona(a)hotmail.com>, "Zhou, Hui" < > zhouh(a)anl.gov>, "devel(a)mpich.org" <devel(a)mpich.org> > *Subject: *Re: [mpich-devel] mpich3 error > > > > We will try MPI_Reduce which will improve our code, but it will not solve > the underlying problem. > > > > Best, > > Brent > > > > On Sat, Jan 16, 2021 at 1:31 PM Thakur, Rajeev <thakur(a)anl.gov> wrote: > > Your mail all the way below says “We are using MPI_Gather collector for > merely calculating the sum of the result of N processes”. Why don’t you use > MPI_Reduce instead then? > > > > Rajeev > > > > > > *From: *Brent Morgan via devel <devel(a)mpich.org> > *Reply-To: *"devel(a)mpich.org" <devel(a)mpich.org> > *Date: *Saturday, January 16, 2021 at 1:38 PM > *To: *"Zhou, Hui" <zhouh(a)anl.gov> > *Cc: *Brent Morgan <brent.taylormorgan(a)gmail.com>, "devel(a)mpich.org" < > devel(a)mpich.org>, Robert Katona <robert.katona(a)hotmail.com> > *Subject: *Re: [mpich-devel] mpich3 error > > > > Hi Hui, Mpich community, > > > > Thanks for the response. You're right, I'll provide a toy program that > replicates the code structure (and results). The toy program is > calculating a sum value from each process- the value isn't too important > for this toy program. The timing, however, is the only thing important in > our demonstration. It exactly replicates what we are observing for our > actual program. This directly relates to the MPI functionality- we can't > find out what the issue is. > > > > I have attached the code. Is something wrong with our implementation? It > starts with the main() function. Thank you very much for any help, > > > > Best, > > Brent > > PS My subscription to discuss(a)mpich.org is pending currently. > > > > On Sat, Jan 16, 2021 at 12:24 PM Brent Morgan < > brent.taylormorgan(a)gmail.com> wrote: > > Hi Hui, Mpich community, > > > > Thanks for the response. You're right, I'll provide a toy program that > replicates the code structure (and results). The toy program is > calculating a sum value from each process- the value isn't too important > for this toy program. The timing, however, is the only thing important in > our demonstration. It exactly replicates what we are observing for our > actual program. This directly relates to the MPI functionality- we can't > find out what the issue is. > > > > I have attached the code. Is something wrong with our implementation? It > starts with the main() function. Thank you very much for any help, > > > > Best, > > Brent > > > > On Fri, Jan 15, 2021 at 10:43 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > > Your description only mentions MPI_Gather. If there is indeed problem with > MPI_Gather, then you should be able to reproduce the issue with a sample > program. Share with us and we can better assist you. If you can’t reproduce > the issue with a simple example, then I suspect there are other problems > that you are not able to fully describe. We really can’t help much without > able to see the code. > > > > That said, I am not even sure what is the issue you are describing. 100 > process MPI_Gather will be slower than 50 process MPI_Gather. And since it > is a collective, if one of your process is delayed due to some computations > or else, the whole collective will take longer to finish just due to > waiting for the late process. You really need tell us what your program is > doing in order for us to even offer an intelligent guess. > > > > -- > Hui Zhou > > > > > > *From: *Brent Morgan <brent.taylormorgan(a)gmail.com> > *Date: *Friday, January 15, 2021 at 10:42 PM > *To: *Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> > *Cc: *Robert Katona <robert.katona(a)hotmail.com> > *Subject: *Re: [mpich-devel] mpich3 error > > Hi MPICH community, > > > > My team has downloaded mpich 3.3.2 (using ch3 as default) and implemented > MPI, and for small # of processes (<50), everything worked fine for our MPI > implementation. For >=50 processes, there was a ch3 error and crashed the > program after a random amount of seconds (sometimes 10seconds, sometimes > 100seconds). So we compiled mpich 3.3.2 with ch4 (instead of default ch3) > using '--with-device=ch4:ofi` flag and this got rid of the error- but for > >12 processes, the speed would slow down to 2x slower suddenly. > > > > Upon Hui's suggestion, we upgraded to mpich 3.4 and compiled with > '--with-device=ch4:ofi` flag (where ch4 is default for mpich 3.4). > Everything worked fine until we hit 20 processes; after >=20 processes, the > 2x slowdown is happening again. > > > > We have tried 1 communicator and multiple communicators in an attempt to > make the MPI implementation faster, but there's no significant difference > in observations. We are using MPI_Gather collector for merely calculating > the sum of the result of N processes, but we can't seem to maintain > stability within MPI as we increase N processes. Is there something we are > missing that is ultimately causing this error? We are at a loss here, > thank you. > > > > Best, > > Brent > > > > > >

1 0

Re: [mpich-devel] mpich3 error
by Thakur, Rajeev 16 Jan '21

16 Jan '21

There is no limitation. Gather concatenates data at the root so all the data needs to be sent. Reduce calculates intermediate sums in a binomial tree fashion, so if you are summing a single number, only a single number is communicated. There are many papers on collective communication algorithms. When you run 27 processes, are there 27 physical cores for them to run concurrently? If not, the performance measurements will not show speedups. From: Brent Morgan <brent.taylormorgan(a)gmail.com> Date: Saturday, January 16, 2021 at 7:37 PM To: "Thakur, Rajeev" <thakur(a)anl.gov> Cc: Robert Katona <robert.katona(a)hotmail.com>, "Zhou, Hui" <zhouh(a)anl.gov>, "devel(a)mpich.org" <devel(a)mpich.org> Subject: Re: [mpich-devel] mpich3 error Why isn’t the sum all computed after doing a gather to the root- is there a limitation of 27 processes (7 devices) for gather? We thought a collector could handle much more, Best, Brent On Sat, Jan 16, 2021 at 2:18 PM Thakur, Rajeev <thakur(a)anl.gov<mailto:[email protected]>> wrote: It uses a different algorithm and communicates much less data. The sum is not all computed after doing a gather to the root. From: Brent Morgan <brent.taylormorgan(a)gmail.com<mailto:[email protected]>> Date: Saturday, January 16, 2021 at 2:58 PM To: "Thakur, Rajeev" <thakur(a)anl.gov<mailto:[email protected]>> Cc: Robert Katona <robert.katona(a)hotmail.com<mailto:[email protected]>>, "Zhou, Hui" <zhouh(a)anl.gov<mailto:[email protected]>>, "devel(a)mpich.org<mailto:[email protected]>" <devel(a)mpich.org<mailto:[email protected]>> Subject: Re: [mpich-devel] mpich3 error We will try MPI_Reduce which will improve our code, but it will not solve the underlying problem. Best, Brent On Sat, Jan 16, 2021 at 1:31 PM Thakur, Rajeev <thakur(a)anl.gov<mailto:[email protected]>> wrote: Your mail all the way below says “We are using MPI_Gather collector for merely calculating the sum of the result of N processes”. Why don’t you use MPI_Reduce instead then? Rajeev From: Brent Morgan via devel <devel(a)mpich.org<mailto:[email protected]>> Reply-To: "devel(a)mpich.org<mailto:[email protected]>" <devel(a)mpich.org<mailto:[email protected]>> Date: Saturday, January 16, 2021 at 1:38 PM To: "Zhou, Hui" <zhouh(a)anl.gov<mailto:[email protected]>> Cc: Brent Morgan <brent.taylormorgan(a)gmail.com<mailto:[email protected]>>, "devel(a)mpich.org<mailto:[email protected]>" <devel(a)mpich.org<mailto:[email protected]>>, Robert Katona <robert.katona(a)hotmail.com<mailto:[email protected]>> Subject: Re: [mpich-devel] mpich3 error Hi Hui, Mpich community, Thanks for the response. You're right, I'll provide a toy program that replicates the code structure (and results). The toy program is calculating a sum value from each process- the value isn't too important for this toy program. The timing, however, is the only thing important in our demonstration. It exactly replicates what we are observing for our actual program. This directly relates to the MPI functionality- we can't find out what the issue is. [cid:[email protected]] I have attached the code. Is something wrong with our implementation? It starts with the main() function. Thank you very much for any help, Best, Brent PS My subscription to discuss(a)mpich.org<mailto:[email protected]> is pending currently. On Sat, Jan 16, 2021 at 12:24 PM Brent Morgan <brent.taylormorgan(a)gmail.com<mailto:[email protected]>> wrote: Hi Hui, Mpich community, Thanks for the response. You're right, I'll provide a toy program that replicates the code structure (and results). The toy program is calculating a sum value from each process- the value isn't too important for this toy program. The timing, however, is the only thing important in our demonstration. It exactly replicates what we are observing for our actual program. This directly relates to the MPI functionality- we can't find out what the issue is. [cid:[email protected]] I have attached the code. Is something wrong with our implementation? It starts with the main() function. Thank you very much for any help, Best, Brent On Fri, Jan 15, 2021 at 10:43 PM Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: Your description only mentions MPI_Gather. If there is indeed problem with MPI_Gather, then you should be able to reproduce the issue with a sample program. Share with us and we can better assist you. If you can’t reproduce the issue with a simple example, then I suspect there are other problems that you are not able to fully describe. We really can’t help much without able to see the code. That said, I am not even sure what is the issue you are describing. 100 process MPI_Gather will be slower than 50 process MPI_Gather. And since it is a collective, if one of your process is delayed due to some computations or else, the whole collective will take longer to finish just due to waiting for the late process. You really need tell us what your program is doing in order for us to even offer an intelligent guess. -- Hui Zhou From: Brent Morgan <brent.taylormorgan(a)gmail.com<mailto:[email protected]>> Date: Friday, January 15, 2021 at 10:42 PM To: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>, discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Robert Katona <robert.katona(a)hotmail.com<mailto:[email protected]>> Subject: Re: [mpich-devel] mpich3 error Hi MPICH community, My team has downloaded mpich 3.3.2 (using ch3 as default) and implemented MPI, and for small # of processes (<50), everything worked fine for our MPI implementation. For >=50 processes, there was a ch3 error and crashed the program after a random amount of seconds (sometimes 10seconds, sometimes 100seconds). So we compiled mpich 3.3.2 with ch4 (instead of default ch3) using '--with-device=ch4:ofi` flag and this got rid of the error- but for >12 processes, the speed would slow down to 2x slower suddenly. Upon Hui's suggestion, we upgraded to mpich 3.4 and compiled with '--with-device=ch4:ofi` flag (where ch4 is default for mpich 3.4). Everything worked fine until we hit 20 processes; after >=20 processes, the 2x slowdown is happening again. We have tried 1 communicator and multiple communicators in an attempt to make the MPI implementation faster, but there's no significant difference in observations. We are using MPI_Gather collector for merely calculating the sum of the result of N processes, but we can't seem to maintain stability within MPI as we increase N processes. Is there something we are missing that is ultimately causing this error? We are at a loss here, thank you. Best, Brent

1 0

Re: [mpich-devel] mpich3 error
by Brent Morgan 16 Jan '21

16 Jan '21

Why isn’t the sum all computed after doing a gather to the root- is there a limitation of 27 processes (7 devices) for gather? We thought a collector could handle much more, Best, Brent On Sat, Jan 16, 2021 at 2:18 PM Thakur, Rajeev <thakur(a)anl.gov> wrote: > It uses a different algorithm and communicates much less data. The sum is > not all computed after doing a gather to the root. > > > > *From: *Brent Morgan <brent.taylormorgan(a)gmail.com> > *Date: *Saturday, January 16, 2021 at 2:58 PM > *To: *"Thakur, Rajeev" <thakur(a)anl.gov> > *Cc: *Robert Katona <robert.katona(a)hotmail.com>, "Zhou, Hui" < > zhouh(a)anl.gov>, "devel(a)mpich.org" <devel(a)mpich.org> > *Subject: *Re: [mpich-devel] mpich3 error > > > > We will try MPI_Reduce which will improve our code, but it will not solve > the underlying problem. > > > > Best, > > Brent > > > > On Sat, Jan 16, 2021 at 1:31 PM Thakur, Rajeev <thakur(a)anl.gov> wrote: > > Your mail all the way below says “We are using MPI_Gather collector for > merely calculating the sum of the result of N processes”. Why don’t you use > MPI_Reduce instead then? > > > > Rajeev > > > > > > *From: *Brent Morgan via devel <devel(a)mpich.org> > *Reply-To: *"devel(a)mpich.org" <devel(a)mpich.org> > *Date: *Saturday, January 16, 2021 at 1:38 PM > *To: *"Zhou, Hui" <zhouh(a)anl.gov> > *Cc: *Brent Morgan <brent.taylormorgan(a)gmail.com>, "devel(a)mpich.org" < > devel(a)mpich.org>, Robert Katona <robert.katona(a)hotmail.com> > *Subject: *Re: [mpich-devel] mpich3 error > > > > Hi Hui, Mpich community, > > > > Thanks for the response. You're right, I'll provide a toy program that > replicates the code structure (and results). The toy program is > calculating a sum value from each process- the value isn't too important > for this toy program. The timing, however, is the only thing important in > our demonstration. It exactly replicates what we are observing for our > actual program. This directly relates to the MPI functionality- we can't > find out what the issue is. > > > > I have attached the code. Is something wrong with our implementation? It > starts with the main() function. Thank you very much for any help, > > > > Best, > > Brent > > PS My subscription to discuss(a)mpich.org is pending currently. > > > > On Sat, Jan 16, 2021 at 12:24 PM Brent Morgan < > brent.taylormorgan(a)gmail.com> wrote: > > Hi Hui, Mpich community, > > > > Thanks for the response. You're right, I'll provide a toy program that > replicates the code structure (and results). The toy program is > calculating a sum value from each process- the value isn't too important > for this toy program. The timing, however, is the only thing important in > our demonstration. It exactly replicates what we are observing for our > actual program. This directly relates to the MPI functionality- we can't > find out what the issue is. > > > > I have attached the code. Is something wrong with our implementation? It > starts with the main() function. Thank you very much for any help, > > > > Best, > > Brent > > > > On Fri, Jan 15, 2021 at 10:43 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > > Your description only mentions MPI_Gather. If there is indeed problem with > MPI_Gather, then you should be able to reproduce the issue with a sample > program. Share with us and we can better assist you. If you can’t reproduce > the issue with a simple example, then I suspect there are other problems > that you are not able to fully describe. We really can’t help much without > able to see the code. > > > > That said, I am not even sure what is the issue you are describing. 100 > process MPI_Gather will be slower than 50 process MPI_Gather. And since it > is a collective, if one of your process is delayed due to some computations > or else, the whole collective will take longer to finish just due to > waiting for the late process. You really need tell us what your program is > doing in order for us to even offer an intelligent guess. > > > > -- > Hui Zhou > > > > > > *From: *Brent Morgan <brent.taylormorgan(a)gmail.com> > *Date: *Friday, January 15, 2021 at 10:42 PM > *To: *Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> > *Cc: *Robert Katona <robert.katona(a)hotmail.com> > *Subject: *Re: [mpich-devel] mpich3 error > > Hi MPICH community, > > > > My team has downloaded mpich 3.3.2 (using ch3 as default) and implemented > MPI, and for small # of processes (<50), everything worked fine for our MPI > implementation. For >=50 processes, there was a ch3 error and crashed the > program after a random amount of seconds (sometimes 10seconds, sometimes > 100seconds). So we compiled mpich 3.3.2 with ch4 (instead of default ch3) > using '--with-device=ch4:ofi` flag and this got rid of the error- but for > >12 processes, the speed would slow down to 2x slower suddenly. > > > > Upon Hui's suggestion, we upgraded to mpich 3.4 and compiled with > '--with-device=ch4:ofi` flag (where ch4 is default for mpich 3.4). > Everything worked fine until we hit 20 processes; after >=20 processes, the > 2x slowdown is happening again. > > > > We have tried 1 communicator and multiple communicators in an attempt to > make the MPI implementation faster, but there's no significant difference > in observations. We are using MPI_Gather collector for merely calculating > the sum of the result of N processes, but we can't seem to maintain > stability within MPI as we increase N processes. Is there something we are > missing that is ultimately causing this error? We are at a loss here, > thank you. > > > > Best, > > Brent > > > > > >

1 0

Re: [mpich-devel] mpich3 error
by Thakur, Rajeev 16 Jan '21

16 Jan '21

It uses a different algorithm and communicates much less data. The sum is not all computed after doing a gather to the root. From: Brent Morgan <brent.taylormorgan(a)gmail.com> Date: Saturday, January 16, 2021 at 2:58 PM To: "Thakur, Rajeev" <thakur(a)anl.gov> Cc: Robert Katona <robert.katona(a)hotmail.com>, "Zhou, Hui" <zhouh(a)anl.gov>, "devel(a)mpich.org" <devel(a)mpich.org> Subject: Re: [mpich-devel] mpich3 error We will try MPI_Reduce which will improve our code, but it will not solve the underlying problem. Best, Brent On Sat, Jan 16, 2021 at 1:31 PM Thakur, Rajeev <thakur(a)anl.gov<mailto:[email protected]>> wrote: Your mail all the way below says “We are using MPI_Gather collector for merely calculating the sum of the result of N processes”. Why don’t you use MPI_Reduce instead then? Rajeev From: Brent Morgan via devel <devel(a)mpich.org<mailto:[email protected]>> Reply-To: "devel(a)mpich.org<mailto:[email protected]>" <devel(a)mpich.org<mailto:[email protected]>> Date: Saturday, January 16, 2021 at 1:38 PM To: "Zhou, Hui" <zhouh(a)anl.gov<mailto:[email protected]>> Cc: Brent Morgan <brent.taylormorgan(a)gmail.com<mailto:[email protected]>>, "devel(a)mpich.org<mailto:[email protected]>" <devel(a)mpich.org<mailto:[email protected]>>, Robert Katona <robert.katona(a)hotmail.com<mailto:[email protected]>> Subject: Re: [mpich-devel] mpich3 error Hi Hui, Mpich community, Thanks for the response. You're right, I'll provide a toy program that replicates the code structure (and results). The toy program is calculating a sum value from each process- the value isn't too important for this toy program. The timing, however, is the only thing important in our demonstration. It exactly replicates what we are observing for our actual program. This directly relates to the MPI functionality- we can't find out what the issue is. [cid:[email protected]] I have attached the code. Is something wrong with our implementation? It starts with the main() function. Thank you very much for any help, Best, Brent PS My subscription to discuss(a)mpich.org<mailto:[email protected]> is pending currently. On Sat, Jan 16, 2021 at 12:24 PM Brent Morgan <brent.taylormorgan(a)gmail.com<mailto:[email protected]>> wrote: Hi Hui, Mpich community, Thanks for the response. You're right, I'll provide a toy program that replicates the code structure (and results). The toy program is calculating a sum value from each process- the value isn't too important for this toy program. The timing, however, is the only thing important in our demonstration. It exactly replicates what we are observing for our actual program. This directly relates to the MPI functionality- we can't find out what the issue is. [cid:[email protected]] I have attached the code. Is something wrong with our implementation? It starts with the main() function. Thank you very much for any help, Best, Brent On Fri, Jan 15, 2021 at 10:43 PM Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: Your description only mentions MPI_Gather. If there is indeed problem with MPI_Gather, then you should be able to reproduce the issue with a sample program. Share with us and we can better assist you. If you can’t reproduce the issue with a simple example, then I suspect there are other problems that you are not able to fully describe. We really can’t help much without able to see the code. That said, I am not even sure what is the issue you are describing. 100 process MPI_Gather will be slower than 50 process MPI_Gather. And since it is a collective, if one of your process is delayed due to some computations or else, the whole collective will take longer to finish just due to waiting for the late process. You really need tell us what your program is doing in order for us to even offer an intelligent guess. -- Hui Zhou From: Brent Morgan <brent.taylormorgan(a)gmail.com<mailto:[email protected]>> Date: Friday, January 15, 2021 at 10:42 PM To: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>, discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Robert Katona <robert.katona(a)hotmail.com<mailto:[email protected]>> Subject: Re: [mpich-devel] mpich3 error Hi MPICH community, My team has downloaded mpich 3.3.2 (using ch3 as default) and implemented MPI, and for small # of processes (<50), everything worked fine for our MPI implementation. For >=50 processes, there was a ch3 error and crashed the program after a random amount of seconds (sometimes 10seconds, sometimes 100seconds). So we compiled mpich 3.3.2 with ch4 (instead of default ch3) using '--with-device=ch4:ofi` flag and this got rid of the error- but for >12 processes, the speed would slow down to 2x slower suddenly. Upon Hui's suggestion, we upgraded to mpich 3.4 and compiled with '--with-device=ch4:ofi` flag (where ch4 is default for mpich 3.4). Everything worked fine until we hit 20 processes; after >=20 processes, the 2x slowdown is happening again. We have tried 1 communicator and multiple communicators in an attempt to make the MPI implementation faster, but there's no significant difference in observations. We are using MPI_Gather collector for merely calculating the sum of the result of N processes, but we can't seem to maintain stability within MPI as we increase N processes. Is there something we are missing that is ultimately causing this error? We are at a loss here, thank you. Best, Brent

1 0

Re: [mpich-devel] mpich3 error
by Brent Morgan 16 Jan '21

16 Jan '21

We will try MPI_Reduce which will improve our code, but it will not solve the underlying problem. Best, Brent On Sat, Jan 16, 2021 at 1:31 PM Thakur, Rajeev <thakur(a)anl.gov> wrote: > Your mail all the way below says “We are using MPI_Gather collector for > merely calculating the sum of the result of N processes”. Why don’t you use > MPI_Reduce instead then? > > > > Rajeev > > > > > > *From: *Brent Morgan via devel <devel(a)mpich.org> > *Reply-To: *"devel(a)mpich.org" <devel(a)mpich.org> > *Date: *Saturday, January 16, 2021 at 1:38 PM > *To: *"Zhou, Hui" <zhouh(a)anl.gov> > *Cc: *Brent Morgan <brent.taylormorgan(a)gmail.com>, "devel(a)mpich.org" < > devel(a)mpich.org>, Robert Katona <robert.katona(a)hotmail.com> > *Subject: *Re: [mpich-devel] mpich3 error > > > > Hi Hui, Mpich community, > > > > Thanks for the response. You're right, I'll provide a toy program that > replicates the code structure (and results). The toy program is > calculating a sum value from each process- the value isn't too important > for this toy program. The timing, however, is the only thing important in > our demonstration. It exactly replicates what we are observing for our > actual program. This directly relates to the MPI functionality- we can't > find out what the issue is. > > > > I have attached the code. Is something wrong with our implementation? It > starts with the main() function. Thank you very much for any help, > > > > Best, > > Brent > > PS My subscription to discuss(a)mpich.org is pending currently. > > > > On Sat, Jan 16, 2021 at 12:24 PM Brent Morgan < > brent.taylormorgan(a)gmail.com> wrote: > > Hi Hui, Mpich community, > > > > Thanks for the response. You're right, I'll provide a toy program that > replicates the code structure (and results). The toy program is > calculating a sum value from each process- the value isn't too important > for this toy program. The timing, however, is the only thing important in > our demonstration. It exactly replicates what we are observing for our > actual program. This directly relates to the MPI functionality- we can't > find out what the issue is. > > > > I have attached the code. Is something wrong with our implementation? It > starts with the main() function. Thank you very much for any help, > > > > Best, > > Brent > > > > On Fri, Jan 15, 2021 at 10:43 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > > Your description only mentions MPI_Gather. If there is indeed problem with > MPI_Gather, then you should be able to reproduce the issue with a sample > program. Share with us and we can better assist you. If you can’t reproduce > the issue with a simple example, then I suspect there are other problems > that you are not able to fully describe. We really can’t help much without > able to see the code. > > > > That said, I am not even sure what is the issue you are describing. 100 > process MPI_Gather will be slower than 50 process MPI_Gather. And since it > is a collective, if one of your process is delayed due to some computations > or else, the whole collective will take longer to finish just due to > waiting for the late process. You really need tell us what your program is > doing in order for us to even offer an intelligent guess. > > > > -- > Hui Zhou > > > > > > *From: *Brent Morgan <brent.taylormorgan(a)gmail.com> > *Date: *Friday, January 15, 2021 at 10:42 PM > *To: *Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> > *Cc: *Robert Katona <robert.katona(a)hotmail.com> > *Subject: *Re: [mpich-devel] mpich3 error > > Hi MPICH community, > > > > My team has downloaded mpich 3.3.2 (using ch3 as default) and implemented > MPI, and for small # of processes (<50), everything worked fine for our MPI > implementation. For >=50 processes, there was a ch3 error and crashed the > program after a random amount of seconds (sometimes 10seconds, sometimes > 100seconds). So we compiled mpich 3.3.2 with ch4 (instead of default ch3) > using '--with-device=ch4:ofi` flag and this got rid of the error- but for > >12 processes, the speed would slow down to 2x slower suddenly. > > > > Upon Hui's suggestion, we upgraded to mpich 3.4 and compiled with > '--with-device=ch4:ofi` flag (where ch4 is default for mpich 3.4). > Everything worked fine until we hit 20 processes; after >=20 processes, the > 2x slowdown is happening again. > > > > We have tried 1 communicator and multiple communicators in an attempt to > make the MPI implementation faster, but there's no significant difference > in observations. We are using MPI_Gather collector for merely calculating > the sum of the result of N processes, but we can't seem to maintain > stability within MPI as we increase N processes. Is there something we are > missing that is ultimately causing this error? We are at a loss here, > thank you. > > > > Best, > > Brent > > > > > >

1 0

Re: [mpich-devel] mpich3 error
by Thakur, Rajeev 16 Jan '21

16 Jan '21

Your mail all the way below says “We are using MPI_Gather collector for merely calculating the sum of the result of N processes”. Why don’t you use MPI_Reduce instead then? Rajeev From: Brent Morgan via devel <devel(a)mpich.org> Reply-To: "devel(a)mpich.org" <devel(a)mpich.org> Date: Saturday, January 16, 2021 at 1:38 PM To: "Zhou, Hui" <zhouh(a)anl.gov> Cc: Brent Morgan <brent.taylormorgan(a)gmail.com>, "devel(a)mpich.org" <devel(a)mpich.org>, Robert Katona <robert.katona(a)hotmail.com> Subject: Re: [mpich-devel] mpich3 error Hi Hui, Mpich community, Thanks for the response. You're right, I'll provide a toy program that replicates the code structure (and results). The toy program is calculating a sum value from each process- the value isn't too important for this toy program. The timing, however, is the only thing important in our demonstration. It exactly replicates what we are observing for our actual program. This directly relates to the MPI functionality- we can't find out what the issue is. [cid:[email protected]] I have attached the code. Is something wrong with our implementation? It starts with the main() function. Thank you very much for any help, Best, Brent PS My subscription to discuss(a)mpich.org<mailto:[email protected]> is pending currently. On Sat, Jan 16, 2021 at 12:24 PM Brent Morgan <brent.taylormorgan(a)gmail.com<mailto:[email protected]>> wrote: Hi Hui, Mpich community, Thanks for the response. You're right, I'll provide a toy program that replicates the code structure (and results). The toy program is calculating a sum value from each process- the value isn't too important for this toy program. The timing, however, is the only thing important in our demonstration. It exactly replicates what we are observing for our actual program. This directly relates to the MPI functionality- we can't find out what the issue is. [cid:[email protected]] I have attached the code. Is something wrong with our implementation? It starts with the main() function. Thank you very much for any help, Best, Brent On Fri, Jan 15, 2021 at 10:43 PM Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: Your description only mentions MPI_Gather. If there is indeed problem with MPI_Gather, then you should be able to reproduce the issue with a sample program. Share with us and we can better assist you. If you can’t reproduce the issue with a simple example, then I suspect there are other problems that you are not able to fully describe. We really can’t help much without able to see the code. That said, I am not even sure what is the issue you are describing. 100 process MPI_Gather will be slower than 50 process MPI_Gather. And since it is a collective, if one of your process is delayed due to some computations or else, the whole collective will take longer to finish just due to waiting for the late process. You really need tell us what your program is doing in order for us to even offer an intelligent guess. -- Hui Zhou From: Brent Morgan <brent.taylormorgan(a)gmail.com<mailto:[email protected]>> Date: Friday, January 15, 2021 at 10:42 PM To: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>, discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Robert Katona <robert.katona(a)hotmail.com<mailto:[email protected]>> Subject: Re: [mpich-devel] mpich3 error Hi MPICH community, My team has downloaded mpich 3.3.2 (using ch3 as default) and implemented MPI, and for small # of processes (<50), everything worked fine for our MPI implementation. For >=50 processes, there was a ch3 error and crashed the program after a random amount of seconds (sometimes 10seconds, sometimes 100seconds). So we compiled mpich 3.3.2 with ch4 (instead of default ch3) using '--with-device=ch4:ofi` flag and this got rid of the error- but for >12 processes, the speed would slow down to 2x slower suddenly. Upon Hui's suggestion, we upgraded to mpich 3.4 and compiled with '--with-device=ch4:ofi` flag (where ch4 is default for mpich 3.4). Everything worked fine until we hit 20 processes; after >=20 processes, the 2x slowdown is happening again. We have tried 1 communicator and multiple communicators in an attempt to make the MPI implementation faster, but there's no significant difference in observations. We are using MPI_Gather collector for merely calculating the sum of the result of N processes, but we can't seem to maintain stability within MPI as we increase N processes. Is there something we are missing that is ultimately causing this error? We are at a loss here, thank you. Best, Brent

1 0

Re: [mpich-devel] mpich3 error
by Brent Morgan 16 Jan '21

16 Jan '21

Hi Hui, Mpich community, Thanks for the response. You're right, I'll provide a toy program that replicates the code structure (and results). The toy program is calculating a sum value from each process- the value isn't too important for this toy program. The timing, however, is the only thing important in our demonstration. It exactly replicates what we are observing for our actual program. This directly relates to the MPI functionality- we can't find out what the issue is. [image: image.png] I have attached the code. Is something wrong with our implementation? It starts with the main() function. Thank you very much for any help, Best, Brent PS My subscription to discuss(a)mpich.org is pending currently. On Sat, Jan 16, 2021 at 12:24 PM Brent Morgan <brent.taylormorgan(a)gmail.com> wrote: > Hi Hui, Mpich community, > > Thanks for the response. You're right, I'll provide a toy program that > replicates the code structure (and results). The toy program is > calculating a sum value from each process- the value isn't too important > for this toy program. The timing, however, is the only thing important in > our demonstration. It exactly replicates what we are observing for our > actual program. This directly relates to the MPI functionality- we can't > find out what the issue is. > > [image: image.png] > I have attached the code. Is something wrong with our implementation? It > starts with the main() function. Thank you very much for any help, > > Best, > Brent > > On Fri, Jan 15, 2021 at 10:43 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > >> Your description only mentions MPI_Gather. If there is indeed problem >> with MPI_Gather, then you should be able to reproduce the issue with a >> sample program. Share with us and we can better assist you. If you can’t >> reproduce the issue with a simple example, then I suspect there are other >> problems that you are not able to fully describe. We really can’t help much >> without able to see the code. >> >> >> >> That said, I am not even sure what is the issue you are describing. 100 >> process MPI_Gather will be slower than 50 process MPI_Gather. And since it >> is a collective, if one of your process is delayed due to some computations >> or else, the whole collective will take longer to finish just due to >> waiting for the late process. You really need tell us what your program is >> doing in order for us to even offer an intelligent guess. >> >> >> >> -- >> Hui Zhou >> >> >> >> >> >> *From: *Brent Morgan <brent.taylormorgan(a)gmail.com> >> *Date: *Friday, January 15, 2021 at 10:42 PM >> *To: *Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> >> *Cc: *Robert Katona <robert.katona(a)hotmail.com> >> *Subject: *Re: [mpich-devel] mpich3 error >> >> Hi MPICH community, >> >> >> >> My team has downloaded mpich 3.3.2 (using ch3 as default) and implemented >> MPI, and for small # of processes (<50), everything worked fine for our MPI >> implementation. For >=50 processes, there was a ch3 error and crashed the >> program after a random amount of seconds (sometimes 10seconds, sometimes >> 100seconds). So we compiled mpich 3.3.2 with ch4 (instead of default ch3) >> using '--with-device=ch4:ofi` flag and this got rid of the error- but for >> >12 processes, the speed would slow down to 2x slower suddenly. >> >> >> >> Upon Hui's suggestion, we upgraded to mpich 3.4 and compiled with >> '--with-device=ch4:ofi` flag (where ch4 is default for mpich 3.4). >> Everything worked fine until we hit 20 processes; after >=20 processes, the >> 2x slowdown is happening again. >> >> >> >> We have tried 1 communicator and multiple communicators in an attempt to >> make the MPI implementation faster, but there's no significant difference >> in observations. We are using MPI_Gather collector for merely calculating >> the sum of the result of N processes, but we can't seem to maintain >> stability within MPI as we increase N processes. Is there something we are >> missing that is ultimately causing this error? We are at a loss here, >> thank you. >> >> >> >> Best, >> >> Brent >> >> >> >> >> >

1 0

Re: [mpich-devel] mpich3 error
by Brent Morgan 15 Jan '21

15 Jan '21

Hi MPICH dev team, We downloaded mpich 3.3.2 (using ch3 as default) and implemented MPI, and for small # of processes (<50), everything worked fine for our MPI implementation. For >=50 processes, there was a ch3 error and crashed the program after a random amount of seconds (sometimes 10seconds, sometimes 100seconds). So we compiled mpich 3.3.2 with ch4 (instead of default ch3) using '--with-device=ch4:ofi` flag and this got rid of the error- but for >12 processes, the speed would slow down to 2x slower suddenly. Upon Hui's suggestion, we upgraded to mpich 3.4 and compiled with '--with-device=ch4:ofi` flag (where ch4 is default for mpich 3.4). Everything worked fine until we hit 20 processes; after >=20 processes, the 2x slowdown is happening again. We have tried 1 communicator and multiple communicators in an attempt to make the MPI implementation faster, but there's no significant difference in observations. We are using MPI_Gather collector for merely calculating the sum of the result of N processes, but we can't seem to maintain stability within MPI as we increase N processes. Is there something we are missing that is ultimately causing this error? We are at a loss here, thank you. Best, Brent PS I am waiting for my subscription acceptance to discuss(a)mpich.org per Hui's suggestion. On Fri, Jan 15, 2021 at 2:57 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > Very glad to hear! > > > > For future reference, consider using discuss(a)mpich.org for usage > questions. We have a larger community there and you may receive faster or > wider helps. > > > > -- > Hui Zhou > > > > > > *From: *Brent Morgan <brent.taylormorgan(a)gmail.com> > *Date: *Friday, January 15, 2021 at 3:43 PM > *To: *Zhou, Hui <zhouh(a)anl.gov> > *Cc: *Robert Katona <robert.katona(a)hotmail.com> > *Subject: *Re: [mpich-devel] mpich3 error > > Hi Hui, > > > > We tried the latest release (3.4) for ch4, and the timing issue is > resolved and is error free so far- I am updating all the nodes still to > confirm for a large number of processes. > > > > Best, > > Brent > > > > On Fri, Jan 15, 2021 at 7:36 AM Zhou, Hui <zhouh(a)anl.gov> wrote: > > Please try with the latest release, for both ch3 and ch4 if you can. > > > > -- > Hui Zhou > > > > > > *From: *Brent Morgan <brent.taylormorgan(a)gmail.com> > *Date: *Friday, January 15, 2021 at 7:58 AM > *To: *Zhou, Hui <zhouh(a)anl.gov>, Robert Katona <robert.katona(a)hotmail.com> > *Subject: *Re: [mpich-devel] mpich3 error > > Hi Hui, > > > > We compiled with `--with-device=ch4:ofi' but now our program is 2x slower, > but it is error-free. > > 1) Is this slowdown of ch4 compared to ch3 expected? > > 2) it looks like there's also a '--with-device=ch4:ucx' flag, will this be > faster? Are there any other ch4 flags that might be faster? > > 3) does MPICH 3.4 have any speed benefit over 3.3.2 > > > > Thank you, this will be very helpful for us. > > > > Best, > > Brent > > > > > > On Fri, Jan 15, 2021 at 4:44 AM Brent Morgan <brent.taylormorgan(a)gmail.com> > wrote: > > Hi Hui, > > > > It looks like compiling MPICH with ch4, with `--with-device=ch4:ofi` , > solved the issue. If you can provide reasons why this may be, that would > be great. Thank you for the help with that syntax, > > > > Best, > > Brent > > > > > > > > On Fri, Jan 15, 2021 at 12:07 AM Brent Morgan < > brent.taylormorgan(a)gmail.com> wrote: > > Hi Hui, > > > > My apologies for confusion. We are calling a collective on 1 thread, for > many processes, so it sounds indeed like 1 communicator should suffice. We > are trying to compile with ch4 to resolve our issue. > > > > Our attempt of using 1 communicator vs. many communicators gave us the > same result, but the version with many communicators was a little faster. > > > > Best, > > Brent > > > > On Thu, Jan 14, 2021 at 10:52 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > > You are confusing processes with threads, or you are confusing me. > Concurrently refers the multiple threads, which you previously mentioned. > All collectives need multiple processes simultaneously participate of > course, thus the name collective. Lock only applies to multiple threads not > process. There is no restrictions are on the scale. You can have 1 million > processes in one communicator if needed. > > > > -- > Hui Zhou > > > > > > *From: *Brent Morgan <brent.taylormorgan(a)gmail.com> > *Date: *Thursday, January 14, 2021 at 11:46 PM > *To: *Zhou, Hui <zhouh(a)anl.gov> > *Subject: *Re: [mpich-devel] mpich3 error > > Hi Hui, > > Thanks for the response, > > "You can’t call collectives (MPI_Gather, MPI_Reduce, etc.) on the same > communicator concurrently. If you use the same communicator, add a > lock/critical section to serialize them. Or you can call collectives on > different communicators concurrently." > > > > We will try this. From what I understand, for large # of processes, it > sounds like MPI_Reduce (or any collective) should be done on smaller scales > serially (with a lock), or with different communicators concurrently. Is > this because a communicator can only handle so many processes? > > > > Best, > > Brent > > > > > > > > On Thu, Jan 14, 2021 at 10:36 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > > You can’t call collectives (MPI_Gather, MPI_Reduce, etc.) on the same > communicator concurrently. If you use the same communicator, add a > lock/critical section to serialize them. Or you can call collectives on > different communicators concurrently. > > > > The best and ultimate documentation is the MPI specification: > https://www.mpi-forum.org/docs/ > > > > -- > Hui Zhou > > > > > > *From: *Brent Morgan <brent.taylormorgan(a)gmail.com> > *Date: *Thursday, January 14, 2021 at 11:23 PM > *To: *Zhou, Hui <zhouh(a)anl.gov> > *Subject: *Re: [mpich-devel] mpich3 error > > Hi Hui, > > > > Thanks; so should we still be doing multiple communicators with All_Reduce > for large N processes? I can't find any documentation on the matter... > We'll be trying things to see if it works. thanks > > > > Best, > > Brent > > > > On Thu, Jan 14, 2021 at 10:11 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > > “Our MPI implementation is merely finding the sum of the results of > the N processes, where N is large. Is MPI_Reduce going to be faster? > > “ > > > > Oh, yeah, if you are doing reduce, you should call `MPI_Reduce`. > > > > However, I suspect there maybe some usage errors involved. Could you post > some sample/pseudo code? > > > > -- > Hui Zhou > > > >

1 0

Re: [mpich-devel] mpich3 error
by Raffenetti, Kenneth J. 15 Jan '21

15 Jan '21

On 1/14/21, 11:02 PM, "Brent Morgan via devel" <devel(a)mpich.org> wrote: Hi Hui Zhou, Robert and I managed to incorporate multiple communicators- we use MPI_Gather() of Send/Receives. However the issue remains- for a small (N<50) # of threads, the calculations work and seem fine. For large (N>=50) # of threads, the issue persists. We will try compiling for ch4 tonight... But I wonder if we're doing something wrong? Should we do Gather of Gathers? Our MPI implementation is merely finding the sum of the results of the N processes, where N is large. Is MPI_Reduce going to be faster? It should *never be slower* to use MPI_Reduce, and it is likely to be faster on most machines you run it on. Ken

1 0