- devel - mailman3.cels.anl.gov

Re: [mpich-devel] mpich3 error
by Brent Morgan 15 Jan '21

15 Jan '21

Hi MPICH dev team, We downloaded mpich 3.3.2 (using ch3 as default) and implemented MPI, and for small # of processes (<50), everything worked fine for our MPI implementation. For >=50 processes, there was a ch3 error and crashed the program after a random amount of seconds (sometimes 10seconds, sometimes 100seconds). So we compiled mpich 3.3.2 with ch4 (instead of default ch3) using '--with-device=ch4:ofi` flag and this got rid of the error- but for >12 processes, the speed would slow down to 2x slower suddenly. Upon Hui's suggestion, we upgraded to mpich 3.4 and compiled with '--with-device=ch4:ofi` flag (where ch4 is default for mpich 3.4). Everything worked fine until we hit 20 processes; after >=20 processes, the 2x slowdown is happening again. We have tried 1 communicator and multiple communicators in an attempt to make the MPI implementation faster, but there's no significant difference in observations. We are using MPI_Gather collector for merely calculating the sum of the result of N processes, but we can't seem to maintain stability within MPI as we increase N processes. Is there something we are missing that is ultimately causing this error? We are at a loss here, thank you. Best, Brent PS I am waiting for my subscription acceptance to discuss(a)mpich.org per Hui's suggestion. On Fri, Jan 15, 2021 at 2:57 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > Very glad to hear! > > > > For future reference, consider using discuss(a)mpich.org for usage > questions. We have a larger community there and you may receive faster or > wider helps. > > > > -- > Hui Zhou > > > > > > *From: *Brent Morgan <brent.taylormorgan(a)gmail.com> > *Date: *Friday, January 15, 2021 at 3:43 PM > *To: *Zhou, Hui <zhouh(a)anl.gov> > *Cc: *Robert Katona <robert.katona(a)hotmail.com> > *Subject: *Re: [mpich-devel] mpich3 error > > Hi Hui, > > > > We tried the latest release (3.4) for ch4, and the timing issue is > resolved and is error free so far- I am updating all the nodes still to > confirm for a large number of processes. > > > > Best, > > Brent > > > > On Fri, Jan 15, 2021 at 7:36 AM Zhou, Hui <zhouh(a)anl.gov> wrote: > > Please try with the latest release, for both ch3 and ch4 if you can. > > > > -- > Hui Zhou > > > > > > *From: *Brent Morgan <brent.taylormorgan(a)gmail.com> > *Date: *Friday, January 15, 2021 at 7:58 AM > *To: *Zhou, Hui <zhouh(a)anl.gov>, Robert Katona <robert.katona(a)hotmail.com> > *Subject: *Re: [mpich-devel] mpich3 error > > Hi Hui, > > > > We compiled with `--with-device=ch4:ofi' but now our program is 2x slower, > but it is error-free. > > 1) Is this slowdown of ch4 compared to ch3 expected? > > 2) it looks like there's also a '--with-device=ch4:ucx' flag, will this be > faster? Are there any other ch4 flags that might be faster? > > 3) does MPICH 3.4 have any speed benefit over 3.3.2 > > > > Thank you, this will be very helpful for us. > > > > Best, > > Brent > > > > > > On Fri, Jan 15, 2021 at 4:44 AM Brent Morgan <brent.taylormorgan(a)gmail.com> > wrote: > > Hi Hui, > > > > It looks like compiling MPICH with ch4, with `--with-device=ch4:ofi` , > solved the issue. If you can provide reasons why this may be, that would > be great. Thank you for the help with that syntax, > > > > Best, > > Brent > > > > > > > > On Fri, Jan 15, 2021 at 12:07 AM Brent Morgan < > brent.taylormorgan(a)gmail.com> wrote: > > Hi Hui, > > > > My apologies for confusion. We are calling a collective on 1 thread, for > many processes, so it sounds indeed like 1 communicator should suffice. We > are trying to compile with ch4 to resolve our issue. > > > > Our attempt of using 1 communicator vs. many communicators gave us the > same result, but the version with many communicators was a little faster. > > > > Best, > > Brent > > > > On Thu, Jan 14, 2021 at 10:52 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > > You are confusing processes with threads, or you are confusing me. > Concurrently refers the multiple threads, which you previously mentioned. > All collectives need multiple processes simultaneously participate of > course, thus the name collective. Lock only applies to multiple threads not > process. There is no restrictions are on the scale. You can have 1 million > processes in one communicator if needed. > > > > -- > Hui Zhou > > > > > > *From: *Brent Morgan <brent.taylormorgan(a)gmail.com> > *Date: *Thursday, January 14, 2021 at 11:46 PM > *To: *Zhou, Hui <zhouh(a)anl.gov> > *Subject: *Re: [mpich-devel] mpich3 error > > Hi Hui, > > Thanks for the response, > > "You can’t call collectives (MPI_Gather, MPI_Reduce, etc.) on the same > communicator concurrently. If you use the same communicator, add a > lock/critical section to serialize them. Or you can call collectives on > different communicators concurrently." > > > > We will try this. From what I understand, for large # of processes, it > sounds like MPI_Reduce (or any collective) should be done on smaller scales > serially (with a lock), or with different communicators concurrently. Is > this because a communicator can only handle so many processes? > > > > Best, > > Brent > > > > > > > > On Thu, Jan 14, 2021 at 10:36 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > > You can’t call collectives (MPI_Gather, MPI_Reduce, etc.) on the same > communicator concurrently. If you use the same communicator, add a > lock/critical section to serialize them. Or you can call collectives on > different communicators concurrently. > > > > The best and ultimate documentation is the MPI specification: > https://www.mpi-forum.org/docs/ > > > > -- > Hui Zhou > > > > > > *From: *Brent Morgan <brent.taylormorgan(a)gmail.com> > *Date: *Thursday, January 14, 2021 at 11:23 PM > *To: *Zhou, Hui <zhouh(a)anl.gov> > *Subject: *Re: [mpich-devel] mpich3 error > > Hi Hui, > > > > Thanks; so should we still be doing multiple communicators with All_Reduce > for large N processes? I can't find any documentation on the matter... > We'll be trying things to see if it works. thanks > > > > Best, > > Brent > > > > On Thu, Jan 14, 2021 at 10:11 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > > “Our MPI implementation is merely finding the sum of the results of > the N processes, where N is large. Is MPI_Reduce going to be faster? > > “ > > > > Oh, yeah, if you are doing reduce, you should call `MPI_Reduce`. > > > > However, I suspect there maybe some usage errors involved. Could you post > some sample/pseudo code? > > > > -- > Hui Zhou > > > >

1 0

Re: [mpich-devel] mpich3 error
by Raffenetti, Kenneth J. 15 Jan '21

15 Jan '21

On 1/14/21, 11:02 PM, "Brent Morgan via devel" <devel(a)mpich.org> wrote: Hi Hui Zhou, Robert and I managed to incorporate multiple communicators- we use MPI_Gather() of Send/Receives. However the issue remains- for a small (N<50) # of threads, the calculations work and seem fine. For large (N>=50) # of threads, the issue persists. We will try compiling for ch4 tonight... But I wonder if we're doing something wrong? Should we do Gather of Gathers? Our MPI implementation is merely finding the sum of the results of the N processes, where N is large. Is MPI_Reduce going to be faster? It should *never be slower* to use MPI_Reduce, and it is likely to be faster on most machines you run it on. Ken

1 0

Re: [mpich-devel] mpich3 error
by Zhou, Hui 14 Jan '21

14 Jan '21

“Our MPI implementation is merely finding the sum of the results of the N processes, where N is large. Is MPI_Reduce going to be faster? “ Oh, yeah, if you are doing reduce, you should call `MPI_Reduce`. However, I suspect there maybe some usage errors involved. Could you post some sample/pseudo code? -- Hui Zhou From: Brent Morgan <brent.taylormorgan(a)gmail.com> Date: Thursday, January 14, 2021 at 11:02 PM To: Robert Katona <robert.katona(a)hotmail.com> Cc: Zhou, Hui <zhouh(a)anl.gov>, devel(a)mpich.org <devel(a)mpich.org> Subject: Re: [mpich-devel] mpich3 error Hi Hui Zhou, Robert and I managed to incorporate multiple communicators- we use MPI_Gather() of Send/Receives. However the issue remains- for a small (N<50) # of threads, the calculations work and seem fine. For large (N>=50) # of threads, the issue persists. We will try compiling for ch4 tonight... But I wonder if we're doing something wrong? Should we do Gather of Gathers? Our MPI implementation is merely finding the sum of the results of the N processes, where N is large. Is MPI_Reduce going to be faster? Best, Brent On Thu, Jan 14, 2021 at 2:15 AM Robert Katona <robert.katona(a)hotmail.com<mailto:[email protected]>> wrote: Hi Hui Zhou, Thanks for the findings. You are right, we used one communicator. And my idea was also to recompile the MPICH with ch4. First I would like to try the MPI_Split() and use multiple communicator. But I stuck on the implementation. In the attachment you can see a toy app where all node want to share a float value with world rank 0. But when I run the app for 20 processes I got the following output: 0, 1, 2, 3, 4, 5, ,6, 7, 8, 9, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, But the expected is: 0, 1, 2, 3, 4, 5, ,6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, What am I missing? Regards, Robert ________________________________ Feladó: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Elküldve: 2021. január 13., szerda 17:39 Címzett: Robert Katona <robert.katona(a)hotmail.com<mailto:[email protected]>>; devel(a)mpich.org<mailto:[email protected]> <devel(a)mpich.org<mailto:[email protected]>> Másolatot kap: Brent Morgan <brent.taylormorgan(a)gmail.com<mailto:[email protected]>> Tárgy: Re: [mpich-devel] mpich3 error Hi Robert, Were you running MPI_Gather in multiple threads concurrently on a same communicator? That is not allowed. You’ll need at least different communicator for different threads. If that was not the issue, could you try compile MPICH with ch4, with `--with-device=ch4:ofi`, assuming you are using the latest release. -- Hui Zhou From: Robert Katona <robert.katona(a)hotmail.com<mailto:[email protected]>> Date: Wednesday, January 13, 2021 at 10:34 AM To: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>, devel(a)mpich.org<mailto:[email protected]> <devel(a)mpich.org<mailto:[email protected]>> Cc: Brent Morgan <brent.taylormorgan(a)gmail.com<mailto:[email protected]>> Subject: Re: [mpich-devel] mpich3 error Hello Hui Zhou, And can you help us in brainstorming? Unfortunately we cannot share the code. But I can give you more details. Couple of computers are connected into the same network and we would like to run a distributed calculation. During testing we run the MPI application over 50 job executer processes/threads and one job evaluter master/root thread. Each job calculate two float value what the master collects. To distribute this data we use the MPI_Gather() function. The execution of the problem runs fine but randomly stops with error message what Brent sent to you. Sometimes it stops after 10 iterative calculation, sometimes after 50 or 70. We do the same calculation, with same input, but the event of the error is very random. And it always stops in the MPI_Gather() function, when the master try to collect the data from all of the jobs. If there is a network issue with any of the computer can it produce this error? We are using Mpich 3.3.2, in the release note of 3.4 it is written the network communication has changed from ch3 to ch4. With ch4 can we expect better behavior? Can you give us any hint where to look and what to check in this topic? Regards, Robert Az Android Outlook<https://aka.ms/ghei36> letöltése ________________________________ Feladó: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Elküldve: 2021. január 13., szerda 15:51 Címzett: devel(a)mpich.org<mailto:[email protected]> Másolat: Brent Morgan; Robert Katona Tárgy: Re: [mpich-devel] mpich3 error Hi Brent, Unfortunately, unless you can provide us with a reproducer, there is little we can do to find the issue. -- Hui Zhou From: Brent Morgan via devel <devel(a)mpich.org<mailto:[email protected]>> Date: Wednesday, January 13, 2021 at 3:20 AM To: devel(a)mpich.org<mailto:[email protected]> <devel(a)mpich.org<mailto:[email protected]>> Cc: Brent Morgan <brent.taylormorgan(a)gmail.com<mailto:[email protected]>>, Robert Katona <robert.katona(a)hotmail.com<mailto:[email protected]>> Subject: [mpich-devel] mpich3 error Hello mpich dev support, I am receiving the following error in my MPI implementation, when I use 110+ threads. Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 567: !vc_ch->recv_active 0x7f7f870937 ??? ???:0 0x7f7f870957 ??? ???:0 0x7f7f8089cf ??? ???:0 0x7f7f854983 ??? ???:0 0x7f7f75a4f3 ??? ???:0 0x7f7f7cd8bf ??? ???:0 0x7f7f7cde0f ??? ???:0 0x7f7f78d577 ??? ???:0 0x7f7f6ea6cb ??? ???:0 0x7f7f6ea8f3 ??? ???:0 0x7f7f78c7ff ??? ???:0 0x7f7f6e9e6f ??? ???:0 0x7f7f6e9eb7 ??? ???:0 0x7f7f6ea033 ??? ???:0 0x55793fe47b ??? ???:0 0x55793f96b3 ??? ???:0 0x7f7f29908f ??? ???:0 0x55793f9a9b ??? ???:0 internal ABORT - process 171 Does this give a clue as to what I may be doing wrong? Thank you, Best, Brent

1 0

Re: [mpich-devel] mpich3 error
by Brent Morgan 14 Jan '21

14 Jan '21

Hi Hui Zhou, Robert and I managed to incorporate multiple communicators- we use MPI_Gather() of Send/Receives. However the issue remains- for a small (N<50) # of threads, the calculations work and seem fine. For large (N>=50) # of threads, the issue persists. We will try compiling for ch4 tonight... But I wonder if we're doing something wrong? Should we do Gather of Gathers? Our MPI implementation is merely finding the sum of the results of the N processes, where N is large. Is MPI_Reduce going to be faster? Best, Brent On Thu, Jan 14, 2021 at 2:15 AM Robert Katona <robert.katona(a)hotmail.com> wrote: > Hi Hui Zhou, > > Thanks for the findings. You are right, we used one communicator. And my > idea was also to recompile the MPICH with ch4. > > First I would like to try the MPI_Split() and use multiple communicator. > But I stuck on the implementation. In the attachment you can see a toy app > where all node want to share a float value with world rank 0. > > But when I run the app for 20 processes I got the following output: > 0, 1, 2, 3, 4, 5, ,6, 7, 8, 9, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > > But the expected is: > 0, 1, 2, 3, 4, 5, ,6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, > > What am I missing? > > Regards, > > Robert > > > > ------------------------------ > *Feladó:* Zhou, Hui <zhouh(a)anl.gov> > *Elküldve:* 2021. január 13., szerda 17:39 > *Címzett:* Robert Katona <robert.katona(a)hotmail.com>; devel(a)mpich.org < > devel(a)mpich.org> > *Másolatot kap:* Brent Morgan <brent.taylormorgan(a)gmail.com> > *Tárgy:* Re: [mpich-devel] mpich3 error > > > Hi Robert, > > > > Were you running MPI_Gather in multiple threads concurrently on a same > communicator? > > That is not allowed. You’ll need at least different communicator for > different threads. > > If that was not the issue, could you try compile MPICH with ch4, with > `--with-device=ch4:ofi`, assuming you are using the latest release. > > > > -- > Hui Zhou > > > > > > *From: *Robert Katona <robert.katona(a)hotmail.com> > *Date: *Wednesday, January 13, 2021 at 10:34 AM > *To: *Zhou, Hui <zhouh(a)anl.gov>, devel(a)mpich.org <devel(a)mpich.org> > *Cc: *Brent Morgan <brent.taylormorgan(a)gmail.com> > *Subject: *Re: [mpich-devel] mpich3 error > > Hello Hui Zhou, > > > > And can you help us in brainstorming? > > > > Unfortunately we cannot share the code. But I can give you more details. > > > > Couple of computers are connected into the same network and we would like > to run a distributed calculation. During testing we run the MPI application > over 50 job executer processes/threads and one job evaluter master/root > thread. Each job calculate two float value what the master collects. To > distribute this data we use the MPI_Gather() function. The execution of the > problem runs fine but randomly stops with error message what Brent sent to > you. > > > > Sometimes it stops after 10 iterative calculation, sometimes after 50 or > 70. We do the same calculation, with same input, but the event of the error > is very random. And it always stops in the MPI_Gather() function, when the > master try to collect the data from all of the jobs. > > > > If there is a network issue with any of the computer can it produce this > error? > > > > We are using Mpich 3.3.2, in the release note of 3.4 it is written the > network communication has changed from ch3 to ch4. With ch4 can we expect > better behavior? > > > > Can you give us any hint where to look and what to check in this topic? > > > > Regards, > > > > Robert > > > > Az Android Outlook <https://aka.ms/ghei36> letöltése > > > ------------------------------ > > *Feladó:* Zhou, Hui <zhouh(a)anl.gov> > *Elküldve:* 2021. január 13., szerda 15:51 > *Címzett:* devel(a)mpich.org > *Másolat:* Brent Morgan; Robert Katona > *Tárgy:* Re: [mpich-devel] mpich3 error > > > > Hi Brent, > > > > Unfortunately, unless you can provide us with a reproducer, there is > little we can do to find the issue. > > > > -- > Hui Zhou > > > > > > *From: *Brent Morgan via devel <devel(a)mpich.org> > *Date: *Wednesday, January 13, 2021 at 3:20 AM > *To: *devel(a)mpich.org <devel(a)mpich.org> > *Cc: *Brent Morgan <brent.taylormorgan(a)gmail.com>, Robert Katona < > robert.katona(a)hotmail.com> > *Subject: *[mpich-devel] mpich3 error > > Hello mpich dev support, > > I am receiving the following error in my MPI implementation, when I use > 110+ threads. > > Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3_progress.c > at line 567: !vc_ch->recv_active > 0x7f7f870937 ??? > ???:0 > 0x7f7f870957 ??? > ???:0 > 0x7f7f8089cf ??? > ???:0 > 0x7f7f854983 ??? > ???:0 > 0x7f7f75a4f3 ??? > ???:0 > 0x7f7f7cd8bf ??? > ???:0 > 0x7f7f7cde0f ??? > ???:0 > 0x7f7f78d577 ??? > ???:0 > 0x7f7f6ea6cb ??? > ???:0 > 0x7f7f6ea8f3 ??? > ???:0 > 0x7f7f78c7ff ??? > ???:0 > 0x7f7f6e9e6f ??? > ???:0 > 0x7f7f6e9eb7 ??? > ???:0 > 0x7f7f6ea033 ??? > ???:0 > 0x55793fe47b ??? > ???:0 > 0x55793f96b3 ??? > ???:0 > 0x7f7f29908f ??? > ???:0 > 0x55793f9a9b ??? > ???:0 > internal ABORT - process 171 > > Does this give a clue as to what I may be doing wrong? Thank you, > > > > Best, > > Brent > > >

1 0

Re: [mpich-devel] mpich3 error
by Zhou, Hui 13 Jan '21

13 Jan '21

Hi Robert, Were you running MPI_Gather in multiple threads concurrently on a same communicator? That is not allowed. You’ll need at least different communicator for different threads. If that was not the issue, could you try compile MPICH with ch4, with `--with-device=ch4:ofi`, assuming you are using the latest release. -- Hui Zhou From: Robert Katona <robert.katona(a)hotmail.com> Date: Wednesday, January 13, 2021 at 10:34 AM To: Zhou, Hui <zhouh(a)anl.gov>, devel(a)mpich.org <devel(a)mpich.org> Cc: Brent Morgan <brent.taylormorgan(a)gmail.com> Subject: Re: [mpich-devel] mpich3 error Hello Hui Zhou, And can you help us in brainstorming? Unfortunately we cannot share the code. But I can give you more details. Couple of computers are connected into the same network and we would like to run a distributed calculation. During testing we run the MPI application over 50 job executer processes/threads and one job evaluter master/root thread. Each job calculate two float value what the master collects. To distribute this data we use the MPI_Gather() function. The execution of the problem runs fine but randomly stops with error message what Brent sent to you. Sometimes it stops after 10 iterative calculation, sometimes after 50 or 70. We do the same calculation, with same input, but the event of the error is very random. And it always stops in the MPI_Gather() function, when the master try to collect the data from all of the jobs. If there is a network issue with any of the computer can it produce this error? We are using Mpich 3.3.2, in the release note of 3.4 it is written the network communication has changed from ch3 to ch4. With ch4 can we expect better behavior? Can you give us any hint where to look and what to check in this topic? Regards, Robert Az Android Outlook<https://aka.ms/ghei36> letöltése ________________________________ Feladó: Zhou, Hui <zhouh(a)anl.gov> Elküldve: 2021. január 13., szerda 15:51 Címzett: devel(a)mpich.org Másolat: Brent Morgan; Robert Katona Tárgy: Re: [mpich-devel] mpich3 error Hi Brent, Unfortunately, unless you can provide us with a reproducer, there is little we can do to find the issue. -- Hui Zhou From: Brent Morgan via devel <devel(a)mpich.org> Date: Wednesday, January 13, 2021 at 3:20 AM To: devel(a)mpich.org <devel(a)mpich.org> Cc: Brent Morgan <brent.taylormorgan(a)gmail.com>, Robert Katona <robert.katona(a)hotmail.com> Subject: [mpich-devel] mpich3 error Hello mpich dev support, I am receiving the following error in my MPI implementation, when I use 110+ threads. Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 567: !vc_ch->recv_active 0x7f7f870937 ??? ???:0 0x7f7f870957 ??? ???:0 0x7f7f8089cf ??? ???:0 0x7f7f854983 ??? ???:0 0x7f7f75a4f3 ??? ???:0 0x7f7f7cd8bf ??? ???:0 0x7f7f7cde0f ??? ???:0 0x7f7f78d577 ??? ???:0 0x7f7f6ea6cb ??? ???:0 0x7f7f6ea8f3 ??? ???:0 0x7f7f78c7ff ??? ???:0 0x7f7f6e9e6f ??? ???:0 0x7f7f6e9eb7 ??? ???:0 0x7f7f6ea033 ??? ???:0 0x55793fe47b ??? ???:0 0x55793f96b3 ??? ???:0 0x7f7f29908f ??? ???:0 0x55793f9a9b ??? ???:0 internal ABORT - process 171 Does this give a clue as to what I may be doing wrong? Thank you, Best, Brent

1 0

Re: [mpich-devel] mpich3 error
by Zhou, Hui 13 Jan '21

13 Jan '21

Hi Brent, Unfortunately, unless you can provide us with a reproducer, there is little we can do to find the issue. -- Hui Zhou From: Brent Morgan via devel <devel(a)mpich.org> Date: Wednesday, January 13, 2021 at 3:20 AM To: devel(a)mpich.org <devel(a)mpich.org> Cc: Brent Morgan <brent.taylormorgan(a)gmail.com>, Robert Katona <robert.katona(a)hotmail.com> Subject: [mpich-devel] mpich3 error Hello mpich dev support, I am receiving the following error in my MPI implementation, when I use 110+ threads. Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 567: !vc_ch->recv_active 0x7f7f870937 ??? ???:0 0x7f7f870957 ??? ???:0 0x7f7f8089cf ??? ???:0 0x7f7f854983 ??? ???:0 0x7f7f75a4f3 ??? ???:0 0x7f7f7cd8bf ??? ???:0 0x7f7f7cde0f ??? ???:0 0x7f7f78d577 ??? ???:0 0x7f7f6ea6cb ??? ???:0 0x7f7f6ea8f3 ??? ???:0 0x7f7f78c7ff ??? ???:0 0x7f7f6e9e6f ??? ???:0 0x7f7f6e9eb7 ??? ???:0 0x7f7f6ea033 ??? ???:0 0x55793fe47b ??? ???:0 0x55793f96b3 ??? ???:0 0x7f7f29908f ??? ???:0 0x55793f9a9b ??? ???:0 internal ABORT - process 171 Does this give a clue as to what I may be doing wrong? Thank you, Best, Brent

1 0

mpich3 error
by Brent Morgan 13 Jan '21

13 Jan '21

Hello mpich dev support, I am receiving the following error in my MPI implementation, when I use 110+ threads. Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 567: !vc_ch->recv_active 0x7f7f870937 ??? ???:0 0x7f7f870957 ??? ???:0 0x7f7f8089cf ??? ???:0 0x7f7f854983 ??? ???:0 0x7f7f75a4f3 ??? ???:0 0x7f7f7cd8bf ??? ???:0 0x7f7f7cde0f ??? ???:0 0x7f7f78d577 ??? ???:0 0x7f7f6ea6cb ??? ???:0 0x7f7f6ea8f3 ??? ???:0 0x7f7f78c7ff ??? ???:0 0x7f7f6e9e6f ??? ???:0 0x7f7f6e9eb7 ??? ???:0 0x7f7f6ea033 ??? ???:0 0x55793fe47b ??? ???:0 0x55793f96b3 ??? ???:0 0x7f7f29908f ??? ???:0 0x55793f9a9b ??? ???:0 internal ABORT - process 171 Does this give a clue as to what I may be doing wrong? Thank you, Best, Brent

1 0

Re: [mpich-devel] [mpich-discuss] Change in SOVERSION of mpich 3.4 ?
by Raffenetti, Kenneth J. 06 Jan '21

06 Jan '21

Hi, We did not bump the major version because MPIX_Mutex interfaces were "experimental" in nature. Our position is that we update the ABI version based on official MPI standard functionality only. Ken On 1/6/21, 3:55 AM, "Alastair McKinstry via discuss" <discuss(a)mpich.org> wrote: Dear MPICH developers, I see that MPICH 3.4 has been released - Happy New Year and congratulations to all involved. I'm one of the Debian Maintainers of MPICH, and have a question though : should the SOMAJOR version not have been increased? From the CHANGES log, the following look concerning: # Change the signature of MPID_Init to include requested and provided thread levels. # Remove support for logical reduction operations on floating point types. # Remove MPIX_Mutex interfaces. Typically when functionality is removed from a library, or signatures change, the major version is incremented (the minor version can be for additions, such as the other functionality added. Are these changes purely internal to MPICH, or has the ABI changed ? This matters to the distributions such as Debian because we ship the shared libs based on SOVERSION; ie the binary package "libmpich12" currently contains libmpich.so.12.18 and libmpich.so.12 , and if mpich 3.4 is packaged as-is, will contain libmpich.so.12.1.9 (ansd symlink libmpich.so.12) and it is necessary for applications to be able to use this as a drop-in replacement. On seeing these changes in the release logs, I had been anticipating that the next version would be libmpich13.0.0 (or similar), shipped in "libmpich13". This enables upgrades; both libmpich12 and libmpich13 can co-exist. Can you confirm whether the ABI changes, and this is a drop-in replacement ? Regards Alastair McKinstry -- Alastair McKinstry, <alastair(a)sceal.ie>, <mckinstry(a)debian.org>, https://diaspora.sceal.ie/u/amckinstry Misentropy: doubting that the Universe is becoming more disordered. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-devel] C99 fixed-width integer types
by Jeff Hammond 13 Aug '20

13 Aug '20

FWIW while MSVC may not support restrict, it supports __restrict [1], which serves the same purpose. It would be straightforward to define a macro MPICH_RESTRICT if you want to use that. [1] https://docs.microsoft.com/en-us/cpp/cpp/extension-restrict?view=vs-2019 Jeff On Thu, Aug 13, 2020 at 1:40 PM Raffenetti, Kenneth J. via devel < devel(a)mpich.org> wrote: > Hi everyone, > > Following a discussion today about C99 types in MPICH, we wanted to give a > heads up. Since version 3.3, MPICH relies on many types which are > technically optional in C99. For example uint64_t, uintptr_t, etc. In case > it was not clear, platforms which do not provide these optional types are > not supported by MPICH. > > The MPICH wiki page on C99 support has been updated to include this > caveat: https://wiki.mpich.org/mpich/index.php/Shifting_toward_C99 > > Please let us know if you have any questions, comments, or concerns. > > Thanks, > Ken > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

C99 fixed-width integer types
by Raffenetti, Kenneth J. 13 Aug '20

13 Aug '20

Hi everyone, Following a discussion today about C99 types in MPICH, we wanted to give a heads up. Since version 3.3, MPICH relies on many types which are technically optional in C99. For example uint64_t, uintptr_t, etc. In case it was not clear, platforms which do not provide these optional types are not supported by MPICH. The MPICH wiki page on C99 support has been updated to include this caveat: https://wiki.mpich.org/mpich/index.php/Shifting_toward_C99 Please let us know if you have any questions, comments, or concerns. Thanks, Ken

1 0