- devel - mailman3.cels.anl.gov

Join the MPICH team at ECP Community BoF Days next week!
by Raffenetti, Ken 06 May '22

06 May '22

All, The MPICH team will host a session as part of next week's ECP Community BoF Days (https://www.exascaleproject.org/event/ecp-community-bof-days-2022/) We will discuss MPICH release updates as well as hear from some of our partners. The event is virtual and registration is free at this link: https://exascaleproject.zoomgov.com/meeting/register/vJIsduuppjosHSWqb9g4oo… Date and time: May 11, 2022 12:00-1:30 PM Central Agenda (subject to change): 12:00-12:05: Intro - Ken Raffenetti (ANL) 12:05-12:35: MPICH Update - Yanfei Guo & Hui Zhou (ANL) 12:35-12:50: Intel Update - Gengbin Zheng (Intel) 12:50-13:05: MVAPICH2 Update - Hari Subramoni (OSU) 13:05-13:20: Pilgrim* Update - Chen Wang (UIUC) 13:20-13:30: Q&A This information is also available on our wiki: https://github.com/pmodels/mpich/wiki/2022-ECP-Community-BoF Cheers, The MPICH team *Pilgrim: https://github.com/pmodels/pilgrim

1 0

Slides from SC21 MPICH BoF posted
by Raffenetti, Ken 30 Nov '21

30 Nov '21

Hi all, Slides from the SC21 MPICH BoF have been posted to the MPICH Github wiki. https://github.com/pmodels/mpich/wiki/2021-Supercomputing-BoF#presentations Additionally, the session recording is available to SC21 attendees via the Hubb platform for a limited time. https://sc21.hubb.me/fe/schedule-builder/sessions/879718 Once again, thanks to all the presenters and the MPICH community for participating! Cheers, The MPICH team

1 0

Join the MPICH BoF this evening at SC21
by Raffenetti, Ken 17 Nov '21

17 Nov '21

All, A reminder that the MPICH team will host a virtual Birds-of-a-Feather session tonight at 5:15pm CST as part of the SC21 proceedings. We hope to see you there to discuss the latest developments in MPICH, and to hear from partners Intel, HPE, ParTec, MVAPICH, and Cornelis Networks. See the link below for join info! https://sc21.hubb.me/fe/schedule-builder/sessions/879718 Cheers, The MPICH team

1 0

Join the MPICH team at SC21
by Raffenetti, Ken 11 Nov '21

11 Nov '21

The MPICH team will participate is several sessions at next week's SC21 conference. We hope you can join us! A FACT-Based Approach: Making ML Collective Autotuning Feasible on Exascale Systems ExaMPI Workshop Paper Presentation Sunday, 14 November 2021 2:30pm - 3pm CST https://sc21.supercomputing.org/presentation/?id=ws_exampi101&sess=sess430 Advanced MPI Programming Tutorial Monday, 15 November 20218am - 5pm CST https://sc21.supercomputing.org/presentation/?id=tut120&sess=sess204 Parallel I/O In Practice Tutorial Monday, 15 November 2021 8am - 5pm CST https://sc21.supercomputing.org/?post_type=page&p=3480&sess=sess203 UCX: Unified Communication X Community Birds of a Feather Tuesday, 16 November 2021 5:15pm - 6:45pm CST https://sc21.supercomputing.org/presentation/?id=bof107&sess=sess355 Runtime Systems for Zettascale Panel Wednesday, 17 November 2021 10:30am - 12pm CST https://sc21.supercomputing.org/presentation/?sess=sess228&id=pan116#038;id… Pilgrim: Scalable and (Near) Lossless MPI Tracing Paper Presentation Wednesday, 17 November 20211:30pm - 2pm CST https://sc21.supercomputing.org/presentation/?sess=sess147&id=pap177#038;id… MPICH: A High Performance Open-Source MPI Implementation Birds of a Feather Wednesday, 17 November 20215:15pm - 6:45pm CST https://sc21.supercomputing.org/presentation/?sess=sess354&id=bof105#038;id…

1 0

Re: [mpich-devel] proposed changes to function enter/exit logging
by Raffenetti, Kenneth J. 04 Aug '21

04 Aug '21

Hi all, I am digging up this old thread to call attention to another PR that proposes changes to the function enter and exit macros. https://github.com/pmodels/mpich/pull/5479 The changes in this new PR are less radical than #4276. It would keep enter/exit macros in the code, but simplify their use. This would help prevent common errors/bugs associate with them. Please let us know if you have thoughts! We would like to merge these for the upcoming 4.0 series. Thanks, Ken On 1/16/20, 10:15 AM, "Raffenetti, Kenneth J." <raffenet(a)mcs.anl.gov> wrote: [UPDATE] An alternative approach to the function enter/exit logging changes has been posted here: https://github.com/pmodels/mpich/pull/4276 The previous -finstrument-functions PR only supported Linux for address->func_name conversion in log files, which was problematic for some. The new approach uses scripts to add/remove the enter/exit macros on-demand. The benefit is that it will work on all systems, and be less error-prone than manually adding the macros by hand. Please comment on the PR or this thread if you have thoughts. Thanks, Ken On 11/7/19 9:35 AM, Raffenetti, Kenneth J. via devel wrote: > I should also note that currently the address->function translation > scripts will only work on Linux. > > Ken > > On 11/7/19 8:30 AM, Ken Raffenetti wrote: >> Here's what I've found so far: >> >> XL compilers support -qfunctrace with slightly different user-supplied >> tracing function signatures. Should be trivial to add support to MPICH. >> >> https://www.ibm.com/support/knowledgecenter/SSGH2K_12.1.0/com.ibm.xlc121.ai… >> >> >> PGI supports -Minstrument with the same tracing function signatures as >> GCC and Clang. >> >> Intel supports -finstrument-functions. >> >> I do not see any support in Oracle suncc. >> >> Ken >> >> On 11/7/19 4:08 AM, Finkel, Hal J. via devel wrote: >>> Clang does support -finstrument-functions (and also provides >>> -finstrument-functions-after-inlining and XRay-based options: >>> https://llvm.org/docs/XRay.html) >>> >>> -Hal >>> >>> On 11/6/19 6:22 PM, Jeff Hammond via devel wrote: >>>> Do you have a list of what compilers support this? Is this just GCC, >>>> just GCC and Clang, or a bigger list? If Clang supports it, do you >>>> know if all of the Clang derivatives (e.g. IBM XLC) support it? >>>> >>>> Thanks, >>>> >>>> Jeff >>>> >>>> On Wed, Nov 6, 2019 at 2:04 PM Raffenetti, Kenneth J. via devel >>>> <devel(a)mpich.org <mailto:[email protected]>> wrote: >>>> >>>> Folks, >>>> >>>> I want to draw attention to a pull request posted to the MPICH >>>> github. >>>> >>>> https://github.com/pmodels/mpich/pull/4139 >>>> >>>> This proposes to remove all MPIR_FUNC_VERBOSE_ENTER and >>>> MPIR_FUNC_VERBOSE_EXIT macros from MPICH and instead use compiler >>>> function instrumentation when configured for debug logging: >>>> https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html. >>>> >>>> There are 2 main benefits: >>>> >>>> 1. De-clutter the code. PR stats show a net deletion of >10,000 >>>> lines! >>>> 2. Less error-prone. The compiler will reliably instrument *all* >>>> functions, and will not suffer from copy/paste errors or typos. >>>> >>>> Drawbacks: >>>> >>>> 1. Requires compiler support for instrumentation. May affect users >>>> debugging certain compiler configurations. >>>> 2. Compiler function instrumentation logs function addresses, not >>>> symbols. An additional step is needed to convert addresses to >>>> function >>>> names via script (included in PR). >>>> >>>> What do people think? Will these changes affect anyone using debug >>>> logging in any regular capacity? We're looking for feedback good >>>> or bad. >>>> Let us know. >>>> >>>> Thanks, >>>> Ken >>>> _______________________________________________ >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/devel >>>> >>>> >>>> >>>> -- >>>> Jeff Hammond >>>> jeff.science(a)gmail.com <mailto:[email protected]> >>>> http://jeffhammond.github.io/ >>>> >>>> _______________________________________________ >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/devel >>> -- >>> Hal Finkel >>> Lead, Compiler Technology and Programming Languages >>> Leadership Computing Facility >>> Argonne National Laboratory >>> >>> >>> _______________________________________________ >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/devel >>> > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel >

1 0

ECP Community BOF Days: MPICH for Exascale: Supporting MPI-4 and ECP
by Raffenetti, Kenneth J. 24 Mar '21

24 Mar '21

Hi everyone, The MPICH team will host a Birds of a Feather session during next week's ECP Community BoF Days. This virtual event will run similar to our annual MPICH BoF at Supercomputing. Come join us to learn about developments in MPICH and its derivatives. Registration is free! Wednesday, March 31st 1:00pm Eastern Registration link: https://exascaleproject.zoomgov.com/meeting/register/vJItduqrpjMsGGTTf2_YAV… More info: https://github.com/pmodels/mpich/wiki/2021-ECP-Community-BoF Best regards, The MPICH Team

1 0

(no subject)
by Lisandro Dalcin 08 Mar '21

08 Mar '21

I'm entangled in a discussion with Open MPI folks about whether the struct datatype constructor should or should not add padding depending on the oldtypes having or not explicit lb/ub markers. https://github.com/open-mpi/ompi/issues/8560 Please see the attached example. It basically creates a simple struct datatype with type signature {unsigned short, unsigned char} in different ways (which I would expect to be all equivalent). Running it with latest MPICH 3.4.1 $ for i in `seq 0 5`; do mpicc -DCASE=$i align.c; ./a.out; done c-sizeof=4 mpi-extent=4 c-sizeof=4 mpi-extent=4 c-sizeof=4 mpi-extent=4 c-sizeof=4 mpi-extent=4 c-sizeof=4 mpi-extent=4 c-sizeof=4 mpi-extent=4 For Open MPI, up to v4.0.x, that was the case, too. However, with latest Open 4.1.0, now I get: $ for i in `seq 0 5`; do mpicc -DCASE=$i align.c; ./a.out; done c-sizeof=4 mpi-extent=4 c-sizeof=4 mpi-extent=4 c-sizeof=4 mpi-extent=4 c-sizeof=4 mpi-extent=4 c-sizeof=4 mpi-extent=3 c-sizeof=4 mpi-extent=2 Open MPI folks claim that their behavior is correct and in line with the MPI standard. Maybe they have a point regarding the ub/lb definition in page 105 of MPI-3.1, despite such behavior being quite unintuitive. On the other hand, MPICH's struct datatype constructor always adjusts the ub to account for padding. Open MPI obviously don't. I don't see any wording in the standard where this could be implementation-defined behavior. If Open MPI folks are right, then MPICH is wrong, or the other way around. But we cannot have the two major competing implementations disagree on this. Regards, -- Lisandro Dalcin ============ Senior Research Scientist Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/

1 0

Re: [mpich-devel] [mpich-discuss] Fatal error in PMPI_Isend: Unknown error class
by Raffenetti, Kenneth J. 19 Feb '21

19 Feb '21

On 2/18/21, 1:03 AM, "Brent Morgan via discuss" <discuss(a)mpich.org> wrote: all the questions we've submitted, none of them were answered or considered at all in neither discuss(a)mpich.org or devel(a)mpich.org <mailto:[email protected]> Can you let us know which emails have gone unanswered? I see that aside from this email which went unanswered for 4 days, the 2 previous questions you posted to discuss and devel had some back and forth. Maybe we are missing some other questions? As others have already said, sample code and/or minimal reproducers would be a sure way to accelerate productive responses. Ken

1 0

Re: [mpich-devel] Fatal error in PMPI_Isend: Unknown error class
by Thakur, Rajeev 18 Feb '21

18 Feb '21

In general, testing your code with more than one MPI implementation (any of MPICH, Open MPI, MVAPICH, Intel MPI, …) will help identify whether the bug is in the user code or the MPI implementation. Rajeev From: Brent Morgan via devel <devel(a)mpich.org> Reply-To: "devel(a)mpich.org" <devel(a)mpich.org> Date: Thursday, February 18, 2021 at 1:02 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org>, "devel(a)mpich.org" <devel(a)mpich.org>, Robert Katona <robert.katona(a)hotmail.com> Cc: Brent Morgan <brent.taylormorgan(a)gmail.com> Subject: Re: [mpich-devel] Fatal error in PMPI_Isend: Unknown error class Hi all, As usual, we resolved our own issue. It turns out there was an overloading of MPI_Isend/MPI_Irecv with sending/receiving a vector of ~100 elements. We improved this by playing with the grid configuration of sub-communicators, but that failed further down the road. We just switched to mpi_send/mpi_recv which we found to be more robust, but our experiments show this is slower (for reasons that differ between MPI_send and MPI_isend). The reason why we are using send/recv and not BCAST, is because BCAST led to undesired timing behavior. It turns out (that we learned on our own), that broadcast sends data down a tree-like arrangement of ranks (when messages are small and communication time is dominated by network latency) or down a ring of ranks (with large messages). If a rank on an intermediate tree level is late, it delays the broadcast to all other ranks beneath it in the sub-tree. Similarly, in the ring arrangement, a late rank delays all ranks after it in the ring. This can lead to delays in subgroups of ranks if the ranks are not aligned in time (like it is for us). Our team is wondering why there isn't any support on these email discussion boards- is this the official place to get help for MPICH? Are there any other places like github discussions? I would imagine NASA or Argonne National Labs or other government laboratories are needing to communicate issues that arise, or need to send/receive much more data than a vector of 100 elements without issues. The more we implement MPICH on our end, the less confident we are about this mpi implementation. Our errors we have observed have no documented error codes or error traces; we have encountered many issues thus far (and reported them, with no responses)- which led to us scanning mpich source code and trying random things to arrive at solutions, which is far from ideal in any software development environment. We've arrived in a position that if we continue to experience more issues, we are being forced to switch to another implementation (openmpi) because of all the questions we've submitted, none of them were answered or considered at all in neither discuss(a)mpich.org<mailto:[email protected]> or devel(a)mpich.org- and I see others' questions go unanswered as well. We don't understand why there isn't any documentation (at all) on some things we've encountered (and others as well). Best, Brent On Sun, Feb 14, 2021 at 12:20 AM Brent Morgan <brent.taylormorgan(a)gmail.com<mailto:[email protected]>> wrote: Hello all, I am seeing this: "Abort(105) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Isend: Unknown error class" This is after my float vector I am sending to the nodes has increased to a size of ~70 elements. I cannot find any documentation about what this means. I tracked down where this fails and upon sending it to the 300th process (of 600), the MPI_ISend() command dies and this error shows. Is there anything I can do to further diagnose the issue? Best, Brent

1 0

Re: [mpich-devel] Fatal error in PMPI_Isend: Unknown error class
by Zhou, Hui 18 Feb '21

18 Feb '21

Dear Brent Morgan, I am glad you resolved your own issue. But I guess you are having the similar after taste that your resolution is less than ideal. Is that the reason for your ranting? It appears the original issue was the MPI_Bcast algorithm, that due to your particular situation, neither the tree algorithm or ring algorithm are not performing? For the record, this is the first time I get a clue on where the issue is at. I recall the past emails were just complaining Isend and Irecv. There are two key steps before a good communication or resolution can happen. First, all the key information need be communicated. I am not shifting the blame to users. This can be very difficult especially when we don’t know what information is critical. Very difficult or impossible before we understand what is the issue – chicken and egg problem. So the best we can do for one, provide as much details as we (means you) think is relevant, and for two, keep finding new details that we omitted, and for three, be understandable. The second key step is, of course, we (whoever you are asking for help from) need know the solution. Trust me, if we *understand* the issue and *know* the solution, we can’t wait to help you. But when we don’t understand your issue (due to lack of information or probes) or don’t know the solution (research problems), you may feel that we are completely ignoring your ask for help. What response would you suggest (when we have no clue)? -- Hui Zhou From: Brent Morgan via devel <devel(a)mpich.org> Date: Thursday, February 18, 2021 at 1:02 AM To: discuss(a)mpich.org <discuss(a)mpich.org>, devel(a)mpich.org <devel(a)mpich.org>, Robert Katona <robert.katona(a)hotmail.com> Cc: Brent Morgan <brent.taylormorgan(a)gmail.com> Subject: Re: [mpich-devel] Fatal error in PMPI_Isend: Unknown error class Hi all, As usual, we resolved our own issue. It turns out there was an overloading of MPI_Isend/MPI_Irecv with sending/receiving a vector of ~100 elements. We improved this by playing with the grid configuration of sub-communicators, but that failed further down the road. We just switched to mpi_send/mpi_recv which we found to be more robust, but our experiments show this is slower (for reasons that differ between MPI_send and MPI_isend). The reason why we are using send/recv and not BCAST, is because BCAST led to undesired timing behavior. It turns out (that we learned on our own), that broadcast sends data down a tree-like arrangement of ranks (when messages are small and communication time is dominated by network latency) or down a ring of ranks (with large messages). If a rank on an intermediate tree level is late, it delays the broadcast to all other ranks beneath it in the sub-tree. Similarly, in the ring arrangement, a late rank delays all ranks after it in the ring. This can lead to delays in subgroups of ranks if the ranks are not aligned in time (like it is for us). Our team is wondering why there isn't any support on these email discussion boards- is this the official place to get help for MPICH? Are there any other places like github discussions? I would imagine NASA or Argonne National Labs or other government laboratories are needing to communicate issues that arise, or need to send/receive much more data than a vector of 100 elements without issues. The more we implement MPICH on our end, the less confident we are about this mpi implementation. Our errors we have observed have no documented error codes or error traces; we have encountered many issues thus far (and reported them, with no responses)- which led to us scanning mpich source code and trying random things to arrive at solutions, which is far from ideal in any software development environment. We've arrived in a position that if we continue to experience more issues, we are being forced to switch to another implementation (openmpi) because of all the questions we've submitted, none of them were answered or considered at all in neither discuss(a)mpich.org<mailto:[email protected]> or devel(a)mpich.org- and I see others' questions go unanswered as well. We don't understand why there isn't any documentation (at all) on some things we've encountered (and others as well). Best, Brent On Sun, Feb 14, 2021 at 12:20 AM Brent Morgan <brent.taylormorgan(a)gmail.com<mailto:[email protected]>> wrote: Hello all, I am seeing this: "Abort(105) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Isend: Unknown error class" This is after my float vector I am sending to the nodes has increased to a size of ~70 elements. I cannot find any documentation about what this means. I tracked down where this fails and upon sending it to the 300th process (of 600), the MPI_ISend() command dies and this error shows. Is there anything I can do to further diagnose the issue? Best, Brent

1 0