September 2023 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] libmpi size unusually large when building with CUDA support
by Lucas Zacchi de Medeiros 18 Sep '23

18 Sep '23

Thanks for your answer Jeff. I’m, going to double check the ROCM5 build later on and get back with the results, but for now the build commands used for the cuda version are: ./configure --prefix=${install_dir}/ucx_1.14.1 --without-bfd --with-cuda= nvidia/Linux_x86_64/21.2/cuda/11.2/ --without-knem --without-rocm --enable-gtest --enable-examples make -j16 make install I see your comment on github (https://github.com/pmodels/mpich/issues/6675) as well, I’ll change the value of if-yaksa-depth and check the results. Thank you for you insight on that. Best, Lucas. From: Jeff Hammond <jeff.science(a)gmail.com> Sent: Monday, 18 September 2023 13:26 To: discuss(a)mpich.org Cc: Lucas Zacchi de Medeiros <lucasd(a)cadence.com> Subject: Re: [mpich-discuss] libmpi size unusually large when building with CUDA support EXTERNAL MAIL If the CPU-only and ROCm builds are the same size, I wonder if ROCm support was compiled in at all. How did you verify that? Can you provide the full build commands for each so I can reproduce these? You might see if -yaksa-depth=1 (or 0 perhaps) changes the results. There is a performance tradeoff, but if your applications don't use interesting noncontiguous datatypes, it shouldn't matter. Jeff On Fri, Sep 15, 2023 at 5:53 PM Lucas Zacchi de Medeiros via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: I work on a project that supports different architectures, so we've built 3 separate versions of MPICH 4.1.1 (cuda11.2, rocm5.4.3 and cpu-only with ucx1.14.1) The cpu-only and the rocm5 libraries are around 50MB each. The CUDA version on the other hand, is close to 1.8GB! There doesn’t seem to be anything out of the ordinary with the builds and all versions are working as expected. This is the output of du -sh performed on the install directories: First the rocm5 build: $ du -sh LINUX_gcc9.3_glibc2.28_rocm5.4.3_ucx1.14.1/lib/* (...) 53M LINUX_gcc9.3_glibc2.28_rocm5.4.3_ucx1.14.1/lib/libmpi.a 40M LINUX_gcc9.3_glibc2.28_rocm5.4.3_ucx1.14.1/lib/libmpi.so.12.3.0 And then the CUDA build: $ du -sh LINUX_gcc9.3_glibc2.17_cuda11.2_ucx1.14.1/lib/* (...) 1.8G LINUX_gcc9.3_glibc2.17_cuda11.2_ucx1.14.1/lib/libmpi.a 1.7G LINUX_gcc9.3_glibc2.17_cuda11.2_ucx1.14.1/lib/libmpi.so.12.3.0 This issue really complicates packaging and distribution since we provide both the archive and the shared libraries, I am looking at more than 3GB for MPICH only. After some investigation, it doesn’t seem the problem is coming from our end. Is it possible that something in mpich’s build process is causing this excessive file size? Thanks for the help. Kind regards, Lucas _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss<https://urldefense.com/v3/__https:/lists.mpich.org/mailman/listinfo/discuss…> -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/<https://urldefense.com/v3/__http:/jeffhammond.github.io/__;!!EHscmS1ygiU1lA…>

1 0

Re: [mpich-discuss] libmpi size unusually large when building with CUDA support
by Jeff Hammond 18 Sep '23

18 Sep '23

If the CPU-only and ROCm builds are the same size, I wonder if ROCm support was compiled in at all. How did you verify that? Can you provide the full build commands for each so I can reproduce these? You might see if -yaksa-depth=1 (or 0 perhaps) changes the results. There is a performance tradeoff, but if your applications don't use interesting noncontiguous datatypes, it shouldn't matter. Jeff On Fri, Sep 15, 2023 at 5:53 PM Lucas Zacchi de Medeiros via discuss < discuss(a)mpich.org> wrote: > I work on a project that supports different architectures, so we've built > 3 separate versions of MPICH 4.1.1 (cuda11.2, rocm5.4.3 and cpu-only with > ucx1.14.1) > > The cpu-only and the rocm5 libraries are around 50MB each. The CUDA > version on the other hand, is close to 1.8GB! There doesn’t seem to be > anything out of the ordinary with the builds and all versions are working > as expected. > > This is the output of du -sh performed on the install directories: > > First the rocm5 build: > > $ du -sh LINUX_gcc9.3_glibc2.28_rocm5.4.3_ucx1.14.1/lib/* > > (...) > > 53M LINUX_gcc9.3_glibc2.28_rocm5.4.3_ucx1.14.1/lib/libmpi.a > > 40M LINUX_gcc9.3_glibc2.28_rocm5.4.3_ucx1.14.1/lib/libmpi.so.12.3.0 > > And then the CUDA build: > > $ du -sh LINUX_gcc9.3_glibc2.17_cuda11.2_ucx1.14.1/lib/* > > (...) > > 1.8G LINUX_gcc9.3_glibc2.17_cuda11.2_ucx1.14.1/lib/libmpi.a > > 1.7G LINUX_gcc9.3_glibc2.17_cuda11.2_ucx1.14.1/lib/libmpi.so.12.3.0 > > > This issue really complicates packaging and distribution since we provide > both the archive and the shared libraries, I am looking at more than 3GB > for MPICH only. > > After some investigation, it doesn’t seem the problem is coming from our > end. Is it possible that something in mpich’s build process is causing this > excessive file size? > > Thanks for the help. > > Kind regards, > Lucas > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] Core dump in parse_args
by kumar.tarun＠siemens.com 15 Sep '23

15 Sep '23

Thanks Rajeev for your reply. I don’t have the exact number as of now. It’s definitely more than 1000. Is there a hard coded limit? Regards Tarun From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Friday, September 15, 2023 2:40 PM To: discuss(a)mpich.org Cc: Kumar, Tarun (DI SW ICS DVT RD QSCE) <kumar.tarun(a)siemens.com> Subject: Re: [mpich-discuss] Core dump in parse_args How many arguments? Rajeev From: "kumar.tarun--- via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Friday, September 15, 2023 at 4:22 PM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "kumar.tarun(a)siemens.com<mailto:[email protected]>" <kumar.tarun(a)siemens.com<mailto:[email protected]>> Subject: [mpich-discuss] Core dump in parse_args Hi, I recently encountered this crash where mpiexec when executed from a bash script and with a large number of arguments crashes. The core dump is as follows: #0 0x0000000000408ec3 in parse_args () #1 0x0000000000409f26 in HYD_uii_mpx_get_parameters () #2 0x000000000040397a in main () If I reduce the number of arguments then I don’t see the crash. Is it a known issue? Regards Tarun

1 0

Re: [mpich-discuss] Core dump in parse_args
by Thakur, Rajeev 15 Sep '23

15 Sep '23

How many arguments? Rajeev From: "kumar.tarun--- via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Friday, September 15, 2023 at 4:22 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "kumar.tarun(a)siemens.com" <kumar.tarun(a)siemens.com> Subject: [mpich-discuss] Core dump in parse_args Hi, I recently encountered this crash where mpiexec when executed from a bash script and with a large number of arguments crashes. The core dump is as follows: #0 0x0000000000408ec3 in parse_args () #1 0x0000000000409f26 in HYD_uii_mpx_get_parameters () #2 0x000000000040397a in main () If I reduce the number of arguments then I don’t see the crash. Is it a known issue? Regards Tarun

1 0

Core dump in parse_args
by kumar.tarun＠siemens.com 15 Sep '23

15 Sep '23

Hi, I recently encountered this crash where mpiexec when executed from a bash script and with a large number of arguments crashes. The core dump is as follows: #0 0x0000000000408ec3 in parse_args () #1 0x0000000000409f26 in HYD_uii_mpx_get_parameters () #2 0x000000000040397a in main () If I reduce the number of arguments then I don't see the crash. Is it a known issue? Regards Tarun

1 0

Re: [mpich-discuss] Buffer corruption due to an excessive number of messages
by George Bosilca 15 Sep '23

15 Sep '23

Kurt, Your flags are correct. With these flags OMPI uses a totally different communication engine, which suggests that if the error persists it might indeed be in the application. Sorry, George. On Fri, Sep 15, 2023 at 3:01 PM Mccall, Kurt E. (MSFC-EV41) < kurt.e.mccall(a)nasa.gov> wrote: > George, thanks for the idea. With those flags, OMPI mpirun said that > “sm” was no longer available, and suggested “vader”. So my flags were > > > > -mca pml ob1 --mca btl vader,self,tcp > > > > Is that still a valid test of OMPI? The errors I have been seeing > continued to occur with the flags. > > > > Kurt > > > > > > *From:* George Bosilca via discuss <discuss(a)mpich.org> > *Sent:* Friday, September 15, 2023 1:34 PM > *To:* discuss(a)mpich.org > *Cc:* George Bosilca <bosilca(a)icl.utk.edu>; Raffenetti, Ken < > raffenet(a)anl.gov> > *Subject:* [EXTERNAL] [BULK] Re: [mpich-discuss] Buffer corruption due to > an excessive number of messages > > > > *CAUTION:* This email originated from outside of NASA. Please take care > when clicking links or opening attachments. Use the "Report Message" > button to report suspicious messages to the NASA SOC. > > > > Kurt, > > > > There is another common component between current MPICH and Open MPI: UCX, > that is handling the low level communications. I suggest to try to change > the communication substrate to see if your issue continues to exist. For > OMPI add `--mca pml ob1 --mca btl self,sm,tcp' to your mpirun command. > > > > George. > > > > > > On Fri, Sep 15, 2023 at 11:20 AM Joachim Jenke via discuss < > discuss(a)mpich.org> wrote: > > Am 15.09.23 um 17:09 schrieb Tony Curtis via discuss: > > > > > >> On Sep 15, 2023, at 11:07 AM, Raffenetti, Ken via discuss > >> <discuss(a)mpich.org> wrote: > >> > >> 1. Is there a way to detect this kind of overload with an MPI call? > >> > >> If MPI detects an error at runtime, the default behavior is to abort > >> the application. If you application does not abort (and you haven't > >> changed the default error handler), then no error was detected by MPI. > >> > > > > There’s a tool called MUST that might help > > > > MUST - RWTH AACHEN UNIVERSITY Lehrstuhl für Informatik 12 - Deutsch > > <https://www.i12.rwth-aachen.de/go/id/nrbe> > > i12.rwth-aachen.de <https://www.i12.rwth-aachen.de/go/id/nrbe> > > apple-touch-icon-180x180.png < > https://www.i12.rwth-aachen.de/go/id/nrbe> > > > > <https://www.i12.rwth-aachen.de/go/id/nrbe> > > > > The current release version can only detect conflicts in buffer usage at > MPI API level. That means, it will only detect buffer conflicts for > in-flight messages as in: > > MPI_Irecv(buf, MPI_INT, 10, ..., req1); > MPI_Irecv(&buf[9], MPI_INT, 10, ..., req2); > MPI_Wait(req1,...); > MPI_Wait(req2,...); > > The upcoming release I was referencing in my other mail would detect > conflicting accesses to in-flight buffers as in: > > MPI_Irecv(buf, MPI_INT, 10, ..., req); > buf[5]=5; > MPI_Wait(req,...); > > > > > (Not affiliated, just happen to have been looking at it) > > Happy to see that people look at the tool :D > > - Joachim > > > > > Tony > > > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > -- > Dr. rer. nat. Joachim Jenke > > IT Center > Group: High Performance Computing > Division: Computational Science and Engineering > RWTH Aachen University > Seffenter Weg 23 > D 52074 Aachen (Germany) > Tel: +49 241 80- 24765 > Fax: +49 241 80-624765 > jenke(a)itc.rwth-aachen.de > www.itc.rwth-aachen.de > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > >

1 0

Re: [mpich-discuss] Buffer corruption due to an excessive number of messages
by Mccall, Kurt E. (MSFC-EV41) 15 Sep '23

15 Sep '23

George, thanks for the idea. With those flags, OMPI mpirun said that “sm” was no longer available, and suggested “vader”. So my flags were -mca pml ob1 --mca btl vader,self,tcp Is that still a valid test of OMPI? The errors I have been seeing continued to occur with the flags. Kurt From: George Bosilca via discuss <discuss(a)mpich.org> Sent: Friday, September 15, 2023 1:34 PM To: discuss(a)mpich.org Cc: George Bosilca <bosilca(a)icl.utk.edu>; Raffenetti, Ken <raffenet(a)anl.gov> Subject: [EXTERNAL] [BULK] Re: [mpich-discuss] Buffer corruption due to an excessive number of messages CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC. Kurt, There is another common component between current MPICH and Open MPI: UCX, that is handling the low level communications. I suggest to try to change the communication substrate to see if your issue continues to exist. For OMPI add `--mca pml ob1 --mca btl self,sm,tcp' to your mpirun command. George. On Fri, Sep 15, 2023 at 11:20 AM Joachim Jenke via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Am 15.09.23 um 17:09 schrieb Tony Curtis via discuss: > > >> On Sep 15, 2023, at 11:07 AM, Raffenetti, Ken via discuss >> <discuss(a)mpich.org<mailto:[email protected]>> wrote: >> >> 1. Is there a way to detect this kind of overload with an MPI call? >> >> If MPI detects an error at runtime, the default behavior is to abort >> the application. If you application does not abort (and you haven't >> changed the default error handler), then no error was detected by MPI. >> > > There’s a tool called MUST that might help > > MUST - RWTH AACHEN UNIVERSITY Lehrstuhl für Informatik 12 - Deutsch > <https://www.i12.rwth-aachen.de/go/id/nrbe> > i12.rwth-aachen.de<http://i12.rwth-aachen.de/> <https://www.i12.rwth-aachen.de/go/id/nrbe> > apple-touch-icon-180x180.png <https://www.i12.rwth-aachen.de/go/id/nrbe> > > <https://www.i12.rwth-aachen.de/go/id/nrbe> > The current release version can only detect conflicts in buffer usage at MPI API level. That means, it will only detect buffer conflicts for in-flight messages as in: MPI_Irecv(buf, MPI_INT, 10, ..., req1); MPI_Irecv(&buf[9], MPI_INT, 10, ..., req2); MPI_Wait(req1,...); MPI_Wait(req2,...); The upcoming release I was referencing in my other mail would detect conflicting accesses to in-flight buffers as in: MPI_Irecv(buf, MPI_INT, 10, ..., req); buf[5]=5; MPI_Wait(req,...); > > (Not affiliated, just happen to have been looking at it) Happy to see that people look at the tool :D - Joachim > > Tony > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org<mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de<mailto:[email protected]> www.itc.rwth-aachen.de<http://www.itc.rwth-aachen.de/> _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Buffer corruption due to an excessive number of messages
by George Bosilca 15 Sep '23

15 Sep '23

Kurt, There is another common component between current MPICH and Open MPI: UCX, that is handling the low level communications. I suggest to try to change the communication substrate to see if your issue continues to exist. For OMPI add `--mca pml ob1 --mca btl self,sm,tcp' to your mpirun command. George. On Fri, Sep 15, 2023 at 11:20 AM Joachim Jenke via discuss < discuss(a)mpich.org> wrote: > Am 15.09.23 um 17:09 schrieb Tony Curtis via discuss: > > > > > >> On Sep 15, 2023, at 11:07 AM, Raffenetti, Ken via discuss > >> <discuss(a)mpich.org> wrote: > >> > >> 1. Is there a way to detect this kind of overload with an MPI call? > >> > >> If MPI detects an error at runtime, the default behavior is to abort > >> the application. If you application does not abort (and you haven't > >> changed the default error handler), then no error was detected by MPI. > >> > > > > There’s a tool called MUST that might help > > > > MUST - RWTH AACHEN UNIVERSITY Lehrstuhl für Informatik 12 - Deutsch > > <https://www.i12.rwth-aachen.de/go/id/nrbe> > > i12.rwth-aachen.de <https://www.i12.rwth-aachen.de/go/id/nrbe> > > apple-touch-icon-180x180.png < > https://www.i12.rwth-aachen.de/go/id/nrbe> > > > > <https://www.i12.rwth-aachen.de/go/id/nrbe> > > > > The current release version can only detect conflicts in buffer usage at > MPI API level. That means, it will only detect buffer conflicts for > in-flight messages as in: > > MPI_Irecv(buf, MPI_INT, 10, ..., req1); > MPI_Irecv(&buf[9], MPI_INT, 10, ..., req2); > MPI_Wait(req1,...); > MPI_Wait(req2,...); > > The upcoming release I was referencing in my other mail would detect > conflicting accesses to in-flight buffers as in: > > MPI_Irecv(buf, MPI_INT, 10, ..., req); > buf[5]=5; > MPI_Wait(req,...); > > > > > (Not affiliated, just happen to have been looking at it) > > Happy to see that people look at the tool :D > > - Joachim > > > > > Tony > > > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > -- > Dr. rer. nat. Joachim Jenke > > IT Center > Group: High Performance Computing > Division: Computational Science and Engineering > RWTH Aachen University > Seffenter Weg 23 > D 52074 Aachen (Germany) > Tel: +49 241 80- 24765 > Fax: +49 241 80-624765 > jenke(a)itc.rwth-aachen.de > www.itc.rwth-aachen.de > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] issue/question
by Michael Deignan 15 Sep '23

15 Sep '23

Ooops, sorry about that. So many mailing lists I subscribe to, I typed the wrong address!

1 0

Re: [mpich-discuss] issue/question
by Panda, Dhabaleswar 15 Sep '23

15 Sep '23

Hi Michael, I am cc’ing this note to the MVAPICH-Core developers. One of the developers will follow-up with you. Thanks, DK From: Thakur, Rajeev via discuss <discuss(a)mpich.org> Sent: Friday, September 15, 2023 11:42 AM To: discuss(a)mpich.org Cc: Thakur, Rajeev <thakur(a)anl.gov> Subject: Re: [mpich-discuss] issue/question The mailing list for MVAPICH2 is mvapich-discuss@ lists. osu. edu . Rajeev From: "Michael P. Deignan via discuss" <discuss@ mpich. org> Reply-To: "discuss@ mpich. org" <discuss@ mpich. org> Date: Friday, September 15, The mailing list for MVAPICH2 is mvapich-discuss(a)lists.osu.edu<mailto:[email protected]> . Rajeev From: "Michael P. Deignan via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Friday, September 15, 2023 at 10:08 AM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "Michael P. Deignan" <michael.p.deignan(a)gmail.com<mailto:[email protected]>> Subject: [mpich-discuss] issue/question I have a user who is running an mpi model using mvapich2, compiled with the Intel OneAPI compiler (2023.1.0) on a Rocky 8.6 OpenHPC cluster. Bear with me for a minute as I lay the foundation to get to rdma-core. Periodically (randomly after X minutes, where X has been as few as 5 and as much as 30) the model will crash and in the error log there will be thousands of messages: [mv2_mcast_resend_window] Failed to post mcast send errno:0 Basically this message repeats until the disk fills up and the job terminates with an out of disk space message. I tracked the error to the mvapich2 source code at src/mpid/ch3/channels/common/include/ibv_mcast.h: int ret; \ ret = ibv_post_send(_mcast_ctx->ud_ctx->qp, \ &(_v->desc.u.sr), &(_v->desc.y.bad_sr)); \ if (ret) { \ PRINT_ERROR("Failed to post mcast send errno:%d\n", \ errno); \ } \ _mcast_ctx->ud_ctx->send_wqes_avail--; \ while (_mcast_ctx->ud_ctx->send_wqes_avail <= 0) { \ MPIDI_CH3I_Progress(FALSE, NULL); \ } which leads me to the ibv_post_send subroutine in include/infiniband/verbs.h: /** * ibv_post_send - Post a list of work requests to a send queue. * * If IBV_SEND_INLINE flag is set, the data buffers can be reused * immediately after the call returns. */ static inline int ibv_post_send(struct ibv_qp *qp, struct ibv_send_wr *wr, struct ibv_send_wr **bad_wr) { return qp->context->ops.post_send(qp, wr, bad_wr); } For some reason, this function call (ibv_post_send) is returning zero back to the caller, and no error code is being set (errno = 0). As the model does run for a random amount of time, this would seem to suggest some type of hardware problem, but everything we've checked (system, ib card, ib switch, etc.,) doesn't show any errors. Can anyone shed some light on under what circumstances a call to ibv_post_send would, in fact, return zero when previously it didn't? I realize I'm probably not giving much data to really help.. I suppose there could be some type of underlying operating system issue but even there I'm not sure how I would determine where the problem is, since everything else is running fine (and it appears only this one model seems to cause this problem) with nothing abnormal in the event logs. I tried running the model on an older Rocks Centos 7.x cluster and likewise received the same error. Having some basic understanding of what would cause the post send call to return zero and not set an error code might help.

1 0