- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Buffer corruption due to an excessive number of messages
by Mccall, Kurt E. (MSFC-EV41) 15 Sep '23

15 Sep '23

Joachim, Unfortunately, using MPI_Improbe/MPI_Mrecv didn't solve the problem -- I still am receiving buffers with invalid data objects near the ends of the buffers. The problem goes away when I reduce the size of the job (number of nodes), making me think it is the large number of messages that is causing the problem. 1. Is there a way to detect this kind of overload with an MPI call? 2. Is there an upper bound on the number of messages that can be "in flight"? 3. Is there a upper bound on message length? Or is there some other possible cause that I haven't thought of? Thanks, Kurt -----Original Message----- From: Joachim Jenke via discuss <discuss(a)mpich.org> Sent: Thursday, September 14, 2023 3:10 PM To: discuss(a)mpich.org Cc: Joachim Jenke <jenke(a)itc.rwth-aachen.de> Subject: [EXTERNAL] [BULK] Re: [mpich-discuss] Buffer corruption due to an excessive number of messages CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC. Hi Kurt, just a thought: do you execute single-threaded or multi-threaded? In case of multi-threaded execution, you should look into MPI_Improbe/MPI_Mrecv just to make sure that you really receive the message you probed for. Even in single-threaded execution you might try whether using these functions instead fixes your issue. Best Joachim Am 14.09.23 um 22:02 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss: > It seems that when I send a process too many non-blocking messages > (with > MPI_Isend) , MPI_Iprobe/MPI_Recv sometimes returns a buffer > > with corrupted data for some of the messages. Usually the corrupted > data objects are at the end of the array that was sent. I checked the > > buffers passed to MPI_Isend, and they are uncorrupted. > > 1. Is there a way to detect this kind of overload with an MPI call? > 2. Is there an upper bound on the number of messages that can be "in > flight"? > 3. Is there a upper bound on message length? > > Thanks, > > Kurt > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://list/ > s.mpich.org%2Fmailman%2Flistinfo%2Fdiscuss&data=05%7C01%7Ckurt.e.mccal > l%40nasa.gov%7C9e8bfd71610243562ce808dbb55eaace%7C7005d45845be48ae8140 > d43da96dd17b%7C0%7C0%7C638303190427564481%7CUnknown%7CTWFpbGZsb3d8eyJW > IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000% > 7C%7C%7C&sdata=yplpEqPO1a8OpC2Nt3z55G2UtfA0lBqV4Zn5OrMRRx0%3D&reserved > =0 -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de http://www.itc.rwth-aachen.de/

1 0

Re: [mpich-discuss] Buffer corruption due to an excessive number of messages
by Joachim Jenke 14 Sep '23

14 Sep '23

Hi Kurt, just a thought: do you execute single-threaded or multi-threaded? In case of multi-threaded execution, you should look into MPI_Improbe/MPI_Mrecv just to make sure that you really receive the message you probed for. Even in single-threaded execution you might try whether using these functions instead fixes your issue. Best Joachim Am 14.09.23 um 22:02 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss: > It seems that when I send a process too many non-blocking messages (with > MPI_Isend) , MPI_Iprobe/MPI_Recv sometimes returns a buffer > > with corrupted data for some of the messages. Usually the corrupted > data objects are at the end of the array that was sent. I checked the > > buffers passed to MPI_Isend, and they are uncorrupted. > > 1. Is there a way to detect this kind of overload with an MPI call? > 2. Is there an upper bound on the number of messages that can be “in > flight”? > 3. Is there a upper bound on message length? > > Thanks, > > Kurt > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Buffer corruption due to an excessive number of messages
by Mccall, Kurt E. (MSFC-EV41) 14 Sep '23

14 Sep '23

It seems that when I send a process too many non-blocking messages (with MPI_Isend) , MPI_Iprobe/MPI_Recv sometimes returns a buffer with corrupted data for some of the messages. Usually the corrupted data objects are at the end of the array that was sent. I checked the buffers passed to MPI_Isend, and they are uncorrupted. 1. Is there a way to detect this kind of overload with an MPI call? 2. Is there an upper bound on the number of messages that can be "in flight"? 3. Is there a upper bound on message length? Thanks, Kurt

1 0

Re: [mpich-discuss] Installing mpich with libfabric
by Zhou, Hui 11 Aug '23

11 Aug '23

Hi Pratik, Could you clarify what do you ment by "utilizing OFI"? Are you looking for libfabric calls such as fi_writemsg? -- Hui ________________________________ From: Pratik Shelar via discuss <discuss(a)mpich.org> Sent: Friday, August 11, 2023 1:32 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Pratik Shelar <pshelar(a)cdac.in> Subject: [mpich-discuss] Installing mpich with libfabric Hi Everyone, specifically aiming to enable RMA (Remote Memory Access) calls for efficient parallel computing. However, I've encountered some challenges in achieving the desired setup. I have attempted multiple configurations, but it seems that the RMA calls of OFI are not being utilized based on my backtracing results. I am seeking your guidance to resolve this issue. Here are the configurations I've tried: 1. ./configure --prefix=/home/<username>/mpich/mpich_exe --with-ch4-ofi-direct-provider=<provider> --with-libfabric-lib=/home/<username>/libfabric/libfabric_exe/lib --with-libfabric-include=/home/<username>/libfabric/libfabric_exe/include 2. ./configure --prefix=/home/<username>/mpich/mpich_exe --enable-ch4-am-only --with-ch4-ofi-direct-provider=<provider> ---with-libfabric-lib=/home/<username>/libfabric/libfabric_exe/lib --with-libfabric-include=/home/<username>/libfabric/libfabric_exe/include 3. ./configure --prefix=/home/<username>/mpich/mpich_exe --enable-mpit-pvars=rma --with-ch4-ofi-direct-provider=<provider> ---with-libfabric-lib=/home/<username>/libfabric/libfabric_exe/lib --with-libfabric-include=/home/<username>/libfabric/libfabric_exe/include 4. ./configure --prefix=/home/<username>/mpich/mpich_exe --enable-mpit-pvars=rma --with-device=ch4:ofi ---with-libfabric-lib=/home/<username>/libfabric/libfabric_exe/lib --with-libfabric-include=/home/<username>/libfabric/libfabric_exe/include Despite these attempts, the RMA calls from OFI are not being utilized as indicated by my analysis with Valgrind. I kindly request your expertise and assistance in resolving this matter. If there are specific configurations or steps that I might be missing, please provide guidance on how to properly enable RMA calls using OFI in the MPICH setup. Thank you for your time and support. I am looking forward to your valuable insights and suggestions to help me overcome this issue. Thanks & Regards, Pratik Baban Shelar Project Engineer HPC-Technologies CDAC-PUNE ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------

1 0

Installing mpich with libfabric
by Pratik Shelar 11 Aug '23

11 Aug '23

Hi Everyone, specifically aiming to enable RMA (Remote Memory Access) calls for efficient parallel computing. However, I've encountered some challenges in achieving the desired setup. I have attempted multiple configurations, but it seems that the RMA calls of OFI are not being utilized based on my backtracing results. I am seeking your guidance to resolve this issue. Here are the configurations I've tried: 1. ./configure --prefix=/home/<username>/mpich/mpich_exe --with-ch4-ofi-direct-provider=<provider> --with-libfabric-lib=/home/<username>/libfabric/libfabric_exe/lib --with-libfabric-include=/home/<username>/libfabric/libfabric_exe/include 2. ./configure --prefix=/home/<username>/mpich/mpich_exe --enable-ch4-am-only --with-ch4-ofi-direct-provider=<provider> ---with-libfabric-lib=/home/<username>/libfabric/libfabric_exe/lib --with-libfabric-include=/home/<username>/libfabric/libfabric_exe/include 3. ./configure --prefix=/home/<username>/mpich/mpich_exe --enable-mpit-pvars=rma --with-ch4-ofi-direct-provider=<provider> ---with-libfabric-lib=/home/<username>/libfabric/libfabric_exe/lib --with-libfabric-include=/home/<username>/libfabric/libfabric_exe/include 4. ./configure --prefix=/home/<username>/mpich/mpich_exe --enable-mpit-pvars=rma --with-device=ch4:ofi ---with-libfabric-lib=/home/<username>/libfabric/libfabric_exe/lib --with-libfabric-include=/home/<username>/libfabric/libfabric_exe/include Despite these attempts, the RMA calls from OFI are not being utilized as indicated by my analysis with Valgrind. I kindly request your expertise and assistance in resolving this matter. If there are specific configurations or steps that I might be missing, please provide guidance on how to properly enable RMA calls using OFI in the MPICH setup. Thank you for your time and support. I am looking forward to your valuable insights and suggestions to help me overcome this issue. Thanks & Regards, Pratik Baban Shelar Project Engineer HPC-Technologies CDAC-PUNE ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------

1 0

Re: [mpich-discuss] reading context error
by Zhou, Hui 08 Aug '23

08 Aug '23

It is saying "An existing connection was forcibly closed by the remote host", which often means the remote process is exiting or being killed (e.g. from segfault). Unfortunately, there isn't much more insight we can offer. -- Hui ________________________________ From: Nazmul Haque via discuss <discuss(a)mpich.org> Sent: Tuesday, August 8, 2023 4:37 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Nazmul Haque <shadhin873(a)outlook.com> Subject: [mpich-discuss] reading context error Hello, While running lammps on mpi using mpich2, we are getting this error and even after doing a thorough search on online and applying diffrent ideas to solve the problem we are getting no solution. May I know why this error is showing? Thank you, Nazmul [cid:7c8cc4e4-2012-45f4-940b-8f7387196b17]

1 0

reading context error
by Nazmul Haque 08 Aug '23

08 Aug '23

Hello, While running lammps on mpi using mpich2, we are getting this error and even after doing a thorough search on online and applying diffrent ideas to solve the problem we are getting no solution. May I know why this error is showing? Thank you, Nazmul [cid:7c8cc4e4-2012-45f4-940b-8f7387196b17]

1 0

reading context error
by Nazmul Haque 08 Aug '23

08 Aug '23

Hello, While running lammps on mpi using mpich2, we are getting this error and even after doing a thorough search on online and applying diffrent ideas to solve the problem we are getting no solution. May I know why this error is showing? Thank you, Nazmul [cid:cadc3264-a77b-41e8-8f2a-2160ffa7b07a]

1 0

Re: [mpich-discuss] Problem with mpich in debian bookworm
by Zhou, Hui 27 Jul '23

27 Jul '23

Hi Andrea, Try install libucx-dev package. -- Hui Zhou ________________________________ From: Andrea Buzzi via discuss <discuss(a)mpich.org> Sent: Thursday, July 27, 2023 3:53 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Andrea Buzzi <A.Buzzi(a)isac.cnr.it> Subject: [mpich-discuss] Problem with mpich in debian bookworm Hi, I have an issue with mpich. I have been using mpich with no problem so far, installed as standard package in debian 11 on my 12-core PC. I found mpich superior to intel mpi. After I have recently upgraded to debian 12 (bookworm), trying to call mpich with the script mpif90, I get a long series of errors (both with gfortran and ifort - in the latter case with FC="ifort"), basically a multiplication of the following sequence: ------ /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-pready.o): in function `PMPI_Pready': (.text+0x7b3): undefined reference to `ucp_am_send_nb' /usr/bin/ld: (.text+0x7e7): undefined reference to `ucp_request_release' /usr/bin/ld: (.text+0xb68): undefined reference to `ucs_status_string' ------- I assume some library or dependency is missing, or maybe the new debian package is broken. I am asking your support since I could not find help online. Best regards Andrea Buzzi Associate senior scientist Istituto di Scienze dell'Atmosfera e del Clima - CNR-ISAC Institute of Atmospheric Sciences and Climate - CNR-ISAC Via Gobetti 101, I-40129 Bologna email: A.Buzzi(a)isac.cnr.it -- Andrea Buzzi Associate senior scientist Istituto di Scienze dell'Atmosfera e del Clima - CNR-ISAC Institute of Atmospheric Sciences and Climate - CNR-ISAC Via Gobetti 101, I-40129 Bologna email: A.Buzzi(a)isac.cnr.it _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Problem with mpich in debian bookworm
by Andrea Buzzi 27 Jul '23

27 Jul '23

Hi, I have an issue with mpich. I have been using mpich with no problem so far, installed as standard package in debian 11 on my 12-core PC. I found mpich superior to intel mpi. After I have recently upgraded to debian 12 (bookworm), trying to call mpich with the script mpif90, I get a long series of errors (both with gfortran and ifort - in the latter case with FC="ifort"), basically a multiplication of the following sequence: ------ /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-pready.o): in function `PMPI_Pready': (.text+0x7b3): undefined reference to `ucp_am_send_nb' /usr/bin/ld: (.text+0x7e7): undefined reference to `ucp_request_release' /usr/bin/ld: (.text+0xb68): undefined reference to `ucs_status_string' ------- I assume some library or dependency is missing, or maybe the new debian package is broken. I am asking your support since I could not find help online. Best regards Andrea Buzzi Associate senior scientist Istituto di Scienze dell'Atmosfera e del Clima - CNR-ISAC Institute of Atmospheric Sciences and Climate - CNR-ISAC Via Gobetti 101, I-40129 Bologna email: A.Buzzi(a)isac.cnr.it -- Andrea Buzzi Associate senior scientist Istituto di Scienze dell'Atmosfera e del Clima - CNR-ISAC Institute of Atmospheric Sciences and Climate - CNR-ISAC Via Gobetti 101, I-40129 Bologna email: A.Buzzi(a)isac.cnr.it

1 0