July 2023 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Problem with mpich in debian bookworm
by Zhou, Hui 27 Jul '23

27 Jul '23

Hi Andrea, Try install libucx-dev package. -- Hui Zhou ________________________________ From: Andrea Buzzi via discuss <discuss(a)mpich.org> Sent: Thursday, July 27, 2023 3:53 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Andrea Buzzi <A.Buzzi(a)isac.cnr.it> Subject: [mpich-discuss] Problem with mpich in debian bookworm Hi, I have an issue with mpich. I have been using mpich with no problem so far, installed as standard package in debian 11 on my 12-core PC. I found mpich superior to intel mpi. After I have recently upgraded to debian 12 (bookworm), trying to call mpich with the script mpif90, I get a long series of errors (both with gfortran and ifort - in the latter case with FC="ifort"), basically a multiplication of the following sequence: ------ /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-pready.o): in function `PMPI_Pready': (.text+0x7b3): undefined reference to `ucp_am_send_nb' /usr/bin/ld: (.text+0x7e7): undefined reference to `ucp_request_release' /usr/bin/ld: (.text+0xb68): undefined reference to `ucs_status_string' ------- I assume some library or dependency is missing, or maybe the new debian package is broken. I am asking your support since I could not find help online. Best regards Andrea Buzzi Associate senior scientist Istituto di Scienze dell'Atmosfera e del Clima - CNR-ISAC Institute of Atmospheric Sciences and Climate - CNR-ISAC Via Gobetti 101, I-40129 Bologna email: A.Buzzi(a)isac.cnr.it -- Andrea Buzzi Associate senior scientist Istituto di Scienze dell'Atmosfera e del Clima - CNR-ISAC Institute of Atmospheric Sciences and Climate - CNR-ISAC Via Gobetti 101, I-40129 Bologna email: A.Buzzi(a)isac.cnr.it _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Problem with mpich in debian bookworm
by Andrea Buzzi 27 Jul '23

27 Jul '23

Hi, I have an issue with mpich. I have been using mpich with no problem so far, installed as standard package in debian 11 on my 12-core PC. I found mpich superior to intel mpi. After I have recently upgraded to debian 12 (bookworm), trying to call mpich with the script mpif90, I get a long series of errors (both with gfortran and ifort - in the latter case with FC="ifort"), basically a multiplication of the following sequence: ------ /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libmpich.a(lib_libmpich_la-pready.o): in function `PMPI_Pready': (.text+0x7b3): undefined reference to `ucp_am_send_nb' /usr/bin/ld: (.text+0x7e7): undefined reference to `ucp_request_release' /usr/bin/ld: (.text+0xb68): undefined reference to `ucs_status_string' ------- I assume some library or dependency is missing, or maybe the new debian package is broken. I am asking your support since I could not find help online. Best regards Andrea Buzzi Associate senior scientist Istituto di Scienze dell'Atmosfera e del Clima - CNR-ISAC Institute of Atmospheric Sciences and Climate - CNR-ISAC Via Gobetti 101, I-40129 Bologna email: A.Buzzi(a)isac.cnr.it -- Andrea Buzzi Associate senior scientist Istituto di Scienze dell'Atmosfera e del Clima - CNR-ISAC Institute of Atmospheric Sciences and Climate - CNR-ISAC Via Gobetti 101, I-40129 Bologna email: A.Buzzi(a)isac.cnr.it

1 0

Re: [mpich-discuss] possible thread bug with MPI_Reduce/MPI_Allreduce
by Zhou, Hui 14 Jul '23

14 Jul '23

Glad I could help. Hui ________________________________ From: Burlen Loring <burlen.loring(a)gmail.com> Sent: Friday, July 14, 2023 12:31 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] possible thread bug with MPI_Reduce/MPI_Allreduce Hi Hui, The wording in the documentation of MPI_Init_thread, and specifically the part that says with MPI_THREAD_MULTIPLE that "multiple threads may call MPI at once with no restrictions", had me thinking along the wrong lines. Following your suggestions, I rewrote my code to give each thread exclusive access to one communicator from a small pool of communicators. This is working well and out performed Iallreduce/Ireduce approach. Thanks so much for the help Burlen On 7/12/23 11:55, Zhou, Hui wrote: Hi Burlen, You are not allowed to issue collectives concurrently on the same communicator. You can use non-blocking collectives, i.e. MPI_Ireduce and MPI_Iallreduce to achieve the overlapping. You still need serialize the issue of MPI_Ireduce/MPI_Iallreduce, but you don't need mutex to wait for the request to complete in concurrent threads. -- Hui ________________________________ From: Burlen Loring via discuss <discuss(a)mpich.org><mailto:[email protected]> Sent: Wednesday, July 12, 2023 1:29 PM To: <discuss(a)mpich.org><mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]> Cc: Burlen Loring <burlen.loring(a)gmail.com><mailto:[email protected]> Subject: [mpich-discuss] possible thread bug with MPI_Reduce/MPI_Allreduce Hi All, I'm using MPICH 4.0.2 on Fedora 37 from the package manager for development. From an MPI parallel simulation I'm spawning a thread that does a number of reductions (MPI_Allreduce and MPI_Reduce). MPI_IN_PLACE option is used. The results are written with POSIX I/O from rank 0. The simulation continues, and can launch the next set of reductions before the previous ones completed. I have called MPI_Init_thread and requested and received MPI_THREAD_MULTIPLE support. However, when multiple threads overlap (in test runs 3-4 threads running concurrently) both MPI_Allreduce and MPI_Reduce calls can produce incorrect results. If instead I serialize the threads, by waiting on them before returning to the simulation, the results are correct. Also, if I use a mutex around my MPI_Allreduce/Reduce sections, the results are correct. I think that MPI_Reduce/Allreduce is not thread safe. I was wondering if this is a known issue? Could it be a mpich build/configure setting not set correctly by the Fedora package maintainer ? Thanks Burlen

1 0

Re: [mpich-discuss] possible thread bug with MPI_Reduce/MPI_Allreduce
by Zhou, Hui 12 Jul '23

12 Jul '23

Hi Burlen, You are not allowed to issue collectives concurrently on the same communicator. You can use non-blocking collectives, i.e. MPI_Ireduce and MPI_Iallreduce to achieve the overlapping. You still need serialize the issue of MPI_Ireduce/MPI_Iallreduce, but you don't need mutex to wait for the request to complete in concurrent threads. -- Hui ________________________________ From: Burlen Loring via discuss <discuss(a)mpich.org> Sent: Wednesday, July 12, 2023 1:29 PM To: <discuss(a)mpich.org> <discuss(a)mpich.org> Cc: Burlen Loring <burlen.loring(a)gmail.com> Subject: [mpich-discuss] possible thread bug with MPI_Reduce/MPI_Allreduce Hi All, I'm using MPICH 4.0.2 on Fedora 37 from the package manager for development. From an MPI parallel simulation I'm spawning a thread that does a number of reductions (MPI_Allreduce and MPI_Reduce). MPI_IN_PLACE option is used. The results are written with POSIX I/O from rank 0. The simulation continues, and can launch the next set of reductions before the previous ones completed. I have called MPI_Init_thread and requested and received MPI_THREAD_MULTIPLE support. However, when multiple threads overlap (in test runs 3-4 threads running concurrently) both MPI_Allreduce and MPI_Reduce calls can produce incorrect results. If instead I serialize the threads, by waiting on them before returning to the simulation, the results are correct. Also, if I use a mutex around my MPI_Allreduce/Reduce sections, the results are correct. I think that MPI_Reduce/Allreduce is not thread safe. I was wondering if this is a known issue? Could it be a mpich build/configure setting not set correctly by the Fedora package maintainer ? Thanks Burlen

1 0

possible thread bug with MPI_Reduce/MPI_Allreduce
by Burlen Loring 12 Jul '23

12 Jul '23

Hi All, I'm using MPICH 4.0.2 on Fedora 37 from the package manager for development. From an MPI parallel simulation I'm spawning a thread that does a number of reductions (MPI_Allreduce and MPI_Reduce). MPI_IN_PLACE option is used. The results are written with POSIX I/O from rank 0. The simulation continues, and can launch the next set of reductions before the previous ones completed. I have called MPI_Init_thread and requested and received MPI_THREAD_MULTIPLE support. However, when multiple threads overlap (in test runs 3-4 threads running concurrently) both MPI_Allreduce and MPI_Reduce calls can produce incorrect results. If instead I serialize the threads, by waiting on them before returning to the simulation, the results are correct. Also, if I use a mutex around my MPI_Allreduce/Reduce sections, the results are correct. I think that MPI_Reduce/Allreduce is not thread safe. I was wondering if this is a known issue? Could it be a mpich build/configure setting not set correctly by the Fedora package maintainer ? Thanks Burlen

1 0