- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] mpich 4.3.0 compilation problem when using --with-hcoll=/opt/mellanox/hcoll
by Zhou, Hui 16 Apr '25

16 Apr '25

Hi Martin, Could you try the patch in https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/7047?__;!!… -- Hui ________________________________ From: Audet, Martin via discuss <discuss(a)mpich.org> Sent: Wednesday, April 16, 2025 11:13 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Subject: [mpich-discuss] mpich 4.3.0 compilation problem when using --with-hcoll=/opt/mellanox/hcoll Hello mpich community, When I try to compile mpich 4. 3. 0 configured with --with-hcoll=/opt/mellanox/hcoll option, I get a compilation error because the hcoll_do_progress() function is defined with two arguments in hcoll_init. c but it is called ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello mpich community, When I try to compile mpich 4.3.0 configured with --with-hcoll=/opt/mellanox/hcoll option, I get a compilation error because the hcoll_do_progress() function is defined with two arguments in hcoll_init.c but it is called only with one in hcoll_rte.c ! Here is the error message I get: src/mpid/common/hcoll/hcoll_rte.c: In function 'progress': src/mpid/common/hcoll/hcoll_rte.c:58:33: warning: passing argument 1 of 'hcoll_do_progress' makes integer from pointer without a cast [-Wint-conversion] 58 | ret = hcoll_do_progress(&made_progress); | ^~~~~~~~~~~~~~ | | | int * In file included from ./src/mpid/ch4/netmod/include/../ucx/ucx_coll.h:11, from ./src/mpid/ch4/netmod/include/../ucx/netmod_inline.h:15, from ./src/mpid/ch4/netmod/include/netmod_impl.h:1589, from ./src/mpid/ch4/include/mpidch4.h:448, from ./src/mpid/ch4/include/mpidpost.h:10, from ./src/include/mpiimpl.h:232, from src/mpid/common/hcoll/hcoll_rte.c:6: ./src/mpid/ch4/netmod/include/../ucx/../../../common/hcoll/hcoll.h:42:27: note: expected 'int' but argument is of type 'int *' 42 | int hcoll_do_progress(int vci, int *made_progress); | ~~~~^~~ src/mpid/common/hcoll/hcoll_rte.c:58:15: error: too few arguments to function 'hcoll_do_progress' 58 | ret = hcoll_do_progress(&made_progress); | ^~~~~~~~~~~~~~~~~ In file included from ./src/mpid/ch4/netmod/include/../ucx/ucx_coll.h:11, from ./src/mpid/ch4/netmod/include/../ucx/netmod_inline.h:15, from ./src/mpid/ch4/netmod/include/netmod_impl.h:1589, from ./src/mpid/ch4/include/mpidch4.h:448, from ./src/mpid/ch4/include/mpidpost.h:10, from ./src/include/mpiimpl.h:232, from src/mpid/common/hcoll/hcoll_rte.c:6: ./src/mpid/ch4/netmod/include/../ucx/../../../common/hcoll/hcoll.h:42:5: note: declared here 42 | int hcoll_do_progress(int vci, int *made_progress); | ^~~~~~~~~~~~~~~~~ I use to compile mpich versions 3.4,x, 4.1.x, and 4.2.x configured with this option (--with-hcoll=) in the past without any problems. It looks like some recent changes in the related files introduced a problem that slip into 4.3.0 and makes compilation impossible. Could it be fixed ? Or could the --with-hcoll option be removed if it is no longer relevant (I guess that if we use ch4:ucx, ucx may itself use hcoll internally to optimize collective operations when running on hierarchical environment) ? Here are some details: arch: x86_64 OS: RHEL 9.5 (up to date except kernel) MOFED: 24.10-2.1.8.0-LTS hcoll: 4.8.3230-1.2410068 ucx: 1.18.0-1.2410068 The complete configuration line: ./configure --with-device=ch4:ucx --with-hcoll=/opt/mellanox/hcoll --prefix=/work/software/x86_64/mpich/mpich-ch4_ucx-4.3.0 --with-xpmem --enable-g=none --enable-fast=all --enable-romio --with-file-system=ufs+nfs+lustre --enable-shared --enable-sharedlibs=gcc Thanks, Martin Audet

1 0

mpich 4.3.0 compilation problem when using --with-hcoll=/opt/mellanox/hcoll
by Audet, Martin 16 Apr '25

16 Apr '25

Hello mpich community, When I try to compile mpich 4.3.0 configured with --with-hcoll=/opt/mellanox/hcoll option, I get a compilation error because the hcoll_do_progress() function is defined with two arguments in hcoll_init.c but it is called only with one in hcoll_rte.c ! Here is the error message I get: src/mpid/common/hcoll/hcoll_rte.c: In function 'progress': src/mpid/common/hcoll/hcoll_rte.c:58:33: warning: passing argument 1 of 'hcoll_do_progress' makes integer from pointer without a cast [-Wint-conversion] 58 | ret = hcoll_do_progress(&made_progress); | ^~~~~~~~~~~~~~ | | | int * In file included from ./src/mpid/ch4/netmod/include/../ucx/ucx_coll.h:11, from ./src/mpid/ch4/netmod/include/../ucx/netmod_inline.h:15, from ./src/mpid/ch4/netmod/include/netmod_impl.h:1589, from ./src/mpid/ch4/include/mpidch4.h:448, from ./src/mpid/ch4/include/mpidpost.h:10, from ./src/include/mpiimpl.h:232, from src/mpid/common/hcoll/hcoll_rte.c:6: ./src/mpid/ch4/netmod/include/../ucx/../../../common/hcoll/hcoll.h:42:27: note: expected 'int' but argument is of type 'int *' 42 | int hcoll_do_progress(int vci, int *made_progress); | ~~~~^~~ src/mpid/common/hcoll/hcoll_rte.c:58:15: error: too few arguments to function 'hcoll_do_progress' 58 | ret = hcoll_do_progress(&made_progress); | ^~~~~~~~~~~~~~~~~ In file included from ./src/mpid/ch4/netmod/include/../ucx/ucx_coll.h:11, from ./src/mpid/ch4/netmod/include/../ucx/netmod_inline.h:15, from ./src/mpid/ch4/netmod/include/netmod_impl.h:1589, from ./src/mpid/ch4/include/mpidch4.h:448, from ./src/mpid/ch4/include/mpidpost.h:10, from ./src/include/mpiimpl.h:232, from src/mpid/common/hcoll/hcoll_rte.c:6: ./src/mpid/ch4/netmod/include/../ucx/../../../common/hcoll/hcoll.h:42:5: note: declared here 42 | int hcoll_do_progress(int vci, int *made_progress); | ^~~~~~~~~~~~~~~~~ I use to compile mpich versions 3.4,x, 4.1.x, and 4.2.x configured with this option (--with-hcoll=) in the past without any problems. It looks like some recent changes in the related files introduced a problem that slip into 4.3.0 and makes compilation impossible. Could it be fixed ? Or could the --with-hcoll option be removed if it is no longer relevant (I guess that if we use ch4:ucx, ucx may itself use hcoll internally to optimize collective operations when running on hierarchical environment) ? Here are some details: arch: x86_64 OS: RHEL 9.5 (up to date except kernel) MOFED: 24.10-2.1.8.0-LTS hcoll: 4.8.3230-1.2410068 ucx: 1.18.0-1.2410068 The complete configuration line: ./configure --with-device=ch4:ucx --with-hcoll=/opt/mellanox/hcoll --prefix=/work/software/x86_64/mpich/mpich-ch4_ucx-4.3.0 --with-xpmem --enable-g=none --enable-fast=all --enable-romio --with-file-system=ufs+nfs+lustre --enable-shared --enable-sharedlibs=gcc Thanks, Martin Audet

1 0

Re: [mpich-discuss] Wrapping routines with choice buffers from Fortran
by Zhou, Hui 24 Feb '25

24 Feb '25

Hi Gregor, Thanks for bring up the Fortran profiling interface issue. Admittedly, we have not discussed such issues thoroughly yet. I believe the truth lies in the consensus and you are the first real user I hear from 😉 . Yes, it appears that the Fortran PMPI_ should call C PMPI_ functions to avoid double profiling. Opinions? -- Hui ________________________________ From: Gregor Corbin via discuss <discuss(a)mpich.org> Sent: Thursday, February 20, 2025 9:56 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Gregor Corbin <g.corbin(a)fz-juelich.de> Subject: [mpich-discuss] Wrapping routines with choice buffers from Fortran Hi everyone, I am developing a set of wrappers for the C and Fortran 2008 ('use mpi_f08') bindings. The F08 wrappers are intercepting the Fortran routines, for instance subroutine mpi_send_f08ts_(buf, count, datatype, dest, tag, comm, ierror) use :: mpi_f08, only: MPI_Comm, MPI_Datatype, PMPI_Send implicit none ! argument declarations ... ! wrapper code ... call PMPI_Send(buf, count, datatype, dest, tag, comm, ierror) end subroutine In MPICH, the call to PMPI_Send is forwarded to MPIR_Send_cdesc which then calls MPI_Send internally. Since I also wrap MPI_Send in C, calls from Fortran go through both wrappers, which is not intended. This only happens for routines with choice buffers, though. Routines without buffers call into the PMPI interface (e.g. mpi_wait_f08_ calls MPIR_Wait_c which binds directly to PMPI_Wait). I attached a small example that demonstrates the issue. Is this behavior intended? How can I correctly wrap these choice buffer routines? Best, Gregor Corbin

1 0

Re: [mpich-discuss] Wrapping routines with choice buffers from Fortran
by Gregor Corbin 21 Feb '25

21 Feb '25

Hi Joachim, Thanks for the hints. I handle the optional ierror argument like this: integer :: my_ierror call PMPI_Send(..., my_ierror) ! tool can use my_ierror if (present(ierror)) ierror = my_ierror If I read the test correctly, 'profile1f90.f90' checks whether mpi_send and mpi_recv are intercepted by the wrappers mpi_send_f08ts and mpi_recv_f08ts. My problem is that the calls to pmpi_send and pmpi_recv in these wrappers redirect to the C MPI_ functions and then also enter my C wrappers. Cheers, Gregor On 20.02.25 17:25, Joachim Jenke via discuss wrote: > Sorry, the mail was intended to be PM :/ Translation inline below > > Am 20.02.25 um 17:16 schrieb Joachim Jenke via discuss: >> Hallo Gregor, >> >> beim gucken, was es dazu in mpich so gibt bin ich darüber gestolpert: >> > grepping trough MPICH source I found: >> https://github.com/pmodels/mpich/blob/main/test/mpi/f08/profile/ >> profile1f90.f90#L70 >> >> Hast du den present check für ierr auf dem Schirm? > > Are you aware of the necessary present check? > > - Joachim >> >> - Joachim >> >> Am 20.02.25 um 16:56 schrieb Gregor Corbin via discuss: >>> Hi everyone, >>> >>> I am developing a set of wrappers for the C and Fortran 2008 ('use >>> mpi_f08') bindings. The F08 wrappers are intercepting the Fortran >>> routines, for instance >>> >>> subroutine mpi_send_f08ts_(buf, count, datatype, dest, tag, >>> comm, ierror) >>> use :: mpi_f08, only: MPI_Comm, MPI_Datatype, PMPI_Send >>> implicit none >>> ! argument declarations ... >>> ! wrapper code ... >>> call PMPI_Send(buf, count, datatype, dest, tag, comm, ierror) >>> end subroutine >>> >>> >>> In MPICH, the call to PMPI_Send is forwarded to MPIR_Send_cdesc >>> which then calls MPI_Send internally. Since I also wrap MPI_Send in >>> C, calls from Fortran go through both wrappers, which is not >>> intended. This only happens for routines with choice buffers, >>> though. Routines without buffers call into the PMPI interface (e.g. >>> mpi_wait_f08_ calls MPIR_Wait_c which binds directly to PMPI_Wait). >>> I attached a small example that demonstrates the issue. >>> >>> Is this behavior intended? How can I correctly wrap these choice >>> buffer routines? >>> >>> Best, >>> >>> Gregor Corbin >>> >>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Wrapping routines with choice buffers from Fortran
by Joachim Jenke 20 Feb '25

20 Feb '25

Sorry, the mail was intended to be PM :/ Translation inline below Am 20.02.25 um 17:16 schrieb Joachim Jenke via discuss: > Hallo Gregor, > > beim gucken, was es dazu in mpich so gibt bin ich darüber gestolpert: > grepping trough MPICH source I found: > https://github.com/pmodels/mpich/blob/main/test/mpi/f08/profile/ > profile1f90.f90#L70 > > Hast du den present check für ierr auf dem Schirm? Are you aware of the necessary present check? - Joachim > > - Joachim > > Am 20.02.25 um 16:56 schrieb Gregor Corbin via discuss: >> Hi everyone, >> >> I am developing a set of wrappers for the C and Fortran 2008 ('use >> mpi_f08') bindings. The F08 wrappers are intercepting the Fortran >> routines, for instance >> >> subroutine mpi_send_f08ts_(buf, count, datatype, dest, tag, comm, >> ierror) >> use :: mpi_f08, only: MPI_Comm, MPI_Datatype, PMPI_Send >> implicit none >> ! argument declarations ... >> ! wrapper code ... >> call PMPI_Send(buf, count, datatype, dest, tag, comm, ierror) >> end subroutine >> >> >> In MPICH, the call to PMPI_Send is forwarded to MPIR_Send_cdesc which >> then calls MPI_Send internally. Since I also wrap MPI_Send in C, calls >> from Fortran go through both wrappers, which is not intended. This >> only happens for routines with choice buffers, though. Routines >> without buffers call into the PMPI interface (e.g. mpi_wait_f08_ calls >> MPIR_Wait_c which binds directly to PMPI_Wait). I attached a small >> example that demonstrates the issue. >> >> Is this behavior intended? How can I correctly wrap these choice >> buffer routines? >> >> Best, >> >> Gregor Corbin >> >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpich-discuss] Wrapping routines with choice buffers from Fortran
by Joachim Jenke 20 Feb '25

20 Feb '25

Hallo Gregor, beim gucken, was es dazu in mpich so gibt bin ich darüber gestolpert: https://github.com/pmodels/mpich/blob/main/test/mpi/f08/profile/profile1f90… Hast du den present check für ierr auf dem Schirm? - Joachim Am 20.02.25 um 16:56 schrieb Gregor Corbin via discuss: > Hi everyone, > > I am developing a set of wrappers for the C and Fortran 2008 ('use > mpi_f08') bindings. The F08 wrappers are intercepting the Fortran > routines, for instance > > subroutine mpi_send_f08ts_(buf, count, datatype, dest, tag, comm, > ierror) > use :: mpi_f08, only: MPI_Comm, MPI_Datatype, PMPI_Send > implicit none > ! argument declarations ... > ! wrapper code ... > call PMPI_Send(buf, count, datatype, dest, tag, comm, ierror) > end subroutine > > > In MPICH, the call to PMPI_Send is forwarded to MPIR_Send_cdesc which > then calls MPI_Send internally. Since I also wrap MPI_Send in C, calls > from Fortran go through both wrappers, which is not intended. This only > happens for routines with choice buffers, though. Routines without > buffers call into the PMPI interface (e.g. mpi_wait_f08_ calls > MPIR_Wait_c which binds directly to PMPI_Wait). I attached a small > example that demonstrates the issue. > > Is this behavior intended? How can I correctly wrap these choice buffer > routines? > > Best, > > Gregor Corbin > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Wrapping routines with choice buffers from Fortran
by Gregor Corbin 20 Feb '25

20 Feb '25

Hi everyone, I am developing a set of wrappers for the C and Fortran 2008 ('use mpi_f08') bindings. The F08 wrappers are intercepting the Fortran routines, for instance subroutine mpi_send_f08ts_(buf, count, datatype, dest, tag, comm, ierror) use :: mpi_f08, only: MPI_Comm, MPI_Datatype, PMPI_Send implicit none ! argument declarations ... ! wrapper code ... call PMPI_Send(buf, count, datatype, dest, tag, comm, ierror) end subroutine In MPICH, the call to PMPI_Send is forwarded to MPIR_Send_cdesc which then calls MPI_Send internally. Since I also wrap MPI_Send in C, calls from Fortran go through both wrappers, which is not intended. This only happens for routines with choice buffers, though. Routines without buffers call into the PMPI interface (e.g. mpi_wait_f08_ calls MPIR_Wait_c which binds directly to PMPI_Wait). I attached a small example that demonstrates the issue. Is this behavior intended? How can I correctly wrap these choice buffer routines? Best, Gregor Corbin

1 0

Re: [mpich-discuss] Yaksa public symbols
by Raffenetti, Ken 19 Feb '25

19 Feb '25

Hi Alastair, When embedded in libmpi.so, these yaksa symbols are not intended to be public. It is an oversight if they are visible, so we will address it. I’ve created an issue on our Github to track https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7306__;!… . Cheers, Ken From: Alastair McKinstry via discuss <discuss(a)mpich.org> Date: Wednesday, February 19, 2025 at 4:35 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Alastair McKinstry <alastair.mckinstry(a)ichec.ie> Subject: [mpich-discuss] Yaksa public symbols Hi, I’m maintaining MPICH in Debian, and enabling HIP and other dependencies. In Debian we supply an ELF symbols file, tracking to make sure there are no unexpected ABI changes. With 4. 3. 0 there are changes and a lot of new yaksa-related symbols, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, I’m maintaining MPICH in Debian, and enabling HIP and other dependencies. In Debian we supply an ELF symbols file, tracking to make sure there are no unexpected ABI changes. With 4.3.0 there are changes and a lot of new yaksa-related symbols, eg _Z33yaksuri_hipi_kernel_pack_LOR_boolPKvPvmPK17yaksuri_hipi_md_s@Base 4.3 _Z34yaksuri_hipi_kernel_pack_LAND_boolPKvPvmPK17yaksuri_hipi_md_s@Base 4.3 … hipFree@Base 4.3 yaksuri_hipi_pack_blkhindx_blkhindx_bool@Base 4.3 yaksuri_hipi_pack_blkhindx_blkhindx_char@Base 4.3 … Approx 8000 new symbols added to the public interface ! Is this expected ? are these part of the public interface or unexpected leakage? Can they be made private safely? Thanks Alastair McKinstry

1 0

Yaksa public symbols
by Alastair McKinstry 19 Feb '25

19 Feb '25

Hi, I’m maintaining MPICH in Debian, and enabling HIP and other dependencies. In Debian we supply an ELF symbols file, tracking to make sure there are no unexpected ABI changes. With 4.3.0 there are changes and a lot of new yaksa-related symbols, eg _Z33yaksuri_hipi_kernel_pack_LOR_boolPKvPvmPK17yaksuri_hipi_md_s@Base 4.3 _Z34yaksuri_hipi_kernel_pack_LAND_boolPKvPvmPK17yaksuri_hipi_md_s@Base 4.3 … hipFree@Base 4.3 yaksuri_hipi_pack_blkhindx_blkhindx_bool@Base 4.3 yaksuri_hipi_pack_blkhindx_blkhindx_char@Base 4.3 … Approx 8000 new symbols added to the public interface ! Is this expected ? are these part of the public interface or unexpected leakage? Can they be made private safely? Thanks Alastair McKinstry

1 0

Re: [mpich-discuss] Only Run as root
by Raffenetti, Ken 17 Feb '25

17 Feb '25

Can you add -v to your mpiexec command? It will enable verbose output and hopefully give some clues to where things are getting stuck. Ken From: Tanner L. Nelson via discuss <discuss(a)mpich.org> Date: Monday, February 17, 2025 at 11:43 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Tanner L. Nelson <Tanner.Nelson(a)inl.gov> Subject: Re: [mpich-discuss] Only Run as root Good morning, We have a small cluster in an isolated environment. Our lead has spearheaded a reinstall to get away from Bright Cluster Manager and CentOS. This is the first time our project has done anything like this, and we have hit a snag ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Good morning, We have a small cluster in an isolated environment. Our lead has spearheaded a reinstall to get away from Bright Cluster Manager and CentOS. This is the first time our project has done anything like this, and we have hit a snag that we believe is MPI-related. As root: · Job runs as expected (verified with 2 nodes) · mpirun with host file, same result As user account: · Single node works · 2 nodes, job hangs with no error output that we can find To us, and our colleagues at our main Computing Center, it seems related to permissions/authentication. But everything we know to check has been checked several times (i.e. user can ssh to nodes 001-002 where root was successful, user was not; permissions on mpi directory 755, even tried 777 just to rule it out). Details I believe may be asked: · RHEL9 · Mpich4.2.3 was compiled with gcc11.5.0 · Consistent results with scheduler and mpirun · Tested HPL and our primary application, both built against the same mpich version Hoping we can get some suggestions. Thanks in advance! Respectfully, Tanner Nelson

1 0