April 2024 - discuss - mailman3.cels.anl.gov

MPI_Reduce on an inter-communicator hangs
by Mccall, Kurt E. (MSFC-EV41) 21 Apr '24

21 Apr '24

I am calling MPI_Reduce on a set of inter-communicators created by MPI_Comm_spawn, each with one process in the local group (the single manager) and two processes in the remote group (the workers). The inter- communicators are visited one at a time in the manager. All workers enter and exit MPI_Reduce without blocking, but the manager enters the first MPI_Reduce for the first inter-communicator and never returns. What am I doing wrong? I am using MPICH 4.1.2. Here is my manager code: #define N_PROC 4 #define N_IN_GROUP 2 int main(int argc, char *argv[]) { int rank, world_size, error_codes[N_PROC]; MPI_Comm intercoms[N_PROC]; char hostname[64]; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &world_size); gethostname(hostname, sizeof(hostname)); char *p = strstr(hostname, "."); *p = '\0'; MPI_Info info; MPI_Info_create(&info); MPI_Info_set(info, "host", hostname); MPI_Info_set(info, "bind_to", "core"); for (int i = 0; i < N_PROC; ++i) { MPI_Comm_spawn("test_reduce_work", argv, N_IN_GROUP, info, 0, MPI_COMM_SELF, &intercoms[i], &error_codes[i]); } sleep(10); unsigned array[100]{0}; for (int i = 0; i < N_PROC; ++i) { cout << "MANAGER: starting reduction " << i << "\n"; MPI_Reduce(NULL, array, 100, MPI_UNSIGNED, MPI_SUM, MPI_ROOT, intercoms[i]); cout << "MANAGER: finished reduction " << i << "\n"; // we never reach this point } for (int i = 0; i < 100; ++i) cout << array[i] << " "; cout << endl; MPI_Finalize(); } And here is my worker code: int main(int argc, char *argv[]) { int rank, world_size; MPI_Comm manager_intercom; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &world_size); MPI_Comm_get_parent(&manager_intercom); unsigned array[100]{1}; cout << "WORKER: starting reduction\n"; MPI_Reduce(array, NULL, 100, MPI_UNSIGNED, MPI_SUM, MPI_PROC_NULL, manager_intercom); cout << "WORKER: finishing reduction\n"; sleep(10); MPI_Finalize(); } Finally, here it the invocation: $ mpiexec -launcher ssh -print-all-exitcodes -wdir /home/kmccall/test_dir -np 1 -ppn 1 test_reduce_man Thanks, Kurt

1 0

Re: [mpich-discuss] [EXTERNAL] Re: Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?)
by Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] 05 Apr '24

05 Apr '24

Ken, Sorry for the late reply. Things blew up at work. I'll work on getting this into a GitHub issue for you next week! Matt -- Matt Thompson, SSAI, Sr Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 https://urldefense.us/v3/__http://science.gsfc.nasa.gov/sed/bio/matthew.tho… From: Raffenetti, Ken <raffenet(a)anl.gov> Date: Tuesday, April 2, 2024 at 9:26 PM To: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [EXTERNAL] Re: [mpich-discuss] Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?) Can you provide details on how to run the application to reproduce the error? Preferably with as few processes as possible. I think we’ll need to do some more digging to get to the cause on our side. It would also be good to transfer these details over to Github so we can better track the issue. Thanks, Ken From: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson(a)nasa.gov> Date: Tuesday, March 26, 2024 at 11:11 AM To: "Raffenetti, Ken" <raffenet(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [EXTERNAL] Re: [mpich-discuss] Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?) Ken, I think so. Given the traceback, my guess is it's dealing with this code from ESMF v8. 6. 0 (https: //github. com/esmf-org/esmf/blob/ec5f18667091090df7e7b716d588955ce9aa4bd5/src/Infrastructure/VM/src/ESMCI_VMKernel. C#L466-L475) #if (MPI_VERSION ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Ken, I think so. Given the traceback, my guess is it's dealing with this code from ESMF v8.6.0 (https://urldefense.us/v3/__https://github.com/esmf-org/esmf/blob/ec5f186670… <https://urldefense.us/v3/__https:/github.com/esmf-org/esmf/blob/ec5f1866709…>) #if (MPI_VERSION >= 3) // set up communicator across single-system-images SSIs MPI_Comm_split_type(mpi_c, MPI_COMM_TYPE_SHARED, 0, MPI_INFO_NULL, &mpi_c_ssi); // set up communicator across root pets of each SSI int color; MPI_Comm_rank(mpi_c_ssi, &color); if (color>0) color = MPI_UNDEFINED; // only root PETs on each SSI MPI_Comm_split(mpi_c, color, 0, &mpi_c_ssi_roots); #endif Indeed, if it's coming from ESMF, that is the *only* MPI_Comm_split_type in the whole code! We also call MPI_Comm_split_type in the overall code (MAPL) and it also only uses that: https://urldefense.us/v3/__https://github.com/search?q=repo*3AGEOS-ESM*2FMA… <https://urldefense.us/v3/__https:/github.com/search?q=repo*3AGEOS-ESM*2FMAP…> -- Matt Thompson, SSAI, Ld Scientific Prog/Analyst/Super NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 https://urldefense.us/v3/__http://science.gsfc.nasa.gov/sed/bio/matthew.tho… <https://urldefense.us/v3/__http:/science.gsfc.nasa.gov/sed/bio/matthew.thom…> From: Raffenetti, Ken <raffenet(a)anl.gov> Date: Tuesday, March 26, 2024 at 10:55 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?) It looks like the crash is happening after shared memory window creation fails. The failure path is getting tripped up removing the window id from the global hash, since it was never added. We will address this in the code so users get a better error message after the failure. Can you confirm that the input communicator to the window creation function is one created with MPI_Comm_split_type(…,MPI_COMM_TYPE_SHARED,…)? Thanks, Ken From: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Tuesday, March 26, 2024 at 9:20 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson(a)nasa.gov> Subject: [mpich-discuss] Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?) All, I've been trying to get a code of mine working with MPICH 4. 2. 0. I can build MPICH just fine and then build our base libraries and then model and all compiles fine. Hello world runs fine on multiple nodes as well. But when I finally try ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd All, I've been trying to get a code of mine working with MPICH 4.2.0. I can build MPICH just fine and then build our base libraries and then model and all compiles fine. Hello world runs fine on multiple nodes as well. But when I finally try and run our complex model: Assertion failed in file ../src/mpid/ch4/src/ch4_impl.h at line 838: map_entry != NULL /discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0/lib/libmpi.so.12(+0x37d211) [0x14bf4f62c211] /discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0/lib/libmpi.so.12(PMPI_Win_allocate_shared+0x3ba) [0x14bf4f3e452a] /discover/swdev/gmao_SIteam/Baselibs/ESMA-Baselibs-7.20.0/x86_64-pc-linux-gnu/ifort_2021.11.0-mpich_4.2.0-SLES15/Linux/lib/libesmf.so(_ZN5ESMCI3VMK14ssishmAllocateERSt6vectorImSaImEEPNS0_9memhandleEb+0x18b) [0x14bf6e91481b] /discover/swdev/gmao_SIteam/Baselibs/ESMA-Baselibs-7.20.0/x86_64-pc-linux-gnu/ifort_2021.11.0-mpich_4.2.0-SLES15/Linux/lib/libesmf.so(_ZN5ESMCI5Array6createEPNS_9ArraySpecEPNS_8DistGridEPNS_10InterArrayIiEES7_S7_S7_S7_S7_S7_P14ESMC_IndexFlagP13ESMC_Pin_FlagS7_S7_S7_PiPNS_2VME+0x2707) [0x14bf6e44a267] What I'm mainly wondering is if anyone has any experience with an error like this? My guess (at the moment) is that I built things wrong for an Infiniband cluster maybe? I'm using Intel Fortran Classic 2021.11.0 with GCC 11.4.0 as a backing C compiler and I built as: mkdir build-ifort-2021.11.0 && cd build-ifort-2021.11.0 ../configure CC=icx CXX=icpx FC=ifort \ --with-ucx=embedded --with-hwloc=embedded --with-libfabric=embedded --with-yaksa=embedded \ --prefix=/discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0 |& tee configure.ifort-2021.11.0.log All those "embedded" flags are mainly because with Open MPI on this system, I have to do something similar with its configure step: --with-hwloc=internal --with-libevent=internal --with-pmix=internal so I figured I should do the same with MPICH> Now, at the end of the configure step I did see: ***************************************************** *** *** device : ch4:ofi (embedded libfabric) *** shm feature : auto *** gpu support : disabled *** MPICH is configured with device ch4:ofi, which should work for TCP networks and any high-bandwidth interconnect supported by libfabric. MPICH can also be configured with "--with-device=ch4:ucx", which should work for TCP networks and any high-bandwidth interconnect supported by the UCX library. In addition, the legacy device ch3 (--with-device=ch3) is also available. ***************************************************** And I did try the `--with-device=ch4:ucx` but that didn't seem to help. And the system I am on is an Infiniband network, so I imagine ofi should work. Note that this code works fine with Intel MPI and Open MPI (which are our "main" MPI stacks), but some of our external users are asking about MPICH support. Matt -- Matt Thompson, SSAI, Ld Scientific Prog/Analyst/Super NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 https://urldefense.us/v3/__http://science.gsfc.nasa.gov/sed/bio/matthew.tho… <https://urldefense.us/v3/__http:/science.gsfc.nasa.gov/sed/bio/matthew.thom…>

1 0

Re: [mpich-discuss] [EXTERNAL] Re: Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?)
by Raffenetti, Ken 02 Apr '24

02 Apr '24

Can you provide details on how to run the application to reproduce the error? Preferably with as few processes as possible. I think we’ll need to do some more digging to get to the cause on our side. It would also be good to transfer these details over to Github so we can better track the issue. Thanks, Ken From: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson(a)nasa.gov> Date: Tuesday, March 26, 2024 at 11:11 AM To: "Raffenetti, Ken" <raffenet(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [EXTERNAL] Re: [mpich-discuss] Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?) Ken, I think so. Given the traceback, my guess is it's dealing with this code from ESMF v8. 6. 0 (https: //github. com/esmf-org/esmf/blob/ec5f18667091090df7e7b716d588955ce9aa4bd5/src/Infrastructure/VM/src/ESMCI_VMKernel. C#L466-L475) #if (MPI_VERSION ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Ken, I think so. Given the traceback, my guess is it's dealing with this code from ESMF v8.6.0 (https://urldefense.us/v3/__https://github.com/esmf-org/esmf/blob/ec5f186670… <https://urldefense.us/v3/__https:/github.com/esmf-org/esmf/blob/ec5f1866709…>) #if (MPI_VERSION >= 3) // set up communicator across single-system-images SSIs MPI_Comm_split_type(mpi_c, MPI_COMM_TYPE_SHARED, 0, MPI_INFO_NULL, &mpi_c_ssi); // set up communicator across root pets of each SSI int color; MPI_Comm_rank(mpi_c_ssi, &color); if (color>0) color = MPI_UNDEFINED; // only root PETs on each SSI MPI_Comm_split(mpi_c, color, 0, &mpi_c_ssi_roots); #endif Indeed, if it's coming from ESMF, that is the *only* MPI_Comm_split_type in the whole code! We also call MPI_Comm_split_type in the overall code (MAPL) and it also only uses that: https://urldefense.us/v3/__https://github.com/search?q=repo*3AGEOS-ESM*2FMA… <https://urldefense.us/v3/__https:/github.com/search?q=repo*3AGEOS-ESM*2FMAP…> -- Matt Thompson, SSAI, Ld Scientific Prog/Analyst/Super NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 https://urldefense.us/v3/__http://science.gsfc.nasa.gov/sed/bio/matthew.tho… <https://urldefense.us/v3/__http:/science.gsfc.nasa.gov/sed/bio/matthew.thom…> From: Raffenetti, Ken <raffenet(a)anl.gov> Date: Tuesday, March 26, 2024 at 10:55 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?) It looks like the crash is happening after shared memory window creation fails. The failure path is getting tripped up removing the window id from the global hash, since it was never added. We will address this in the code so users get a better error message after the failure. Can you confirm that the input communicator to the window creation function is one created with MPI_Comm_split_type(…,MPI_COMM_TYPE_SHARED,…)? Thanks, Ken From: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Tuesday, March 26, 2024 at 9:20 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson(a)nasa.gov> Subject: [mpich-discuss] Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?) All, I've been trying to get a code of mine working with MPICH 4. 2. 0. I can build MPICH just fine and then build our base libraries and then model and all compiles fine. Hello world runs fine on multiple nodes as well. But when I finally try ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd All, I've been trying to get a code of mine working with MPICH 4.2.0. I can build MPICH just fine and then build our base libraries and then model and all compiles fine. Hello world runs fine on multiple nodes as well. But when I finally try and run our complex model: Assertion failed in file ../src/mpid/ch4/src/ch4_impl.h at line 838: map_entry != NULL /discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0/lib/libmpi.so.12(+0x37d211) [0x14bf4f62c211] /discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0/lib/libmpi.so.12(PMPI_Win_allocate_shared+0x3ba) [0x14bf4f3e452a] /discover/swdev/gmao_SIteam/Baselibs/ESMA-Baselibs-7.20.0/x86_64-pc-linux-gnu/ifort_2021.11.0-mpich_4.2.0-SLES15/Linux/lib/libesmf.so(_ZN5ESMCI3VMK14ssishmAllocateERSt6vectorImSaImEEPNS0_9memhandleEb+0x18b) [0x14bf6e91481b] /discover/swdev/gmao_SIteam/Baselibs/ESMA-Baselibs-7.20.0/x86_64-pc-linux-gnu/ifort_2021.11.0-mpich_4.2.0-SLES15/Linux/lib/libesmf.so(_ZN5ESMCI5Array6createEPNS_9ArraySpecEPNS_8DistGridEPNS_10InterArrayIiEES7_S7_S7_S7_S7_S7_P14ESMC_IndexFlagP13ESMC_Pin_FlagS7_S7_S7_PiPNS_2VME+0x2707) [0x14bf6e44a267] What I'm mainly wondering is if anyone has any experience with an error like this? My guess (at the moment) is that I built things wrong for an Infiniband cluster maybe? I'm using Intel Fortran Classic 2021.11.0 with GCC 11.4.0 as a backing C compiler and I built as: mkdir build-ifort-2021.11.0 && cd build-ifort-2021.11.0 ../configure CC=icx CXX=icpx FC=ifort \ --with-ucx=embedded --with-hwloc=embedded --with-libfabric=embedded --with-yaksa=embedded \ --prefix=/discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0 |& tee configure.ifort-2021.11.0.log All those "embedded" flags are mainly because with Open MPI on this system, I have to do something similar with its configure step: --with-hwloc=internal --with-libevent=internal --with-pmix=internal so I figured I should do the same with MPICH> Now, at the end of the configure step I did see: ***************************************************** *** *** device : ch4:ofi (embedded libfabric) *** shm feature : auto *** gpu support : disabled *** MPICH is configured with device ch4:ofi, which should work for TCP networks and any high-bandwidth interconnect supported by libfabric. MPICH can also be configured with "--with-device=ch4:ucx", which should work for TCP networks and any high-bandwidth interconnect supported by the UCX library. In addition, the legacy device ch3 (--with-device=ch3) is also available. ***************************************************** And I did try the `--with-device=ch4:ucx` but that didn't seem to help. And the system I am on is an Infiniband network, so I imagine ofi should work. Note that this code works fine with Intel MPI and Open MPI (which are our "main" MPI stacks), but some of our external users are asking about MPICH support. Matt -- Matt Thompson, SSAI, Ld Scientific Prog/Analyst/Super NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 https://urldefense.us/v3/__http://science.gsfc.nasa.gov/sed/bio/matthew.tho… <https://urldefense.us/v3/__http:/science.gsfc.nasa.gov/sed/bio/matthew.thom…>

1 0