- discuss - mailman3.cels.anl.gov

Environment variable forwarding using Hydra and "-launcher ssh"
by Edric Ellis 22 Apr '24

22 Apr '24

We're in the process of moving from mpich-3.x to mpich-4.1.2. We've run into some odd behaviour on SLURM related to environment variable forwarding by mpiexec. It looks like mpiexec now propagates only SLURM_* environment variables, instead of filtering them out (or intending to). Consider something like this: $ mpiexec -launcher slurm printenv HOME SLURM_JOBID Using mpich-3.x, the HOME variable gets forward. Using mpich-4.1.2, it does not. I believe that mpich-3.x intends to filter out SLURM_JOBID, but the value still seems to be present, maybe srun forwards that. It's the fact that HOME doesn't get through using mpich-4.1.2 that is causing us problems. Running mpich- Here's what I think is the relevant change for SLURM: https://urldefense.us/v3/__https://github.com/pmodels/mpich/commit/95ba4ddc… . Am I reading that correctly? The doc here https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/main/doc/w… states that SLURM_ things should be filtered out, but that doesn't appear to be happening? For reference, here's what mpich-4.1.2 "mpiexec -verbose -launcher slurm" prints: mpiexec options: ---------------- Base path: /path/to/mpich-4.1.2 Launcher: slurm Debug level: 1 Enable X: -1 Global environment: ------------------- SLURM_JOBID=102437 SLURM_JOB_USER=eellis SLURM_JOB_QOS=normal SLURM_JOB_NUM_NODES=2 SLURM_TASKS_PER_NODE=1(x2) SLURM_TOPOLOGY_ADDR_PATTERN=node ... many more SLURM_* And here's what mpich-3.x prints: mpiexec options: ---------------- Base path: /path/to/mpich-3.x Launcher: slurm Debug level: 1 Enable X: -1 Global environment: ------------------- ALTERNATE_EDITOR=emacs MAIL=/var/mail/eellis USER=eellis SLURM_JOB_USER=eellis l=/local/eellis XDG_SESSION_TYPE=unspecified SLURM_JOB_QOS=normal ... no SLURM_JOBID Cheers, Edric.

1 0

Re: [mpich-discuss] MPI_Reduce on an inter-communicator hangs
by Thakur, Rajeev 21 Apr '24

21 Apr '24

I haven’t run the code, but shouldn’t the workers pass 0 (rank of master) as the root instead of MPI_PROC_NULL? Rajeev From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Sunday, April 21, 2024 at 4:53 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] MPI_Reduce on an inter-communicator hangs I am calling MPI_Reduce on a set of inter-communicators created by MPI_Comm_spawn, each with one process in the local group (the single manager) and two processes in the remote group (the workers). The inter- communicators are visited one at ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I am calling MPI_Reduce on a set of inter-communicators created by MPI_Comm_spawn, each with one process in the local group (the single manager) and two processes in the remote group (the workers). The inter- communicators are visited one at a time in the manager. All workers enter and exit MPI_Reduce without blocking, but the manager enters the first MPI_Reduce for the first inter-communicator and never returns. What am I doing wrong? I am using MPICH 4.1.2. Here is my manager code: #define N_PROC 4 #define N_IN_GROUP 2 int main(int argc, char *argv[]) { int rank, world_size, error_codes[N_PROC]; MPI_Comm intercoms[N_PROC]; char hostname[64]; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &world_size); gethostname(hostname, sizeof(hostname)); char *p = strstr(hostname, "."); *p = '\0'; MPI_Info info; MPI_Info_create(&info); MPI_Info_set(info, "host", hostname); MPI_Info_set(info, "bind_to", "core"); for (int i = 0; i < N_PROC; ++i) { MPI_Comm_spawn("test_reduce_work", argv, N_IN_GROUP, info, 0, MPI_COMM_SELF, &intercoms[i], &error_codes[i]); } sleep(10); unsigned array[100]{0}; for (int i = 0; i < N_PROC; ++i) { cout << "MANAGER: starting reduction " << i << "\n"; MPI_Reduce(NULL, array, 100, MPI_UNSIGNED, MPI_SUM, MPI_ROOT, intercoms[i]); cout << "MANAGER: finished reduction " << i << "\n"; // we never reach this point } for (int i = 0; i < 100; ++i) cout << array[i] << " "; cout << endl; MPI_Finalize(); } And here is my worker code: int main(int argc, char *argv[]) { int rank, world_size; MPI_Comm manager_intercom; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &world_size); MPI_Comm_get_parent(&manager_intercom); unsigned array[100]{1}; cout << "WORKER: starting reduction\n"; MPI_Reduce(array, NULL, 100, MPI_UNSIGNED, MPI_SUM, MPI_PROC_NULL, manager_intercom); cout << "WORKER: finishing reduction\n"; sleep(10); MPI_Finalize(); } Finally, here it the invocation: $ mpiexec -launcher ssh -print-all-exitcodes -wdir /home/kmccall/test_dir -np 1 -ppn 1 test_reduce_man Thanks, Kurt

1 0

Re: [mpich-discuss] MPI_Reduce on an inter-communicator hangs
by Jenke, Joachim 21 Apr '24

21 Apr '24

Hi Kurt, The workers need to pass root=0. Mpi_proc_null indicates that the local process is not involved in the communication (like in p2p). This would be the appropriate argument for non-root arguments in the manager group, i.e., if you start with more than one process. > all other processes in the same group as the root use MPI_PROC_NULL https://urldefense.us/v3/__https://www.mpi-forum.org/docs/mpi-2.2/mpi22-rep… It also explains your observation of falling through the call without waiting. Best Joachim ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Sunday, April 21, 2024 11:52:48 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] MPI_Reduce on an inter-communicator hangs I am calling MPI_Reduce on a set of inter-communicators created by MPI_Comm_spawn, each with one process in the local group (the single manager) and two processes in the remote group (the workers). The inter- communicators are visited one at ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I am calling MPI_Reduce on a set of inter-communicators created by MPI_Comm_spawn, each with one process in the local group (the single manager) and two processes in the remote group (the workers). The inter- communicators are visited one at a time in the manager. All workers enter and exit MPI_Reduce without blocking, but the manager enters the first MPI_Reduce for the first inter-communicator and never returns. What am I doing wrong? I am using MPICH 4.1.2. Here is my manager code: #define N_PROC 4 #define N_IN_GROUP 2 int main(int argc, char *argv[]) { int rank, world_size, error_codes[N_PROC]; MPI_Comm intercoms[N_PROC]; char hostname[64]; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &world_size); gethostname(hostname, sizeof(hostname)); char *p = strstr(hostname, "."); *p = '\0'; MPI_Info info; MPI_Info_create(&info); MPI_Info_set(info, "host", hostname); MPI_Info_set(info, "bind_to", "core"); for (int i = 0; i < N_PROC; ++i) { MPI_Comm_spawn("test_reduce_work", argv, N_IN_GROUP, info, 0, MPI_COMM_SELF, &intercoms[i], &error_codes[i]); } sleep(10); unsigned array[100]{0}; for (int i = 0; i < N_PROC; ++i) { cout << "MANAGER: starting reduction " << i << "\n"; MPI_Reduce(NULL, array, 100, MPI_UNSIGNED, MPI_SUM, MPI_ROOT, intercoms[i]); cout << "MANAGER: finished reduction " << i << "\n"; // we never reach this point } for (int i = 0; i < 100; ++i) cout << array[i] << " "; cout << endl; MPI_Finalize(); } And here is my worker code: int main(int argc, char *argv[]) { int rank, world_size; MPI_Comm manager_intercom; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &world_size); MPI_Comm_get_parent(&manager_intercom); unsigned array[100]{1}; cout << "WORKER: starting reduction\n"; MPI_Reduce(array, NULL, 100, MPI_UNSIGNED, MPI_SUM, MPI_PROC_NULL, manager_intercom); cout << "WORKER: finishing reduction\n"; sleep(10); MPI_Finalize(); } Finally, here it the invocation: $ mpiexec -launcher ssh -print-all-exitcodes -wdir /home/kmccall/test_dir -np 1 -ppn 1 test_reduce_man Thanks, Kurt

1 0

MPI_Reduce on an inter-communicator hangs
by Mccall, Kurt E. (MSFC-EV41) 21 Apr '24

21 Apr '24

I am calling MPI_Reduce on a set of inter-communicators created by MPI_Comm_spawn, each with one process in the local group (the single manager) and two processes in the remote group (the workers). The inter- communicators are visited one at a time in the manager. All workers enter and exit MPI_Reduce without blocking, but the manager enters the first MPI_Reduce for the first inter-communicator and never returns. What am I doing wrong? I am using MPICH 4.1.2. Here is my manager code: #define N_PROC 4 #define N_IN_GROUP 2 int main(int argc, char *argv[]) { int rank, world_size, error_codes[N_PROC]; MPI_Comm intercoms[N_PROC]; char hostname[64]; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &world_size); gethostname(hostname, sizeof(hostname)); char *p = strstr(hostname, "."); *p = '\0'; MPI_Info info; MPI_Info_create(&info); MPI_Info_set(info, "host", hostname); MPI_Info_set(info, "bind_to", "core"); for (int i = 0; i < N_PROC; ++i) { MPI_Comm_spawn("test_reduce_work", argv, N_IN_GROUP, info, 0, MPI_COMM_SELF, &intercoms[i], &error_codes[i]); } sleep(10); unsigned array[100]{0}; for (int i = 0; i < N_PROC; ++i) { cout << "MANAGER: starting reduction " << i << "\n"; MPI_Reduce(NULL, array, 100, MPI_UNSIGNED, MPI_SUM, MPI_ROOT, intercoms[i]); cout << "MANAGER: finished reduction " << i << "\n"; // we never reach this point } for (int i = 0; i < 100; ++i) cout << array[i] << " "; cout << endl; MPI_Finalize(); } And here is my worker code: int main(int argc, char *argv[]) { int rank, world_size; MPI_Comm manager_intercom; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &world_size); MPI_Comm_get_parent(&manager_intercom); unsigned array[100]{1}; cout << "WORKER: starting reduction\n"; MPI_Reduce(array, NULL, 100, MPI_UNSIGNED, MPI_SUM, MPI_PROC_NULL, manager_intercom); cout << "WORKER: finishing reduction\n"; sleep(10); MPI_Finalize(); } Finally, here it the invocation: $ mpiexec -launcher ssh -print-all-exitcodes -wdir /home/kmccall/test_dir -np 1 -ppn 1 test_reduce_man Thanks, Kurt

1 0

Re: [mpich-discuss] [EXTERNAL] Re: Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?)
by Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] 05 Apr '24

05 Apr '24

Ken, Sorry for the late reply. Things blew up at work. I'll work on getting this into a GitHub issue for you next week! Matt -- Matt Thompson, SSAI, Sr Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 https://urldefense.us/v3/__http://science.gsfc.nasa.gov/sed/bio/matthew.tho… From: Raffenetti, Ken <raffenet(a)anl.gov> Date: Tuesday, April 2, 2024 at 9:26 PM To: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [EXTERNAL] Re: [mpich-discuss] Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?) Can you provide details on how to run the application to reproduce the error? Preferably with as few processes as possible. I think we’ll need to do some more digging to get to the cause on our side. It would also be good to transfer these details over to Github so we can better track the issue. Thanks, Ken From: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson(a)nasa.gov> Date: Tuesday, March 26, 2024 at 11:11 AM To: "Raffenetti, Ken" <raffenet(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [EXTERNAL] Re: [mpich-discuss] Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?) Ken, I think so. Given the traceback, my guess is it's dealing with this code from ESMF v8. 6. 0 (https: //github. com/esmf-org/esmf/blob/ec5f18667091090df7e7b716d588955ce9aa4bd5/src/Infrastructure/VM/src/ESMCI_VMKernel. C#L466-L475) #if (MPI_VERSION ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Ken, I think so. Given the traceback, my guess is it's dealing with this code from ESMF v8.6.0 (https://urldefense.us/v3/__https://github.com/esmf-org/esmf/blob/ec5f186670… <https://urldefense.us/v3/__https:/github.com/esmf-org/esmf/blob/ec5f1866709…>) #if (MPI_VERSION >= 3) // set up communicator across single-system-images SSIs MPI_Comm_split_type(mpi_c, MPI_COMM_TYPE_SHARED, 0, MPI_INFO_NULL, &mpi_c_ssi); // set up communicator across root pets of each SSI int color; MPI_Comm_rank(mpi_c_ssi, &color); if (color>0) color = MPI_UNDEFINED; // only root PETs on each SSI MPI_Comm_split(mpi_c, color, 0, &mpi_c_ssi_roots); #endif Indeed, if it's coming from ESMF, that is the *only* MPI_Comm_split_type in the whole code! We also call MPI_Comm_split_type in the overall code (MAPL) and it also only uses that: https://urldefense.us/v3/__https://github.com/search?q=repo*3AGEOS-ESM*2FMA… <https://urldefense.us/v3/__https:/github.com/search?q=repo*3AGEOS-ESM*2FMAP…> -- Matt Thompson, SSAI, Ld Scientific Prog/Analyst/Super NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 https://urldefense.us/v3/__http://science.gsfc.nasa.gov/sed/bio/matthew.tho… <https://urldefense.us/v3/__http:/science.gsfc.nasa.gov/sed/bio/matthew.thom…> From: Raffenetti, Ken <raffenet(a)anl.gov> Date: Tuesday, March 26, 2024 at 10:55 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?) It looks like the crash is happening after shared memory window creation fails. The failure path is getting tripped up removing the window id from the global hash, since it was never added. We will address this in the code so users get a better error message after the failure. Can you confirm that the input communicator to the window creation function is one created with MPI_Comm_split_type(…,MPI_COMM_TYPE_SHARED,…)? Thanks, Ken From: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Tuesday, March 26, 2024 at 9:20 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson(a)nasa.gov> Subject: [mpich-discuss] Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?) All, I've been trying to get a code of mine working with MPICH 4. 2. 0. I can build MPICH just fine and then build our base libraries and then model and all compiles fine. Hello world runs fine on multiple nodes as well. But when I finally try ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd All, I've been trying to get a code of mine working with MPICH 4.2.0. I can build MPICH just fine and then build our base libraries and then model and all compiles fine. Hello world runs fine on multiple nodes as well. But when I finally try and run our complex model: Assertion failed in file ../src/mpid/ch4/src/ch4_impl.h at line 838: map_entry != NULL /discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0/lib/libmpi.so.12(+0x37d211) [0x14bf4f62c211] /discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0/lib/libmpi.so.12(PMPI_Win_allocate_shared+0x3ba) [0x14bf4f3e452a] /discover/swdev/gmao_SIteam/Baselibs/ESMA-Baselibs-7.20.0/x86_64-pc-linux-gnu/ifort_2021.11.0-mpich_4.2.0-SLES15/Linux/lib/libesmf.so(_ZN5ESMCI3VMK14ssishmAllocateERSt6vectorImSaImEEPNS0_9memhandleEb+0x18b) [0x14bf6e91481b] /discover/swdev/gmao_SIteam/Baselibs/ESMA-Baselibs-7.20.0/x86_64-pc-linux-gnu/ifort_2021.11.0-mpich_4.2.0-SLES15/Linux/lib/libesmf.so(_ZN5ESMCI5Array6createEPNS_9ArraySpecEPNS_8DistGridEPNS_10InterArrayIiEES7_S7_S7_S7_S7_S7_P14ESMC_IndexFlagP13ESMC_Pin_FlagS7_S7_S7_PiPNS_2VME+0x2707) [0x14bf6e44a267] What I'm mainly wondering is if anyone has any experience with an error like this? My guess (at the moment) is that I built things wrong for an Infiniband cluster maybe? I'm using Intel Fortran Classic 2021.11.0 with GCC 11.4.0 as a backing C compiler and I built as: mkdir build-ifort-2021.11.0 && cd build-ifort-2021.11.0 ../configure CC=icx CXX=icpx FC=ifort \ --with-ucx=embedded --with-hwloc=embedded --with-libfabric=embedded --with-yaksa=embedded \ --prefix=/discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0 |& tee configure.ifort-2021.11.0.log All those "embedded" flags are mainly because with Open MPI on this system, I have to do something similar with its configure step: --with-hwloc=internal --with-libevent=internal --with-pmix=internal so I figured I should do the same with MPICH> Now, at the end of the configure step I did see: ***************************************************** *** *** device : ch4:ofi (embedded libfabric) *** shm feature : auto *** gpu support : disabled *** MPICH is configured with device ch4:ofi, which should work for TCP networks and any high-bandwidth interconnect supported by libfabric. MPICH can also be configured with "--with-device=ch4:ucx", which should work for TCP networks and any high-bandwidth interconnect supported by the UCX library. In addition, the legacy device ch3 (--with-device=ch3) is also available. ***************************************************** And I did try the `--with-device=ch4:ucx` but that didn't seem to help. And the system I am on is an Infiniband network, so I imagine ofi should work. Note that this code works fine with Intel MPI and Open MPI (which are our "main" MPI stacks), but some of our external users are asking about MPICH support. Matt -- Matt Thompson, SSAI, Ld Scientific Prog/Analyst/Super NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 https://urldefense.us/v3/__http://science.gsfc.nasa.gov/sed/bio/matthew.tho… <https://urldefense.us/v3/__http:/science.gsfc.nasa.gov/sed/bio/matthew.thom…>

1 0

Re: [mpich-discuss] [EXTERNAL] Re: Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?)
by Raffenetti, Ken 02 Apr '24

02 Apr '24

Can you provide details on how to run the application to reproduce the error? Preferably with as few processes as possible. I think we’ll need to do some more digging to get to the cause on our side. It would also be good to transfer these details over to Github so we can better track the issue. Thanks, Ken From: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson(a)nasa.gov> Date: Tuesday, March 26, 2024 at 11:11 AM To: "Raffenetti, Ken" <raffenet(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [EXTERNAL] Re: [mpich-discuss] Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?) Ken, I think so. Given the traceback, my guess is it's dealing with this code from ESMF v8. 6. 0 (https: //github. com/esmf-org/esmf/blob/ec5f18667091090df7e7b716d588955ce9aa4bd5/src/Infrastructure/VM/src/ESMCI_VMKernel. C#L466-L475) #if (MPI_VERSION ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Ken, I think so. Given the traceback, my guess is it's dealing with this code from ESMF v8.6.0 (https://urldefense.us/v3/__https://github.com/esmf-org/esmf/blob/ec5f186670… <https://urldefense.us/v3/__https:/github.com/esmf-org/esmf/blob/ec5f1866709…>) #if (MPI_VERSION >= 3) // set up communicator across single-system-images SSIs MPI_Comm_split_type(mpi_c, MPI_COMM_TYPE_SHARED, 0, MPI_INFO_NULL, &mpi_c_ssi); // set up communicator across root pets of each SSI int color; MPI_Comm_rank(mpi_c_ssi, &color); if (color>0) color = MPI_UNDEFINED; // only root PETs on each SSI MPI_Comm_split(mpi_c, color, 0, &mpi_c_ssi_roots); #endif Indeed, if it's coming from ESMF, that is the *only* MPI_Comm_split_type in the whole code! We also call MPI_Comm_split_type in the overall code (MAPL) and it also only uses that: https://urldefense.us/v3/__https://github.com/search?q=repo*3AGEOS-ESM*2FMA… <https://urldefense.us/v3/__https:/github.com/search?q=repo*3AGEOS-ESM*2FMAP…> -- Matt Thompson, SSAI, Ld Scientific Prog/Analyst/Super NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 https://urldefense.us/v3/__http://science.gsfc.nasa.gov/sed/bio/matthew.tho… <https://urldefense.us/v3/__http:/science.gsfc.nasa.gov/sed/bio/matthew.thom…> From: Raffenetti, Ken <raffenet(a)anl.gov> Date: Tuesday, March 26, 2024 at 10:55 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?) It looks like the crash is happening after shared memory window creation fails. The failure path is getting tripped up removing the window id from the global hash, since it was never added. We will address this in the code so users get a better error message after the failure. Can you confirm that the input communicator to the window creation function is one created with MPI_Comm_split_type(…,MPI_COMM_TYPE_SHARED,…)? Thanks, Ken From: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Tuesday, March 26, 2024 at 9:20 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson(a)nasa.gov> Subject: [mpich-discuss] Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?) All, I've been trying to get a code of mine working with MPICH 4. 2. 0. I can build MPICH just fine and then build our base libraries and then model and all compiles fine. Hello world runs fine on multiple nodes as well. But when I finally try ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd All, I've been trying to get a code of mine working with MPICH 4.2.0. I can build MPICH just fine and then build our base libraries and then model and all compiles fine. Hello world runs fine on multiple nodes as well. But when I finally try and run our complex model: Assertion failed in file ../src/mpid/ch4/src/ch4_impl.h at line 838: map_entry != NULL /discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0/lib/libmpi.so.12(+0x37d211) [0x14bf4f62c211] /discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0/lib/libmpi.so.12(PMPI_Win_allocate_shared+0x3ba) [0x14bf4f3e452a] /discover/swdev/gmao_SIteam/Baselibs/ESMA-Baselibs-7.20.0/x86_64-pc-linux-gnu/ifort_2021.11.0-mpich_4.2.0-SLES15/Linux/lib/libesmf.so(_ZN5ESMCI3VMK14ssishmAllocateERSt6vectorImSaImEEPNS0_9memhandleEb+0x18b) [0x14bf6e91481b] /discover/swdev/gmao_SIteam/Baselibs/ESMA-Baselibs-7.20.0/x86_64-pc-linux-gnu/ifort_2021.11.0-mpich_4.2.0-SLES15/Linux/lib/libesmf.so(_ZN5ESMCI5Array6createEPNS_9ArraySpecEPNS_8DistGridEPNS_10InterArrayIiEES7_S7_S7_S7_S7_S7_P14ESMC_IndexFlagP13ESMC_Pin_FlagS7_S7_S7_PiPNS_2VME+0x2707) [0x14bf6e44a267] What I'm mainly wondering is if anyone has any experience with an error like this? My guess (at the moment) is that I built things wrong for an Infiniband cluster maybe? I'm using Intel Fortran Classic 2021.11.0 with GCC 11.4.0 as a backing C compiler and I built as: mkdir build-ifort-2021.11.0 && cd build-ifort-2021.11.0 ../configure CC=icx CXX=icpx FC=ifort \ --with-ucx=embedded --with-hwloc=embedded --with-libfabric=embedded --with-yaksa=embedded \ --prefix=/discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0 |& tee configure.ifort-2021.11.0.log All those "embedded" flags are mainly because with Open MPI on this system, I have to do something similar with its configure step: --with-hwloc=internal --with-libevent=internal --with-pmix=internal so I figured I should do the same with MPICH> Now, at the end of the configure step I did see: ***************************************************** *** *** device : ch4:ofi (embedded libfabric) *** shm feature : auto *** gpu support : disabled *** MPICH is configured with device ch4:ofi, which should work for TCP networks and any high-bandwidth interconnect supported by libfabric. MPICH can also be configured with "--with-device=ch4:ucx", which should work for TCP networks and any high-bandwidth interconnect supported by the UCX library. In addition, the legacy device ch3 (--with-device=ch3) is also available. ***************************************************** And I did try the `--with-device=ch4:ucx` but that didn't seem to help. And the system I am on is an Infiniband network, so I imagine ofi should work. Note that this code works fine with Intel MPI and Open MPI (which are our "main" MPI stacks), but some of our external users are asking about MPICH support. Matt -- Matt Thompson, SSAI, Ld Scientific Prog/Analyst/Super NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 https://urldefense.us/v3/__http://science.gsfc.nasa.gov/sed/bio/matthew.tho… <https://urldefense.us/v3/__http:/science.gsfc.nasa.gov/sed/bio/matthew.thom…>

1 0

Re: [mpich-discuss] Unexpected behaviour of MPI_Probe + MPI_Get_count
by Zhou, Hui 27 Mar '24

27 Mar '24

It means the libfabric psm3 provider in 1.16.1 has a bug. :) Since the emebedded libfabric worked, it means they have fixed the bug in the recent versions. Different providers may have different performance. -- Hui ________________________________ From: Iker Martín Álvarez <martini(a)uji.es> Sent: Wednesday, March 27, 2024 6:55 AM To: Zhou, Hui <zhouh(a)anl.gov> Cc: discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] Unexpected behaviour of MPI_Probe + MPI_Get_count Hello Hui, After testing this provider, the issue has been resolved. I am not sure what the difference is between the two providers, but we are grateful that it works now. Thank you very much for your time. Kind regards, Iker Missatge de Zhou, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello Hui, After testing this provider, the issue has been resolved. I am not sure what the difference is between the two providers, but we are grateful that it works now. Thank you very much for your time. Kind regards, Iker Missatge de Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> del dia dt., 26 de març 2024 a les 15:37: Hi Iker, Could you try FI_PROVIDER=verbs? -- Hui ________________________________ From: Iker Martín Álvarez <martini(a)uji.es<mailto:[email protected]>> Sent: Tuesday, March 26, 2024 2:35 AM To: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Cc: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: [mpich-discuss] Unexpected behaviour of MPI_Probe + MPI_Get_count Hello Zhou, I just tried using both providers and in both cases the execution hangs at MPI_Init for those processes that are on a different node than the main one. This happens even for an MPI "Hello World" code for both providers. So ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello Zhou, I just tried using both providers and in both cases the execution hangs at MPI_Init for those processes that are on a different node than the main one. This happens even for an MPI "Hello World" code for both providers. So I can't even check if it has the same problem. Thank you very much for your time. Best regards, Iker Missatge de Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> del dia dl., 25 de març 2024 a les 23:05: Hi Iker, Could you try setting FI_PROVIDER=sockets or FI_PROVIDER=tcp to see if the issue persists? -- Hui ________________________________ From: Iker Martín Álvarez <martini(a)uji.es<mailto:[email protected]>> Sent: Monday, March 25, 2024 12:38 PM To: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Cc: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: [mpich-discuss] Unexpected behaviour of MPI_Probe + MPI_Get_count Hello Zhou, Thanks for the quick reply. In the attached file you can see the result of running the code with the environment variable you gave. Kind regards, Iker Missatge de Zhou, Hui <zhouh@ anl. gov> del dia dl. , 25 de març 2024 a les ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello Zhou, Thanks for the quick reply. In the attached file you can see the result of running the code with the environment variable you gave. Kind regards, Iker Missatge de Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> del dia dl., 25 de març 2024 a les 17:03: Hi Iker, Could you try reproduce the issue by setting MPIR_CVAR_DEBUG_SUMMARY=1, and report the console output? The issue may be in specific provider. The log should show that. -- Hui ________________________________ From: Iker Martín Álvarez via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Monday, March 25, 2024 6:23 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Iker Martín Álvarez <martini(a)uji.es<mailto:[email protected]>> Subject: [mpich-discuss] Unexpected behaviour of MPI_Probe + MPI_Get_count Hello, I recently encountered an unexpected behaviour of the MPI_Probe + MPI_Get_count functions under specific conditions. I was hoping that this forum could advise me on a solution. Specifically, the application performs an MPI_Send communication ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, I recently encountered an unexpected behaviour of the MPI_Probe + MPI_Get_count functions under specific conditions. I was hoping that this forum could advise me on a solution. Specifically, the application performs an MPI_Send communication from the root process to process B. Process B doesn't know the size of the message, so I use MPI_Probe + MPI_Get_count to discover it. However, as an example, if the size of the message is 1000 bytes, process B expects with MPI_Get_count function a total of 20 bytes. The problem only occurs with a specific installation of MPICH and when the following conditions are met in my code: - The problem only occurs in internode communications. - The problem only appears if derived types are used in the communication. Specifically a derived type to communicate a vector of integers and a vector of reals, both with the same number of elements. - None of the MPI functions give an error code. They all return MPI_Sucess. - If instead of allocating the amount of bytes returned by MPI_Get_count(=20), I allocate the expected value (1000), the message is received correctly. - The size returned by MPI_Get_count seems to be variable depending on the total number of addresses with which the derived type is created. I have attached the file to reproduce the problem. It can also be accessed via the GitLab link below: https://urldefense.us/v3/__https://lorca.act.uji.es/gitlab/martini/mpich_of… <https://urldefense.us/v3/__https://lorca.act.uji.es/gitlab/martini/mpich_of…> It is designed to be run with 3 processes, two of them hosted on one node and the third on a different one. As previously mentioned, this problem occurs when using MPICH with ch4:ofi without using the embedded option. Specifically, I have tested the following installations in which the error appears: - MPICH 4.2.0 with config options: '--with-device=ch4:ofi' '--with-libfabric=/home/martini/Instalaciones/libfabric-1.16.1' - MPICH 4.0.3 with config options: '--with-device=ch4:ofi' '--with-libfabric=/home/martini/Instalaciones/libfabric-1.16.1' - MPICH 4.0.3 with config options: '--with-device=ch4:ofi' '--with-libfabric=/home/martini/Instalaciones/libfabric-1.16.1' '--disable-psm3' - MPICH 3.4.1 with config options: '--with-device=ch4:ofi' '--with-libfabric=/home/martini/Instalaciones/libfabric-1.16.1' However, it does work as expected for the following MPICH installations: - MPICH 4.0.3 with config options: '--with-device=ch4:ofi' '--with-libfabric=embedded' - MPICH 4.0.3 with config options: '--with-device=ch4:ucx' '--with-ucx=/soft/gnu/ucx-1.11' - MPICH 3.4.1 with config options: '--with-device=ch4:ucx' '--with-ucx=/soft/gnu/ucx-1.11' Although for these installations the code does work, we would like to use a different libfabric installation than the embedded one because we get better networking performance. In the case of UCX, it is because the application in question uses the MPI_Comm_spawn call and MPICH does not currently support it with UCX. Thank you for your help. Best regards, Iker

1 0

Re: [mpich-discuss] Unexpected behaviour of MPI_Probe + MPI_Get_count
by Iker Martín Álvarez 27 Mar '24

27 Mar '24

Hello Hui, After testing this provider, the issue has been resolved. I am not sure what the difference is between the two providers, but we are grateful that it works now. Thank you very much for your time. Kind regards, Iker Missatge de Zhou, Hui <zhouh(a)anl.gov> del dia dt., 26 de març 2024 a les 15:37: > Hi Iker, > > Could you try FI_PROVIDER=verbs? > > -- > Hui > ------------------------------ > *From:* Iker Martín Álvarez <martini(a)uji.es> > *Sent:* Tuesday, March 26, 2024 2:35 AM > *To:* Zhou, Hui <zhouh(a)anl.gov> > *Cc:* discuss(a)mpich.org <discuss(a)mpich.org> > *Subject:* Re: [mpich-discuss] Unexpected behaviour of MPI_Probe + > MPI_Get_count > > Hello Zhou, I just tried using both providers and in both cases the > execution hangs at MPI_Init for those processes that are on a different > node than the main one. This happens even for an MPI "Hello World" code for > both providers. So > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > Hello Zhou, > > I just tried using both providers and in both cases the execution hangs at > MPI_Init for those processes that are on a different node than the main > one. This happens even for an MPI "Hello World" code for both providers. > > So I can't even check if it has the same problem. > > Thank you very much for your time. > Best regards, > Iker > > Missatge de Zhou, Hui <zhouh(a)anl.gov> del dia dl., 25 de març 2024 a les > 23:05: > > Hi Iker, > > Could you try setting FI_PROVIDER=sockets or FI_PROVIDER=tcp to see if > the issue persists? > > -- > Hui > ------------------------------ > *From:* Iker Martín Álvarez <martini(a)uji.es> > *Sent:* Monday, March 25, 2024 12:38 PM > *To:* Zhou, Hui <zhouh(a)anl.gov> > *Cc:* discuss(a)mpich.org <discuss(a)mpich.org> > *Subject:* Re: [mpich-discuss] Unexpected behaviour of MPI_Probe + > MPI_Get_count > > Hello Zhou, Thanks for the quick reply. In the attached file you can see > the result of running the code with the environment variable you gave. Kind > regards, Iker Missatge de Zhou, Hui <zhouh@ anl. gov> del dia dl. , 25 de > març 2024 a les > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > Hello Zhou, > > Thanks for the quick reply. > In the attached file you can see the result of running the code with the > environment variable you gave. > > Kind regards, > Iker > > Missatge de Zhou, Hui <zhouh(a)anl.gov> del dia dl., 25 de març 2024 a les > 17:03: > > Hi Iker, > > Could you try reproduce the issue by setting MPIR_CVAR_DEBUG_SUMMARY=1, > and report the console output? The issue may be in specific provider. The > log should show that. > > -- > Hui > ------------------------------ > *From:* Iker Martín Álvarez via discuss <discuss(a)mpich.org> > *Sent:* Monday, March 25, 2024 6:23 AM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* Iker Martín Álvarez <martini(a)uji.es> > *Subject:* [mpich-discuss] Unexpected behaviour of MPI_Probe + > MPI_Get_count > > Hello, I recently encountered an unexpected behaviour of the MPI_Probe + > MPI_Get_count functions under specific conditions. I was hoping that this > forum could advise me on a solution. Specifically, the application > performs an MPI_Send communication > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > Hello, > > I recently encountered an unexpected behaviour of the MPI_Probe + > MPI_Get_count functions under specific conditions. I was hoping that this > forum could advise me on a solution. > > Specifically, the application performs an MPI_Send communication from the > root process to process B. Process B doesn't know the size of the message, > so I use MPI_Probe + MPI_Get_count to discover it. However, as an example, > if the size of the message is 1000 bytes, process B expects with > MPI_Get_count function a total of 20 bytes. > > The problem only occurs with a specific installation of MPICH and when the > following conditions are met in my code: > - The problem only occurs in internode communications. > - The problem only appears if derived types are used in the communication. > Specifically a derived type to communicate a vector of integers and a > vector of reals, both with the same number of elements. > - None of the MPI functions give an error code. They all return MPI_Sucess. > - If instead of allocating the amount of bytes returned by > MPI_Get_count(=20), I allocate the expected value (1000), the message is > received correctly. > - The size returned by MPI_Get_count seems to be variable depending on the > total number of addresses with which the derived type is created. > > I have attached the file to reproduce the problem. It can also be accessed > via the GitLab link below: > https://urldefense.us/v3/__https://lorca.act.uji.es/gitlab/martini/mpich_of… > <https://urldefense.us/v3/__https://lorca.act.uji.es/gitlab/martini/mpich_of…> > It is designed to be run with 3 processes, two of them hosted on one node > and the third on a different one. > > As previously mentioned, this problem occurs when using MPICH with ch4:ofi > without using the embedded option. Specifically, I have tested the > following installations in which the error appears: > - MPICH 4.2.0 with config options: '--with-device=ch4:ofi' > '--with-libfabric=/home/martini/Instalaciones/libfabric-1.16.1' > - MPICH 4.0.3 with config options: '--with-device=ch4:ofi' > '--with-libfabric=/home/martini/Instalaciones/libfabric-1.16.1' > - MPICH 4.0.3 with config options: '--with-device=ch4:ofi' > '--with-libfabric=/home/martini/Instalaciones/libfabric-1.16.1' '--disable-psm3' > - MPICH 3.4.1 with config options: '--with-device=ch4:ofi' > '--with-libfabric=/home/martini/Instalaciones/libfabric-1.16.1' > > However, it does work as expected for the following MPICH installations: > - MPICH 4.0.3 with config options: '--with-device=ch4:ofi' > '--with-libfabric=embedded' > - MPICH 4.0.3 with config options: '--with-device=ch4:ucx' > '--with-ucx=/soft/gnu/ucx-1.11' > - MPICH 3.4.1 with config options: '--with-device=ch4:ucx' > '--with-ucx=/soft/gnu/ucx-1.11' > > Although for these installations the code does work, we would like to use > a different libfabric installation than the embedded one because we get > better networking performance. In the case of UCX, it is because the > application in question uses the MPI_Comm_spawn call and MPICH does not > currently support it with UCX. > > Thank you for your help. > Best regards, > Iker > >

1 0

Re: [mpich-discuss] [EXTERNAL] Re: Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?)
by Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] 26 Mar '24

26 Mar '24

Ken, I think so. Given the traceback, my guess is it's dealing with this code from ESMF v8.6.0 (https://urldefense.us/v3/__https://github.com/esmf-org/esmf/blob/ec5f186670… ) #if (MPI_VERSION >= 3) // set up communicator across single-system-images SSIs MPI_Comm_split_type(mpi_c, MPI_COMM_TYPE_SHARED, 0, MPI_INFO_NULL, &mpi_c_ssi); // set up communicator across root pets of each SSI int color; MPI_Comm_rank(mpi_c_ssi, &color); if (color>0) color = MPI_UNDEFINED; // only root PETs on each SSI MPI_Comm_split(mpi_c, color, 0, &mpi_c_ssi_roots); #endif Indeed, if it's coming from ESMF, that is the *only* MPI_Comm_split_type in the whole code! We also call MPI_Comm_split_type in the overall code (MAPL) and it also only uses that: https://urldefense.us/v3/__https://github.com/search?q=repo*3AGEOS-ESM*2FMA… -- Matt Thompson, SSAI, Ld Scientific Prog/Analyst/Super NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 https://urldefense.us/v3/__http://science.gsfc.nasa.gov/sed/bio/matthew.tho… From: Raffenetti, Ken <raffenet(a)anl.gov> Date: Tuesday, March 26, 2024 at 10:55 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?) It looks like the crash is happening after shared memory window creation fails. The failure path is getting tripped up removing the window id from the global hash, since it was never added. We will address this in the code so users get a better error message after the failure. Can you confirm that the input communicator to the window creation function is one created with MPI_Comm_split_type(…,MPI_COMM_TYPE_SHARED,…)? Thanks, Ken From: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Tuesday, March 26, 2024 at 9:20 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson(a)nasa.gov> Subject: [mpich-discuss] Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?) All, I've been trying to get a code of mine working with MPICH 4. 2. 0. I can build MPICH just fine and then build our base libraries and then model and all compiles fine. Hello world runs fine on multiple nodes as well. But when I finally try ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd All, I've been trying to get a code of mine working with MPICH 4.2.0. I can build MPICH just fine and then build our base libraries and then model and all compiles fine. Hello world runs fine on multiple nodes as well. But when I finally try and run our complex model: Assertion failed in file ../src/mpid/ch4/src/ch4_impl.h at line 838: map_entry != NULL /discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0/lib/libmpi.so.12(+0x37d211) [0x14bf4f62c211] /discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0/lib/libmpi.so.12(PMPI_Win_allocate_shared+0x3ba) [0x14bf4f3e452a] /discover/swdev/gmao_SIteam/Baselibs/ESMA-Baselibs-7.20.0/x86_64-pc-linux-gnu/ifort_2021.11.0-mpich_4.2.0-SLES15/Linux/lib/libesmf.so(_ZN5ESMCI3VMK14ssishmAllocateERSt6vectorImSaImEEPNS0_9memhandleEb+0x18b) [0x14bf6e91481b] /discover/swdev/gmao_SIteam/Baselibs/ESMA-Baselibs-7.20.0/x86_64-pc-linux-gnu/ifort_2021.11.0-mpich_4.2.0-SLES15/Linux/lib/libesmf.so(_ZN5ESMCI5Array6createEPNS_9ArraySpecEPNS_8DistGridEPNS_10InterArrayIiEES7_S7_S7_S7_S7_S7_P14ESMC_IndexFlagP13ESMC_Pin_FlagS7_S7_S7_PiPNS_2VME+0x2707) [0x14bf6e44a267] What I'm mainly wondering is if anyone has any experience with an error like this? My guess (at the moment) is that I built things wrong for an Infiniband cluster maybe? I'm using Intel Fortran Classic 2021.11.0 with GCC 11.4.0 as a backing C compiler and I built as: mkdir build-ifort-2021.11.0 && cd build-ifort-2021.11.0 ../configure CC=icx CXX=icpx FC=ifort \ --with-ucx=embedded --with-hwloc=embedded --with-libfabric=embedded --with-yaksa=embedded \ --prefix=/discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0 |& tee configure.ifort-2021.11.0.log All those "embedded" flags are mainly because with Open MPI on this system, I have to do something similar with its configure step: --with-hwloc=internal --with-libevent=internal --with-pmix=internal so I figured I should do the same with MPICH> Now, at the end of the configure step I did see: ***************************************************** *** *** device : ch4:ofi (embedded libfabric) *** shm feature : auto *** gpu support : disabled *** MPICH is configured with device ch4:ofi, which should work for TCP networks and any high-bandwidth interconnect supported by libfabric. MPICH can also be configured with "--with-device=ch4:ucx", which should work for TCP networks and any high-bandwidth interconnect supported by the UCX library. In addition, the legacy device ch3 (--with-device=ch3) is also available. ***************************************************** And I did try the `--with-device=ch4:ucx` but that didn't seem to help. And the system I am on is an Infiniband network, so I imagine ofi should work. Note that this code works fine with Intel MPI and Open MPI (which are our "main" MPI stacks), but some of our external users are asking about MPICH support. Matt -- Matt Thompson, SSAI, Ld Scientific Prog/Analyst/Super NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 https://urldefense.us/v3/__http://science.gsfc.nasa.gov/sed/bio/matthew.tho… <https://urldefense.us/v3/__http:/science.gsfc.nasa.gov/sed/bio/matthew.thom…>

1 0

Re: [mpich-discuss] Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?)
by Raffenetti, Ken 26 Mar '24

26 Mar '24

It looks like the crash is happening after shared memory window creation fails. The failure path is getting tripped up removing the window id from the global hash, since it was never added. We will address this in the code so users get a better error message after the failure. Can you confirm that the input communicator to the window creation function is one created with MPI_Comm_split_type(…,MPI_COMM_TYPE_SHARED,…)? Thanks, Ken From: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Tuesday, March 26, 2024 at 9:20 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson(a)nasa.gov> Subject: [mpich-discuss] Help with MPICH 4.2.0 and win_allocate_shared (or maybe infiniband?) All, I've been trying to get a code of mine working with MPICH 4. 2. 0. I can build MPICH just fine and then build our base libraries and then model and all compiles fine. Hello world runs fine on multiple nodes as well. But when I finally try ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd All, I've been trying to get a code of mine working with MPICH 4.2.0. I can build MPICH just fine and then build our base libraries and then model and all compiles fine. Hello world runs fine on multiple nodes as well. But when I finally try and run our complex model: Assertion failed in file ../src/mpid/ch4/src/ch4_impl.h at line 838: map_entry != NULL /discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0/lib/libmpi.so.12(+0x37d211) [0x14bf4f62c211] /discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0/lib/libmpi.so.12(PMPI_Win_allocate_shared+0x3ba) [0x14bf4f3e452a] /discover/swdev/gmao_SIteam/Baselibs/ESMA-Baselibs-7.20.0/x86_64-pc-linux-gnu/ifort_2021.11.0-mpich_4.2.0-SLES15/Linux/lib/libesmf.so(_ZN5ESMCI3VMK14ssishmAllocateERSt6vectorImSaImEEPNS0_9memhandleEb+0x18b) [0x14bf6e91481b] /discover/swdev/gmao_SIteam/Baselibs/ESMA-Baselibs-7.20.0/x86_64-pc-linux-gnu/ifort_2021.11.0-mpich_4.2.0-SLES15/Linux/lib/libesmf.so(_ZN5ESMCI5Array6createEPNS_9ArraySpecEPNS_8DistGridEPNS_10InterArrayIiEES7_S7_S7_S7_S7_S7_P14ESMC_IndexFlagP13ESMC_Pin_FlagS7_S7_S7_PiPNS_2VME+0x2707) [0x14bf6e44a267] What I'm mainly wondering is if anyone has any experience with an error like this? My guess (at the moment) is that I built things wrong for an Infiniband cluster maybe? I'm using Intel Fortran Classic 2021.11.0 with GCC 11.4.0 as a backing C compiler and I built as: mkdir build-ifort-2021.11.0 && cd build-ifort-2021.11.0 ../configure CC=icx CXX=icpx FC=ifort \ --with-ucx=embedded --with-hwloc=embedded --with-libfabric=embedded --with-yaksa=embedded \ --prefix=/discover/swdev/gmao_SIteam/MPI/mpich/4.2.0-SLES15/ifort-2021.11.0 |& tee configure.ifort-2021.11.0.log All those "embedded" flags are mainly because with Open MPI on this system, I have to do something similar with its configure step: --with-hwloc=internal --with-libevent=internal --with-pmix=internal so I figured I should do the same with MPICH> Now, at the end of the configure step I did see: ***************************************************** *** *** device : ch4:ofi (embedded libfabric) *** shm feature : auto *** gpu support : disabled *** MPICH is configured with device ch4:ofi, which should work for TCP networks and any high-bandwidth interconnect supported by libfabric. MPICH can also be configured with "--with-device=ch4:ucx", which should work for TCP networks and any high-bandwidth interconnect supported by the UCX library. In addition, the legacy device ch3 (--with-device=ch3) is also available. ***************************************************** And I did try the `--with-device=ch4:ucx` but that didn't seem to help. And the system I am on is an Infiniband network, so I imagine ofi should work. Note that this code works fine with Intel MPI and Open MPI (which are our "main" MPI stacks), but some of our external users are asking about MPICH support. Matt -- Matt Thompson, SSAI, Ld Scientific Prog/Analyst/Super NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 https://urldefense.us/v3/__http://science.gsfc.nasa.gov/sed/bio/matthew.tho… <https://urldefense.us/v3/__http:/science.gsfc.nasa.gov/sed/bio/matthew.thom…>

1 0