- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Why stuck in MPI_Finalize?
by Zhou, Hui 29 Nov '21

29 Nov '21

Hi Erik, Shouldn't MPI_Finalize run by the garbage collector as well by treating the entire MPI as an object? -- Hui Zhou ________________________________ From: Erik Schnetter <schnetter(a)gmail.com> Sent: Monday, November 29, 2021 7:23 PM To: Zhou, Hui <zhouh(a)anl.gov> Cc: discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] Why stuck in MPI_Finalize? Hui Julia is garbage collected, and our MPI wrappers free MPI handles automatically when the wrapped handles become garbage collected. I thus don't have direct control over when and which handles are freed before MPI_Finalize is called at the end of the test case. Our approach is to let things be as they are, ensuring that no MPI calls are made any more when MPI_Finalized returns true. That means that different MPI processes might (or might not ) free handles before calling MPI_Finalize. I notice that this seems to work fine on Linux, but not always on macOS. I don't have a simple reproducer in C. Pointers to setting up Julia: See e.g. <https://github.com/JuliaParallel/MPI.jl/blob/master/.github/workflows/UnitT…> which runs the unit tests for the Julia package `MPI.jl` which wraps MPI implementations. There are several stanzas; the one called `test-system-brew` runs MPI.jl on a system-provided (external MPI) installation that has been installed via Homebrew on macOS. You would replace the step "Install MPI via homebrew" by a step that downloads and installs MPICH into /usr/local. I'll be happy to discuss details, possibly over Zoom. -erik On Mon, Nov 29, 2021 at 5:12 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > > Hi Erik, > > It is not just a barrier in MPI_Finalize. It also needs to make sure pending communications are completed. Sometimes this is the user's fault, such as mismatched communication or partially freed communicators. But sometimes this is due to lower-level network library. For example, some libfabric provider does not always flush send. We tried to detect and prevent all cases that result in hanging, but there always seem to be cases that escape our solution. If you can drill down some simple reproducible (even with 5% chance) cases, please create a github issue and we'll track it down. > > By the way, if you have pointers on setting up Julia CI testing, we can try to setup nightly testing on our end as well. Catching the errors when it appear always makes troubleshooting easier. > > -- > Hui > ________________________________ > From: Erik Schnetter via discuss <discuss(a)mpich.org> > Sent: Monday, November 29, 2021 12:30 PM > To: discuss(a)mpich.org <discuss(a)mpich.org> > Cc: Erik Schnetter <schnetter(a)gmail.com> > Subject: [mpich-discuss] Why stuck in MPI_Finalize? > > I have a Julia test case on macOS where MPICH randomly gets stuck in > MPI_Finalize (with about a 5% chance). See e.g. > https://github.com/JuliaParallel/MPI.jl/runs/4357341818 > > Can you advise under what circumstances MPICH could get stuck there? > The respective run uses 3 processes, and all 3 processes call into > MPI_Finalize, but no process returns. > > I assume that MPI_Finalize contains internally the equivalent to an > MPI_Barrier, but that should succeed here. Are there other actions > taken in MPI_Finalize that would require some kind of consistent state > across the application? For example, if a communicator was created on > all processes, but freed only on some processes, could this cause such > a deadlock? > > -erik > > -- > Erik Schnetter <schnetter(a)gmail.com> > http://www.perimeterinstitute.ca/personal/eschnetter/ > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Erik Schnetter <schnetter(a)gmail.com> http://www.perimeterinstitute.ca/personal/eschnetter/

1 0

Re: [mpich-discuss] Why stuck in MPI_Finalize?
by Erik Schnetter 29 Nov '21

29 Nov '21

Hui Julia is garbage collected, and our MPI wrappers free MPI handles automatically when the wrapped handles become garbage collected. I thus don't have direct control over when and which handles are freed before MPI_Finalize is called at the end of the test case. Our approach is to let things be as they are, ensuring that no MPI calls are made any more when MPI_Finalized returns true. That means that different MPI processes might (or might not ) free handles before calling MPI_Finalize. I notice that this seems to work fine on Linux, but not always on macOS. I don't have a simple reproducer in C. Pointers to setting up Julia: See e.g. <https://github.com/JuliaParallel/MPI.jl/blob/master/.github/workflows/UnitT…> which runs the unit tests for the Julia package `MPI.jl` which wraps MPI implementations. There are several stanzas; the one called `test-system-brew` runs MPI.jl on a system-provided (external MPI) installation that has been installed via Homebrew on macOS. You would replace the step "Install MPI via homebrew" by a step that downloads and installs MPICH into /usr/local. I'll be happy to discuss details, possibly over Zoom. -erik On Mon, Nov 29, 2021 at 5:12 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > > Hi Erik, > > It is not just a barrier in MPI_Finalize. It also needs to make sure pending communications are completed. Sometimes this is the user's fault, such as mismatched communication or partially freed communicators. But sometimes this is due to lower-level network library. For example, some libfabric provider does not always flush send. We tried to detect and prevent all cases that result in hanging, but there always seem to be cases that escape our solution. If you can drill down some simple reproducible (even with 5% chance) cases, please create a github issue and we'll track it down. > > By the way, if you have pointers on setting up Julia CI testing, we can try to setup nightly testing on our end as well. Catching the errors when it appear always makes troubleshooting easier. > > -- > Hui > ________________________________ > From: Erik Schnetter via discuss <discuss(a)mpich.org> > Sent: Monday, November 29, 2021 12:30 PM > To: discuss(a)mpich.org <discuss(a)mpich.org> > Cc: Erik Schnetter <schnetter(a)gmail.com> > Subject: [mpich-discuss] Why stuck in MPI_Finalize? > > I have a Julia test case on macOS where MPICH randomly gets stuck in > MPI_Finalize (with about a 5% chance). See e.g. > https://github.com/JuliaParallel/MPI.jl/runs/4357341818 > > Can you advise under what circumstances MPICH could get stuck there? > The respective run uses 3 processes, and all 3 processes call into > MPI_Finalize, but no process returns. > > I assume that MPI_Finalize contains internally the equivalent to an > MPI_Barrier, but that should succeed here. Are there other actions > taken in MPI_Finalize that would require some kind of consistent state > across the application? For example, if a communicator was created on > all processes, but freed only on some processes, could this cause such > a deadlock? > > -erik > > -- > Erik Schnetter <schnetter(a)gmail.com> > http://www.perimeterinstitute.ca/personal/eschnetter/ > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Erik Schnetter <schnetter(a)gmail.com> http://www.perimeterinstitute.ca/personal/eschnetter/

1 0

Re: [mpich-discuss] Why stuck in MPI_Finalize?
by Zhou, Hui 29 Nov '21

29 Nov '21

Hi Erik, It is not just a barrier in MPI_Finalize. It also needs to make sure pending communications are completed. Sometimes this is the user's fault, such as mismatched communication or partially freed communicators. But sometimes this is due to lower-level network library. For example, some libfabric provider does not always flush send. We tried to detect and prevent all cases that result in hanging, but there always seem to be cases that escape our solution. If you can drill down some simple reproducible (even with 5% chance) cases, please create a github issue and we'll track it down. By the way, if you have pointers on setting up Julia CI testing, we can try to setup nightly testing on our end as well. Catching the errors when it appear always makes troubleshooting easier. -- Hui ________________________________ From: Erik Schnetter via discuss <discuss(a)mpich.org> Sent: Monday, November 29, 2021 12:30 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Erik Schnetter <schnetter(a)gmail.com> Subject: [mpich-discuss] Why stuck in MPI_Finalize? I have a Julia test case on macOS where MPICH randomly gets stuck in MPI_Finalize (with about a 5% chance). See e.g. https://github.com/JuliaParallel/MPI.jl/runs/4357341818 Can you advise under what circumstances MPICH could get stuck there? The respective run uses 3 processes, and all 3 processes call into MPI_Finalize, but no process returns. I assume that MPI_Finalize contains internally the equivalent to an MPI_Barrier, but that should succeed here. Are there other actions taken in MPI_Finalize that would require some kind of consistent state across the application? For example, if a communicator was created on all processes, but freed only on some processes, could this cause such a deadlock? -erik -- Erik Schnetter <schnetter(a)gmail.com> http://www.perimeterinstitute.ca/personal/eschnetter/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Why stuck in MPI_Finalize?
by Erik Schnetter 29 Nov '21

29 Nov '21

I have a Julia test case on macOS where MPICH randomly gets stuck in MPI_Finalize (with about a 5% chance). See e.g. https://github.com/JuliaParallel/MPI.jl/runs/4357341818 Can you advise under what circumstances MPICH could get stuck there? The respective run uses 3 processes, and all 3 processes call into MPI_Finalize, but no process returns. I assume that MPI_Finalize contains internally the equivalent to an MPI_Barrier, but that should succeed here. Are there other actions taken in MPI_Finalize that would require some kind of consistent state across the application? For example, if a communicator was created on all processes, but freed only on some processes, could this cause such a deadlock? -erik -- Erik Schnetter <schnetter(a)gmail.com> http://www.perimeterinstitute.ca/personal/eschnetter/

1 0

Re: [mpich-discuss] MPI_Comm_spawn and SLURM
by Mccall, Kurt E. (MSFC-EV41) 27 Nov '21

27 Nov '21

Hui, Looks like the child process failed to launch, so it may be a process manager issue. I ran "scontrol show hosts" and it showed a number of possible errors in the compute node SLURM configuration (like "CoresPerSocket=1", which is wrong). Looks like our compute nodes are not configured correctly for SLURM. Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Friday, November 26, 2021 10:13 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: MPI_Comm_spawn and SLURM Kurt, Could you verify whether the children process is launched? The way `MPI_Comm_spawn_multiple` works is - 1. Parent root open a port 2. Parent root call PMI routine to spawn children processes - here the process manager is SLURM. If the children processes failed to spawn, then it is likely the process manager issue. 3. Children processes run MPI_Init, and at the end internally call MPI_Comm_connect 4. Meanwhile, the parent root process wait in `MPI_Comm_accept` If root process is hanging in `MPI_Comm_accept`, then either the process manager failed to launch children processes, or children processes are misconfigured by process manager and didn't call `MPI_Comm_connect`. Let's determine where children are at. -- Hui Zhou From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Friday, November 26, 2021 at 5:05 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [mpich-discuss] MPI_Comm_spawn and SLURM I am attempting to run MPICH under SLURM for the first time, and there could be a lot of things I am doing wrong. All processes are getting launched but the master process is freezing in MPI_Comm_spawn. The stack trace is below, followed by the SLURM command I use to start the job. Even though all 20 processes are running, one per node as desired, salloc reports that "Requested node configuration is not available". Not sure if that is why MPI_Comm_spawn is frozen. Thanks for any help. #0 0x00007f0183d08a08 in poll () from /lib64/libc.so.6 #1 0x00007f0185aed611 in MPID_nem_tcp_connpoll (in_blocking_poll=<optimized out>) at ../mpich-4.0b1/src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c:1765 #2 0x00007f0185ad8186 in MPID_nem_mpich_blocking_recv (completions=<optimized out>, in_fbox=<synthetic pointer>, cell=<synthetic pointer>) at ../mpich-4.0b1/src/mpid/ch3/channels/nemesis/include/mpid_nem_inline.h:947 #3 MPIDI_CH3I_Progress (progress_state=progress_state@entry=0x7ffcc4839760, is_blocking=is_blocking@entry=1) at ../mpich-4.0b1/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:360 #4 0x00007f0185a8811a in MPIDI_Create_inter_root_communicator_accept (vc_pptr=<synthetic pointer>, comm_pptr=<synthetic pointer>, port_name=<optimized out>) at ../mpich-4.0b1/src/mpid/ch3/src/ch3u_port.c:417 #5 MPIDI_Comm_accept (port_name=<optimized out>, info=<optimized out>, root=0, comm_ptr=0x7f0185ef1540 <MPIR_Comm_builtin+832>, newcomm=0x7ffcc4839bb8) at ../mpich-4.0b1/src/mpid/ch3/src/ch3u_port.c:1176 #6 0x00007f0185ac3d45 in MPID_Comm_accept ( port_name=port_name@entry=0x7ffcc4839a00 "tag#0$description#n001$port#55907$ifname#172.16.56.1$", info=info@entry=0x0, root=root@entry=0, comm=comm@entry=0x7f0185ef1540 <MPIR_Comm_builtin+832>, newcomm_ptr=newcomm_ptr@entry=0x7ffcc4839bb8) at ../mpich-4.0b1/src/mpid/ch3/src/mpid_port.c:130 #7 0x00007f0185a73285 in MPIDI_Comm_spawn_multiple (count=<optimized out>, commands=0x7ffcc4839b78, argvs=0x7ffcc4839b70, maxprocs=0x7ffcc4839b6c, info_ptrs=<optimized out>, root=<optimized out>, comm_ptr=0x7f0185ef1540 <MPIR_Comm_builtin+832>, intercomm=0x7ffcc4839bb8, errcodes=<optimized out>) at ../mpich-4.0b1/src/mpid/ch3/src/ch3u_comm_spawn_multiple.c:258 #8 0x00007f0185abec99 in MPID_Comm_spawn_multiple (count=count@entry=1, array_of_commands=array_of_commands@entry=0x7ffcc4839b78, array_of_argv=array_of_argv@entry=0x7ffcc4839b70, array_of_maxprocs=array_of_maxprocs@entry=0x7ffcc4839b6c, array_of_info_ptrs=array_of_info_ptrs@entry=0x7ffcc4839b60, root=root@entry=0, comm_ptr=0x7f0185ef1540 <MPIR_Comm_builtin+832>, intercomm=0x7ffcc4839bb8, array_of_errcodes=0x7ffcc4839c98) at ../mpich-4.0b1/src/mpid/ch3/src/mpid_comm_spawn_multiple.c:49 #9 0x00007f0185a34895 in MPIR_Comm_spawn_impl (command=<optimized out>, command@entry=0x995428 "NeedlesMpiMM", argv=<optimized out>, argv@entry=0x993f50, maxprocs=<optimized out>, maxprocs@entry=1, info_ptr=<optimized out>, root=root@entry=0, comm_ptr=comm_ptr@entry=0x7f0185ef1540 <MPIR_Comm_builtin+832>, p_intercomm_ptr=0x7ffcc4839bb8, array_of_errcodes=0x7ffcc4839c98) at ../mpich-4.0b1/src/mpi/spawn/spawn_impl.c:168 #10 0x00007f0185953637 in internal_Comm_spawn (array_of_errcodes=0x7ffcc4839c98, intercomm=0x7ffcc4839d7c, comm=1140850689, root=0, info=-1677721600, maxprocs=1, argv=<optimized out>, command=0x995428 "NeedlesMpiMM") at ../mpich-4.0b1/src/binding/c/spawn/comm_spawn.c:83 #11 PMPI_Comm_spawn (command=0x995428 "NeedlesMpiMM", argv=0x993f50, maxprocs=1, info=-1677721600, root=0, comm=1140850689, intercomm=0x7ffcc4839d7c, array_of_errcodes=0x7ffcc4839c98) at ../mpich-4.0b1/src/binding/c/spawn/comm_spawn.c:169 #12 0x000000000040cec3 in needles::NeedlesMpiMaster::spawnNewManager (this=0x995380, nodenum=0, host_name="n001.cluster.pssclabs.com", intercom=@0x7ffcc4839d7c: 67108864) at src/NeedlesMpiMaster.cpp:1432 #13 0x00000000004084cb in needles::NeedlesMpiMaster::init (this=0x995380, argc=23, argv=0x7ffcc483a548, rank=0, world_size=20) at src/NeedlesMpiMaster.cpp:246 #14 0x0000000000406799 in main (argc=23, argv=0x7ffcc483a548) at src/NeedlesMpiManagerMain.cpp:96 Here is my salloc command to start the job. I want one task per node, reserving the rest of the cores on the node for spawning of additional processes. $ salloc -ntasks=20 --cpus-per-task=24 -verbose Here is what salloc reports: salloc: -------------------- -------------------- salloc: cpus-per-task : 24 salloc: ntasks : 20 salloc: verbose : 1 salloc: -------------------- -------------------- salloc: end of defined options salloc: Linear node selection plugin loaded with argument 4 salloc: select/cons_res loaded with argument 4 salloc: Cray/Aries node selection plugin loaded salloc: select/cons_tres loaded with argument 4 salloc: Granted job allocation 34311 srun: error: Unable to create step for job 34311: Requested node configuration is not available

1 0

Re: [mpich-discuss] MPI_Comm_spawn and SLURM
by Zhou, Hui 26 Nov '21

26 Nov '21

Kurt, Could you verify whether the children process is launched? The way `MPI_Comm_spawn_multiple` works is – 1. Parent root open a port 2. Parent root call PMI routine to spawn children processes – here the process manager is SLURM. If the children processes failed to spawn, then it is likely the process manager issue. 3. Children processes run MPI_Init, and at the end internally call MPI_Comm_connect 4. Meanwhile, the parent root process wait in `MPI_Comm_accept` If root process is hanging in `MPI_Comm_accept`, then either the process manager failed to launch children processes, or children processes are misconfigured by process manager and didn’t call `MPI_Comm_connect`. Let’s determine where children are at. -- Hui Zhou From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Date: Friday, November 26, 2021 at 5:05 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] MPI_Comm_spawn and SLURM I am attempting to run MPICH under SLURM for the first time, and there could be a lot of things I am doing wrong. All processes are getting launched but the master process is freezing in MPI_Comm_spawn. The stack trace is below, followed by the SLURM command I use to start the job. Even though all 20 processes are running, one per node as desired, salloc reports that “Requested node configuration is not available". Not sure if that is why MPI_Comm_spawn is frozen. Thanks for any help. #0 0x00007f0183d08a08 in poll () from /lib64/libc.so.6 #1 0x00007f0185aed611 in MPID_nem_tcp_connpoll (in_blocking_poll=<optimized out>) at ../mpich-4.0b1/src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c:1765 #2 0x00007f0185ad8186 in MPID_nem_mpich_blocking_recv (completions=<optimized out>, in_fbox=<synthetic pointer>, cell=<synthetic pointer>) at ../mpich-4.0b1/src/mpid/ch3/channels/nemesis/include/mpid_nem_inline.h:947 #3 MPIDI_CH3I_Progress (progress_state=progress_state@entry=0x7ffcc4839760, is_blocking=is_blocking@entry=1) at ../mpich-4.0b1/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:360 #4 0x00007f0185a8811a in MPIDI_Create_inter_root_communicator_accept (vc_pptr=<synthetic pointer>, comm_pptr=<synthetic pointer>, port_name=<optimized out>) at ../mpich-4.0b1/src/mpid/ch3/src/ch3u_port.c:417 #5 MPIDI_Comm_accept (port_name=<optimized out>, info=<optimized out>, root=0, comm_ptr=0x7f0185ef1540 <MPIR_Comm_builtin+832>, newcomm=0x7ffcc4839bb8) at ../mpich-4.0b1/src/mpid/ch3/src/ch3u_port.c:1176 #6 0x00007f0185ac3d45 in MPID_Comm_accept ( port_name=port_name@entry=0x7ffcc4839a00 "tag#0$description#n001$port#55907$ifname#172.16.56.1$", info=info@entry=0x0, root=root@entry=0, comm=comm@entry=0x7f0185ef1540 <MPIR_Comm_builtin+832>, newcomm_ptr=newcomm_ptr@entry=0x7ffcc4839bb8) at ../mpich-4.0b1/src/mpid/ch3/src/mpid_port.c:130 #7 0x00007f0185a73285 in MPIDI_Comm_spawn_multiple (count=<optimized out>, commands=0x7ffcc4839b78, argvs=0x7ffcc4839b70, maxprocs=0x7ffcc4839b6c, info_ptrs=<optimized out>, root=<optimized out>, comm_ptr=0x7f0185ef1540 <MPIR_Comm_builtin+832>, intercomm=0x7ffcc4839bb8, errcodes=<optimized out>) at ../mpich-4.0b1/src/mpid/ch3/src/ch3u_comm_spawn_multiple.c:258 #8 0x00007f0185abec99 in MPID_Comm_spawn_multiple (count=count@entry=1, array_of_commands=array_of_commands@entry=0x7ffcc4839b78, array_of_argv=array_of_argv@entry=0x7ffcc4839b70, array_of_maxprocs=array_of_maxprocs@entry=0x7ffcc4839b6c, array_of_info_ptrs=array_of_info_ptrs@entry=0x7ffcc4839b60, root=root@entry=0, comm_ptr=0x7f0185ef1540 <MPIR_Comm_builtin+832>, intercomm=0x7ffcc4839bb8, array_of_errcodes=0x7ffcc4839c98) at ../mpich-4.0b1/src/mpid/ch3/src/mpid_comm_spawn_multiple.c:49 #9 0x00007f0185a34895 in MPIR_Comm_spawn_impl (command=<optimized out>, command@entry=0x995428 "NeedlesMpiMM", argv=<optimized out>, argv@entry=0x993f50, maxprocs=<optimized out>, maxprocs@entry=1, info_ptr=<optimized out>, root=root@entry=0, comm_ptr=comm_ptr@entry=0x7f0185ef1540 <MPIR_Comm_builtin+832>, p_intercomm_ptr=0x7ffcc4839bb8, array_of_errcodes=0x7ffcc4839c98) at ../mpich-4.0b1/src/mpi/spawn/spawn_impl.c:168 #10 0x00007f0185953637 in internal_Comm_spawn (array_of_errcodes=0x7ffcc4839c98, intercomm=0x7ffcc4839d7c, comm=1140850689, root=0, info=-1677721600, maxprocs=1, argv=<optimized out>, command=0x995428 "NeedlesMpiMM") at ../mpich-4.0b1/src/binding/c/spawn/comm_spawn.c:83 #11 PMPI_Comm_spawn (command=0x995428 "NeedlesMpiMM", argv=0x993f50, maxprocs=1, info=-1677721600, root=0, comm=1140850689, intercomm=0x7ffcc4839d7c, array_of_errcodes=0x7ffcc4839c98) at ../mpich-4.0b1/src/binding/c/spawn/comm_spawn.c:169 #12 0x000000000040cec3 in needles::NeedlesMpiMaster::spawnNewManager (this=0x995380, nodenum=0, host_name="n001.cluster.pssclabs.com", intercom=@0x7ffcc4839d7c: 67108864) at src/NeedlesMpiMaster.cpp:1432 #13 0x00000000004084cb in needles::NeedlesMpiMaster::init (this=0x995380, argc=23, argv=0x7ffcc483a548, rank=0, world_size=20) at src/NeedlesMpiMaster.cpp:246 #14 0x0000000000406799 in main (argc=23, argv=0x7ffcc483a548) at src/NeedlesMpiManagerMain.cpp:96 Here is my salloc command to start the job. I want one task per node, reserving the rest of the cores on the node for spawning of additional processes. $ salloc –ntasks=20 --cpus-per-task=24 –verbose Here is what salloc reports: salloc: -------------------- -------------------- salloc: cpus-per-task : 24 salloc: ntasks : 20 salloc: verbose : 1 salloc: -------------------- -------------------- salloc: end of defined options salloc: Linear node selection plugin loaded with argument 4 salloc: select/cons_res loaded with argument 4 salloc: Cray/Aries node selection plugin loaded salloc: select/cons_tres loaded with argument 4 salloc: Granted job allocation 34311 srun: error: Unable to create step for job 34311: Requested node configuration is not available

1 0

MPI_Comm_spawn and SLURM
by Mccall, Kurt E. (MSFC-EV41) 26 Nov '21

26 Nov '21

I am attempting to run MPICH under SLURM for the first time, and there could be a lot of things I am doing wrong. All processes are getting launched but the master process is freezing in MPI_Comm_spawn. The stack trace is below, followed by the SLURM command I use to start the job. Even though all 20 processes are running, one per node as desired, salloc reports that "Requested node configuration is not available". Not sure if that is why MPI_Comm_spawn is frozen. Thanks for any help. #0 0x00007f0183d08a08 in poll () from /lib64/libc.so.6 #1 0x00007f0185aed611 in MPID_nem_tcp_connpoll (in_blocking_poll=<optimized out>) at ../mpich-4.0b1/src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c:1765 #2 0x00007f0185ad8186 in MPID_nem_mpich_blocking_recv (completions=<optimized out>, in_fbox=<synthetic pointer>, cell=<synthetic pointer>) at ../mpich-4.0b1/src/mpid/ch3/channels/nemesis/include/mpid_nem_inline.h:947 #3 MPIDI_CH3I_Progress (progress_state=progress_state@entry=0x7ffcc4839760, is_blocking=is_blocking@entry=1) at ../mpich-4.0b1/src/mpid/ch3/channels/nemesis/src/ch3_progress.c:360 #4 0x00007f0185a8811a in MPIDI_Create_inter_root_communicator_accept (vc_pptr=<synthetic pointer>, comm_pptr=<synthetic pointer>, port_name=<optimized out>) at ../mpich-4.0b1/src/mpid/ch3/src/ch3u_port.c:417 #5 MPIDI_Comm_accept (port_name=<optimized out>, info=<optimized out>, root=0, comm_ptr=0x7f0185ef1540 <MPIR_Comm_builtin+832>, newcomm=0x7ffcc4839bb8) at ../mpich-4.0b1/src/mpid/ch3/src/ch3u_port.c:1176 #6 0x00007f0185ac3d45 in MPID_Comm_accept ( port_name=port_name@entry=0x7ffcc4839a00 "tag#0$description#n001$port#55907$ifname#172.16.56.1$", info=info@entry=0x0, root=root@entry=0, comm=comm@entry=0x7f0185ef1540 <MPIR_Comm_builtin+832>, newcomm_ptr=newcomm_ptr@entry=0x7ffcc4839bb8) at ../mpich-4.0b1/src/mpid/ch3/src/mpid_port.c:130 #7 0x00007f0185a73285 in MPIDI_Comm_spawn_multiple (count=<optimized out>, commands=0x7ffcc4839b78, argvs=0x7ffcc4839b70, maxprocs=0x7ffcc4839b6c, info_ptrs=<optimized out>, root=<optimized out>, comm_ptr=0x7f0185ef1540 <MPIR_Comm_builtin+832>, intercomm=0x7ffcc4839bb8, errcodes=<optimized out>) at ../mpich-4.0b1/src/mpid/ch3/src/ch3u_comm_spawn_multiple.c:258 #8 0x00007f0185abec99 in MPID_Comm_spawn_multiple (count=count@entry=1, array_of_commands=array_of_commands@entry=0x7ffcc4839b78, array_of_argv=array_of_argv@entry=0x7ffcc4839b70, array_of_maxprocs=array_of_maxprocs@entry=0x7ffcc4839b6c, array_of_info_ptrs=array_of_info_ptrs@entry=0x7ffcc4839b60, root=root@entry=0, comm_ptr=0x7f0185ef1540 <MPIR_Comm_builtin+832>, intercomm=0x7ffcc4839bb8, array_of_errcodes=0x7ffcc4839c98) at ../mpich-4.0b1/src/mpid/ch3/src/mpid_comm_spawn_multiple.c:49 #9 0x00007f0185a34895 in MPIR_Comm_spawn_impl (command=<optimized out>, command@entry=0x995428 "NeedlesMpiMM", argv=<optimized out>, argv@entry=0x993f50, maxprocs=<optimized out>, maxprocs@entry=1, info_ptr=<optimized out>, root=root@entry=0, comm_ptr=comm_ptr@entry=0x7f0185ef1540 <MPIR_Comm_builtin+832>, p_intercomm_ptr=0x7ffcc4839bb8, array_of_errcodes=0x7ffcc4839c98) at ../mpich-4.0b1/src/mpi/spawn/spawn_impl.c:168 #10 0x00007f0185953637 in internal_Comm_spawn (array_of_errcodes=0x7ffcc4839c98, intercomm=0x7ffcc4839d7c, comm=1140850689, root=0, info=-1677721600, maxprocs=1, argv=<optimized out>, command=0x995428 "NeedlesMpiMM") at ../mpich-4.0b1/src/binding/c/spawn/comm_spawn.c:83 #11 PMPI_Comm_spawn (command=0x995428 "NeedlesMpiMM", argv=0x993f50, maxprocs=1, info=-1677721600, root=0, comm=1140850689, intercomm=0x7ffcc4839d7c, array_of_errcodes=0x7ffcc4839c98) at ../mpich-4.0b1/src/binding/c/spawn/comm_spawn.c:169 #12 0x000000000040cec3 in needles::NeedlesMpiMaster::spawnNewManager (this=0x995380, nodenum=0, host_name="n001.cluster.pssclabs.com", intercom=@0x7ffcc4839d7c: 67108864) at src/NeedlesMpiMaster.cpp:1432 #13 0x00000000004084cb in needles::NeedlesMpiMaster::init (this=0x995380, argc=23, argv=0x7ffcc483a548, rank=0, world_size=20) at src/NeedlesMpiMaster.cpp:246 #14 0x0000000000406799 in main (argc=23, argv=0x7ffcc483a548) at src/NeedlesMpiManagerMain.cpp:96 Here is my salloc command to start the job. I want one task per node, reserving the rest of the cores on the node for spawning of additional processes. $ salloc -ntasks=20 --cpus-per-task=24 -verbose Here is what salloc reports: salloc: -------------------- -------------------- salloc: cpus-per-task : 24 salloc: ntasks : 20 salloc: verbose : 1 salloc: -------------------- -------------------- salloc: end of defined options salloc: Linear node selection plugin loaded with argument 4 salloc: select/cons_res loaded with argument 4 salloc: Cray/Aries node selection plugin loaded salloc: select/cons_tres loaded with argument 4 salloc: Granted job allocation 34311 srun: error: Unable to create step for job 34311: Requested node configuration is not available

1 0

Re: [mpich-discuss] MPI_Waitany got abort signal.
by Zhou, Hui 24 Nov '21

24 Nov '21

Hi Anatoly, Without a reproducible simple case, it will be difficult to pin down the issue. To debug, I would start injecting prints right above the assertion to see what is actually in the pkt_type. But since mpich-3.1 is very old, can you try a newer release? The latest is mpich 4.0b1 -- https://www.mpich.org/downloads/ -- Hui Zhou [https://www.mpich.org/files/2012/10/rnd100_home.jpg1zB]<https://www.mpich.org/downloads/> Downloads | MPICH<https://www.mpich.org/downloads/> Downloads MPICH is distributed under a BSD-like license. NOTE: MPICH binary packages are available in many UNIX distributions and for Windows. For example, you can search for it using “yum” (on Fedora), “apt” (Debian/Ubuntu), “pkg_add” (FreeBSD) or “port”/”brew” (Mac OS). www.mpich.org ________________________________ From: Anatoly G via discuss <discuss(a)mpich.org> Sent: Wednesday, November 24, 2021 3:49 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Anatoly G <anatolyrishon(a)gmail.com> Subject: [mpich-discuss] MPI_Waitany got abort signal. Hello Mpich, I use mpich-3.1 on Ubuntu 14. Each process has complicated logic except process 0. Process 0 is used as a router to communicate with an application and broadcast/collect results from other processes. During night runs, sometimes I see a single failure of process with rank 0. From process 0 I get wallowing print: Assertion failed in file src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c at line 596: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO internal ABORT - process 0 *** Error in `/export/home/fpd/versions/current_ver/third_party/MPI_Scheduler': double free or corruption (fasttop): 0x00007f27003f59a0 *** Stack trace: 2d40 /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f2712ff2d40] gsignal + 57 abort + 328 f394 /lib/x86_64-linux-gnu/libc.so.6(+0x73394) [0x7f271302f394] b66e /lib/x86_64-linux-gnu/libc.so.6(+0x7f66e) [0x7f271303b66e] std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() + 31 8259 /lib/x86_64-linux-gnu/libc.so.6(+0x3c259) [0x7f2712ff8259] 82a5 /lib/x86_64-linux-gnu/libc.so.6(+0x3c2a5) [0x7f2712ff82a5] d049 /export/home/fpd/versions/current_ver/third_party/libMPIServices.so(+0x224049) [0x7f2714bdd049] MPID_Abort + 103 MPIR_Assert_fail + 37 d598 /export/home/fpd/versions/current_ver/third_party/libMPIServices.so(+0x244598) [0x7f2714bfd598] MPID_nem_tcp_connpoll + 366 MPIDI_CH3I_Progress + 1408 MPI_Waitany + 1072 From the rest of processes I get: terminate called after throwing an instance of 'int' s/netmod/tcp/socksm.c at line 596: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO internal ABORT - process 1 *** Error in `Scheduler': double free or corruption (fasttop): 0x0000000003e16d60 *** Unfortunately, I can't reproduce this failure on a simplified system. Even on a real system failure can happen once in at night. We have a memory monitor which shows that we have free memory on the computer. Can you please advise me, what can be the reason for failure? Regards, Anatoly.

1 0

MPI_Waitany got abort signal.
by Anatoly G 24 Nov '21

24 Nov '21

Hello Mpich, I use mpich-3.1 on Ubuntu 14. Each process has complicated logic except process 0. Process 0 is used as a router to communicate with an application and broadcast/collect results from other processes. During night runs, sometimes I see a single failure of process with rank 0. >From process 0 I get wallowing print: Assertion failed in file src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c at line 596: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO internal ABORT - process 0 *** Error in `/export/home/fpd/versions/current_ver/third_party/MPI_Scheduler': double free or corruption (fasttop): 0x00007f27003f59a0 *** *Stack trace:* 2d40 /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f2712ff2d40] gsignal + 57 abort + 328 f394 /lib/x86_64-linux-gnu/libc.so.6(+0x73394) [0x7f271302f394] b66e /lib/x86_64-linux-gnu/libc.so.6(+0x7f66e) [0x7f271303b66e] std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() + 31 8259 /lib/x86_64-linux-gnu/libc.so.6(+0x3c259) [0x7f2712ff8259] 82a5 /lib/x86_64-linux-gnu/libc.so.6(+0x3c2a5) [0x7f2712ff82a5] d049 /export/home/fpd/versions/current_ver/third_party/libMPIServices.so(+0x224049) [0x7f2714bdd049] MPID_Abort + 103 MPIR_Assert_fail + 37 d598 /export/home/fpd/versions/current_ver/third_party/libMPIServices.so(+0x244598) [0x7f2714bfd598] MPID_nem_tcp_connpoll + 366 MPIDI_CH3I_Progress + 1408 MPI_Waitany + 1072 >From the rest of processes I get: terminate called after throwing an instance of 'int' s/netmod/tcp/socksm.c at line 596: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO internal ABORT - process 1 *** Error in `Scheduler': double free or corruption (fasttop): 0x0000000003e16d60 *** Unfortunately, I can't reproduce this failure on a simplified system. Even on a real system failure can happen once in at night. We have a memory monitor which shows that we have free memory on the computer. Can you please advise me, what can be the reason for failure? Regards, Anatoly.

1 0

Re: [mpich-discuss] Maximum number of inter-communicators?
by Zhou, Hui 22 Nov '21

22 Nov '21

We just released 4.0b1 last week! We will release 4.0rc1 and 4.0 GA in roughly two weeks apart if no major issues are discovered during this period. -- Hui Zhou From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Date: Monday, November 22, 2021 at 9:13 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: Re: [mpich-discuss] Maximum number of inter-communicators? Hui, A reboot of that failing node fixed the problem. My original question was if there was a limit on the number of inter-communicators. With 4.0a2, my job runs fine with the increased number of inter-communicators, filling up each node nicely. Should I just step up to 3.4.2, or wait until 4.0 is released? Do you know when 4.0 will be released? Thanks for your help, Kurt Hi Kurt, There is indeed a limit on maximum number of communicators that you can have, including both intra communicators and inter-communicators. Try free the communicators that you no longer need. In older version of MPICH, there may be additional limit on how many dynamic processes one can connect. If you still hit crash after making sure there isn't too many simultaneous active communicators, could you try the latest release -- http://www.mpich.org/static/downloads/4.0a2/mpich-4.0a2.tar.gz<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.mpich.…>, and see if the issue persist? -- Hui From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Monday, November 22, 2021 8:10 AM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: Re: [mpich-discuss] [EXTERNAL] Re: Maximum number of inter-communicators? Hui, just one process on a particular node failed (without any error message). We just had an O/S upgrade on the whole cluster, and may an error was made on that node in the upgrade. I’m trying to figure out what is different about that node. Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Friday, November 19, 2021 4:29 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: Maximum number of inter-communicators? Kurt, That usually means the process went away without telling the process manager (hydra) first. Those messages are from hydra when the connection to the process disconnected unexpectedly. Did all processes call `MPI_Finalize` before exit? -- Hui Zhou From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Friday, November 19, 2021 at 2:26 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] Maximum number of inter-communicators? Thanks Hui, replacing MPI_LOGICAL with MPI_CXX_BOOL enabled the code to run without the segfault (using 4.0a2). Do you have any clue what this error message means? [mpiexec(a)n022.cluster.com] control_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:206): assert (!closed) failed [mpiexec(a)n022.cluster.com] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [mpiexec(a)n022.cluster.com] HYD_pmci_wait_for_completion (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:160): error waiting for event [mpiexec(a)n022.cluster.com] main (../../../../mpich-4.0a2/src/pm/hydra/ui/mpich/mpiexec.c:325): process manager error waiting for completion [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:0:[email protected]] main (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip.c:169): de[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_ Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Monday, November 15, 2021 9:44 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: Maximum number of inter-communicators? Hi Kurt, Did you build mpich with Fortran disabled? MPI_LOGICAL is a Fortran datatype and is unavailable when Fortran binding is disabled. Try use MPI_C_BOOL instead. If you didn't disable cxx, you may also try use MPI_CXX_BOOL since you are programming in C++. Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Saturday, November 13, 2021 2:56 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] Maximum number of inter-communicators? Hui, I built MPICH 4.0a2 with gcc 4.8.5, and included the -enable-g=all flag to “configure” so that debugging symbols would be present. The code is crashing my call to MPI_Type_commit, in libpthreads.so. gdb give this stack trace below. Since MPICH 3.3.2, has there been changes in how custom types are created (the code worked in 3.3.2)? I included my type-creating code after the stack trace. Program received signal SIGSEGV, Segmentation fault. MPIR_Typerep_create_struct (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x128b6b0, array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, array_of_types=array_of_types@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fec8bedd258 <MPIR_Datatype_direct+1400>) at ../mpich-4.0a2/src/mpi/datatype/typerep/src/typerep_dataloop_create.c:659 659 MPIR_Ensure_Aint_fits_in_int(old_dtp->builtin_element_size); (gdb) where #0 MPIR_Typerep_create_struct (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x128b6b0, array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, array_of_types=array_of_types@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fec8bedd258 <MPIR_Datatype_direct+1400>) at ../mpich-4.0a2/src/mpi/datatype/typerep/src/typerep_dataloop_create.c:659 #1 0x00007fec8b9b2608 in type_struct (count=count@entry=8, blocklength_array=blocklength_array@entry=0x128b6b0, displacement_array=displacement_array@entry=0x7fffcaa243c0, oldtype_array=oldtype_array@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fffcaa242dc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:206 #2 0x00007fec8b9b4b9e in type_struct (newtype=0x7fffcaa242dc, oldtype_array=0x7fffcaa24410, displacement_array=0x7fffcaa243c0, blocklength_array=0x128b6b0, count=8) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:227 #3 MPIR_Type_struct (count=count@entry=8, blocklength_array=0x128b6b0, displacement_array=displacement_array@entry=0x7fffcaa243c0, oldtype_array=oldtype_array@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fffcaa242dc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:235 #4 0x00007fec8b9b7b08 in MPIR_Type_create_struct_impl (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x7fffcaa24440, array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, array_of_types=array_of_types@entry=0x7fffcaa24410, newtype=newtype@entry=0x12853fc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:908 #5 0x00007fec8b85ad26 in internal_Type_create_struct (newtype=0x12853fc, array_of_types=0x7fffcaa24410, array_of_displacements=<optimized out>, array_of_blocklengths=0x7fffcaa24440, count=8) at ../mpich-4.0a2/src/binding/c/datatype/type_create_struct.c:79 #6 PMPI_Type_create_struct (count=8, array_of_blocklengths=0x7fffcaa24440, array_of_displacements=0x7fffcaa243c0, array_of_types=0x7fffcaa24410, newtype=0x12853fc) at ../mpich-4.0a2/src/binding/c/datatype/type_create_struct.c:164 #7 0x0000000000438dfb in needles::MpiMsgBasic::createMsgDataType (this=0x12853fc) at src/MsgBasic.cpp:97 #8 0x0000000000412b77 in needles::NeedlesMpiManager::init (this=0x12853a0, argc=23, argv=0x7fffcaa24e08, rank=20, world_size=21) at src/NeedlesMpiManager.cpp:204 #9 0x000000000040605f in main (argc=23, argv=0x7fffcaa24e08) at src/NeedlesMpiManagerMain.cpp:142 (gdb) Here is my code that creates the custom type and then calls MPI_Type_commit: MsgBasic obj; int struct_len = 8, i; int block_len[struct_len]; MPI_Datatype types[struct_len]; MPI_Aint displacements[struct_len]; i = 0; block_len[i] = 1; types[i] = MPI_LOGICAL; displacements[i] = (size_t) &obj.tuple_valid_ - (size_t) &obj; ++i; block_len[i] = 1; types[i] = MPI_LOGICAL; displacements[i] = (size_t) &obj.tuple_seq_valid_ - (size_t) &obj; // the int array "start_" member ++i; block_len[i] = Tuple::N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.start_ - (size_t) &obj; // the int array "end_" member ++i; block_len[i] = Tuple::N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.end_ - (size_t) &obj; // the integer "opcode_" member ++i; block_len[i] = 1; types[i] = MPI_INT; displacements[i] = (size_t) &obj.opcode_ - (size_t) &obj; // the boolean "success_" member ++i; block_len[i] = 1; types[i] = MPI_LOGICAL; // NOTE: might be MPI_BOOLEAN in later version displacements[i] = (size_t) &obj.success_ - (size_t) &obj; // the double "run_time_sec_" member ++i; block_len[i] = 1; types[i] = MPI_DOUBLE; displacements[i] = (size_t) &obj.run_time_sec_ - (size_t) &obj; // the char array "error_msg_" member ++i; block_len[i] = NeedlesMpi::ERROR_MSG_LEN_ + 1; types[i] = MPI_CHAR; displacements[i] = (size_t) &obj.error_msg_[0] - (size_t) &obj; MPI_Type_create_struct(struct_len, block_len, displacements, types, &msg_data_type_); MPI_Type_commit(&msg_data_type_); Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Sunday, October 24, 2021 6:46 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: Maximum number of inter-communicators? Hi Kurt, There is indeed a limit on maximum number of communicators that you can have, including both intra communicators and inter-communicators. Try free the communicators that you no longer need. In older version of MPICH, there may be additional limit on how many dynamic processes one can connect. If you still hit crash after making sure there isn't too many simultaneous active communicators, could you try the latest release -- http://www.mpich.org/static/downloads/4.0a2/mpich-4.0a2.tar.gz<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.mpich.…>, and see if the issue persist? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Sunday, October 24, 2021 2:37 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [mpich-discuss] Maximum number of inter-communicators? Hi, Based on a paper I read about giving an MPI job some fault tolerance, I’m exclusively connecting my processes with inter-communicators. I’ve found that if I increase the number of processes beyond a certain point, many processes don’t get created at all and the whole job crashes. Am I running up against an operating system limit (like the number of open file descriptors – it is set at 1024), or some sort of MPICH limit? If it matters, my process architecture (a tree) is as follows: one master process connected to 21 manager processes on 21 other nodes, and each manager connected to 8 worker processes on the manager’s own node. This is the largest job I’ve been able to create without it crashing. Attempting to increase the number of workers beyond 8 results in a crash. I’m using MPICH 3.3.2 on Centos 3.10.0. MPICH was compiled with the Portland Group compiler pgc++ 19.5-0. Thanks, Kurt

1 0