- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Spawned process hanging in MPI_Finalize
by Joachim Protze 03 Mar '21

03 Mar '21

Hi Kurt, did you call MPI_Comm_disconnect on all processes connected with the inter-communicator? Also the parent process needs to disconnect from the inter-comm before the MPI_Comm_disconnect can return. - Joachim Am 03.03.21 um 01:54 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss: > I have a parent process that creates a child via MPI_Comm_spawn(). When the child decides it has to exit, it is hanging in MPI_Finalize(). It does the same if it calls MPI_Comm_disconnect() before MPI_Finalize. > > Here is the stack trace in the child: > > (gdb) where > #0 0x00007fc6f2fedde0 in __poll_nocancel () from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/libc.so.6 > #1 0x00007fc6f4dc840e in MPID_nem_tcp_connpoll () at src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c:1819 > #2 0x00007fc6f4db857e in MPID_nem_network_poll () at src/mpid/ch3/channels/nemesis/src/mpid_nem_network_poll.c:16 > #3 0x00007fc6f4dafc43 in MPIDI_CH3I_Progress () at src/mpid/ch3/channels/nemesis/src/ch3_progress.c:1019 > #4 0x00007fc6f4d5094d in MPIDI_CH3U_VC_WaitForClose () at src/mpid/ch3/src/ch3u_handle_connection.c:383 > #5 0x00007fc6f4d94efa in MPID_Finalize () at src/mpid/ch3/src/mpid_finalize.c:110 > #6 0x00007fc6f4c432ca in PMPI_Finalize () at src/mpi/init/finalize.c:260 > #7 0x0000000000408a85 in needles::MpiWorker::finalize () at src/MpiWorker.cpp:470 > > Maybe I have a communication that hasn't completed, or the child is waiting for the parent to call MPI_Finalize. I believe that you (Ken, Hui) told me that it shouldn't do the latter. > > Is there a way for the child to cleanly exit without hanging in MPI_Finalize? I tried calling MPI_Cancel() in the child on the only possible communication request that I knew of, but it didn't help. > It just occurred to me that I haven't tried calling MPI_Cancel on the requests in the parent... > > Thanks, > Kurt > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Error with mpich-3.3.2 and ucx-1.8.0
by Junchao Zhang 02 Mar '21

02 Mar '21

Hello, We met an error with mpich-3.3.2 and ucx-1.8.0. See the attached example, which uses a user defined data type in MPI_Startall. MPICH was configured with --with-device=ch4:ucx --with-ucx=/path/to/ucx-1.8.0. The error stack is $ mpirun -n 2 ./dtype Assertion failed in file src/mpi/datatype/type_free.c at line 38: (((datatype_ptr)))->ref_count >= 0 /home/jczhang/soft/lib/libmpi.so.12(+0x48fa1f) [0x7f4019b6ba1f] /home/jczhang/soft/lib/libmpi.so.12(MPL_backtrace_show+0x18) [0x7f4019b6baff] /home/jczhang/soft/lib/libmpi.so.12(+0x441e27) [0x7f4019b1de27] /home/jczhang/soft/lib/libmpi.so.12(+0xfebc2) [0x7f40197dabc2] /home/jczhang/soft/lib/libmpi.so.12(MPI_Type_free+0x5fd) [0x7f40197db239] It does not happen with mpich-3.4.1. --Junchao Zhang

1 0

Spawned process hanging in MPI_Finalize
by Mccall, Kurt E. (MSFC-EV41) 02 Mar '21

02 Mar '21

I have a parent process that creates a child via MPI_Comm_spawn(). When the child decides it has to exit, it is hanging in MPI_Finalize(). It does the same if it calls MPI_Comm_disconnect() before MPI_Finalize. Here is the stack trace in the child: (gdb) where #0 0x00007fc6f2fedde0 in __poll_nocancel () from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/libc.so.6 #1 0x00007fc6f4dc840e in MPID_nem_tcp_connpoll () at src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c:1819 #2 0x00007fc6f4db857e in MPID_nem_network_poll () at src/mpid/ch3/channels/nemesis/src/mpid_nem_network_poll.c:16 #3 0x00007fc6f4dafc43 in MPIDI_CH3I_Progress () at src/mpid/ch3/channels/nemesis/src/ch3_progress.c:1019 #4 0x00007fc6f4d5094d in MPIDI_CH3U_VC_WaitForClose () at src/mpid/ch3/src/ch3u_handle_connection.c:383 #5 0x00007fc6f4d94efa in MPID_Finalize () at src/mpid/ch3/src/mpid_finalize.c:110 #6 0x00007fc6f4c432ca in PMPI_Finalize () at src/mpi/init/finalize.c:260 #7 0x0000000000408a85 in needles::MpiWorker::finalize () at src/MpiWorker.cpp:470 Maybe I have a communication that hasn't completed, or the child is waiting for the parent to call MPI_Finalize. I believe that you (Ken, Hui) told me that it shouldn't do the latter. Is there a way for the child to cleanly exit without hanging in MPI_Finalize? I tried calling MPI_Cancel() in the child on the only possible communication request that I knew of, but it didn't help. It just occurred to me that I haven't tried calling MPI_Cancel on the requests in the parent... Thanks, Kurt

1 0

Re: [mpich-discuss] MPI_Info key host error in SLURM
by Raffenetti, Kenneth J. 02 Mar '21

02 Mar '21

Thanks for sending. I haven't found time to look closely yet. Will update you asap. Ken On 3/1/21, 1:38 PM, "Iker Martín Álvarez" <martini(a)uji.es> wrote: Hi Ken, Thanks for your reply. Here are attached two files with the arg "-v". BatchOutput.txt has the output which is giving the error message when the code is executed with the command sbatch, while the file InteractiveOutput.txt is the one executed interactively in the node and work as expected. Both of them has been compiled and executed with mpich 3.4.1. Thanks, Iker El vie, 26 feb 2021 a las 16:15, Raffenetti, Kenneth J. (<raffenet(a)mcs.anl.gov>) escribió: Hi, Could you add "-v" to your mpiexec command and provide the output? The "host" info key is handled by the process manager when executing the spawn command. Ken On 2/24/21, 11:55 AM, "Iker Martín Álvarez via discuss" <discuss(a)mpich.org> wrote: Hi, I have been working around the MPI_Comm_spawn function with the key "host" along a value in a system with the SLURM resource manager (slurm-wlm 17.11.2). The function works as expected, but when I send to execute the code with the sbatch command, an error arises. This does not happen when I execute directly the code in the same machine SLURM decided to execute it when it was sended with sbatch. In both cases with the key "host", as when I do not use the key, it works just fine. The same code has been tested with MPICH 3.3.2 and 3.4.1, which gives different errors. Also, I tried it with other implementations (OpenMPI and Intel MPI), which works as expected creating the processes in the indicated host. I would like to create processes by MPI_Comm_spawn in an assigned host, so if there are other key values for the Info argument, I could try them, but I have not found any looking through MPICH documentation. Here is the code I have been using: int main(int argc, char ** argv) { int myId, numP; MPI_Info info; MPI_Comm comm; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &myId); MPI_Comm_size(MPI_COMM_WORLD, &numP); MPI_Comm comm_par; MPI_Comm_get_parent(&comm_par); if(comm_par != MPI_COMM_NULL ) { if(myId == ROOT) { printf("SONS\n"); fflush(stdout); } } else { if(myId == ROOT) { printf("FATHERS\n"); fflush(stdout); } MPI_Info_create(&info); MPI_Info_set(info, "host", "n00"); MPI_Comm_spawn(argv[0], MPI_ARGV_NULL, numP, info, ROOT, MPI_COMM_WORLD, &comm, MPI_ERRCODES_IGNORE); } MPI_Finalize(); return 0; } For MPICH 3.4.1 there is sometimes no error and the code stops working at MPI_Comm_spawn function, other times this error is shown: Abort(1615120) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Internal MPI error!, error stack: MPIR_Init_thread(152)...: MPID_Init(562)..........: MPIDU_Init_shm_init(195): Init_shm_barrier(94)....: Internal MPI error! barrier not initialized Also, the error code for MPICH 3.3.2: Assertion failed in file src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank < pg->size Assertion failed in file src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank < pg->size Assertion failed in file src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank < pg->size libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable internal ABORT - process 0 Thanks, Iker

1 0

Re: [mpich-discuss] MPI_Info key host error in SLURM
by Iker Martín Álvarez 01 Mar '21

01 Mar '21

Hi Ken, Thanks for your reply. Here are attached two files with the arg "-v". *BatchOutput.txt *has the output which is giving the error message when the code is executed with the command *sbatch*, while the file *InteractiveOutput.txt* is the one executed interactively in the node and work as expected. Both of them has been compiled and executed with mpich 3.4.1. Thanks, Iker El vie, 26 feb 2021 a las 16:15, Raffenetti, Kenneth J. (< raffenet(a)mcs.anl.gov>) escribió: > Hi, > > Could you add "-v" to your mpiexec command and provide the output? The > "host" info key is handled by the process manager when executing the spawn > command. > > Ken > > On 2/24/21, 11:55 AM, "Iker Martín Álvarez via discuss" < > discuss(a)mpich.org> wrote: > > Hi, > I have been working around the MPI_Comm_spawn function with the key > "host" along a value in a system with the SLURM resource manager (slurm-wlm > 17.11.2). The function works as expected, but when I send to execute the > code with the sbatch command, an error arises. This does not happen when I > execute directly the code in the same machine SLURM decided to execute it > when it was sended with sbatch. In both cases with the key "host", as when > I do not use the key, it works just fine. > > The same code has been tested with MPICH 3.3.2 and 3.4.1, which gives > different errors. Also, I tried it with other implementations (OpenMPI and > Intel MPI), which works as expected creating the processes in the indicated > host. > > I would like to create processes by MPI_Comm_spawn in an assigned > host, so if there are other key values for the Info argument, I could try > them, but I have not found any looking through MPICH documentation. > > Here is the code I have been using: > > int main(int argc, char ** argv) { > > int myId, numP; > MPI_Info info; > MPI_Comm comm; > > MPI_Init(&argc, &argv); > MPI_Comm_rank(MPI_COMM_WORLD, &myId); > MPI_Comm_size(MPI_COMM_WORLD, &numP); > > MPI_Comm comm_par; > MPI_Comm_get_parent(&comm_par); > if(comm_par != MPI_COMM_NULL ) { > if(myId == ROOT) { > printf("SONS\n"); fflush(stdout); > } > } else { > if(myId == ROOT) { > printf("FATHERS\n"); fflush(stdout); > } > MPI_Info_create(&info); > MPI_Info_set(info, "host", "n00"); > MPI_Comm_spawn(argv[0], MPI_ARGV_NULL, numP, info, ROOT, > MPI_COMM_WORLD, &comm, MPI_ERRCODES_IGNORE); > } > MPI_Finalize(); > return 0; > } > > > For MPICH 3.4.1 there is sometimes no error and the code stops working > at MPI_Comm_spawn function, other times this error is shown: > Abort(1615120) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: > Internal MPI error!, error stack: > MPIR_Init_thread(152)...: > MPID_Init(562)..........: > MPIDU_Init_shm_init(195): > Init_shm_barrier(94)....: Internal MPI error! barrier not initialized > > > Also, the error code for MPICH 3.3.2: > > Assertion failed in file > src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank > < pg->size > Assertion failed in file > src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank > < pg->size > Assertion failed in file > src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank > < pg->size > libbacktrace: no debug info in ELF executable > libbacktrace: no debug info in ELF executable > libbacktrace: no debug info in ELF executable > libbacktrace: no debug info in ELF executable > libbacktrace: no debug info in ELF executable > libbacktrace: no debug info in ELF executable > libbacktrace: no debug info in ELF executable > libbacktrace: no debug info in ELF executable > libbacktrace: no debug info in ELF executable > libbacktrace: no debug info in ELF executable > libbacktrace: no debug info in ELF executable > libbacktrace: no debug info in ELF executable > internal ABORT - process 0 > > Thanks, Iker > >

1 0

Re: [mpich-discuss] Status of PMI in mpich
by Zhou, Hui 01 Mar '21

01 Mar '21

The PMI interface is not exposed to directly use outside mpich library. You’ll need directly copy the pmi code from the MPICH source distribution. It is located in `src/pmi` folder. -- Hui Zhou From: kevin--- via discuss <discuss(a)mpich.org> Date: Monday, March 1, 2021 at 9:23 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: kevin(a)continuum-dynamics.com <kevin(a)continuum-dynamics.com> Subject: [mpich-discuss] Status of PMI in mpich Hello, I'm interested in using PMI from mpich programs. Can someone tell me what the status of PMI is? Is it usable or is it in an experimental state? Do I need to build mpich with any special flags to take advantage of it? I have a program developed on a Cray that I would like to port to other machines, but I get an error when compiling (using mpicc) about not finding pmi.h. Kevin Olson _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Status of PMI in mpich
by kevin＠continuum-dynamics.com 01 Mar '21

01 Mar '21

Hello, I'm interested in using PMI from mpich programs. Can someone tell me what the status of PMI is? Is it usable or is it in an experimental state? Do I need to build mpich with any special flags to take advantage of it? I have a program developed on a Cray that I would like to port to other machines, but I get an error when compiling (using mpicc) about not finding pmi.h. Kevin Olson

1 0

Re: [mpich-discuss] MPI_Info key host error in SLURM
by Raffenetti, Kenneth J. 26 Feb '21

26 Feb '21

Hi, Could you add "-v" to your mpiexec command and provide the output? The "host" info key is handled by the process manager when executing the spawn command. Ken On 2/24/21, 11:55 AM, "Iker Martín Álvarez via discuss" <discuss(a)mpich.org> wrote: Hi, I have been working around the MPI_Comm_spawn function with the key "host" along a value in a system with the SLURM resource manager (slurm-wlm 17.11.2). The function works as expected, but when I send to execute the code with the sbatch command, an error arises. This does not happen when I execute directly the code in the same machine SLURM decided to execute it when it was sended with sbatch. In both cases with the key "host", as when I do not use the key, it works just fine. The same code has been tested with MPICH 3.3.2 and 3.4.1, which gives different errors. Also, I tried it with other implementations (OpenMPI and Intel MPI), which works as expected creating the processes in the indicated host. I would like to create processes by MPI_Comm_spawn in an assigned host, so if there are other key values for the Info argument, I could try them, but I have not found any looking through MPICH documentation. Here is the code I have been using: int main(int argc, char ** argv) { int myId, numP; MPI_Info info; MPI_Comm comm; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &myId); MPI_Comm_size(MPI_COMM_WORLD, &numP); MPI_Comm comm_par; MPI_Comm_get_parent(&comm_par); if(comm_par != MPI_COMM_NULL ) { if(myId == ROOT) { printf("SONS\n"); fflush(stdout); } } else { if(myId == ROOT) { printf("FATHERS\n"); fflush(stdout); } MPI_Info_create(&info); MPI_Info_set(info, "host", "n00"); MPI_Comm_spawn(argv[0], MPI_ARGV_NULL, numP, info, ROOT, MPI_COMM_WORLD, &comm, MPI_ERRCODES_IGNORE); } MPI_Finalize(); return 0; } For MPICH 3.4.1 there is sometimes no error and the code stops working at MPI_Comm_spawn function, other times this error is shown: Abort(1615120) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Internal MPI error!, error stack: MPIR_Init_thread(152)...: MPID_Init(562)..........: MPIDU_Init_shm_init(195): Init_shm_barrier(94)....: Internal MPI error! barrier not initialized Also, the error code for MPICH 3.3.2: Assertion failed in file src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank < pg->size Assertion failed in file src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank < pg->size Assertion failed in file src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank < pg->size libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable internal ABORT - process 0 Thanks, Iker

1 0

MPI_Info key host error in SLURM
by Iker Martín Álvarez 24 Feb '21

24 Feb '21

Hi, I have been working around the MPI_Comm_spawn function with the key "host" along a value in a system with the SLURM resource manager (slurm-wlm 17.11.2). The function works as expected, but when I send to execute the code with the sbatch command, an error arises. This does not happen when I execute directly the code in the same machine SLURM decided to execute it when it was sended with *sbatch*. In both cases with the key "host", as when I do not use the key, it works just fine. The same code has been tested with MPICH 3.3.2 and 3.4.1, which gives different errors. Also, I tried it with other implementations (OpenMPI and Intel MPI), which works as expected creating the processes in the indicated host. I would like to create processes by MPI_Comm_spawn in an assigned host, so if there are other key values for the Info argument, I could try them, but I have not found any looking through MPICH documentation. Here is the code I have been using: int main(int argc, char ** argv) { int myId, numP; MPI_Info info; MPI_Comm comm; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &myId); MPI_Comm_size(MPI_COMM_WORLD, &numP); MPI_Comm comm_par; MPI_Comm_get_parent(&comm_par); if(comm_par != MPI_COMM_NULL ) { if(myId == ROOT) { printf("SONS\n"); fflush(stdout); } } else { if(myId == ROOT) { printf("FATHERS\n"); fflush(stdout); } MPI_Info_create(&info); MPI_Info_set(info, "host", "n00"); MPI_Comm_spawn(argv[0], MPI_ARGV_NULL, numP, info, ROOT, MPI_COMM_WORLD, &comm, MPI_ERRCODES_IGNORE); } MPI_Finalize(); return 0; } For MPICH 3.4.1 there is sometimes no error and the code stops working at MPI_Comm_spawn function, other times this error is shown: Abort(1615120) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Internal MPI error!, error stack: MPIR_Init_thread(152)...: MPID_Init(562)..........: MPIDU_Init_shm_init(195): Init_shm_barrier(94)....: Internal MPI error! barrier not initialized Also, the error code for MPICH 3.3.2: Assertion failed in file src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank < pg->size Assertion failed in file src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank < pg->size Assertion failed in file src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank < pg->size libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable internal ABORT - process 0 Thanks, Iker

1 0

Re: [mpich-discuss] [EXTERNAL] Re: Calling MPI_Finalize or not
by Raffenetti, Kenneth J. 21 Feb '21

21 Feb '21

Kurt, Yes, that is correct. Spawned processes should be sure to call MPI_COMM_DISCONNECT on their parent communicator before MPI_FINALIZE. Ken On 2/21/21, 4:13 PM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Hui, So are you saying that the worker can call MPI_Finalize and it won’t wait for the manager that spawned it to do the same? That would be ideal. Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Sunday, February 21, 2021 4:07 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: Calling MPI_Finalize or not The worker should call `MPI_Finalize` before exit. If somehow it hangs, assuming all the resources have been freed/disconnected, it must be a bug, please file an issue and we’ll track it down. Each spawned process has a different `MPI_COMM_WORLD` from the process who spawned it. -- Hui Zhou From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Date: Sunday, February 21, 2021 at 3:06 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] Calling MPI_Finalize or not Two questions: I have a group of manager processes, each of which uses MPI_Comm_spawn() to create individual worker processes one at a time. Occasionally, a worker has to exit and is replaced by another worker via MPI_Comm_spawn(). If the exiting worker calls MPI_Finalize(), will it sit there waiting for all other managers and workers in MPI_COMM_WORLD to do the same, or will it wait only for the manager that created its inter-communicator to call MPI_Finalize? In either case, maybe it is better for the worker to NOT call MPI_Finalize at all so that it doesn’t sit there taking up a core’s resources while it waits for the others. What are the implications of not calling MPI_Finalize in the workers? Thanks, Kurt

1 0