Hi Kurt, as a mental model, we can see MPI_Comm_spawn and MPI_Intercomm_create as collective constructors of the inter-comm object. All collectively created MPI objects also need a collective destructor call for full cleanup. MPI_Comm_disconnect is this collective destructor. MPI_Finalize implies MPI_Comm_disconnect for all undestroyed communicators of the process. From that perspective, it might even be sufficient to call MPI_Comm_disconnect on the parent process and let MPI_Finalize in the child process do the job. If MPI_Intercomm_create creates an inter-comm in a process you want to stop early, all involved processes will need to call MPI_Comm_disconnect before the process can terminate. Best Joachim Am 04.03.21 um 01:55 schrieb Mccall, Kurt E. (MSFC-EV41):

Joachim,

Thanks, that helped! Is it necessary to call MPI_Comm_disconnect on inter-communicators that are created by MPI_Intercomm_create?

Kurt

-----Original Message----- From: Joachim Protze <[email protected]> Sent: Wednesday, March 3, 2021 5:52 AM To: [email protected] Cc: Mccall, Kurt E. (MSFC-EV41) <[email protected]> Subject: [EXTERNAL] Re: [mpich-discuss] Spawned process hanging in MPI_Finalize

Hi Kurt,

did you call MPI_Comm_disconnect on all processes connected with the inter-communicator? Also the parent process needs to disconnect from the inter-comm before the MPI_Comm_disconnect can return.

- Joachim

Am 03.03.21 um 01:54 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss:

...

I have a parent process that creates a child via MPI_Comm_spawn(). When the child decides it has to exit, it is hanging in MPI_Finalize(). It does the same if it calls MPI_Comm_disconnect() before MPI_Finalize.

Here is the stack trace in the child:

(gdb) where #0 0x00007fc6f2fedde0 in __poll_nocancel () from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/libc.so.6 #1 0x00007fc6f4dc840e in MPID_nem_tcp_connpoll () at src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c:1819 #2 0x00007fc6f4db857e in MPID_nem_network_poll () at src/mpid/ch3/channels/nemesis/src/mpid_nem_network_poll.c:16 #3 0x00007fc6f4dafc43 in MPIDI_CH3I_Progress () at src/mpid/ch3/channels/nemesis/src/ch3_progress.c:1019 #4 0x00007fc6f4d5094d in MPIDI_CH3U_VC_WaitForClose () at src/mpid/ch3/src/ch3u_handle_connection.c:383 #5 0x00007fc6f4d94efa in MPID_Finalize () at src/mpid/ch3/src/mpid_finalize.c:110 #6 0x00007fc6f4c432ca in PMPI_Finalize () at src/mpi/init/finalize.c:260 #7 0x0000000000408a85 in needles::MpiWorker::finalize () at src/MpiWorker.cpp:470

Maybe I have a communication that hasn't completed, or the child is waiting for the parent to call MPI_Finalize. I believe that you (Ken, Hui) told me that it shouldn't do the latter.

Is there a way for the child to cleanly exit without hanging in MPI_Finalize? I tried calling MPI_Cancel() in the child on the only possible communication request that I knew of, but it didn't help. It just occurred to me that I haven't tried calling MPI_Cancel on the requests in the parent...

Thanks, Kurt

_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

-- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 [email protected] www.itc.rwth-aachen.de