Hi Kurt, MPICH should gracefully cleanup all of the remaining processes, when one process dies. If it is not doing that, then it might a bug. But before we go digging into it, can you try the “mpiexec” from the latest mpich version? There have been a few bug fixes. In fact, we just fixed one more bug a couple of days ago (https://github.com/pmodels/mpich/pull/4862/commits/7f362d74ee), although I think this last bug might be unrelated to the error that you are seeing. — Pavan

On Nov 5, 2020, at 12:42 AM, Mccall, Kurt E. (MSFC-EV41) via discuss <[email protected]> wrote:

I’m using MPICH 3.3.2, Torque 5.1.1.

I’ve discussed something like this with you guys before – is there any way to have MPICH shut down the entire job after a seg fault, rather than running forever? You’ve said that what really needs to be done is to locate and fix the bug causing it, which is right, but I’m just worried that a non-terminating job would be confusing to the user that encounters it, if some new unanticipated seg fault were generated in the future.

I know that I can’t put MPI_Finalize() or any non-async safe code in my signal handler. It helps a little bit to print a backtrace from the signal handler (I know, I/O is also not safe).

Is there anything else that can be done? Here are the MPICH error messages written some minutes after the seg fault occurs, when the job is ending and all of the good processes are calling MPI_Finalize().

Thanks, Kurt

[proxy:0:[email protected]] [proxy:1:[email protected]] [proxy:3:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): assert (!closed) failed [proxy:4:[email protected]] assert (!closed) failed assert (!closed) failed [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): [proxy:1:[email protected]] [proxy:3:[email protected]] [proxy:5:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): assert (!closed) failed HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): [proxy:6:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): callback returned error status [proxy:4:[email protected]] [proxy:8:[email protected]] callback returned error status callback returned error status [proxy:9:[email protected]] assert (!closed) failed HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): [proxy:1:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): [proxy:3:[email protected]] [proxy:10:[email protected]] [proxy:5:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): assert (!closed) failed main (pm/pmiserv/pmip.c:200): callback returned error status main (pm/pmiserv/pmip.c:200): assert (!closed) failed main (pm/pmiserv/pmip.c:200): HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): assert (!closed) failed [proxy:6:[email protected]] demux engine error waiting for event [proxy:4:[email protected]] demux engine error waiting for event [proxy:8:[email protected]] demux engine error waiting for event callback returned error status assert (!closed) failed [proxy:9:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): main (pm/pmiserv/pmip.c:200): HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): [proxy:5:[email protected]] callback returned error status [proxy:10:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status demux engine error waiting for event main (pm/pmiserv/pmip.c:200): [proxy:8:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): [proxy:7:[email protected]] callback returned error status [proxy:6:[email protected]] demux engine error waiting for event main (pm/pmiserv/pmip.c:200): callback returned error status [proxy:9:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): main (pm/pmiserv/pmip.c:200): demux engine error waiting for event [proxy:10:[email protected]] main (pm/pmiserv/pmip.c:200): assert (!closed) failed demux engine error waiting for event main (pm/pmiserv/pmip.c:200): demux engine error waiting for event [proxy:7:[email protected]] demux engine error waiting for event HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status [proxy:7:[email protected]] main (pm/pmiserv/pmip.c:200): demux engine error waiting for event _______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss