Thanks Hui. I’ll investigate further. Kurt From: Zhou, Hui <[email protected]> Sent: Monday, October 5, 2020 6:16 PM To: [email protected] Cc: Mccall, Kurt E. (MSFC-EV41) <[email protected]> Subject: [EXTERNAL] Re: [mpich-discuss] Intermittent MPICH error I believe it means one of the node process was terminated unexpectedly. We’ll need more clue to make a guess on what is the actual cause. -- Hui Zhou From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Monday, October 5, 2020 at 4:47 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Cc: "Mccall, Kurt E. (MSFC-EV41)" <[email protected]<mailto:[email protected]>> Subject: [mpich-discuss] Intermittent MPICH error I am using MPICH 3.3.2 on Centos 3.10, Torque 5.1.1. I have never seen this error when running my MPI jobs as myself, but when other users run the same code, this error may or may not occur. It seems to happen when the job is shutting down. [[email protected]] HYDU_sock_write (utils/sock/sock.c:256): write error (Bad file descriptor) [[email protected]] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:178): unable to write data to proxy [[email protected]] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:77): unable to send signal downstream [[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status [[email protected]] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:196): error waiting for event [[email protected]] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion Does anyone have a clue what these errors mean? I hope this is enough info to go on. If not, please let me know what else I should provide. Thanks, Kurt