October 2020 - discuss - mailman3.cels.anl.gov

ULFM and timeouts
by Shead, Timothy 13 Oct '20

13 Oct '20

Folks: I’m interested in running some fault tolerance experiments using the ULFM extensions in MPICH, but I’m stumbling right out of the gate. When I run the following Python code, rank 0 broadcasts a monotonically increasing value to the other ranks, and it gets printed to stdout: import itertools import time from mpi4py import MPI comm = MPI.COMM_WORLD for count in itertools.count(): value = comm.bcast(count, root=0) print(f"{comm.rank}: {value}", flush=True) time.sleep(1) So far, so good. If I run the code using mpiexec -n 3 --disable-auto-cleanup python test.py I get the expected output. If I kill one of the three processes, the others keep running for a few more iterations thanks to --disable-auto-cleanup, until bcast() blocks. My assumption was that MPICH would eventually return an error code, rather than blocking. Otherwise, it doesn’t seem like my code will ever have the chance to use revoke() and shrink(). What am I missing here? Is there a way to specify timeouts for blocking operations? Am I limited to using async operations with ULFM? Could you refer me to any ULFM tests or examples that you used in development? Thanks in advance, Tim Timothy M. Shead Sandia National Laboratories tshead(a)sandia.gov<mailto:[email protected]>

1 0

Re: [mpich-discuss] Regarding issue with MPICH 3.2
by Zhou, Hui 09 Oct '20

09 Oct '20

Thanks, Neeraj singh. To confirm, you always have this error with the profiler, but intermittent without profiler? When the error shows up, does the error show up at the beginning of the application, middle of the application, or near the end of the application? -- Hui Zhou From: Neeraj singh <neeraj844singh(a)gmail.com> Date: Friday, October 9, 2020 at 9:22 AM To: "Zhou, Hui" <zhouh(a)anl.gov> Cc: "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] Regarding issue with MPICH 3.2 Yes, but sometimes without profiler also, I am also facing this issue with some applications also. Thanks and regards On Fri, Oct 9, 2020 at 7:27 PM Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: Hi Neeraj singh, To clarify, does this error only happen when you try to use the profiler tool? -- Hui Zhou From: Neeraj singh via discuss <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Friday, October 9, 2020 at 8:41 AM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: Neeraj singh <neeraj844singh(a)gmail.com<mailto:[email protected]>> Subject: [mpich-discuss] Regarding issue with MPICH 3.2 I am using the MAQAO profiler tool to collect data based on sampling. While launching an application for multinode some issues I'm facing. Kindly check the attached problem lines and clarify the same. Thanks & Regards Problem lines: [mpiexec@gpu7] HYDU_sock_write (utils/sock/sock.c:294): write error (Bad file descriptor) [mpiexec@gpu7] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:177): unable to write data to proxy [mpiexec@gpu7] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:79): unable to send signal downstream [mpiexec@gpu7] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec@gpu7] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec@gpu7] main (ui/mpich/mpiexec.c:340): process manager error waiting for completion

1 0

Re: [mpich-discuss] Regarding issue with MPICH 3.2
by Neeraj singh 09 Oct '20

09 Oct '20

Yes, but sometimes without profiler also, I am also facing this issue with some applications also. Thanks and regards On Fri, Oct 9, 2020 at 7:27 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > Hi Neeraj singh, > > > > To clarify, does this error only happen when you try to use the profiler > tool? > > > > -- > Hui Zhou > > > > > > *From: *Neeraj singh via discuss <discuss(a)mpich.org> > *Reply-To: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Date: *Friday, October 9, 2020 at 8:41 AM > *To: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Cc: *Neeraj singh <neeraj844singh(a)gmail.com> > *Subject: *[mpich-discuss] Regarding issue with MPICH 3.2 > > > > I am using the MAQAO profiler tool to collect data based on sampling. > While launching an application for multinode some issues I'm facing. > > Kindly check the attached problem lines and clarify the same. > > Thanks & Regards > > > > > > Problem lines: > > [mpiexec@gpu7] HYDU_sock_write (utils/sock/sock.c:294): write error (Bad > file descriptor) > [mpiexec@gpu7] HYD_pmcd_pmiserv_send_signal > (pm/pmiserv/pmiserv_cb.c:177): unable to write data to proxy > [mpiexec@gpu7] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:79): unable to send > signal downstream > [mpiexec@gpu7] HYDT_dmxu_poll_wait_for_event > (tools/demux/demux_poll.c:76): callback returned error status > [mpiexec@gpu7] HYD_pmci_wait_for_completion > (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event > [mpiexec@gpu7] main (ui/mpich/mpiexec.c:340): process manager error > waiting for completion > > >

1 0

Re: [mpich-discuss] Regarding issue with MPICH 3.2
by Zhou, Hui 09 Oct '20

09 Oct '20

Hi Neeraj singh, To clarify, does this error only happen when you try to use the profiler tool? -- Hui Zhou From: Neeraj singh via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Friday, October 9, 2020 at 8:41 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: Neeraj singh <neeraj844singh(a)gmail.com> Subject: [mpich-discuss] Regarding issue with MPICH 3.2 I am using the MAQAO profiler tool to collect data based on sampling. While launching an application for multinode some issues I'm facing. Kindly check the attached problem lines and clarify the same. Thanks & Regards Problem lines: [mpiexec@gpu7] HYDU_sock_write (utils/sock/sock.c:294): write error (Bad file descriptor) [mpiexec@gpu7] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:177): unable to write data to proxy [mpiexec@gpu7] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:79): unable to send signal downstream [mpiexec@gpu7] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec@gpu7] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec@gpu7] main (ui/mpich/mpiexec.c:340): process manager error waiting for completion

1 0

Regarding issue with MPICH 3.2
by Neeraj singh 09 Oct '20

09 Oct '20

I am using the MAQAO profiler tool to collect data based on sampling. While launching an application for multinode some issues I'm facing. Kindly check the attached problem lines and clarify the same. Thanks & Regards Problem lines: [mpiexec@gpu7] HYDU_sock_write (utils/sock/sock.c:294): write error (Bad file descriptor) [mpiexec@gpu7] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:177): unable to write data to proxy [mpiexec@gpu7] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:79): unable to send signal downstream [mpiexec@gpu7] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec@gpu7] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec@gpu7] main (ui/mpich/mpiexec.c:340): process manager error waiting for completion

1 0

Re: [mpich-discuss] Intermittent MPICH error
by Mccall, Kurt E. (MSFC-EV41) 06 Oct '20

06 Oct '20

Thanks Hui. I’ll investigate further. Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Monday, October 5, 2020 6:16 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] Intermittent MPICH error I believe it means one of the node process was terminated unexpectedly. We’ll need more clue to make a guess on what is the actual cause. -- Hui Zhou From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Monday, October 5, 2020 at 4:47 PM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [mpich-discuss] Intermittent MPICH error I am using MPICH 3.3.2 on Centos 3.10, Torque 5.1.1. I have never seen this error when running my MPI jobs as myself, but when other users run the same code, this error may or may not occur. It seems to happen when the job is shutting down. [mpiexec(a)n006.cluster.com] HYDU_sock_write (utils/sock/sock.c:256): write error (Bad file descriptor) [mpiexec(a)n006.cluster.com] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:178): unable to write data to proxy [mpiexec(a)n006.cluster.com] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:77): unable to send signal downstream [mpiexec(a)n006.cluster.com] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status [mpiexec(a)n006.cluster.com] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:196): error waiting for event [mpiexec(a)n006.cluster.com] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion Does anyone have a clue what these errors mean? I hope this is enough info to go on. If not, please let me know what else I should provide. Thanks, Kurt

1 0

Re: [mpich-discuss] Intermittent MPICH error
by Zhou, Hui 05 Oct '20

05 Oct '20

I believe it means one of the node process was terminated unexpectedly. We’ll need more clue to make a guess on what is the actual cause. -- Hui Zhou From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, October 5, 2020 at 4:47 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] Intermittent MPICH error I am using MPICH 3.3.2 on Centos 3.10, Torque 5.1.1. I have never seen this error when running my MPI jobs as myself, but when other users run the same code, this error may or may not occur. It seems to happen when the job is shutting down. [mpiexec(a)n006.cluster.com] HYDU_sock_write (utils/sock/sock.c:256): write error (Bad file descriptor) [mpiexec(a)n006.cluster.com] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:178): unable to write data to proxy [mpiexec(a)n006.cluster.com] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:77): unable to send signal downstream [mpiexec(a)n006.cluster.com] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status [mpiexec(a)n006.cluster.com] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:196): error waiting for event [mpiexec(a)n006.cluster.com] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion Does anyone have a clue what these errors mean? I hope this is enough info to go on. If not, please let me know what else I should provide. Thanks, Kurt

1 0

Intermittent MPICH error
by Mccall, Kurt E. (MSFC-EV41) 05 Oct '20

05 Oct '20

I am using MPICH 3.3.2 on Centos 3.10, Torque 5.1.1. I have never seen this error when running my MPI jobs as myself, but when other users run the same code, this error may or may not occur. It seems to happen when the job is shutting down. [mpiexec(a)n006.cluster.com] HYDU_sock_write (utils/sock/sock.c:256): write error (Bad file descriptor) [mpiexec(a)n006.cluster.com] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:178): unable to write data to proxy [mpiexec(a)n006.cluster.com] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:77): unable to send signal downstream [mpiexec(a)n006.cluster.com] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status [mpiexec(a)n006.cluster.com] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:196): error waiting for event [mpiexec(a)n006.cluster.com] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion Does anyone have a clue what these errors mean? I hope this is enough info to go on. If not, please let me know what else I should provide. Thanks, Kurt

1 0