On May 10, 2020, at 3:16 AM, Martin Ivanov <[email protected]> wrote:
Hello Tony, Thank you very much for your reply. I am posting you the output of 'ps x', when 'mpirun -n 5 mpich-3.0.4/examples/hellow' freezes:
After that freeze, I killed mpirun with Ctrl + C: " marto@dragonfly% mpirun -n 5 mpich-3.0.4/examples/hellow ^C[mpiexec@dragonfly] Sending Ctrl-C to processes as requested [mpiexec@dragonfly] Press Ctrl-C again to force abort [proxy:0:0@dragonfly] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:71): assert (!(pollfds[i].revents & ~POLLIN & ~POLLOUT & ~POLLHUP & ~POLLERR)) failed [proxy:0:0@dragonfly] main (./pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@dragonfly] control_cb (./pm/pmiserv/pmiserv_cb.c:202): assert (!closed) failed [mpiexec@dragonfly] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status [mpiexec@dragonfly] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:197): error waiting for event [mpiexec@dragonfly] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion "
The next relaunch of mpirun with 5 cores was successful. For completeness, I am attaching the output of 'ps x' after mpirun with 2 cores freezes, which it actually with 2 cores always does.
I hope this was helpful. I am looking forward to your reply.
Hi, Well, I took this as an opportunity to play with dfly again, so I replicated mpich 3.0.4 and see much the same behavior. Mpirun -n 2 hangs consistently, -n 4 works reliably (= # system cores), -n 5 sometimes. I’m not an mpich developer so I will leave further prognostication to their capable hands, but your suspicions about hwloc from the warning messages seem to be well-founded. Tony