- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] [EXTERNAL] Re: mpiexec fails to launch any processes
by Mccall, Kurt E. (MSFC-EV41) 13 Jun '22

13 Jun '22

Hui, That worked too. I guess I’ll have to find a way to pass a “verbose” argument to sbatch and see why Slurm is killing my application. Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Monday, June 13, 2022 4:11 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: Re: [EXTERNAL] Re: mpiexec fails to launch any processes Kurt, Could you try launch hostname with the same command? mpiexec -launcher ssh -verbose -print-all-exitcodes -wdir <directory> -np 20 -ppn 1 hostname If that went okay, it then seems to point to your application. Something in your code made Slurm kill the job. -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Sent: Monday, June 13, 2022 4:02 PM To: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Subject: RE: [EXTERNAL] Re: mpiexec fails to launch any processes Hui, $ mpiexec -N 10 -hostfile MySlurmNodeFile2 hostname works properly, reporting from each of 10 nodes. Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Monday, June 13, 2022 2:44 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: mpiexec fails to launch any processes Hi Kurt, I don't have much clue. Are you able to launch some trivial applications, for example, "hostname"? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Monday, June 13, 2022 12:29 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] mpiexec fails to launch any processes Outlook blocked the output file slurm.out that I had attached. Trying to send it again as slurm.txt. Kurt Hi, My mpiexec command fails to launch any processes. I ran it with the -verbose option but didn’t see any obvious errors in the output (attached). The command is: mpiexec -launcher ssh -verbose -print-all-exitcodes -wdir <directory> -np 20 -ppn 1 <more args…> I am running MPICH 4.0.1 under Slurm 20.11.8. Thanks for any help. Kurt

1 0

Re: [mpich-discuss] [EXTERNAL] Re: mpiexec fails to launch any processes
by Zhou, Hui 13 Jun '22

13 Jun '22

Kurt, Could you try launch hostname with the same command? mpiexec -launcher ssh -verbose -print-all-exitcodes -wdir <directory> -np 20 -ppn 1 hostname If that went okay, it then seems to point to your application. Something in your code made Slurm kill the job. -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Sent: Monday, June 13, 2022 4:02 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: RE: [EXTERNAL] Re: mpiexec fails to launch any processes Hui, $ mpiexec -N 10 -hostfile MySlurmNodeFile2 hostname works properly, reporting from each of 10 nodes. Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Monday, June 13, 2022 2:44 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: mpiexec fails to launch any processes Hi Kurt, I don't have much clue. Are you able to launch some trivial applications, for example, "hostname"? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Monday, June 13, 2022 12:29 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] mpiexec fails to launch any processes Outlook blocked the output file slurm.out that I had attached. Trying to send it again as slurm.txt. Kurt Hi, My mpiexec command fails to launch any processes. I ran it with the -verbose option but didn’t see any obvious errors in the output (attached). The command is: mpiexec -launcher ssh -verbose -print-all-exitcodes -wdir <directory> -np 20 -ppn 1 <more args…> I am running MPICH 4.0.1 under Slurm 20.11.8. Thanks for any help. Kurt

1 0

Re: [mpich-discuss] [EXTERNAL] Re: mpiexec fails to launch any processes
by Mccall, Kurt E. (MSFC-EV41) 13 Jun '22

13 Jun '22

Hui, $ mpiexec -N 10 -hostfile MySlurmNodeFile2 hostname works properly, reporting from each of 10 nodes. Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Monday, June 13, 2022 2:44 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: mpiexec fails to launch any processes Hi Kurt, I don't have much clue. Are you able to launch some trivial applications, for example, "hostname"? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Monday, June 13, 2022 12:29 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] mpiexec fails to launch any processes Outlook blocked the output file slurm.out that I had attached. Trying to send it again as slurm.txt. Kurt Hi, My mpiexec command fails to launch any processes. I ran it with the -verbose option but didn’t see any obvious errors in the output (attached). The command is: mpiexec -launcher ssh -verbose -print-all-exitcodes -wdir <directory> -np 20 -ppn 1 <more args…> I am running MPICH 4.0.1 under Slurm 20.11.8. Thanks for any help. Kurt

1 0

Re: [mpich-discuss] mpiexec fails to launch any processes
by Zhou, Hui 13 Jun '22

13 Jun '22

Hi Kurt, I don't have much clue. Are you able to launch some trivial applications, for example, "hostname"? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Monday, June 13, 2022 12:29 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: Re: [mpich-discuss] mpiexec fails to launch any processes Outlook blocked the output file slurm.out that I had attached. Trying to send it again as slurm.txt. Kurt Hi, My mpiexec command fails to launch any processes. I ran it with the -verbose option but didn’t see any obvious errors in the output (attached). The command is: mpiexec -launcher ssh -verbose -print-all-exitcodes -wdir <directory> -np 20 -ppn 1 <more args…> I am running MPICH 4.0.1 under Slurm 20.11.8. Thanks for any help. Kurt

1 0

Re: [mpich-discuss] mpiexec fails to launch any processes
by Mccall, Kurt E. (MSFC-EV41) 13 Jun '22

13 Jun '22

Outlook blocked the output file slurm.out that I had attached. Trying to send it again as slurm.txt. Kurt Hi, My mpiexec command fails to launch any processes. I ran it with the -verbose option but didn't see any obvious errors in the output (attached). The command is: mpiexec -launcher ssh -verbose -print-all-exitcodes -wdir <directory> -np 20 -ppn 1 <more args...> I am running MPICH 4.0.1 under Slurm 20.11.8. Thanks for any help. Kurt

1 0

mpiexec fails to launch any processes
by Mccall, Kurt E. (MSFC-EV41) 13 Jun '22

13 Jun '22

Hi, My mpiexec command fails to launch any processes. I ran it with the -verbose option but didn't see any obvious errors in the output (attached). The command is: mpiexec -launcher ssh -verbose -print-all-exitcodes -wdir <directory> -np 20 -ppn 1 <more args...> I am running MPICH 4.0.1 under Slurm 20.11.8. Thanks for any help. Kurt

1 0

Re: [mpich-discuss] Join the MPICH team at ECP Community BoF Days next week!
by Raffenetti, Ken 06 May '22

06 May '22

Quick update to the agenda. Krishna Kandalla will give an update for HPE. Agenda (subject to change): [5 mins] Intro - Ken Raffenetti ANL [30 mins] MPICH Update - Yanfei Guo & Hui Zhou (ANL) [12 mins] Intel Update - Gengbin Zheng (Intel) [12 mins] MVAPICH2 Update - Hari Subramoni (OSU) [12 mins] HPE Update - Krishna Kandalla (HPE) [12 mins] Pilgrim Update - Chen Wang (UIUC) [~5 mins] Q&A Cheers, The MPICH team On 5/6/22, 2:12 PM, "Raffenetti, Ken" <raffenet(a)anl.gov> wrote: All, The MPICH team will host a session as part of next week's ECP Community BoF Days (https://www.exascaleproject.org/event/ecp-community-bof-days-2022/) We will discuss MPICH release updates as well as hear from some of our partners. The event is virtual and registration is free at this link: https://exascaleproject.zoomgov.com/meeting/register/vJIsduuppjosHSWqb9g4oo… Date and time: May 11, 2022 12:00-1:30 PM Central Agenda (subject to change): 12:00-12:05: Intro - Ken Raffenetti (ANL) 12:05-12:35: MPICH Update - Yanfei Guo & Hui Zhou (ANL) 12:35-12:50: Intel Update - Gengbin Zheng (Intel) 12:50-13:05: MVAPICH2 Update - Hari Subramoni (OSU) 13:05-13:20: Pilgrim* Update - Chen Wang (UIUC) 13:20-13:30: Q&A This information is also available on our wiki: https://github.com/pmodels/mpich/wiki/2022-ECP-Community-BoF Cheers, The MPICH team *Pilgrim: https://github.com/pmodels/pilgrim

1 0

Join the MPICH team at ECP Community BoF Days next week!
by Raffenetti, Ken 06 May '22

06 May '22

All, The MPICH team will host a session as part of next week's ECP Community BoF Days (https://www.exascaleproject.org/event/ecp-community-bof-days-2022/) We will discuss MPICH release updates as well as hear from some of our partners. The event is virtual and registration is free at this link: https://exascaleproject.zoomgov.com/meeting/register/vJIsduuppjosHSWqb9g4oo… Date and time: May 11, 2022 12:00-1:30 PM Central Agenda (subject to change): 12:00-12:05: Intro - Ken Raffenetti (ANL) 12:05-12:35: MPICH Update - Yanfei Guo & Hui Zhou (ANL) 12:35-12:50: Intel Update - Gengbin Zheng (Intel) 12:50-13:05: MVAPICH2 Update - Hari Subramoni (OSU) 13:05-13:20: Pilgrim* Update - Chen Wang (UIUC) 13:20-13:30: Q&A This information is also available on our wiki: https://github.com/pmodels/mpich/wiki/2022-ECP-Community-BoF Cheers, The MPICH team *Pilgrim: https://github.com/pmodels/pilgrim

1 0

Re: [mpich-discuss] [EXTERNAL] Re: mpiexec error
by Mccall, Kurt E. (MSFC-EV41) 06 May '22

06 May '22

Ken, That was it -- looks like Torque requires FQDNs for MPI_Comm_spawn. I had the code set up for Slurm, which requires short host names. Thanks for your help! Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, May 6, 2022 12:23 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: Re: [EXTERNAL] Re: [mpich-discuss] mpiexec error Are the hostnames in the file all fully qualified? For example "n022.cluster.com". This error message suggests its looking for host "n022". [mpiexec(a)n022.cluster.com] HYDT_bscd_pbs_launch_procs (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/external/pbs_launch.c:74): error finding PBS node ID for host n022 I am trying to understand if its failing to match because somewhere fully qualified part is being stripped. Ken On 5/6/22, 12:17 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, The node file has 420 lines, with 20 lines for each host, including n022.cluster.com (21 nodes, 20 cores per node) Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, May 6, 2022 12:13 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] mpiexec error Hi Kurt, Before running mpiexec, can you print out the hostfile to confirm the contents? Something like this: cat $PBS_HOSTFILE Ken On 5/6/22, 11:55 AM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Running MPICH 4.0.1 under Torque 5.1, I’m getting the mpiexec error “user specified host not in the PBS allocated list”. My qsub command is: qsub -V -j oe -e stdio -o stdio -f -X -l nodes=21:ppn=20 <bash_script> My mpiexec command is: mpiexec -print-all-exitcodes -enable-x -np 21 -wdir ${work_dir} -env DISPLAY localhost:10.0 --ppn 1 <more args> … Here is the full error message. Thanks for any help. [mpiexec(a)n022.cluster.com] find_pbs_node_id (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/external/pbs_launch.c:27): user specified host not in the PBS allocated list [mpiexec(a)n022.cluster.com] HYDT_bscd_pbs_launch_procs (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/external/pbs_launch.c:74): error finding PBS node ID for host n022 [mpiexec(a)n022.cluster.com] HYDT_bsci_launch_procs (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/src/bsci_launch.c:17): launcher returned error while launching processes [mpiexec(a)n022.cluster.com] fn_spawn (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_pmi_v1.c:580): launcher cannot launch processes [mpiexec(a)n022.cluster.com] handle_pmi_cmd (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:48): PMI handler returned error [mpiexec(a)n022.cluster.com] control_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:284): unable to process PMI command [mpiexec(a)n022.cluster.com] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [mpiexec(a)n022.cluster.com] HYD_pmci_wait_for_completion (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:160): error waiting for event [mpiexec(a)n022.cluster.com] main (../../../../mpich-4.0.1/src/pm/hydra/ui/mpich/mpiexec.c:325): process manager error waiting for completion [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demudemux_poll.c:76): callback returned error status [proxy:0:[email protected]] main (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip.c:169): demux engine error waiting for event demux_poll.c:76): callback returned error status [proxy:0:[email protected]] main (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip.c:169): demux engine error waiting for event [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed

1 0

Re: [mpich-discuss] [EXTERNAL] Re: mpiexec error
by Raffenetti, Ken 06 May '22

06 May '22

Are the hostnames in the file all fully qualified? For example "n022.cluster.com". This error message suggests its looking for host "n022". [mpiexec(a)n022.cluster.com] HYDT_bscd_pbs_launch_procs (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/external/pbs_launch.c:74): error finding PBS node ID for host n022 I am trying to understand if its failing to match because somewhere fully qualified part is being stripped. Ken On 5/6/22, 12:17 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, The node file has 420 lines, with 20 lines for each host, including n022.cluster.com (21 nodes, 20 cores per node) Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, May 6, 2022 12:13 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] mpiexec error Hi Kurt, Before running mpiexec, can you print out the hostfile to confirm the contents? Something like this: cat $PBS_HOSTFILE Ken On 5/6/22, 11:55 AM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Running MPICH 4.0.1 under Torque 5.1, I’m getting the mpiexec error “user specified host not in the PBS allocated list”. My qsub command is: qsub -V -j oe -e stdio -o stdio -f -X -l nodes=21:ppn=20 <bash_script> My mpiexec command is: mpiexec -print-all-exitcodes -enable-x -np 21 -wdir ${work_dir} -env DISPLAY localhost:10.0 --ppn 1 <more args> … Here is the full error message. Thanks for any help. [mpiexec(a)n022.cluster.com] find_pbs_node_id (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/external/pbs_launch.c:27): user specified host not in the PBS allocated list [mpiexec(a)n022.cluster.com] HYDT_bscd_pbs_launch_procs (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/external/pbs_launch.c:74): error finding PBS node ID for host n022 [mpiexec(a)n022.cluster.com] HYDT_bsci_launch_procs (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/src/bsci_launch.c:17): launcher returned error while launching processes [mpiexec(a)n022.cluster.com] fn_spawn (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_pmi_v1.c:580): launcher cannot launch processes [mpiexec(a)n022.cluster.com] handle_pmi_cmd (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:48): PMI handler returned error [mpiexec(a)n022.cluster.com] control_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:284): unable to process PMI command [mpiexec(a)n022.cluster.com] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [mpiexec(a)n022.cluster.com] HYD_pmci_wait_for_completion (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:160): error waiting for event [mpiexec(a)n022.cluster.com] main (../../../../mpich-4.0.1/src/pm/hydra/ui/mpich/mpiexec.c:325): process manager error waiting for completion [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demudemux_poll.c:76): callback returned error status [proxy:0:[email protected]] main (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip.c:169): demux engine error waiting for event demux_poll.c:76): callback returned error status [proxy:0:[email protected]] main (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip.c:169): demux engine error waiting for event [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed

1 0