May 2022 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Join the MPICH team at ECP Community BoF Days next week!
by Raffenetti, Ken 06 May '22

06 May '22

Quick update to the agenda. Krishna Kandalla will give an update for HPE. Agenda (subject to change): [5 mins] Intro - Ken Raffenetti ANL [30 mins] MPICH Update - Yanfei Guo & Hui Zhou (ANL) [12 mins] Intel Update - Gengbin Zheng (Intel) [12 mins] MVAPICH2 Update - Hari Subramoni (OSU) [12 mins] HPE Update - Krishna Kandalla (HPE) [12 mins] Pilgrim Update - Chen Wang (UIUC) [~5 mins] Q&A Cheers, The MPICH team On 5/6/22, 2:12 PM, "Raffenetti, Ken" <raffenet(a)anl.gov> wrote: All, The MPICH team will host a session as part of next week's ECP Community BoF Days (https://www.exascaleproject.org/event/ecp-community-bof-days-2022/) We will discuss MPICH release updates as well as hear from some of our partners. The event is virtual and registration is free at this link: https://exascaleproject.zoomgov.com/meeting/register/vJIsduuppjosHSWqb9g4oo… Date and time: May 11, 2022 12:00-1:30 PM Central Agenda (subject to change): 12:00-12:05: Intro - Ken Raffenetti (ANL) 12:05-12:35: MPICH Update - Yanfei Guo & Hui Zhou (ANL) 12:35-12:50: Intel Update - Gengbin Zheng (Intel) 12:50-13:05: MVAPICH2 Update - Hari Subramoni (OSU) 13:05-13:20: Pilgrim* Update - Chen Wang (UIUC) 13:20-13:30: Q&A This information is also available on our wiki: https://github.com/pmodels/mpich/wiki/2022-ECP-Community-BoF Cheers, The MPICH team *Pilgrim: https://github.com/pmodels/pilgrim

1 0

Join the MPICH team at ECP Community BoF Days next week!
by Raffenetti, Ken 06 May '22

06 May '22

All, The MPICH team will host a session as part of next week's ECP Community BoF Days (https://www.exascaleproject.org/event/ecp-community-bof-days-2022/) We will discuss MPICH release updates as well as hear from some of our partners. The event is virtual and registration is free at this link: https://exascaleproject.zoomgov.com/meeting/register/vJIsduuppjosHSWqb9g4oo… Date and time: May 11, 2022 12:00-1:30 PM Central Agenda (subject to change): 12:00-12:05: Intro - Ken Raffenetti (ANL) 12:05-12:35: MPICH Update - Yanfei Guo & Hui Zhou (ANL) 12:35-12:50: Intel Update - Gengbin Zheng (Intel) 12:50-13:05: MVAPICH2 Update - Hari Subramoni (OSU) 13:05-13:20: Pilgrim* Update - Chen Wang (UIUC) 13:20-13:30: Q&A This information is also available on our wiki: https://github.com/pmodels/mpich/wiki/2022-ECP-Community-BoF Cheers, The MPICH team *Pilgrim: https://github.com/pmodels/pilgrim

1 0

Re: [mpich-discuss] [EXTERNAL] Re: mpiexec error
by Mccall, Kurt E. (MSFC-EV41) 06 May '22

06 May '22

Ken, That was it -- looks like Torque requires FQDNs for MPI_Comm_spawn. I had the code set up for Slurm, which requires short host names. Thanks for your help! Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, May 6, 2022 12:23 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: Re: [EXTERNAL] Re: [mpich-discuss] mpiexec error Are the hostnames in the file all fully qualified? For example "n022.cluster.com". This error message suggests its looking for host "n022". [mpiexec(a)n022.cluster.com] HYDT_bscd_pbs_launch_procs (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/external/pbs_launch.c:74): error finding PBS node ID for host n022 I am trying to understand if its failing to match because somewhere fully qualified part is being stripped. Ken On 5/6/22, 12:17 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, The node file has 420 lines, with 20 lines for each host, including n022.cluster.com (21 nodes, 20 cores per node) Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, May 6, 2022 12:13 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] mpiexec error Hi Kurt, Before running mpiexec, can you print out the hostfile to confirm the contents? Something like this: cat $PBS_HOSTFILE Ken On 5/6/22, 11:55 AM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Running MPICH 4.0.1 under Torque 5.1, I’m getting the mpiexec error “user specified host not in the PBS allocated list”. My qsub command is: qsub -V -j oe -e stdio -o stdio -f -X -l nodes=21:ppn=20 <bash_script> My mpiexec command is: mpiexec -print-all-exitcodes -enable-x -np 21 -wdir ${work_dir} -env DISPLAY localhost:10.0 --ppn 1 <more args> … Here is the full error message. Thanks for any help. [mpiexec(a)n022.cluster.com] find_pbs_node_id (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/external/pbs_launch.c:27): user specified host not in the PBS allocated list [mpiexec(a)n022.cluster.com] HYDT_bscd_pbs_launch_procs (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/external/pbs_launch.c:74): error finding PBS node ID for host n022 [mpiexec(a)n022.cluster.com] HYDT_bsci_launch_procs (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/src/bsci_launch.c:17): launcher returned error while launching processes [mpiexec(a)n022.cluster.com] fn_spawn (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_pmi_v1.c:580): launcher cannot launch processes [mpiexec(a)n022.cluster.com] handle_pmi_cmd (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:48): PMI handler returned error [mpiexec(a)n022.cluster.com] control_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:284): unable to process PMI command [mpiexec(a)n022.cluster.com] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [mpiexec(a)n022.cluster.com] HYD_pmci_wait_for_completion (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:160): error waiting for event [mpiexec(a)n022.cluster.com] main (../../../../mpich-4.0.1/src/pm/hydra/ui/mpich/mpiexec.c:325): process manager error waiting for completion [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demudemux_poll.c:76): callback returned error status [proxy:0:[email protected]] main (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip.c:169): demux engine error waiting for event demux_poll.c:76): callback returned error status [proxy:0:[email protected]] main (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip.c:169): demux engine error waiting for event [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed

1 0

Re: [mpich-discuss] [EXTERNAL] Re: mpiexec error
by Raffenetti, Ken 06 May '22

06 May '22

Are the hostnames in the file all fully qualified? For example "n022.cluster.com". This error message suggests its looking for host "n022". [mpiexec(a)n022.cluster.com] HYDT_bscd_pbs_launch_procs (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/external/pbs_launch.c:74): error finding PBS node ID for host n022 I am trying to understand if its failing to match because somewhere fully qualified part is being stripped. Ken On 5/6/22, 12:17 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, The node file has 420 lines, with 20 lines for each host, including n022.cluster.com (21 nodes, 20 cores per node) Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, May 6, 2022 12:13 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] mpiexec error Hi Kurt, Before running mpiexec, can you print out the hostfile to confirm the contents? Something like this: cat $PBS_HOSTFILE Ken On 5/6/22, 11:55 AM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Running MPICH 4.0.1 under Torque 5.1, I’m getting the mpiexec error “user specified host not in the PBS allocated list”. My qsub command is: qsub -V -j oe -e stdio -o stdio -f -X -l nodes=21:ppn=20 <bash_script> My mpiexec command is: mpiexec -print-all-exitcodes -enable-x -np 21 -wdir ${work_dir} -env DISPLAY localhost:10.0 --ppn 1 <more args> … Here is the full error message. Thanks for any help. [mpiexec(a)n022.cluster.com] find_pbs_node_id (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/external/pbs_launch.c:27): user specified host not in the PBS allocated list [mpiexec(a)n022.cluster.com] HYDT_bscd_pbs_launch_procs (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/external/pbs_launch.c:74): error finding PBS node ID for host n022 [mpiexec(a)n022.cluster.com] HYDT_bsci_launch_procs (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/src/bsci_launch.c:17): launcher returned error while launching processes [mpiexec(a)n022.cluster.com] fn_spawn (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_pmi_v1.c:580): launcher cannot launch processes [mpiexec(a)n022.cluster.com] handle_pmi_cmd (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:48): PMI handler returned error [mpiexec(a)n022.cluster.com] control_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:284): unable to process PMI command [mpiexec(a)n022.cluster.com] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [mpiexec(a)n022.cluster.com] HYD_pmci_wait_for_completion (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:160): error waiting for event [mpiexec(a)n022.cluster.com] main (../../../../mpich-4.0.1/src/pm/hydra/ui/mpich/mpiexec.c:325): process manager error waiting for completion [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demudemux_poll.c:76): callback returned error status [proxy:0:[email protected]] main (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip.c:169): demux engine error waiting for event demux_poll.c:76): callback returned error status [proxy:0:[email protected]] main (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip.c:169): demux engine error waiting for event [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed

1 0

Re: [mpich-discuss] [EXTERNAL] Re: mpiexec error
by Mccall, Kurt E. (MSFC-EV41) 06 May '22

06 May '22

Ken, The node file has 420 lines, with 20 lines for each host, including n022.cluster.com (21 nodes, 20 cores per node) Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, May 6, 2022 12:13 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] mpiexec error Hi Kurt, Before running mpiexec, can you print out the hostfile to confirm the contents? Something like this: cat $PBS_HOSTFILE Ken On 5/6/22, 11:55 AM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Running MPICH 4.0.1 under Torque 5.1, I’m getting the mpiexec error “user specified host not in the PBS allocated list”. My qsub command is: qsub -V -j oe -e stdio -o stdio -f -X -l nodes=21:ppn=20 <bash_script> My mpiexec command is: mpiexec -print-all-exitcodes -enable-x -np 21 -wdir ${work_dir} -env DISPLAY localhost:10.0 --ppn 1 <more args> … Here is the full error message. Thanks for any help. [mpiexec(a)n022.cluster.com] find_pbs_node_id (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/external/pbs_launch.c:27): user specified host not in the PBS allocated list [mpiexec(a)n022.cluster.com] HYDT_bscd_pbs_launch_procs (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/external/pbs_launch.c:74): error finding PBS node ID for host n022 [mpiexec(a)n022.cluster.com] HYDT_bsci_launch_procs (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/src/bsci_launch.c:17): launcher returned error while launching processes [mpiexec(a)n022.cluster.com] fn_spawn (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_pmi_v1.c:580): launcher cannot launch processes [mpiexec(a)n022.cluster.com] handle_pmi_cmd (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:48): PMI handler returned error [mpiexec(a)n022.cluster.com] control_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:284): unable to process PMI command [mpiexec(a)n022.cluster.com] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [mpiexec(a)n022.cluster.com] HYD_pmci_wait_for_completion (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:160): error waiting for event [mpiexec(a)n022.cluster.com] main (../../../../mpich-4.0.1/src/pm/hydra/ui/mpich/mpiexec.c:325): process manager error waiting for completion [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demudemux_poll.c:76): callback returned error status [proxy:0:[email protected]] main (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip.c:169): demux engine error waiting for event demux_poll.c:76): callback returned error status [proxy:0:[email protected]] main (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip.c:169): demux engine error waiting for event [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed

1 0

Re: [mpich-discuss] mpiexec error
by Raffenetti, Ken 06 May '22

06 May '22

Hi Kurt, Before running mpiexec, can you print out the hostfile to confirm the contents? Something like this: cat $PBS_HOSTFILE Ken On 5/6/22, 11:55 AM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Running MPICH 4.0.1 under Torque 5.1, I’m getting the mpiexec error “user specified host not in the PBS allocated list”. My qsub command is: qsub -V -j oe -e stdio -o stdio -f -X -l nodes=21:ppn=20 <bash_script> My mpiexec command is: mpiexec -print-all-exitcodes -enable-x -np 21 -wdir ${work_dir} -env DISPLAY localhost:10.0 --ppn 1 <more args> … Here is the full error message. Thanks for any help. [mpiexec(a)n022.cluster.com] find_pbs_node_id (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/external/pbs_launch.c:27): user specified host not in the PBS allocated list [mpiexec(a)n022.cluster.com] HYDT_bscd_pbs_launch_procs (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/external/pbs_launch.c:74): error finding PBS node ID for host n022 [mpiexec(a)n022.cluster.com] HYDT_bsci_launch_procs (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/src/bsci_launch.c:17): launcher returned error while launching processes [mpiexec(a)n022.cluster.com] fn_spawn (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_pmi_v1.c:580): launcher cannot launch processes [mpiexec(a)n022.cluster.com] handle_pmi_cmd (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:48): PMI handler returned error [mpiexec(a)n022.cluster.com] control_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:284): unable to process PMI command [mpiexec(a)n022.cluster.com] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [mpiexec(a)n022.cluster.com] HYD_pmci_wait_for_completion (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:160): error waiting for event [mpiexec(a)n022.cluster.com] main (../../../../mpich-4.0.1/src/pm/hydra/ui/mpich/mpiexec.c:325): process manager error waiting for completion [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demudemux_poll.c:76): callback returned error status [proxy:0:[email protected]] main (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip.c:169): demux engine error waiting for event demux_poll.c:76): callback returned error status [proxy:0:[email protected]] main (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip.c:169): demux engine error waiting for event [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed

1 0

mpiexec error
by Mccall, Kurt E. (MSFC-EV41) 06 May '22

06 May '22

Running MPICH 4.0.1 under Torque 5.1, I'm getting the mpiexec error "user specified host not in the PBS allocated list". My qsub command is: qsub -V -j oe -e stdio -o stdio -f -X -l nodes=21:ppn=20 <bash_script> My mpiexec command is: mpiexec -print-all-exitcodes -enable-x -np 21 -wdir ${work_dir} -env DISPLAY localhost:10.0 --ppn 1 <more args> ... Here is the full error message. Thanks for any help. [mpiexec(a)n022.cluster.com] find_pbs_node_id (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/external/pbs_launch.c:27): user specified host not in the PBS allocated list [mpiexec(a)n022.cluster.com] HYDT_bscd_pbs_launch_procs (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/external/pbs_launch.c:74): error finding PBS node ID for host n022 [mpiexec(a)n022.cluster.com] HYDT_bsci_launch_procs (../../../../mpich-4.0.1/src/pm/hydra/tools/bootstrap/src/bsci_launch.c:17): launcher returned error while launching processes [mpiexec(a)n022.cluster.com] fn_spawn (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_pmi_v1.c:580): launcher cannot launch processes [mpiexec(a)n022.cluster.com] handle_pmi_cmd (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:48): PMI handler returned error [mpiexec(a)n022.cluster.com] control_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:284): unable to process PMI command [mpiexec(a)n022.cluster.com] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [mpiexec(a)n022.cluster.com] HYD_pmci_wait_for_completion (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:160): error waiting for event [mpiexec(a)n022.cluster.com] main (../../../../mpich-4.0.1/src/pm/hydra/ui/mpich/mpiexec.c:325): process manager error waiting for completion [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demux/[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0.1/src/pm/hydra/tools/demudemux_poll.c:76): callback returned error status [proxy:0:[email protected]] main (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip.c:169): demux engine error waiting for event demux_poll.c:76): callback returned error status [proxy:0:[email protected]] main (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip.c:169): demux engine error waiting for event [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:899): assert (!closed) failed

1 0

Re: [mpich-discuss] mpiexec : multi nodes and multi-cores questions
by Zhou, Hui 06 May '22

06 May '22

Hi Nicholas, Assuming ParaView is using the same behavior as upstream hydra * -np 8 should launch 8 processes, 4 on each node * -np 4 should launch 4 processes, 2 on each node * To launch 4 processes on one node, try -hosts compute0:4,compute1:4 -np 4 * The number 4 behind each host instructs hydra to assign the first 4 ranks to the host before moving on to another assignment, the default is 1. Hydra uses round-robin assignment policy Hope it helps, -- Hui ________________________________ From: Nicholas Yue via discuss <discuss(a)mpich.org> Sent: Friday, May 6, 2022 11:11 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Nicholas Yue <yue.nicholas(a)gmail.com> Subject: [mpich-discuss] mpiexec : multi nodes and multi-cores questions Hi, I am using mpiexec from Paraview which from the `help` print out, indicates that it is derived from MPICH ``` Please see the intructions provided at http://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager for further details ``` so I thought I might ask some questions on this mailing list. I have set up a two node cluster, each compute node has 4 cores To fully utilise all the cores in both nodes, is this the correct way to launch pvserver ? ``` /shared/ParaView-5.10.1-osmesa-MPI-Linux-Python3.9-x86_64/bin/mpiexec -hosts compute0,compute1 -np 8 /shared/ParaView-5.10.1-osmesa-MPI-Linux-Python3.9-x86_64/bin/pvserver ``` If I change to `-np 4` would mpiexec use only compute0 or would it still run on both compute0 and compute1 but use only 2 cores of each node ? Cheers -- Nicholas Yue https://www.linkedin.com/in/nicholasyue/

1 0

mpiexec : multi nodes and multi-cores questions
by Nicholas Yue 06 May '22

06 May '22

Hi, I am using mpiexec from Paraview which from the `help` print out, indicates that it is derived from MPICH ``` Please see the intructions provided at http://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager for further details ``` so I thought I might ask some questions on this mailing list. I have set up a two node cluster, each compute node has 4 cores To fully utilise all the cores in both nodes, is this the correct way to launch pvserver ? ``` /shared/ParaView-5.10.1-osmesa-MPI-Linux-Python3.9-x86_64/bin/mpiexec -hosts compute0,compute1 -np 8 /shared/ParaView-5.10.1-osmesa-MPI-Linux-Python3.9-x86_64/bin/pvserver ``` If I change to `-np 4` would mpiexec use only compute0 or would it still run on both compute0 and compute1 but use only 2 cores of each node ? Cheers -- Nicholas Yue https://www.linkedin.com/in/nicholasyue/

1 0

Re: [mpich-discuss] libfabric failure during mpich install
by Zhou, Hui 06 May '22

06 May '22

Try configure libfabric with --enable-direct=sockets. You can pass the option directly to MPICH's configure and it should pass it down to libfabric -- Hui ________________________________ From: Marshall Tabetah via discuss <discuss(a)mpich.org> Sent: Friday, May 6, 2022 10:08 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Marshall Tabetah <met4t(a)virginia.edu> Subject: [mpich-discuss] libfabric failure during mpich install Hello, Team As I attempted to configure mpich-4.0.2 in my cygwin terminal, everything went on well until it failed to configure libfabric. The following error message was thus released upon termination. error: libfabric only builds on Linux, OS X, and FreeBSD. When I find out more information about libfabric, it seems it has support for Windows, so I do not surely know where the error comes from. Anyway, I hereby attach the files that communicate the status of the attempted configuration. Files that are not attached are those that were not produced. I appreciate further discussion/orientation. Cheers, Marshall

1 0