- discuss - mailman3.cels.anl.gov

Question about non-blocking MPI - 3
by teivml＠gmail.com 04 Dec '20

04 Dec '20

Hello everyone, What is the environment setting for the use of non-blocking MPI-3 * collectives in mpich3.3? In Intel MPI, the setting is as follows $ export I_MPI_ASYNC_PROGRESS=1 $ export I_MPI_ASYNC_PROGRESS_PIN=<CPU list> Is there an equivalent setting in mpich3.3? Thank you, Viet.

1 0

Re: [mpich-discuss] "Graceful" recovery from segmentation fault
by Balaji, Pavan 05 Nov '20

05 Nov '20

Hi Kurt, MPICH should gracefully cleanup all of the remaining processes, when one process dies. If it is not doing that, then it might a bug. But before we go digging into it, can you try the “mpiexec” from the latest mpich version? There have been a few bug fixes. In fact, we just fixed one more bug a couple of days ago (https://github.com/pmodels/mpich/pull/4862/commits/7f362d74ee) although I think this last bug might be unrelated to the error that you are seeing. — Pavan > On Nov 5, 2020, at 12:42 AM, Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> wrote: > > I’m using MPICH 3.3.2, Torque 5.1.1. > > I’ve discussed something like this with you guys before – is there any way to have MPICH shut down the > entire job after a seg fault, rather than running forever? You’ve said that what really needs to be done > is to locate and fix the bug causing it, which is right, but I’m just worried that a non-terminating job would > be confusing to the user that encounters it, if some new unanticipated seg fault were generated in the future. > > I know that I can’t put MPI_Finalize() or any non-async safe code in my signal handler. It helps a little bit > to print a backtrace from the signal handler (I know, I/O is also not safe). > > Is there anything else that can be done? Here are the MPICH error messages written some minutes after the > seg fault occurs, when the job is ending and all of the good processes are calling MPI_Finalize(). > > Thanks, Kurt > > [proxy:0:[email protected]] [proxy:1:[email protected]] [proxy:3:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): assert (!closed) failed > [proxy:4:[email protected]] assert (!closed) failed > assert (!closed) failed > [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): [proxy:1:[email protected]] [proxy:3:[email protected]] [proxy:5:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): assert (!closed) failed > HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): [proxy:6:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): callback returned error status > [proxy:4:[email protected]] [proxy:8:[email protected]] callback returned error status > callback returned error status > [proxy:9:[email protected]] assert (!closed) failed > HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): [proxy:1:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): [proxy:3:[email protected]] [proxy:10:[email protected]] [proxy:5:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): assert (!closed) failed > main (pm/pmiserv/pmip.c:200): callback returned error status > main (pm/pmiserv/pmip.c:200): assert (!closed) failed > main (pm/pmiserv/pmip.c:200): HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): assert (!closed) failed > [proxy:6:[email protected]] demux engine error waiting for event > [proxy:4:[email protected]] demux engine error waiting for event > [proxy:8:[email protected]] demux engine error waiting for event > callback returned error status > assert (!closed) failed > [proxy:9:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): main (pm/pmiserv/pmip.c:200): HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): [proxy:5:[email protected]] callback returned error status > [proxy:10:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status > demux engine error waiting for event > main (pm/pmiserv/pmip.c:200): [proxy:8:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): [proxy:7:[email protected]] callback returned error status > [proxy:6:[email protected]] demux engine error waiting for event > main (pm/pmiserv/pmip.c:200): callback returned error status > [proxy:9:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): main (pm/pmiserv/pmip.c:200): demux engine error waiting for event > [proxy:10:[email protected]] main (pm/pmiserv/pmip.c:200): assert (!closed) failed > demux engine error waiting for event > main (pm/pmiserv/pmip.c:200): demux engine error waiting for event > [proxy:7:[email protected]] demux engine error waiting for event > HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status > [proxy:7:[email protected]] main (pm/pmiserv/pmip.c:200): demux engine error waiting for event > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

"Graceful" recovery from segmentation fault
by Mccall, Kurt E. (MSFC-EV41) 05 Nov '20

05 Nov '20

I'm using MPICH 3.3.2, Torque 5.1.1. I've discussed something like this with you guys before - is there any way to have MPICH shut down the entire job after a seg fault, rather than running forever? You've said that what really needs to be done is to locate and fix the bug causing it, which is right, but I'm just worried that a non-terminating job would be confusing to the user that encounters it, if some new unanticipated seg fault were generated in the future. I know that I can't put MPI_Finalize() or any non-async safe code in my signal handler. It helps a little bit to print a backtrace from the signal handler (I know, I/O is also not safe). Is there anything else that can be done? Here are the MPICH error messages written some minutes after the seg fault occurs, when the job is ending and all of the good processes are calling MPI_Finalize(). Thanks, Kurt [proxy:0:[email protected]] [proxy:1:[email protected]] [proxy:3:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): assert (!closed) failed [proxy:4:[email protected]] assert (!closed) failed assert (!closed) failed [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): [proxy:1:[email protected]] [proxy:3:[email protected]] [proxy:5:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): assert (!closed) failed HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): [proxy:6:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): callback returned error status [proxy:4:[email protected]] [proxy:8:[email protected]] callback returned error status callback returned error status [proxy:9:[email protected]] assert (!closed) failed HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): [proxy:1:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): [proxy:3:[email protected]] [proxy:10:[email protected]] [proxy:5:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): assert (!closed) failed main (pm/pmiserv/pmip.c:200): callback returned error status main (pm/pmiserv/pmip.c:200): assert (!closed) failed main (pm/pmiserv/pmip.c:200): HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): assert (!closed) failed [proxy:6:[email protected]] demux engine error waiting for event [proxy:4:[email protected]] demux engine error waiting for event [proxy:8:[email protected]] demux engine error waiting for event callback returned error status assert (!closed) failed [proxy:9:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): main (pm/pmiserv/pmip.c:200): HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): [proxy:5:[email protected]] callback returned error status [proxy:10:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status demux engine error waiting for event main (pm/pmiserv/pmip.c:200): [proxy:8:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): [proxy:7:[email protected]] callback returned error status [proxy:6:[email protected]] demux engine error waiting for event main (pm/pmiserv/pmip.c:200): callback returned error status [proxy:9:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:878): main (pm/pmiserv/pmip.c:200): demux engine error waiting for event [proxy:10:[email protected]] main (pm/pmiserv/pmip.c:200): assert (!closed) failed demux engine error waiting for event main (pm/pmiserv/pmip.c:200): demux engine error waiting for event [proxy:7:[email protected]] demux engine error waiting for event HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status [proxy:7:[email protected]] main (pm/pmiserv/pmip.c:200): demux engine error waiting for event

1 0

MPICH not working for multiple nodes over ssh
by Shubham Sahoo 04 Nov '20

04 Nov '20

Hello, I am trying to use mpich for a beowulf cluster. I am able to do passwordless ssh from one node to another. I have made a common directory using nfs mount for both the nodes. So I don't think there are any issues on the ssh or networking side, but still mpirun is not working. The output of the following line is : # mpiuser@ip-172-31-38-210:~/cloud$ mpirun -np 2 -hosts node1,node0 -v --launcher ssh ./cpi host: node1 host: node0 ================================================================================================== mpiexec options: ---------------- Base path: /usr/bin/ Launcher: ssh Debug level: 1 Enable X: -1 Global environment: ------------------- SHELL=/bin/bash PWD=/home/mpiuser/cloud LOGNAME=mpiuser HOME=/home/mpiuser LANG=C.UTF-8 LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36: LESSCLOSE=/usr/bin/lesspipe %s %s TERM=xterm LESSOPEN=| /usr/bin/lesspipe %s USER=mpiuser SHLVL=1 LD_LIBRARY_PATH=”/usr/lib:” XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop PATH=/usr/bin:/usr/bin:/usr/lib:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin MAIL=/var/mail/mpiuser _=/usr/bin/mpirun OLDPWD=/home/mpiuser Hydra internal environment: --------------------------- GFORTRAN_UNBUFFERED_PRECONNECTED=y Proxy information: ********************* [1] proxy: node1 (1 cores) Exec list: ./cpi (1 processes); [2] proxy: node0 (1 cores) Exec list: ./cpi (1 processes); ================================================================================================== [mpiexec@ip-172-31-38-210] Timeout set to -1 (-1 means infinite) [mpiexec@ip-172-31-38-210] Got a control port string of ip-172-31-38-210:39041 Proxy launch args: /usr/bin/hydra_pmi_proxy --control-port ip-172-31-38-210:39041 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id Arguments being passed to proxy 0: --version 3.3.2 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname node1 --global-core-map 0,1,2 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_8191_0_1424805634_ip-172-31-38-210 --pmi-process-mapping (vector,(0,2,1)) --ckpoint-num -1 --global-inherited-env 17 'SHELL=/bin/bash' 'PWD=/home/mpiuser/cloud' 'LOGNAME=mpiuser' 'HOME=/home/mpiuser' 'LANG=C.UTF-8' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:' 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'TERM=xterm' 'LESSOPEN=| /usr/bin/lesspipe %s' 'USER=mpiuser' 'SHLVL=1' 'LD_LIBRARY_PATH=”/usr/lib:”' 'XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop' 'PATH=/usr/bin:/usr/bin:/usr/lib:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin' 'MAIL=/var/mail/mpiuser' '_=/usr/bin/mpirun' 'OLDPWD=/home/mpiuser' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/mpiuser/cloud --exec-args 1 ./cpi Arguments being passed to proxy 1: --version 3.3.2 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname node0 --global-core-map 0,1,2 --pmi-id-map 0,1 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_8191_0_1424805634_ip-172-31-38-210 --pmi-process-mapping (vector,(0,2,1)) --ckpoint-num -1 --global-inherited-env 17 'SHELL=/bin/bash' 'PWD=/home/mpiuser/cloud' 'LOGNAME=mpiuser' 'HOME=/home/mpiuser' 'LANG=C.UTF-8' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:' 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'TERM=xterm' 'LESSOPEN=| /usr/bin/lesspipe %s' 'USER=mpiuser' 'SHLVL=1' 'LD_LIBRARY_PATH=”/usr/lib:”' 'XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop' 'PATH=/usr/bin:/usr/bin:/usr/lib:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin' 'MAIL=/var/mail/mpiuser' '_=/usr/bin/mpirun' 'OLDPWD=/home/mpiuser' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/mpiuser/cloud --exec-args 1 ./cpi [mpiexec@ip-172-31-38-210] Launch arguments: /usr/bin/ssh -x node1 "/usr/bin/hydra_pmi_proxy" --control-port ip-172-31-38-210:39041 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 0 [mpiexec@ip-172-31-38-210] Launch arguments: /usr/bin/ssh -x node0 "/usr/bin/hydra_pmi_proxy" --control-port ip-172-31-38-210:39041 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 1 [proxy:0:1@ip-172-31-38-210] got pmi command (from 4): init pmi_version=1 pmi_subversion=1 [proxy:0:1@ip-172-31-38-210] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@ip-172-31-38-210] got pmi command (from 4): get_maxes [proxy:0:1@ip-172-31-38-210] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@ip-172-31-38-210] got pmi command (from 4): get_appnum [proxy:0:1@ip-172-31-38-210] PMI response: cmd=appnum appnum=0 [proxy:0:1@ip-172-31-38-210] got pmi command (from 4): get_my_kvsname [proxy:0:1@ip-172-31-38-210] PMI response: cmd=my_kvsname kvsname=kvs_8191_0_1424805634_ip-172-31-38-210 [proxy:0:1@ip-172-31-38-210] got pmi command (from 4): get_my_kvsname [proxy:0:1@ip-172-31-38-210] PMI response: cmd=my_kvsname kvsname=kvs_8191_0_1424805634_ip-172-31-38-210 [proxy:0:1@ip-172-31-38-210] got pmi command (from 4): get_my_kvsname [proxy:0:1@ip-172-31-38-210] PMI response: cmd=my_kvsname kvsname=kvs_8191_0_1424805634_ip-172-31-38-210 [proxy:0:1@ip-172-31-38-210] got pmi command (from 4): get kvsname=kvs_8191_0_1424805634_ip-172-31-38-210 key=PMI_process_mapping [proxy:0:1@ip-172-31-38-210] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:1@ip-172-31-38-210] [mpiexec@ip-172-31-38-210] [pgid: 0] got PMI command: cmd=barrier_in got pmi command (from 4): barrier_in [proxy:0:1@ip-172-31-38-210] forwarding command (cmd=barrier_in) upstream ^C[mpiexec@ip-172-31-38-210] Sending Ctrl-C to processes as requested [mpiexec@ip-172-31-38-210] Press Ctrl-C again to force abort [mpiexec@ip-172-31-38-210] HYDU_sock_write (utils/sock/sock.c:256): write error (Bad file descriptor) [mpiexec@ip-172-31-38-210] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:178): unable to write data to proxy [mpiexec@ip-172-31-38-210] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:77): unable to send signal downstream [mpiexec@ip-172-31-38-210] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status [mpiexec@ip-172-31-38-210] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:196): error waiting for event [mpiexec@ip-172-31-38-210] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion

1 0

ULFM and timeouts
by Shead, Timothy 13 Oct '20

13 Oct '20

Folks: I’m interested in running some fault tolerance experiments using the ULFM extensions in MPICH, but I’m stumbling right out of the gate. When I run the following Python code, rank 0 broadcasts a monotonically increasing value to the other ranks, and it gets printed to stdout: import itertools import time from mpi4py import MPI comm = MPI.COMM_WORLD for count in itertools.count(): value = comm.bcast(count, root=0) print(f"{comm.rank}: {value}", flush=True) time.sleep(1) So far, so good. If I run the code using mpiexec -n 3 --disable-auto-cleanup python test.py I get the expected output. If I kill one of the three processes, the others keep running for a few more iterations thanks to --disable-auto-cleanup, until bcast() blocks. My assumption was that MPICH would eventually return an error code, rather than blocking. Otherwise, it doesn’t seem like my code will ever have the chance to use revoke() and shrink(). What am I missing here? Is there a way to specify timeouts for blocking operations? Am I limited to using async operations with ULFM? Could you refer me to any ULFM tests or examples that you used in development? Thanks in advance, Tim Timothy M. Shead Sandia National Laboratories tshead(a)sandia.gov<mailto:[email protected]>

1 0

Re: [mpich-discuss] Regarding issue with MPICH 3.2
by Zhou, Hui 09 Oct '20

09 Oct '20

Thanks, Neeraj singh. To confirm, you always have this error with the profiler, but intermittent without profiler? When the error shows up, does the error show up at the beginning of the application, middle of the application, or near the end of the application? -- Hui Zhou From: Neeraj singh <neeraj844singh(a)gmail.com> Date: Friday, October 9, 2020 at 9:22 AM To: "Zhou, Hui" <zhouh(a)anl.gov> Cc: "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] Regarding issue with MPICH 3.2 Yes, but sometimes without profiler also, I am also facing this issue with some applications also. Thanks and regards On Fri, Oct 9, 2020 at 7:27 PM Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: Hi Neeraj singh, To clarify, does this error only happen when you try to use the profiler tool? -- Hui Zhou From: Neeraj singh via discuss <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Friday, October 9, 2020 at 8:41 AM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: Neeraj singh <neeraj844singh(a)gmail.com<mailto:[email protected]>> Subject: [mpich-discuss] Regarding issue with MPICH 3.2 I am using the MAQAO profiler tool to collect data based on sampling. While launching an application for multinode some issues I'm facing. Kindly check the attached problem lines and clarify the same. Thanks & Regards Problem lines: [mpiexec@gpu7] HYDU_sock_write (utils/sock/sock.c:294): write error (Bad file descriptor) [mpiexec@gpu7] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:177): unable to write data to proxy [mpiexec@gpu7] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:79): unable to send signal downstream [mpiexec@gpu7] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec@gpu7] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec@gpu7] main (ui/mpich/mpiexec.c:340): process manager error waiting for completion

1 0

Re: [mpich-discuss] Regarding issue with MPICH 3.2
by Neeraj singh 09 Oct '20

09 Oct '20

Yes, but sometimes without profiler also, I am also facing this issue with some applications also. Thanks and regards On Fri, Oct 9, 2020 at 7:27 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > Hi Neeraj singh, > > > > To clarify, does this error only happen when you try to use the profiler > tool? > > > > -- > Hui Zhou > > > > > > *From: *Neeraj singh via discuss <discuss(a)mpich.org> > *Reply-To: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Date: *Friday, October 9, 2020 at 8:41 AM > *To: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Cc: *Neeraj singh <neeraj844singh(a)gmail.com> > *Subject: *[mpich-discuss] Regarding issue with MPICH 3.2 > > > > I am using the MAQAO profiler tool to collect data based on sampling. > While launching an application for multinode some issues I'm facing. > > Kindly check the attached problem lines and clarify the same. > > Thanks & Regards > > > > > > Problem lines: > > [mpiexec@gpu7] HYDU_sock_write (utils/sock/sock.c:294): write error (Bad > file descriptor) > [mpiexec@gpu7] HYD_pmcd_pmiserv_send_signal > (pm/pmiserv/pmiserv_cb.c:177): unable to write data to proxy > [mpiexec@gpu7] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:79): unable to send > signal downstream > [mpiexec@gpu7] HYDT_dmxu_poll_wait_for_event > (tools/demux/demux_poll.c:76): callback returned error status > [mpiexec@gpu7] HYD_pmci_wait_for_completion > (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event > [mpiexec@gpu7] main (ui/mpich/mpiexec.c:340): process manager error > waiting for completion > > >

1 0

Re: [mpich-discuss] Regarding issue with MPICH 3.2
by Zhou, Hui 09 Oct '20

09 Oct '20

Hi Neeraj singh, To clarify, does this error only happen when you try to use the profiler tool? -- Hui Zhou From: Neeraj singh via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Friday, October 9, 2020 at 8:41 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: Neeraj singh <neeraj844singh(a)gmail.com> Subject: [mpich-discuss] Regarding issue with MPICH 3.2 I am using the MAQAO profiler tool to collect data based on sampling. While launching an application for multinode some issues I'm facing. Kindly check the attached problem lines and clarify the same. Thanks & Regards Problem lines: [mpiexec@gpu7] HYDU_sock_write (utils/sock/sock.c:294): write error (Bad file descriptor) [mpiexec@gpu7] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:177): unable to write data to proxy [mpiexec@gpu7] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:79): unable to send signal downstream [mpiexec@gpu7] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec@gpu7] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec@gpu7] main (ui/mpich/mpiexec.c:340): process manager error waiting for completion

1 0

Regarding issue with MPICH 3.2
by Neeraj singh 09 Oct '20

09 Oct '20

I am using the MAQAO profiler tool to collect data based on sampling. While launching an application for multinode some issues I'm facing. Kindly check the attached problem lines and clarify the same. Thanks & Regards Problem lines: [mpiexec@gpu7] HYDU_sock_write (utils/sock/sock.c:294): write error (Bad file descriptor) [mpiexec@gpu7] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:177): unable to write data to proxy [mpiexec@gpu7] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:79): unable to send signal downstream [mpiexec@gpu7] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec@gpu7] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec@gpu7] main (ui/mpich/mpiexec.c:340): process manager error waiting for completion

1 0

Re: [mpich-discuss] Intermittent MPICH error
by Mccall, Kurt E. (MSFC-EV41) 06 Oct '20

06 Oct '20

Thanks Hui. I’ll investigate further. Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Monday, October 5, 2020 6:16 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] Intermittent MPICH error I believe it means one of the node process was terminated unexpectedly. We’ll need more clue to make a guess on what is the actual cause. -- Hui Zhou From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Monday, October 5, 2020 at 4:47 PM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [mpich-discuss] Intermittent MPICH error I am using MPICH 3.3.2 on Centos 3.10, Torque 5.1.1. I have never seen this error when running my MPI jobs as myself, but when other users run the same code, this error may or may not occur. It seems to happen when the job is shutting down. [mpiexec(a)n006.cluster.com] HYDU_sock_write (utils/sock/sock.c:256): write error (Bad file descriptor) [mpiexec(a)n006.cluster.com] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:178): unable to write data to proxy [mpiexec(a)n006.cluster.com] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:77): unable to send signal downstream [mpiexec(a)n006.cluster.com] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status [mpiexec(a)n006.cluster.com] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:196): error waiting for event [mpiexec(a)n006.cluster.com] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion Does anyone have a clue what these errors mean? I hope this is enough info to go on. If not, please let me know what else I should provide. Thanks, Kurt

1 0