host: svr host: pc5 ================================================================================================== mpiexec options: ---------------- Base path: /global/usr/bin/ Launcher: (null) Debug level: 1 Enable X: -1 Global environment: ------------------- BIBINPUTS=.:/home/tladd/localTex/BibTex//: MANPATH=/global/usr/apps/texlive/2013/texmf/doc/man:/global/usr/share/man:/usr/share/man SSH_AGENT_PID=2819 HOSTNAME=svr.che.ufl.edu BSTINPUTS=.:/home/tladd/localTex/LaTex//: GPG_AGENT_INFO=/tmp/gpg-dFOyuV/S.gpg-agent:2819:1 GLADE_PIXMAP_PATH=: TERM=xterm SHELL=/bin/bash XDG_MENU_PREFIX=xfce- XDG_SESSION_COOKIE=71fff1a0e249855d5388855b00000016-1395782357.971651-1051300291 HISTSIZE=1000 TMPDIR=/tmp WINDOWID=50422136 OLDPWD=/home/tladd/tests/benchmarks/netbench/src PYTHONUSERBASE=/home/tladd/Enthought/Canopy_64bit/User USER=tladd LD_LIBRARY_PATH=/global/usr/lib LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36: GLADE_MODULE_PATH=: SSH_AUTH_SOCK=/tmp/gpg-ErYGot/S.gpg-agent.ssh SESSION_MANAGER=local/unix:@/tmp/.ICE-unix/2830,unix/unix:/tmp/.ICE-unix/2830 XDG_CONFIG_DIRS=/etc/xdg PATH=.:/home/tladd/Enthought/Canopy_64bit/User/bin:/global/usr/apps/texlive/2013/bin/x86_64-linux:/home/tladd/bin:/global/usr/bin:/usr/java/default/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin DESKTOP_SESSION=xfce MAIL=/var/spool/mail/tladd C_INCLUDE_PATH=/home/tladd/Enthought/Canopy_64bit/User/include PWD=/home/tladd/tests/benchmarks/netbench LANG=en_US MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles LOADEDMODULES= TEXMFCONFIG=/home/tladd/.texlive2013/texmf-config history=50 PS1=\h:\u(\W)> umask=027 SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass HISTCONTROL=ignoredups HYDRA_DEBUG=1 TEXINPUTS=.:/home/tladd/localTex/LaTex//: HOME=/home/tladd SHLVL=5 PYTHONPATH=/home/tladd/Enthought/Canopy_64bit/User/lib/python2.7/site_packages LOGNAME=tladd PRINTER=lj4000 CVS_RSH=ssh XDG_DATA_DIRS=/usr/local/share:/usr/share DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-HBJ4HMTTAQ,guid=52dbd52ea170467c23d5cf390000002c MODULESHOME=/usr/share/Modules LESSOPEN=|/usr/bin/lesspipe.sh %s TEXDIR=/global/usr/apps/texlive/2013 INFOPATH=/global/usr/apps/texlive/2013/texmf/doc/info TEXMFVAR=/home/tladd/.texlive2013/texmf-var WINDOWPATH=7 LIBGLADE_MODULE_PATH=: GLADE_CATALOG_PATH=: G_BROKEN_FILENAMES=1 XAUTHORITY=/home/tladd/.Xauthority COLORTERM=Terminal module=() { eval `/usr/bin/modulecmd bash $*` } _=/global/usr/bin/mpirun Hydra internal environment: --------------------------- GFORTRAN_UNBUFFERED_PRECONNECTED=y Proxy information: ********************* [1] proxy: svr (1 cores) Exec list: src/netbench (1 processes); [2] proxy: pc5 (1 cores) Exec list: src/netbench (1 processes); ================================================================================================== [mpiexec@svr.che.ufl.edu] Timeout set to -1 (-1 means infinite) [mpiexec@svr.che.ufl.edu] Got a control port string of svr.che.ufl.edu:44477 Proxy launch args: /global/usr/bin/hydra_pmi_proxy --control-port svr.che.ufl.edu:44477 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id Arguments being passed to proxy 0: --version 3.0.4 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME --hostname svr --global-core-map 0,1,2 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_17241_0 --pmi-process-mapping (vector,(0,2,1)) --ckpoint-num -1 --global-inherited-env 60 'BIBINPUTS=.:/home/tladd/localTex/BibTex//:' 'MANPATH=/global/usr/apps/texlive/2013/texmf/doc/man:/global/usr/share/man:/usr/share/man' 'SSH_AGENT_PID=2819' 'HOSTNAME=svr.che.ufl.edu' 'BSTINPUTS=.:/home/tladd/localTex/LaTex//:' 'GPG_AGENT_INFO=/tmp/gpg-dFOyuV/S.gpg-agent:2819:1' 'GLADE_PIXMAP_PATH=:' 'TERM=xterm' 'SHELL=/bin/bash' 'XDG_MENU_PREFIX=xfce-' 'XDG_SESSION_COOKIE=71fff1a0e249855d5388855b00000016-1395782357.971651-1051300291' 'HISTSIZE=1000' 'TMPDIR=/tmp' 'WINDOWID=50422136' 'OLDPWD=/home/tladd/tests/benchmarks/netbench/src' 'PYTHONUSERBASE=/home/tladd/Enthought/Canopy_64bit/User' 'USER=tladd' 'LD_LIBRARY_PATH=/global/usr/lib' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:' 'GLADE_MODULE_PATH=:' 'SSH_AUTH_SOCK=/tmp/gpg-ErYGot/S.gpg-agent.ssh' 'SESSION_MANAGER=local/unix:@/tmp/.ICE-unix/2830,unix/unix:/tmp/.ICE-unix/2830' 'XDG_CONFIG_DIRS=/etc/xdg' 'PATH=.:/home/tladd/Enthought/Canopy_64bit/User/bin:/global/usr/apps/texlive/2013/bin/x86_64-linux:/home/tladd/bin:/global/usr/bin:/usr/java/default/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin' 'DESKTOP_SESSION=xfce' 'MAIL=/var/spool/mail/tladd' 'C_INCLUDE_PATH=/home/tladd/Enthought/Canopy_64bit/User/include' 'PWD=/home/tladd/tests/benchmarks/netbench' 'LANG=en_US' 'MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles' 'LOADEDMODULES=' 'TEXMFCONFIG=/home/tladd/.texlive2013/texmf-config' 'history=50' 'PS1=\h:\u(\W)> ' 'umask=027' 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass' 'HISTCONTROL=ignoredups' 'HYDRA_DEBUG=1' 'TEXINPUTS=.:/home/tladd/localTex/LaTex//:' 'HOME=/home/tladd' 'SHLVL=5' 'PYTHONPATH=/home/tladd/Enthought/Canopy_64bit/User/lib/python2.7/site_packages' 'LOGNAME=tladd' 'PRINTER=lj4000' 'CVS_RSH=ssh' 'XDG_DATA_DIRS=/usr/local/share:/usr/share' 'DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-HBJ4HMTTAQ,guid=52dbd52ea170467c23d5cf390000002c' 'MODULESHOME=/usr/share/Modules' 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'TEXDIR=/global/usr/apps/texlive/2013' 'INFOPATH=/global/usr/apps/texlive/2013/texmf/doc/info' 'TEXMFVAR=/home/tladd/.texlive2013/texmf-var' 'WINDOWPATH=7' 'LIBGLADE_MODULE_PATH=:' 'GLADE_CATALOG_PATH=:' 'G_BROKEN_FILENAMES=1' 'XAUTHORITY=/home/tladd/.Xauthority' 'COLORTERM=Terminal' 'module=() { eval `/usr/bin/modulecmd bash $*` }' '_=/global/usr/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/tladd/tests/benchmarks/netbench --exec-args 3 src/netbench 1024 128 Arguments being passed to proxy 1: --version 3.0.4 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME --hostname pc5 --global-core-map 0,1,2 --pmi-id-map 0,1 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_17241_0 --pmi-process-mapping (vector,(0,2,1)) --ckpoint-num -1 --global-inherited-env 60 'BIBINPUTS=.:/home/tladd/localTex/BibTex//:' 'MANPATH=/global/usr/apps/texlive/2013/texmf/doc/man:/global/usr/share/man:/usr/share/man' 'SSH_AGENT_PID=2819' 'HOSTNAME=svr.che.ufl.edu' 'BSTINPUTS=.:/home/tladd/localTex/LaTex//:' 'GPG_AGENT_INFO=/tmp/gpg-dFOyuV/S.gpg-agent:2819:1' 'GLADE_PIXMAP_PATH=:' 'TERM=xterm' 'SHELL=/bin/bash' 'XDG_MENU_PREFIX=xfce-' 'XDG_SESSION_COOKIE=71fff1a0e249855d5388855b00000016-1395782357.971651-1051300291' 'HISTSIZE=1000' 'TMPDIR=/tmp' 'WINDOWID=50422136' 'OLDPWD=/home/tladd/tests/benchmarks/netbench/src' 'PYTHONUSERBASE=/home/tladd/Enthought/Canopy_64bit/User' 'USER=tladd' 'LD_LIBRARY_PATH=/global/usr/lib' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:' 'GLADE_MODULE_PATH=:' 'SSH_AUTH_SOCK=/tmp/gpg-ErYGot/S.gpg-agent.ssh' 'SESSION_MANAGER=local/unix:@/tmp/.ICE-unix/2830,unix/unix:/tmp/.ICE-unix/2830' 'XDG_CONFIG_DIRS=/etc/xdg' 'PATH=.:/home/tladd/Enthought/Canopy_64bit/User/bin:/global/usr/apps/texlive/2013/bin/x86_64-linux:/home/tladd/bin:/global/usr/bin:/usr/java/default/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin' 'DESKTOP_SESSION=xfce' 'MAIL=/var/spool/mail/tladd' 'C_INCLUDE_PATH=/home/tladd/Enthought/Canopy_64bit/User/include' 'PWD=/home/tladd/tests/benchmarks/netbench' 'LANG=en_US' 'MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles' 'LOADEDMODULES=' 'TEXMFCONFIG=/home/tladd/.texlive2013/texmf-config' 'history=50' 'PS1=\h:\u(\W)> ' 'umask=027' 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass' 'HISTCONTROL=ignoredups' 'HYDRA_DEBUG=1' 'TEXINPUTS=.:/home/tladd/localTex/LaTex//:' 'HOME=/home/tladd' 'SHLVL=5' 'PYTHONPATH=/home/tladd/Enthought/Canopy_64bit/User/lib/python2.7/site_packages' 'LOGNAME=tladd' 'PRINTER=lj4000' 'CVS_RSH=ssh' 'XDG_DATA_DIRS=/usr/local/share:/usr/share' 'DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-HBJ4HMTTAQ,guid=52dbd52ea170467c23d5cf390000002c' 'MODULESHOME=/usr/share/Modules' 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'TEXDIR=/global/usr/apps/texlive/2013' 'INFOPATH=/global/usr/apps/texlive/2013/texmf/doc/info' 'TEXMFVAR=/home/tladd/.texlive2013/texmf-var' 'WINDOWPATH=7' 'LIBGLADE_MODULE_PATH=:' 'GLADE_CATALOG_PATH=:' 'G_BROKEN_FILENAMES=1' 'XAUTHORITY=/home/tladd/.Xauthority' 'COLORTERM=Terminal' 'module=() { eval `/usr/bin/modulecmd bash $*` }' '_=/global/usr/bin/mpirun' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/tladd/tests/benchmarks/netbench --exec-args 3 src/netbench 1024 128 [mpiexec@svr.che.ufl.edu] Launch arguments: /global/usr/bin/hydra_pmi_proxy --control-port svr.che.ufl.edu:44477 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 0 [mpiexec@svr.che.ufl.edu] Launch arguments: /usr/bin/ssh -x pc5 "/global/usr/bin/hydra_pmi_proxy" --control-port svr.che.ufl.edu:44477 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 1 [proxy:0:0@svr.che.ufl.edu] got pmi command (from 0): init pmi_version=1 pmi_subversion=1 [proxy:0:0@svr.che.ufl.edu] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@svr.che.ufl.edu] got pmi command (from 0): get_maxes [proxy:0:0@svr.che.ufl.edu] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@svr.che.ufl.edu] got pmi command (from 0): get_appnum [proxy:0:0@svr.che.ufl.edu] PMI response: cmd=appnum appnum=0 [proxy:0:0@svr.che.ufl.edu] got pmi command (from 0): get_my_kvsname [proxy:0:0@svr.che.ufl.edu] PMI response: cmd=my_kvsname kvsname=kvs_17241_0 [proxy:0:0@svr.che.ufl.edu] got pmi command (from 0): get_my_kvsname [proxy:0:0@svr.che.ufl.edu] PMI response: cmd=my_kvsname kvsname=kvs_17241_0 [proxy:0:0@svr.che.ufl.edu] got pmi command (from 0): get kvsname=kvs_17241_0 key=PMI_process_mapping [proxy:0:0@svr.che.ufl.edu] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@svr.che.ufl.edu] got pmi command (from 0): barrier_in [proxy:0:0@svr.che.ufl.edu] forwarding command (cmd=barrier_in) upstream [mpiexec@svr.che.ufl.edu] [pgid: 0] got PMI command: cmd=barrier_in [proxy:0:1@pc5] got pmi command (from 4): init pmi_version=1 pmi_subversion=1 [proxy:0:1@pc5] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@pc5] got pmi command (from 4): get_maxes [proxy:0:1@pc5] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@pc5] got pmi command (from 4): get_appnum [proxy:0:1@pc5] PMI response: cmd=appnum appnum=0 [proxy:0:1@pc5] got pmi command (from 4): get_my_kvsname [proxy:0:1@pc5] PMI response: cmd=my_kvsname kvsname=kvs_17241_0 [proxy:0:1@pc5] got pmi command (from 4): get_my_kvsname [proxy:0:1@pc5] PMI response: cmd=my_kvsname kvsname=kvs_17241_0 [proxy:0:1@pc5] got pmi command (from 4): get kvsname=kvs_17241_0 key=PMI_process_mapping [proxy:0:1@pc5] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [mpiexec@svr.che.ufl.edu] [pgid: 0] got PMI command: cmd=barrier_in [mpiexec@svr.che.ufl.edu] PMI response to fd 6 pid 4: cmd=barrier_out [mpiexec@svr.che.ufl.edu] PMI response to fd 7 pid 4: cmd=barrier_out [proxy:0:1@pc5] got pmi command (from 4): barrier_in [proxy:0:1@pc5] forwarding command (cmd=barrier_in) upstream [proxy:0:0@svr.che.ufl.edu] PMI response: cmd=barrier_out [proxy:0:1@pc5] PMI response: cmd=barrier_out [proxy:0:0@svr.che.ufl.edu] got pmi command (from 0): put kvsname=kvs_17241_0 key=P0-businesscard value=description#svr$port#56372$ifname#127.0.0.1$ [proxy:0:0@svr.che.ufl.edu] cached command: P0-businesscard=description#svr$port#56372$ifname#127.0.0.1$ [proxy:0:0@svr.che.ufl.edu] PMI response: cmd=put_result rc=0 msg=success [proxy:0:0@svr.che.ufl.edu] got pmi command (from 0): barrier_in [proxy:0:0@svr.che.ufl.edu] flushing 1 put command(s) out [proxy:0:0@svr.che.ufl.edu] forwarding command (cmd=put P0-businesscard=description#svr$port#56372$ifname#127.0.0.1$) upstream [proxy:0:0@svr.che.ufl.edu] forwarding command (cmd=barrier_in) upstream [mpiexec@svr.che.ufl.edu] [pgid: 0] got PMI command: cmd=put P0-businesscard=description#svr$port#56372$ifname#127.0.0.1$ [mpiexec@svr.che.ufl.edu] [pgid: 0] got PMI command: cmd=barrier_in [mpiexec@svr.che.ufl.edu] [pgid: 0] got PMI command: cmd=put P1-businesscard=description#pc5$port#39666$ifname#127.0.0.1$ [mpiexec@svr.che.ufl.edu] [pgid: 0] got PMI command: cmd=barrier_in [mpiexec@svr.che.ufl.edu] PMI response to fd 6 pid 4: cmd=keyval_cache P0-businesscard=description#svr$port#56372$ifname#127.0.0.1$ P1-businesscard=description#pc5$port#39666$ifname#127.0.0.1$ [mpiexec@svr.che.ufl.edu] PMI response to fd 7 pid 4: cmd=keyval_cache P0-businesscard=description#svr$port#56372$ifname#127.0.0.1$ P1-businesscard=description#pc5$port#39666$ifname#127.0.0.1$ [mpiexec@svr.che.ufl.edu] PMI response to fd 6 pid 4: cmd=barrier_out [mpiexec@svr.che.ufl.edu] PMI response to fd 7 pid 4: cmd=barrier_out [proxy:0:1@pc5] got pmi command (from 4): put kvsname=kvs_17241_0 key=P1-businesscard value=description#pc5$port#39666$ifname#127.0.0.1$ [proxy:0:1@pc5] cached command: P1-businesscard=description#pc5$port#39666$ifname#127.0.0.1$ [proxy:0:1@pc5] PMI response: cmd=put_result rc=0 msg=success [proxy:0:1@pc5] got pmi command (from 4): barrier_in [proxy:0:1@pc5] flushing 1 put command(s) out [proxy:0:1@pc5] forwarding command (cmd=put P1-businesscard=description#pc5$port#39666$ifname#127.0.0.1$) upstream [proxy:0:1@pc5] forwarding command (cmd=barrier_in) upstream [proxy:0:0@svr.che.ufl.edu] PMI response: cmd=barrier_out [proxy:0:0@svr.che.ufl.edu] got pmi command (from 0): get kvsname=kvs_17241_0 key=P1-businesscard [proxy:0:0@svr.che.ufl.edu] PMI response: cmd=get_result rc=0 msg=success value=description#pc5$port#39666$ifname#127.0.0.1$ Fatal error in PMPI_Barrier: A process has failed, error stack: PMPI_Barrier(426).........: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(333)....: Failure during collective MPIR_Barrier_impl(315)....: MPIR_Barrier_intra(83)....: dequeue_and_set_error(888): Communication error with rank 1 [proxy:0:1@pc5] PMI response: cmd=barrier_out =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:1@pc5] got pmi command (from 4): get kvsname=kvs_17241_0 key=P0-businesscard [proxy:0:1@pc5] PMI response: cmd=get_result rc=0 msg=success value=description#svr$port#56372$ifname#127.0.0.1$ [proxy:0:1@pc5] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed [proxy:0:1@pc5] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status [proxy:0:1@pc5] main (./pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@svr.che.ufl.edu] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@svr.che.ufl.edu] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@svr.che.ufl.edu] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for completion [mpiexec@svr.che.ufl.edu] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion