[mpich-discuss] MPICH process frozen without any error message nor warning

27 Feb 2018

      Hi everyone and thanks for your attention.

I am trying to run a meteorological simulation with RAMS model (
http://vandenheever.atmos.colostate.edu/vdhpage/rams.php) in a new cluster
with CentOs7 in all nodes. We have other applications running fine in the
cluster, sending processes to each node,... everything seemed to run fine.

But the RAMS model freezes at its first stage on the parallel run. RAMS
developers recommend using mpich2-1.4.1 as they have successfully tested.
If I try to run a parallel simulation on the master node it runs fine and
starts the required number of processes. If I try to use the other nodes
the simulation freezes, do not stop with an error message; RAMS model
processes appear in all used nodes and in the master node but any output is
created and the usual status messages from RAMS do not appear on the screen.

The command line to run the model is

[paco@Llamp RUN]$ time ../misc/mpich2-1.4.1/bin/mpirun -verbose
-machinefile mpd.hosts -n 20 ./rams-6.2.03 -f RAMSIN

The only warning I can find in the log file (see below) is
"we don't understand the response get_result; forwarding downstream"
 but as I'm not an expert with mpich I can't figure out the reason of
the freezing.

Thanks in advance for your help

Log file:
[paco@l4 paco]$ ../mpiexec -verbose -outfile-pattern mpi.log
-errfile-pattern mpierror.log -machinefile machs -np 40 ../rams-6.2.06 -f
RAMSIN
host: l3
host: l4

==================================================================================================
mpiexec options:
----------------
  Base path: /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco/../
  Launcher: (null)
  Debug level: 1
  Enable X: -1

  Global environment:
  -------------------
    XDG_SESSION_ID=6511
    HOSTNAME=l4
    SELINUX_ROLE_REQUESTED=
    TERM=xterm-256color
    SHELL=/bin/bash
    HISTSIZE=1000
    SSH_CLIENT=193.144.100.32 49462 22
    SELINUX_USE_CURRENT_RANGE=
    SSH_TTY=/dev/pts/0
    USER=paco

LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
    MAIL=/var/spool/mail/paco

PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/paco/.local/bin:/home/paco/bin
    PWD=/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco
    LANG=es_ES.UTF-8
    MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles
    LOADEDMODULES=
    SELINUX_LEVEL_REQUESTED=
    HISTCONTROL=ignoredups
    SHLVL=1
    HOME=/home/paco
    LOGNAME=paco
    SSH_CONNECTION=193.144.100.32 49462 193.144.100.46 22
    MODULESHOME=/usr/share/Modules
    LESSOPEN=||/usr/bin/lesspipe.sh %s
    XDG_RUNTIME_DIR=/run/user/1003
    BASH_FUNC_module()=() {  eval `/usr/bin/modulecmd bash $*`
}
    _=../mpiexec
    OLDPWD=/home/paco

  Hydra internal environment:
  ---------------------------
    GFORTRAN_UNBUFFERED_PRECONNECTED=y

    Proxy information:
    *********************
      [1] proxy: l3 (1 cores)
      Exec list: ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1
processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes);
../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06
(1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes);
../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06
(1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes);
../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06
(1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes);
../rams-6.2.06 (1 processes);

      [2] proxy: l4 (1 cores)
      Exec list: ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1
processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes);
../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06
(1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes);
../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06
(1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes);
../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06
(1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes);
../rams-6.2.06 (1 processes);

==================================================================================================

[mpiexec@l4] Timeout set to -1 (-1 means infinite)
[mpiexec@l4] Got a control port string of l4:44483

Proxy launch args:
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco/../hydra_pmi_proxy
--control-port l4:44483 --debug --rmk user --launcher ssh --demux poll
--pgid 0 --retries 10 --proxy-id

[mpiexec@l4] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
Arguments being passed to proxy 0:
--version 1.4.1 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME --hostname l3
--global-core-map 0,1,1 --filler-process-map 0,1,1 --global-process-count
40 --auto-cleanup 1 --pmi-rank -1 --pmi-kvsname kvs_32745_0
--pmi-process-mapping (vector,(0,2,1)) --ckpoint-num -1
--global-inherited-env 29 'XDG_SESSION_ID=6511' 'HOSTNAME=l4'
'SELINUX_ROLE_REQUESTED=' 'TERM=xterm-256color' 'SHELL=/bin/bash'
'HISTSIZE=1000' 'SSH_CLIENT=193.144.100.32 49462 22'
'SELINUX_USE_CURRENT_RANGE=' 'SSH_TTY=/dev/pts/0' 'USER=paco'
'LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:'
'MAIL=/var/spool/mail/paco'
'PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/paco/.local/bin:/home/paco/bin'
'PWD=/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco'
'LANG=es_ES.UTF-8'
'MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles'
'LOADEDMODULES=' 'SELINUX_LEVEL_REQUESTED=' 'HISTCONTROL=ignoredups'
'SHLVL=1' 'HOME=/home/paco' 'LOGNAME=paco' 'SSH_CONNECTION=193.144.100.32
49462 193.144.100.46 22' 'MODULESHOME=/usr/share/Modules'
'LESSOPEN=||/usr/bin/lesspipe.sh %s' 'XDG_RUNTIME_DIR=/run/user/1003'
'BASH_FUNC_module()=() {  eval `/usr/bin/modulecmd bash $*`
}' '_=../mpiexec' 'OLDPWD=/home/paco' --global-user-env 0
--global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y'
--proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN

[mpiexec@l4] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
Arguments being passed to proxy 1:
--version 1.4.1 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME --hostname l4
--global-core-map 1,1,0 --filler-process-map 1,1,0 --global-process-count
40 --auto-cleanup 1 --pmi-rank -1 --pmi-kvsname kvs_32745_0
--pmi-process-mapping (vector,(0,2,1)) --ckpoint-num -1
--global-inherited-env 29 'XDG_SESSION_ID=6511' 'HOSTNAME=l4'
'SELINUX_ROLE_REQUESTED=' 'TERM=xterm-256color' 'SHELL=/bin/bash'
'HISTSIZE=1000' 'SSH_CLIENT=193.144.100.32 49462 22'
'SELINUX_USE_CURRENT_RANGE=' 'SSH_TTY=/dev/pts/0' 'USER=paco'
'LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:'
'MAIL=/var/spool/mail/paco'
'PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/paco/.local/bin:/home/paco/bin'
'PWD=/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco'
'LANG=es_ES.UTF-8'
'MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles'
'LOADEDMODULES=' 'SELINUX_LEVEL_REQUESTED=' 'HISTCONTROL=ignoredups'
'SHLVL=1' 'HOME=/home/paco' 'LOGNAME=paco' 'SSH_CONNECTION=193.144.100.32
49462 193.144.100.46 22' 'MODULESHOME=/usr/share/Modules'
'LESSOPEN=||/usr/bin/lesspipe.sh %s' 'XDG_RUNTIME_DIR=/run/user/1003'
'BASH_FUNC_module()=() {  eval `/usr/bin/modulecmd bash $*`
}' '_=../mpiexec' 'OLDPWD=/home/paco' --global-user-env 0
--global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y'
--proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1
--exec-local-env 0 --exec-wdir
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3
../rams-6.2.06 -f RAMSIN

[mpiexec@l4] Launch arguments: /usr/bin/ssh -x l3
"/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco/../hydra_pmi_proxy"
--control-port l4:44483 --debug --rmk user --launcher ssh --demux poll
--pgid 0 --retries 10 --proxy-id 0
[mpiexec@l4] Launch arguments:
/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco/../hydra_pmi_proxy
--control-port l4:44483 --debug --rmk user --launcher ssh --demux poll
--pgid 0 --retries 10 --proxy-id 1
[proxy:0:1@l4] got pmi command (from 0): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:1@l4] got pmi command (from 6): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:1@l4] got pmi command (from 11): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:1@l4] got pmi command (from 16): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:1@l4] got pmi command (from 19): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:1@l4] got pmi command (from 25): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:1@l4] got pmi command (from 31): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:1@l4] got pmi command (from 37): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:1@l4] got pmi command (from 0): get_maxes

[proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:1@l4] got pmi command (from 6): get_maxes

[proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:1@l4] got pmi command (from 11): get_maxes

[proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:1@l4] got pmi command (from 16): get_maxes

[proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:1@l4] got pmi command (from 19): get_maxes

[proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:1@l4] got pmi command (from 25): get_maxes

[proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:1@l4] got pmi command (from 46): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:1@l4] got pmi command (from 49): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:1@l4] got pmi command (from 0): get_appnum

[proxy:0:1@l4] PMI response: cmd=appnum appnum=0
[proxy:0:1@l4] got pmi command (from 6): get_appnum

[proxy:0:1@l4] PMI response: cmd=appnum appnum=0
[proxy:0:1@l4] got pmi command (from 11): get_appnum

[proxy:0:1@l4] PMI response: cmd=appnum appnum=0
[proxy:0:1@l4] got pmi command (from 16): get_appnum

[proxy:0:1@l4] PMI response: cmd=appnum appnum=0
[proxy:0:1@l4] got pmi command (from 19): get_appnum

[proxy:0:1@l4] PMI response: cmd=appnum appnum=0
[proxy:0:1@l4] got pmi command (from 25): get_appnum

[proxy:0:1@l4] PMI response: cmd=appnum appnum=0
[proxy:0:1@l4] got pmi command (from 31): get_maxes

[proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:1@l4] got pmi command (from 37): get_maxes

[proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:1@l4] got pmi command (from 0): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 6): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 11): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 16): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 19): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 25): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 31): get_appnum

[proxy:0:1@l4] PMI response: cmd=appnum appnum=0
[proxy:0:1@l4] got pmi command (from 46): get_maxes

[proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:1@l4] got pmi command (from 49): get_maxes

[proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:1@l4] got pmi command (from 0): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 6): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 11): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 19): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 25): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 31): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 37): get_appnum

[proxy:0:1@l4] PMI response: cmd=appnum appnum=0
[proxy:0:1@l4] got pmi command (from 46): get_appnum

[proxy:0:1@l4] PMI response: cmd=appnum appnum=0
[proxy:0:1@l4] got pmi command (from 0): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:1@l4] got pmi command (from 6): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:1@l4] got pmi command (from 11): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:1@l4] got pmi command (from 19): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:1@l4] got pmi command (from 25): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:1@l4] got pmi command (from 37): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 49): get_appnum

[proxy:0:1@l4] PMI response: cmd=appnum appnum=0
[proxy:0:1@l4] got pmi command (from 6): barrier_in

[proxy:0:1@l4] got pmi command (from 11): barrier_in

[proxy:0:1@l4] got pmi command (from 19): barrier_in

[proxy:0:1@l4] got pmi command (from 25): barrier_in

[proxy:0:1@l4] got pmi command (from 31): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 46): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 31): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:1@l4] got pmi command (from 37): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 49): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 31): barrier_in

[proxy:0:1@l4] got pmi command (from 37): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:1@l4] got pmi command (from 46): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 52): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:1@l4] got pmi command (from 37): barrier_in

[proxy:0:1@l4] got pmi command (from 46): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:1@l4] got pmi command (from 49): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 46): barrier_in

[proxy:0:1@l4] got pmi command (from 52): get_maxes

[proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:1@l4] got pmi command (from 55): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:1@l4] got pmi command (from 49): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:1@l4] got pmi command (from 52): get_appnum

[proxy:0:1@l4] PMI response: cmd=appnum appnum=0
[proxy:0:1@l4] got pmi command (from 52): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 55): get_maxes

[proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:1@l4] got pmi command (from 49): barrier_in

[proxy:0:1@l4] got pmi command (from 52): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 55): get_appnum

[proxy:0:1@l4] PMI response: cmd=appnum appnum=0
[proxy:0:1@l4] got pmi command (from 16): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 52): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:1@l4] got pmi command (from 58): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:1@l4] got pmi command (from 16): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:1@l4] got pmi command (from 52): barrier_in

[proxy:0:1@l4] got pmi command (from 55): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 16): barrier_in

[proxy:0:1@l4] got pmi command (from 58): get_maxes

[proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:1@l4] got pmi command (from 55): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 0): put
kvsname=kvs_32745_0 key=sharedFilename[1]
value=/dev/shm/mpich_shar_tmp42EbvZ
[proxy:0:1@l4] we don't understand this command put; forwarding upstream
[proxy:0:1@l4] got pmi command (from 58): get_appnum

[mpiexec@l4] [pgid: 0] got PMI command: cmd=put kvsname=kvs_32745_0
key=sharedFilename[1] value=/dev/shm/mpich_shar_tmp42EbvZ
[mpiexec@l4] PMI response to fd 6 pid 0: cmd=put_result rc=0 msg=success
[proxy:0:1@l4] PMI response: cmd=appnum appnum=0
[proxy:0:1@l4] got pmi command (from 55): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:1@l4] got pmi command (from 58): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] we don't understand the response put_result; forwarding
downstream
[proxy:0:1@l4] got pmi command (from 55): barrier_in

[proxy:0:1@l4] got pmi command (from 58): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 64): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:1@l4] got pmi command (from 0): barrier_in

[proxy:0:1@l4] got pmi command (from 58): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:1@l4] got pmi command (from 64): get_maxes

[proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:1@l4] got pmi command (from 58): barrier_in

[proxy:0:1@l4] got pmi command (from 64): get_appnum

[proxy:0:1@l4] PMI response: cmd=appnum appnum=0
[proxy:0:1@l4] got pmi command (from 64): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 64): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 64): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:1@l4] got pmi command (from 64): barrier_in

[proxy:0:1@l4] got pmi command (from 28): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:1@l4] got pmi command (from 28): get_maxes

[proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:1@l4] got pmi command (from 28): get_appnum

[proxy:0:1@l4] PMI response: cmd=appnum appnum=0
[proxy:0:1@l4] got pmi command (from 28): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 28): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 28): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:1@l4] got pmi command (from 28): barrier_in

[proxy:0:1@l4] got pmi command (from 43): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:1@l4] got pmi command (from 43): get_maxes

[proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:1@l4] got pmi command (from 43): get_appnum

[proxy:0:1@l4] PMI response: cmd=appnum appnum=0
[proxy:0:1@l4] got pmi command (from 43): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 43): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 43): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:1@l4] got pmi command (from 43): barrier_in

[proxy:0:1@l4] got pmi command (from 40): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:1@l4] got pmi command (from 40): get_maxes

[proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:1@l4] got pmi command (from 40): get_appnum

[proxy:0:1@l4] PMI response: cmd=appnum appnum=0
[proxy:0:1@l4] got pmi command (from 40): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 40): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 40): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:1@l4] got pmi command (from 40): barrier_in

[proxy:0:1@l4] got pmi command (from 22): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:1@l4] got pmi command (from 22): get_maxes

[proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:1@l4] got pmi command (from 22): get_appnum

[proxy:0:1@l4] PMI response: cmd=appnum appnum=0
[proxy:0:1@l4] got pmi command (from 22): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 22): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 22): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:1@l4] got pmi command (from 22): barrier_in

[proxy:0:1@l4] got pmi command (from 34): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:1@l4] got pmi command (from 61): init
pmi_version=1 pmi_subversion=1
[proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:1@l4] got pmi command (from 34): get_maxes

[proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:1@l4] got pmi command (from 61): get_maxes

[proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:1@l4] got pmi command (from 34): get_appnum

[proxy:0:1@l4] PMI response: cmd=appnum appnum=0
[proxy:0:1@l4] got pmi command (from 61): get_appnum

[proxy:0:1@l4] PMI response: cmd=appnum appnum=0
[proxy:0:1@l4] got pmi command (from 34): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 61): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 34): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 61): get_my_kvsname

[proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:1@l4] got pmi command (from 34): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:1@l4] got pmi command (from 61): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:1@l4] got pmi command (from 34): barrier_in

[proxy:0:1@l4] got pmi command (from 61): barrier_in

[mpiexec@l4] [pgid: 0] got PMI command: cmd=barrier_in
[proxy:0:1@l4] forwarding command (cmd=barrier_in) upstream
[proxy:0:0@l3] got pmi command (from 4): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0@l3] got pmi command (from 5): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0@l3] got pmi command (from 6): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0@l3] got pmi command (from 11): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0@l3] got pmi command (from 14): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0@l3] got pmi command (from 17): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0@l3] got pmi command (from 20): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0@l3] got pmi command (from 23): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0@l3] got pmi command (from 26): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0@l3] got pmi command (from 29): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0@l3] got pmi command (from 32): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0@l3] got pmi command (from 4): get_maxes

[proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:0@l3] got pmi command (from 5): get_maxes

[proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:0@l3] got pmi command (from 6): get_maxes

[proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:0@l3] got pmi command (from 11): get_maxes

[proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:0@l3] got pmi command (from 14): get_maxes

[proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:0@l3] got pmi command (from 17): get_maxes

[proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:0@l3] got pmi command (from 20): get_maxes

[proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:0@l3] got pmi command (from 23): get_maxes

[proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:0@l3] got pmi command (from 26): get_maxes

[proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:0@l3] got pmi command (from 35): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0@l3] got pmi command (from 38): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0@l3] got pmi command (from 4): get_appnum

[proxy:0:0@l3] PMI response: cmd=appnum appnum=0
[proxy:0:0@l3] got pmi command (from 5): get_appnum

[proxy:0:0@l3] PMI response: cmd=appnum appnum=0
[proxy:0:0@l3] got pmi command (from 6): get_appnum

[proxy:0:0@l3] PMI response: cmd=appnum appnum=0
[proxy:0:0@l3] got pmi command (from 11): get_appnum

[proxy:0:0@l3] PMI response: cmd=appnum appnum=0
[proxy:0:0@l3] got pmi command (from 14): get_appnum

[proxy:0:0@l3] PMI response: cmd=appnum appnum=0
[proxy:0:0@l3] got pmi command (from 17): get_appnum

[proxy:0:0@l3] PMI response: cmd=appnum appnum=0
[proxy:0:0@l3] got pmi command (from 20): get_appnum

[proxy:0:0@l3] PMI response: cmd=appnum appnum=0
[proxy:0:0@l3] got pmi command (from 23): get_appnum

[proxy:0:0@l3] PMI response: cmd=appnum appnum=0
[proxy:0:0@l3] got pmi command (from 26): get_appnum

[proxy:0:0@l3] PMI response: cmd=appnum appnum=0
[proxy:0:0@l3] got pmi command (from 29): get_maxes

[proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:0@l3] got pmi command (from 32): get_maxes

[proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:0@l3] got pmi command (from 4): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 5): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 6): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 11): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 14): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 17): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 20): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 23): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 26): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 29): get_appnum

[proxy:0:0@l3] PMI response: cmd=appnum appnum=0
[proxy:0:0@l3] got pmi command (from 35): get_maxes

[proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:0@l3] got pmi command (from 38): get_maxes

[proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:0@l3] got pmi command (from 41): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0@l3] got pmi command (from 4): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 5): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 6): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 11): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 14): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 17): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 20): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 23): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 26): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 29): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 32): get_appnum

[proxy:0:0@l3] PMI response: cmd=appnum appnum=0
[proxy:0:0@l3] got pmi command (from 35): get_appnum

[proxy:0:0@l3] PMI response: cmd=appnum appnum=0
[proxy:0:0@l3] got pmi command (from 38): get_appnum

[proxy:0:0@l3] PMI response: cmd=appnum appnum=0
[proxy:0:0@l3] got pmi command (from 4): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:0@l3] got pmi command (from 5): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:0@l3] got pmi command (from 6): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:0@l3] got pmi command (from 11): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:0@l3] got pmi command (from 14): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:0@l3] got pmi command (from 17): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:0@l3] got pmi command (from 20): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:0@l3] got pmi command (from 23): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:0@l3] got pmi command (from 26): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:0@l3] got pmi command (from 32): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 35): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 41): get_maxes

[proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:0@l3] got pmi command (from 5): barrier_in

[proxy:0:0@l3] got pmi command (from 6): barrier_in

[proxy:0:0@l3] got pmi command (from 11): barrier_in

[proxy:0:0@l3] got pmi command (from 14): barrier_in

[proxy:0:0@l3] got pmi command (from 17): barrier_in

[proxy:0:0@l3] got pmi command (from 20): barrier_in

[proxy:0:0@l3] got pmi command (from 23): barrier_in

[proxy:0:0@l3] got pmi command (from 26): barrier_in

[proxy:0:0@l3] got pmi command (from 29): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 38): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 29): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:0@l3] got pmi command (from 32): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 35): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 41): get_appnum

[proxy:0:0@l3] PMI response: cmd=appnum appnum=0
[proxy:0:0@l3] got pmi command (from 29): barrier_in

[proxy:0:0@l3] got pmi command (from 32): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:0@l3] got pmi command (from 35): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:0@l3] got pmi command (from 38): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 32): barrier_in

[proxy:0:0@l3] got pmi command (from 41): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 35): barrier_in

[proxy:0:0@l3] got pmi command (from 38): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:0@l3] got pmi command (from 38): barrier_in

[proxy:0:0@l3] got pmi command (from 41): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 41): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:0@l3] got pmi command (from 41): barrier_in

[mpiexec@l4] [pgid: 0] got PMI command: cmd=put kvsname=kvs_32745_0
key=sharedFilename[0] value=/dev/shm/mpich_shar_tmphAojti
[mpiexec@l4] PMI response to fd 9 pid 4: cmd=put_result rc=0 msg=success
[proxy:0:0@l3] got pmi command (from 4): put
kvsname=kvs_32745_0 key=sharedFilename[0]
value=/dev/shm/mpich_shar_tmphAojti
[proxy:0:0@l3] we don't understand this command put; forwarding upstream
[proxy:0:0@l3] we don't understand the response put_result; forwarding
downstream
[proxy:0:0@l3] got pmi command (from 4): barrier_in

[proxy:0:0@l3] got pmi command (from 56): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0@l3] got pmi command (from 50): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0@l3] got pmi command (from 56): get_maxes

[proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:0@l3] got pmi command (from 59): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0@l3] got pmi command (from 56): get_appnum

[proxy:0:0@l3] PMI response: cmd=appnum appnum=0
[proxy:0:0@l3] got pmi command (from 50): get_maxes

[proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:0@l3] got pmi command (from 56): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 59): get_maxes

[proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:0@l3] got pmi command (from 50): get_appnum

[proxy:0:0@l3] PMI response: cmd=appnum appnum=0
[proxy:0:0@l3] got pmi command (from 59): get_appnum

[proxy:0:0@l3] PMI response: cmd=appnum appnum=0
[proxy:0:0@l3] got pmi command (from 50): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 56): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 59): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 56): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:0@l3] got pmi command (from 50): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 56): barrier_in

[proxy:0:0@l3] got pmi command (from 59): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 50): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:0@l3] got pmi command (from 59): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:0@l3] got pmi command (from 50): barrier_in

[proxy:0:0@l3] got pmi command (from 59): barrier_in

[proxy:0:0@l3] got pmi command (from 44): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0@l3] got pmi command (from 53): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0@l3] got pmi command (from 44): get_maxes

[proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:0@l3] got pmi command (from 53): get_maxes

[proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:0@l3] got pmi command (from 44): get_appnum

[proxy:0:0@l3] PMI response: cmd=appnum appnum=0
[proxy:0:0@l3] got pmi command (from 53): get_appnum

[proxy:0:0@l3] PMI response: cmd=appnum appnum=0
[proxy:0:0@l3] got pmi command (from 44): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 53): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 44): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 53): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 44): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:0@l3] got pmi command (from 53): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:0@l3] got pmi command (from 44): barrier_in

[proxy:0:0@l3] got pmi command (from 53): barrier_in

[proxy:0:0@l3] got pmi command (from 47): init
pmi_version=1 pmi_subversion=1
[proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1
pmi_subversion=1 rc=0
[proxy:0:0@l3] got pmi command (from 47): get_maxes

[proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
vallen_max=1024
[proxy:0:0@l3] got pmi command (from 47): get_appnum

[proxy:0:0@l3] PMI response: cmd=appnum appnum=0
[proxy:0:0@l3] got pmi command (from 47): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 47): get_my_kvsname

[proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0
[proxy:0:0@l3] got pmi command (from 47): get
kvsname=kvs_32745_0 key=PMI_process_mapping
[proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success
value=(vector,(0,2,1))
[proxy:0:0@l3] got pmi command (from 47): barrier_in

[proxy:0:0@l3] forwarding command (cmd=barrier_in) upstream
[mpiexec@l4] [pgid: 0] got PMI command: cmd=barrier_in
[mpiexec@l4] PMI response to fd 9 pid 47: cmd=barrier_out
[mpiexec@l4] PMI response to fd 6 pid 47: cmd=barrier_out
[proxy:0:1@l4] PMI response: cmd=barrier_out
[proxy:0:1@l4] PMI response: cmd=barrier_out
[proxy:0:1@l4] PMI response: cmd=barrier_out
[proxy:0:1@l4] PMI response: cmd=barrier_out
[proxy:0:1@l4] PMI response: cmd=barrier_out
[proxy:0:1@l4] PMI response: cmd=barrier_out
[proxy:0:1@l4] PMI response: cmd=barrier_out
[proxy:0:1@l4] PMI response: cmd=barrier_out
[proxy:0:1@l4] PMI response: cmd=barrier_out
[proxy:0:1@l4] PMI response: cmd=barrier_out
[proxy:0:1@l4] PMI response: cmd=barrier_out
[proxy:0:1@l4] PMI response: cmd=barrier_out
[proxy:0:1@l4] PMI response: cmd=barrier_out
[proxy:0:1@l4] PMI response: cmd=barrier_out
[proxy:0:1@l4] PMI response: cmd=barrier_out
[proxy:0:1@l4] PMI response: cmd=barrier_out
[proxy:0:1@l4] PMI response: cmd=barrier_out
[proxy:0:1@l4] PMI response: cmd=barrier_out
[proxy:0:1@l4] PMI response: cmd=barrier_out
[mpiexec@l4] [pgid: 0] got PMI command: cmd=get kvsname=kvs_32745_0
key=sharedFilename[0]
[mpiexec@l4] PMI response to fd 9 pid 5: cmd=get_result rc=0 msg=success
value=/dev/shm/mpich_shar_tmphAojti
[proxy:0:0@l3] PMI response: cmd=barrier_out
[proxy:0:0@l3] PMI response: cmd=barrier_out
[proxy:0:0@l3] PMI response: cmd=barrier_out
[proxy:0:0@l3] PMI response: cmd=barrier_out
[proxy:0:0@l3] PMI response: cmd=barrier_out
[proxy:0:0@l3] PMI response: cmd=barrier_out
[proxy:0:0@l3] PMI response: cmd=barrier_out
[proxy:0:0@l3] PMI response: cmd=barrier_out
[proxy:0:0@l3] PMI response: cmd=barrier_out
[proxy:0:0@l3] PMI response: cmd=barrier_out
[proxy:0:0@l3] PMI response: cmd=barrier_out
[proxy:0:0@l3] PMI response: cmd=barrier_out
[proxy:0:0@l3] PMI response: cmd=barrier_out
[proxy:0:0@l3] PMI response: cmd=barrier_out
[proxy:0:0@l3] PMI response: cmd=barrier_out
[proxy:0:0@l3] PMI response: cmd=barrier_out
[proxy:0:0@l3] PMI response: cmd=barrier_out
[proxy:0:0@l3] PMI response: cmd=barrier_out
[proxy:0:0@l3] PMI response: cmd=barrier_out
[proxy:0:0@l3] PMI response: cmd=barrier_out
[proxy:0:0@l3] got pmi command (from 5): get
kvsname=kvs_32745_0 key=sharedFilename[0]
[proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[0]) upstream
[proxy:0:0@l3] got pmi command (from 6): get
kvsname=kvs_32745_0 key=sharedFilename[0]
[proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[0]) upstream
[proxy:0:0@l3] got pmi command (from 11): get
kvsname=kvs_32745_0 key=sharedFilename[0]
[proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[0]) upstream
[proxy:0:0@l3] got pmi command (from 14): get
kvsname=kvs_32745_0 key=sharedFilename[0]
[proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[0]) upstream
[proxy:0:0@l3] got pmi command (from 17): get
kvsname=kvs_32745_0 key=sharedFilename[0]
[proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[0]) upstream
[proxy:0:0@l3] got pmi command (from 20): get
kvsname=kvs_32745_0 key=sharedFilename[0]
[proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[0]) upstream
[proxy:0:0@l3] got pmi command (from 23): get
kvsname=kvs_32745_0 key=sharedFilename[0]
[proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[0]) upstream
[proxy:0:0@l3] got pmi command (from 26): get
kvsname=kvs_32745_0 key=sharedFilename[0]
[proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[0]) upstream
[proxy:0:0@l3] got pmi command (from 29): get
kvsname=kvs_32745_0 key=sharedFilename[0]
[proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[0]) upstream
[proxy:0:0@l3] got pmi command (from 32): get
kvsname=kvs_32745_0 key=sharedFilename[0]
[proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[0]) upstream
[proxy:0:0@l3] got pmi command (from 35): get
kvsname=kvs_32745_0 key=sharedFilename[0]
[proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[0]) upstream
[proxy:0:1@l4] PMI response: cmd=barrier_out
[proxy:0:1@l4] got pmi command (from 6): get
kvsname=kvs_32745_0 key=sharedFilename[1]
[proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[1]) upstream
[proxy:0:1@l4] got pmi command (from 11): get
kvsname=kvs_32745_0 key=sharedFilename[1]
[proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[1]) upstream
[proxy:0:1@l4] got pmi command (from 16): get
kvsname=kvs_32745_0 key=sharedFilename[1]
[proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[1]) upstream
[proxy:0:1@l4] got pmi command (from 19): get
kvsname=kvs_32745_0 key=sharedFilename[1]
[proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[1]) upstream
[proxy:0:1@l4] got pmi command (from 25): get
kvsname=kvs_32745_0 key=sharedFilename[1]
[proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[1]) upstream
[proxy:0:1@l4] got pmi command (from 28): get
kvsname=kvs_32745_0 key=sharedFilename[1]
[proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[1]) upstream
[proxy:0:1@l4] got pmi command (from 31): get
kvsname=kvs_32745_0 key=sharedFilename[1]
[proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[1]) upstream
[proxy:0:1@l4] got pmi command (from 34): get
kvsname=kvs_32745_0 key=sharedFilename[1]
[proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[1]) upstream
[proxy:0:1@l4] got pmi command (from 43): get
kvsname=kvs_32745_0 key=sharedFilename[1]
[proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[1]) upstream
[proxy:0:1@l4] got pmi command (from 46): get
kvsname=kvs_32745_0 key=sharedFilename[1]
[proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[1]) upstream
[proxy:0:1@l4] got pmi command (from 49): get
kvsname=kvs_32745_0 key=sharedFilename[1]
[proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[1]) upstream
[proxy:0:1@l4] got pmi command (from 52): get
kvsname=kvs_32745_0 key=sharedFilename[1]
[proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[1]) upstream
[proxy:0:1@l4] got pmi command (from 55): get
kvsname=kvs_32745_0 key=sharedFilename[1]
[proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[1]) upstream
[proxy:0:1@l4] got pmi command (from 22): get
kvsname=kvs_32745_0 key=sharedFilename[1]
[proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[1]) upstream
[proxy:0:1@l4] got pmi command (from 37): get
kvsname=kvs_32745_0 key=sharedFilename[1]
[proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[1]) upstream
[proxy:0:1@l4] got pmi command (from 40): get
kvsname=kvs_32745_0 key=sharedFilename[1]
[proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[1]) upstream
[proxy:0:1@l4] got pmi command (from 58): get
kvsname=kvs_32745_0 key=sharedFilename[1]
[proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[1]) upstream
[proxy:0:1@l4] got pmi command (from 61): get
kvsname=kvs_32745_0 key=sharedFilename[1]
[proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[1]) upstream
[proxy:0:1@l4] got pmi command (from 64): get
kvsname=kvs_32745_0 key=sharedFilename[1]
[proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[1]) upstream
[mpiexec@l4] [pgid: 0] got PMI command: cmd=get kvsname=kvs_32745_0
key=sharedFilename[1]
[mpiexec@l4] PMI response to fd 6 pid 6: cmd=get_result rc=0 msg=success
value=/dev/shm/mpich_shar_tmp42EbvZ
[mpiexec@l4] [pgid: 0] got PMI command: cmd=get kvsname=kvs_32745_0
key=sharedFilename[0]
[mpiexec@l4] PMI response to fd 9 pid 6: cmd=get_result rc=0 msg=success
value=/dev/shm/mpich_shar_tmphAojti
[proxy:0:0@l3] got pmi command (from 38): get
kvsname=kvs_32745_0 key=sharedFilename[0]
[proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[0]) upstream
[proxy:0:0@l3] got pmi command (from 41): get
kvsname=kvs_32745_0 key=sharedFilename[0]
[proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[0]) upstream
[proxy:0:0@l3] got pmi command (from 44): get
kvsname=kvs_32745_0 key=sharedFilename[0]
[proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[0]) upstream
[proxy:0:0@l3] got pmi command (from 47): get
kvsname=kvs_32745_0 key=sharedFilename[0]
[proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[0]) upstream
[proxy:0:0@l3] got pmi command (from 50): get
kvsname=kvs_32745_0 key=sharedFilename[0]
[proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[0]) upstream
[proxy:0:0@l3] got pmi command (from 53): get
kvsname=kvs_32745_0 key=sharedFilename[0]
[proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[0]) upstream
[proxy:0:0@l3] we don't understand the response get_result; forwarding
downstream
[proxy:0:0@l3] got pmi command (from 56): get
kvsname=kvs_32745_0 key=sharedFilename[0]
[proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[0]) upstream
[proxy:0:0@l3] got pmi command (from 59): get
kvsname=kvs_32745_0 key=sharedFilename[0]
[proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0
key=sharedFilename[0]) upstream
[proxy:0:1@l4] we don't understand the response get_result; forwarding
downstream
[mpiexec@l4] [pgid: 0] got PMI command: cmd=get kvsname=kvs_32745_0
key=sharedFilename[1]
[mpiexec@l4] PMI response to fd 6 pid 11: cmd=get_result rc=0 msg=success
value=/dev

_______________________________________________
discuss mailing list     [email protected]
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss

[mpich-discuss] MPICH process frozen without any error message nor warning

CEAM Meteorología