Hi everyone and thanks for your attention. I am trying to run a meteorological simulation with RAMS model ( http://vandenheever.atmos.colostate.edu/vdhpage/rams.php) in a new cluster with CentOs7 in all nodes. We have other applications running fine in the cluster, sending processes to each node,... everything seemed to run fine. But the RAMS model freezes at its first stage on the parallel run. RAMS developers recommend using mpich2-1.4.1 as they have successfully tested. If I try to run a parallel simulation on the master node it runs fine and starts the required number of processes. If I try to use the other nodes the simulation freezes, do not stop with an error message; RAMS model processes appear in all used nodes and in the master node but any output is created and the usual status messages from RAMS do not appear on the screen. The command line to run the model is [paco@Llamp RUN]$ time ../misc/mpich2-1.4.1/bin/mpirun -verbose -machinefile mpd.hosts -n 20 ./rams-6.2.03 -f RAMSIN The only warning I can find in the log file (see below) is "we don't understand the response get_result; forwarding downstream" but as I'm not an expert with mpich I can't figure out the reason of the freezing. Thanks in advance for your help Log file: [paco@l4 paco]$ ../mpiexec -verbose -outfile-pattern mpi.log -errfile-pattern mpierror.log -machinefile machs -np 40 ../rams-6.2.06 -f RAMSIN host: l3 host: l4 ================================================================================================== mpiexec options: ---------------- Base path: /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco/../ Launcher: (null) Debug level: 1 Enable X: -1 Global environment: ------------------- XDG_SESSION_ID=6511 HOSTNAME=l4 SELINUX_ROLE_REQUESTED= TERM=xterm-256color SHELL=/bin/bash HISTSIZE=1000 SSH_CLIENT=193.144.100.32 49462 22 SELINUX_USE_CURRENT_RANGE= SSH_TTY=/dev/pts/0 USER=paco LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45: MAIL=/var/spool/mail/paco PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/paco/.local/bin:/home/paco/bin PWD=/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco LANG=es_ES.UTF-8 MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles LOADEDMODULES= SELINUX_LEVEL_REQUESTED= HISTCONTROL=ignoredups SHLVL=1 HOME=/home/paco LOGNAME=paco SSH_CONNECTION=193.144.100.32 49462 193.144.100.46 22 MODULESHOME=/usr/share/Modules LESSOPEN=||/usr/bin/lesspipe.sh %s XDG_RUNTIME_DIR=/run/user/1003 BASH_FUNC_module()=() { eval `/usr/bin/modulecmd bash $*` } _=../mpiexec OLDPWD=/home/paco Hydra internal environment: --------------------------- GFORTRAN_UNBUFFERED_PRECONNECTED=y Proxy information: ********************* [1] proxy: l3 (1 cores) Exec list: ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); [2] proxy: l4 (1 cores) Exec list: ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ../rams-6.2.06 (1 processes); ================================================================================================== [mpiexec@l4] Timeout set to -1 (-1 means infinite) [mpiexec@l4] Got a control port string of l4:44483 Proxy launch args: /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco/../hydra_pmi_proxy --control-port l4:44483 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --proxy-id [mpiexec@l4] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1 Arguments being passed to proxy 0: --version 1.4.1 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME --hostname l3 --global-core-map 0,1,1 --filler-process-map 0,1,1 --global-process-count 40 --auto-cleanup 1 --pmi-rank -1 --pmi-kvsname kvs_32745_0 --pmi-process-mapping (vector,(0,2,1)) --ckpoint-num -1 --global-inherited-env 29 'XDG_SESSION_ID=6511' 'HOSTNAME=l4' 'SELINUX_ROLE_REQUESTED=' 'TERM=xterm-256color' 'SHELL=/bin/bash' 'HISTSIZE=1000' 'SSH_CLIENT=193.144.100.32 49462 22' 'SELINUX_USE_CURRENT_RANGE=' 'SSH_TTY=/dev/pts/0' 'USER=paco' 'LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:' 'MAIL=/var/spool/mail/paco' 'PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/paco/.local/bin:/home/paco/bin' 'PWD=/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco' 'LANG=es_ES.UTF-8' 'MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles' 'LOADEDMODULES=' 'SELINUX_LEVEL_REQUESTED=' 'HISTCONTROL=ignoredups' 'SHLVL=1' 'HOME=/home/paco' 'LOGNAME=paco' 'SSH_CONNECTION=193.144.100.32 49462 193.144.100.46 22' 'MODULESHOME=/usr/share/Modules' 'LESSOPEN=||/usr/bin/lesspipe.sh %s' 'XDG_RUNTIME_DIR=/run/user/1003' 'BASH_FUNC_module()=() { eval `/usr/bin/modulecmd bash $*` }' '_=../mpiexec' 'OLDPWD=/home/paco' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN [mpiexec@l4] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1 Arguments being passed to proxy 1: --version 1.4.1 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME --hostname l4 --global-core-map 1,1,0 --filler-process-map 1,1,0 --global-process-count 40 --auto-cleanup 1 --pmi-rank -1 --pmi-kvsname kvs_32745_0 --pmi-process-mapping (vector,(0,2,1)) --ckpoint-num -1 --global-inherited-env 29 'XDG_SESSION_ID=6511' 'HOSTNAME=l4' 'SELINUX_ROLE_REQUESTED=' 'TERM=xterm-256color' 'SHELL=/bin/bash' 'HISTSIZE=1000' 'SSH_CLIENT=193.144.100.32 49462 22' 'SELINUX_USE_CURRENT_RANGE=' 'SSH_TTY=/dev/pts/0' 'USER=paco' 'LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:' 'MAIL=/var/spool/mail/paco' 'PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/paco/.local/bin:/home/paco/bin' 'PWD=/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco' 'LANG=es_ES.UTF-8' 'MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles' 'LOADEDMODULES=' 'SELINUX_LEVEL_REQUESTED=' 'HISTCONTROL=ignoredups' 'SHLVL=1' 'HOME=/home/paco' 'LOGNAME=paco' 'SSH_CONNECTION=193.144.100.32 49462 193.144.100.46 22' 'MODULESHOME=/usr/share/Modules' 'LESSOPEN=||/usr/bin/lesspipe.sh %s' 'XDG_RUNTIME_DIR=/run/user/1003' 'BASH_FUNC_module()=() { eval `/usr/bin/modulecmd bash $*` }' '_=../mpiexec' 'OLDPWD=/home/paco' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco --exec-args 3 ../rams-6.2.06 -f RAMSIN [mpiexec@l4] Launch arguments: /usr/bin/ssh -x l3 "/export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco/../hydra_pmi_proxy" --control-port l4:44483 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --proxy-id 0 [mpiexec@l4] Launch arguments: /export4/RAMS/rams_20170906_release_6.2.06/bin.rams/paco/../hydra_pmi_proxy --control-port l4:44483 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --proxy-id 1 [proxy:0:1@l4] got pmi command (from 0): init pmi_version=1 pmi_subversion=1 [proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@l4] got pmi command (from 6): init pmi_version=1 pmi_subversion=1 [proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@l4] got pmi command (from 11): init pmi_version=1 pmi_subversion=1 [proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@l4] got pmi command (from 16): init pmi_version=1 pmi_subversion=1 [proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@l4] got pmi command (from 19): init pmi_version=1 pmi_subversion=1 [proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@l4] got pmi command (from 25): init pmi_version=1 pmi_subversion=1 [proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@l4] got pmi command (from 31): init pmi_version=1 pmi_subversion=1 [proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@l4] got pmi command (from 37): init pmi_version=1 pmi_subversion=1 [proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@l4] got pmi command (from 0): get_maxes [proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@l4] got pmi command (from 6): get_maxes [proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@l4] got pmi command (from 11): get_maxes [proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@l4] got pmi command (from 16): get_maxes [proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@l4] got pmi command (from 19): get_maxes [proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@l4] got pmi command (from 25): get_maxes [proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@l4] got pmi command (from 46): init pmi_version=1 pmi_subversion=1 [proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@l4] got pmi command (from 49): init pmi_version=1 pmi_subversion=1 [proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@l4] got pmi command (from 0): get_appnum [proxy:0:1@l4] PMI response: cmd=appnum appnum=0 [proxy:0:1@l4] got pmi command (from 6): get_appnum [proxy:0:1@l4] PMI response: cmd=appnum appnum=0 [proxy:0:1@l4] got pmi command (from 11): get_appnum [proxy:0:1@l4] PMI response: cmd=appnum appnum=0 [proxy:0:1@l4] got pmi command (from 16): get_appnum [proxy:0:1@l4] PMI response: cmd=appnum appnum=0 [proxy:0:1@l4] got pmi command (from 19): get_appnum [proxy:0:1@l4] PMI response: cmd=appnum appnum=0 [proxy:0:1@l4] got pmi command (from 25): get_appnum [proxy:0:1@l4] PMI response: cmd=appnum appnum=0 [proxy:0:1@l4] got pmi command (from 31): get_maxes [proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@l4] got pmi command (from 37): get_maxes [proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@l4] got pmi command (from 0): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 6): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 11): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 16): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 19): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 25): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 31): get_appnum [proxy:0:1@l4] PMI response: cmd=appnum appnum=0 [proxy:0:1@l4] got pmi command (from 46): get_maxes [proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@l4] got pmi command (from 49): get_maxes [proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@l4] got pmi command (from 0): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 6): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 11): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 19): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 25): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 31): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 37): get_appnum [proxy:0:1@l4] PMI response: cmd=appnum appnum=0 [proxy:0:1@l4] got pmi command (from 46): get_appnum [proxy:0:1@l4] PMI response: cmd=appnum appnum=0 [proxy:0:1@l4] got pmi command (from 0): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:1@l4] got pmi command (from 6): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:1@l4] got pmi command (from 11): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:1@l4] got pmi command (from 19): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:1@l4] got pmi command (from 25): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:1@l4] got pmi command (from 37): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 49): get_appnum [proxy:0:1@l4] PMI response: cmd=appnum appnum=0 [proxy:0:1@l4] got pmi command (from 6): barrier_in [proxy:0:1@l4] got pmi command (from 11): barrier_in [proxy:0:1@l4] got pmi command (from 19): barrier_in [proxy:0:1@l4] got pmi command (from 25): barrier_in [proxy:0:1@l4] got pmi command (from 31): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 46): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 31): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:1@l4] got pmi command (from 37): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 49): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 31): barrier_in [proxy:0:1@l4] got pmi command (from 37): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:1@l4] got pmi command (from 46): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 52): init pmi_version=1 pmi_subversion=1 [proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@l4] got pmi command (from 37): barrier_in [proxy:0:1@l4] got pmi command (from 46): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:1@l4] got pmi command (from 49): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 46): barrier_in [proxy:0:1@l4] got pmi command (from 52): get_maxes [proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@l4] got pmi command (from 55): init pmi_version=1 pmi_subversion=1 [proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@l4] got pmi command (from 49): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:1@l4] got pmi command (from 52): get_appnum [proxy:0:1@l4] PMI response: cmd=appnum appnum=0 [proxy:0:1@l4] got pmi command (from 52): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 55): get_maxes [proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@l4] got pmi command (from 49): barrier_in [proxy:0:1@l4] got pmi command (from 52): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 55): get_appnum [proxy:0:1@l4] PMI response: cmd=appnum appnum=0 [proxy:0:1@l4] got pmi command (from 16): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 52): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:1@l4] got pmi command (from 58): init pmi_version=1 pmi_subversion=1 [proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@l4] got pmi command (from 16): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:1@l4] got pmi command (from 52): barrier_in [proxy:0:1@l4] got pmi command (from 55): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 16): barrier_in [proxy:0:1@l4] got pmi command (from 58): get_maxes [proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@l4] got pmi command (from 55): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 0): put kvsname=kvs_32745_0 key=sharedFilename[1] value=/dev/shm/mpich_shar_tmp42EbvZ [proxy:0:1@l4] we don't understand this command put; forwarding upstream [proxy:0:1@l4] got pmi command (from 58): get_appnum [mpiexec@l4] [pgid: 0] got PMI command: cmd=put kvsname=kvs_32745_0 key=sharedFilename[1] value=/dev/shm/mpich_shar_tmp42EbvZ [mpiexec@l4] PMI response to fd 6 pid 0: cmd=put_result rc=0 msg=success [proxy:0:1@l4] PMI response: cmd=appnum appnum=0 [proxy:0:1@l4] got pmi command (from 55): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:1@l4] got pmi command (from 58): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] we don't understand the response put_result; forwarding downstream [proxy:0:1@l4] got pmi command (from 55): barrier_in [proxy:0:1@l4] got pmi command (from 58): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 64): init pmi_version=1 pmi_subversion=1 [proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@l4] got pmi command (from 0): barrier_in [proxy:0:1@l4] got pmi command (from 58): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:1@l4] got pmi command (from 64): get_maxes [proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@l4] got pmi command (from 58): barrier_in [proxy:0:1@l4] got pmi command (from 64): get_appnum [proxy:0:1@l4] PMI response: cmd=appnum appnum=0 [proxy:0:1@l4] got pmi command (from 64): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 64): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 64): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:1@l4] got pmi command (from 64): barrier_in [proxy:0:1@l4] got pmi command (from 28): init pmi_version=1 pmi_subversion=1 [proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@l4] got pmi command (from 28): get_maxes [proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@l4] got pmi command (from 28): get_appnum [proxy:0:1@l4] PMI response: cmd=appnum appnum=0 [proxy:0:1@l4] got pmi command (from 28): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 28): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 28): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:1@l4] got pmi command (from 28): barrier_in [proxy:0:1@l4] got pmi command (from 43): init pmi_version=1 pmi_subversion=1 [proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@l4] got pmi command (from 43): get_maxes [proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@l4] got pmi command (from 43): get_appnum [proxy:0:1@l4] PMI response: cmd=appnum appnum=0 [proxy:0:1@l4] got pmi command (from 43): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 43): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 43): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:1@l4] got pmi command (from 43): barrier_in [proxy:0:1@l4] got pmi command (from 40): init pmi_version=1 pmi_subversion=1 [proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@l4] got pmi command (from 40): get_maxes [proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@l4] got pmi command (from 40): get_appnum [proxy:0:1@l4] PMI response: cmd=appnum appnum=0 [proxy:0:1@l4] got pmi command (from 40): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 40): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 40): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:1@l4] got pmi command (from 40): barrier_in [proxy:0:1@l4] got pmi command (from 22): init pmi_version=1 pmi_subversion=1 [proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@l4] got pmi command (from 22): get_maxes [proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@l4] got pmi command (from 22): get_appnum [proxy:0:1@l4] PMI response: cmd=appnum appnum=0 [proxy:0:1@l4] got pmi command (from 22): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 22): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 22): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:1@l4] got pmi command (from 22): barrier_in [proxy:0:1@l4] got pmi command (from 34): init pmi_version=1 pmi_subversion=1 [proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@l4] got pmi command (from 61): init pmi_version=1 pmi_subversion=1 [proxy:0:1@l4] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@l4] got pmi command (from 34): get_maxes [proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@l4] got pmi command (from 61): get_maxes [proxy:0:1@l4] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@l4] got pmi command (from 34): get_appnum [proxy:0:1@l4] PMI response: cmd=appnum appnum=0 [proxy:0:1@l4] got pmi command (from 61): get_appnum [proxy:0:1@l4] PMI response: cmd=appnum appnum=0 [proxy:0:1@l4] got pmi command (from 34): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 61): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 34): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 61): get_my_kvsname [proxy:0:1@l4] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:1@l4] got pmi command (from 34): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:1@l4] got pmi command (from 61): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:1@l4] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:1@l4] got pmi command (from 34): barrier_in [proxy:0:1@l4] got pmi command (from 61): barrier_in [mpiexec@l4] [pgid: 0] got PMI command: cmd=barrier_in [proxy:0:1@l4] forwarding command (cmd=barrier_in) upstream [proxy:0:0@l3] got pmi command (from 4): init pmi_version=1 pmi_subversion=1 [proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@l3] got pmi command (from 5): init pmi_version=1 pmi_subversion=1 [proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@l3] got pmi command (from 6): init pmi_version=1 pmi_subversion=1 [proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@l3] got pmi command (from 11): init pmi_version=1 pmi_subversion=1 [proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@l3] got pmi command (from 14): init pmi_version=1 pmi_subversion=1 [proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@l3] got pmi command (from 17): init pmi_version=1 pmi_subversion=1 [proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@l3] got pmi command (from 20): init pmi_version=1 pmi_subversion=1 [proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@l3] got pmi command (from 23): init pmi_version=1 pmi_subversion=1 [proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@l3] got pmi command (from 26): init pmi_version=1 pmi_subversion=1 [proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@l3] got pmi command (from 29): init pmi_version=1 pmi_subversion=1 [proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@l3] got pmi command (from 32): init pmi_version=1 pmi_subversion=1 [proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@l3] got pmi command (from 4): get_maxes [proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@l3] got pmi command (from 5): get_maxes [proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@l3] got pmi command (from 6): get_maxes [proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@l3] got pmi command (from 11): get_maxes [proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@l3] got pmi command (from 14): get_maxes [proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@l3] got pmi command (from 17): get_maxes [proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@l3] got pmi command (from 20): get_maxes [proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@l3] got pmi command (from 23): get_maxes [proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@l3] got pmi command (from 26): get_maxes [proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@l3] got pmi command (from 35): init pmi_version=1 pmi_subversion=1 [proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@l3] got pmi command (from 38): init pmi_version=1 pmi_subversion=1 [proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@l3] got pmi command (from 4): get_appnum [proxy:0:0@l3] PMI response: cmd=appnum appnum=0 [proxy:0:0@l3] got pmi command (from 5): get_appnum [proxy:0:0@l3] PMI response: cmd=appnum appnum=0 [proxy:0:0@l3] got pmi command (from 6): get_appnum [proxy:0:0@l3] PMI response: cmd=appnum appnum=0 [proxy:0:0@l3] got pmi command (from 11): get_appnum [proxy:0:0@l3] PMI response: cmd=appnum appnum=0 [proxy:0:0@l3] got pmi command (from 14): get_appnum [proxy:0:0@l3] PMI response: cmd=appnum appnum=0 [proxy:0:0@l3] got pmi command (from 17): get_appnum [proxy:0:0@l3] PMI response: cmd=appnum appnum=0 [proxy:0:0@l3] got pmi command (from 20): get_appnum [proxy:0:0@l3] PMI response: cmd=appnum appnum=0 [proxy:0:0@l3] got pmi command (from 23): get_appnum [proxy:0:0@l3] PMI response: cmd=appnum appnum=0 [proxy:0:0@l3] got pmi command (from 26): get_appnum [proxy:0:0@l3] PMI response: cmd=appnum appnum=0 [proxy:0:0@l3] got pmi command (from 29): get_maxes [proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@l3] got pmi command (from 32): get_maxes [proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@l3] got pmi command (from 4): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 5): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 6): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 11): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 14): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 17): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 20): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 23): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 26): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 29): get_appnum [proxy:0:0@l3] PMI response: cmd=appnum appnum=0 [proxy:0:0@l3] got pmi command (from 35): get_maxes [proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@l3] got pmi command (from 38): get_maxes [proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@l3] got pmi command (from 41): init pmi_version=1 pmi_subversion=1 [proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@l3] got pmi command (from 4): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 5): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 6): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 11): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 14): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 17): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 20): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 23): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 26): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 29): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 32): get_appnum [proxy:0:0@l3] PMI response: cmd=appnum appnum=0 [proxy:0:0@l3] got pmi command (from 35): get_appnum [proxy:0:0@l3] PMI response: cmd=appnum appnum=0 [proxy:0:0@l3] got pmi command (from 38): get_appnum [proxy:0:0@l3] PMI response: cmd=appnum appnum=0 [proxy:0:0@l3] got pmi command (from 4): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@l3] got pmi command (from 5): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@l3] got pmi command (from 6): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@l3] got pmi command (from 11): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@l3] got pmi command (from 14): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@l3] got pmi command (from 17): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@l3] got pmi command (from 20): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@l3] got pmi command (from 23): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@l3] got pmi command (from 26): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@l3] got pmi command (from 32): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 35): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 41): get_maxes [proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@l3] got pmi command (from 5): barrier_in [proxy:0:0@l3] got pmi command (from 6): barrier_in [proxy:0:0@l3] got pmi command (from 11): barrier_in [proxy:0:0@l3] got pmi command (from 14): barrier_in [proxy:0:0@l3] got pmi command (from 17): barrier_in [proxy:0:0@l3] got pmi command (from 20): barrier_in [proxy:0:0@l3] got pmi command (from 23): barrier_in [proxy:0:0@l3] got pmi command (from 26): barrier_in [proxy:0:0@l3] got pmi command (from 29): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 38): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 29): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@l3] got pmi command (from 32): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 35): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 41): get_appnum [proxy:0:0@l3] PMI response: cmd=appnum appnum=0 [proxy:0:0@l3] got pmi command (from 29): barrier_in [proxy:0:0@l3] got pmi command (from 32): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@l3] got pmi command (from 35): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@l3] got pmi command (from 38): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 32): barrier_in [proxy:0:0@l3] got pmi command (from 41): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 35): barrier_in [proxy:0:0@l3] got pmi command (from 38): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@l3] got pmi command (from 38): barrier_in [proxy:0:0@l3] got pmi command (from 41): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 41): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@l3] got pmi command (from 41): barrier_in [mpiexec@l4] [pgid: 0] got PMI command: cmd=put kvsname=kvs_32745_0 key=sharedFilename[0] value=/dev/shm/mpich_shar_tmphAojti [mpiexec@l4] PMI response to fd 9 pid 4: cmd=put_result rc=0 msg=success [proxy:0:0@l3] got pmi command (from 4): put kvsname=kvs_32745_0 key=sharedFilename[0] value=/dev/shm/mpich_shar_tmphAojti [proxy:0:0@l3] we don't understand this command put; forwarding upstream [proxy:0:0@l3] we don't understand the response put_result; forwarding downstream [proxy:0:0@l3] got pmi command (from 4): barrier_in [proxy:0:0@l3] got pmi command (from 56): init pmi_version=1 pmi_subversion=1 [proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@l3] got pmi command (from 50): init pmi_version=1 pmi_subversion=1 [proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@l3] got pmi command (from 56): get_maxes [proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@l3] got pmi command (from 59): init pmi_version=1 pmi_subversion=1 [proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@l3] got pmi command (from 56): get_appnum [proxy:0:0@l3] PMI response: cmd=appnum appnum=0 [proxy:0:0@l3] got pmi command (from 50): get_maxes [proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@l3] got pmi command (from 56): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 59): get_maxes [proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@l3] got pmi command (from 50): get_appnum [proxy:0:0@l3] PMI response: cmd=appnum appnum=0 [proxy:0:0@l3] got pmi command (from 59): get_appnum [proxy:0:0@l3] PMI response: cmd=appnum appnum=0 [proxy:0:0@l3] got pmi command (from 50): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 56): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 59): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 56): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@l3] got pmi command (from 50): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 56): barrier_in [proxy:0:0@l3] got pmi command (from 59): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 50): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@l3] got pmi command (from 59): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@l3] got pmi command (from 50): barrier_in [proxy:0:0@l3] got pmi command (from 59): barrier_in [proxy:0:0@l3] got pmi command (from 44): init pmi_version=1 pmi_subversion=1 [proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@l3] got pmi command (from 53): init pmi_version=1 pmi_subversion=1 [proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@l3] got pmi command (from 44): get_maxes [proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@l3] got pmi command (from 53): get_maxes [proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@l3] got pmi command (from 44): get_appnum [proxy:0:0@l3] PMI response: cmd=appnum appnum=0 [proxy:0:0@l3] got pmi command (from 53): get_appnum [proxy:0:0@l3] PMI response: cmd=appnum appnum=0 [proxy:0:0@l3] got pmi command (from 44): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 53): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 44): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 53): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 44): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@l3] got pmi command (from 53): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@l3] got pmi command (from 44): barrier_in [proxy:0:0@l3] got pmi command (from 53): barrier_in [proxy:0:0@l3] got pmi command (from 47): init pmi_version=1 pmi_subversion=1 [proxy:0:0@l3] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@l3] got pmi command (from 47): get_maxes [proxy:0:0@l3] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@l3] got pmi command (from 47): get_appnum [proxy:0:0@l3] PMI response: cmd=appnum appnum=0 [proxy:0:0@l3] got pmi command (from 47): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 47): get_my_kvsname [proxy:0:0@l3] PMI response: cmd=my_kvsname kvsname=kvs_32745_0 [proxy:0:0@l3] got pmi command (from 47): get kvsname=kvs_32745_0 key=PMI_process_mapping [proxy:0:0@l3] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@l3] got pmi command (from 47): barrier_in [proxy:0:0@l3] forwarding command (cmd=barrier_in) upstream [mpiexec@l4] [pgid: 0] got PMI command: cmd=barrier_in [mpiexec@l4] PMI response to fd 9 pid 47: cmd=barrier_out [mpiexec@l4] PMI response to fd 6 pid 47: cmd=barrier_out [proxy:0:1@l4] PMI response: cmd=barrier_out [proxy:0:1@l4] PMI response: cmd=barrier_out [proxy:0:1@l4] PMI response: cmd=barrier_out [proxy:0:1@l4] PMI response: cmd=barrier_out [proxy:0:1@l4] PMI response: cmd=barrier_out [proxy:0:1@l4] PMI response: cmd=barrier_out [proxy:0:1@l4] PMI response: cmd=barrier_out [proxy:0:1@l4] PMI response: cmd=barrier_out [proxy:0:1@l4] PMI response: cmd=barrier_out [proxy:0:1@l4] PMI response: cmd=barrier_out [proxy:0:1@l4] PMI response: cmd=barrier_out [proxy:0:1@l4] PMI response: cmd=barrier_out [proxy:0:1@l4] PMI response: cmd=barrier_out [proxy:0:1@l4] PMI response: cmd=barrier_out [proxy:0:1@l4] PMI response: cmd=barrier_out [proxy:0:1@l4] PMI response: cmd=barrier_out [proxy:0:1@l4] PMI response: cmd=barrier_out [proxy:0:1@l4] PMI response: cmd=barrier_out [proxy:0:1@l4] PMI response: cmd=barrier_out [mpiexec@l4] [pgid: 0] got PMI command: cmd=get kvsname=kvs_32745_0 key=sharedFilename[0] [mpiexec@l4] PMI response to fd 9 pid 5: cmd=get_result rc=0 msg=success value=/dev/shm/mpich_shar_tmphAojti [proxy:0:0@l3] PMI response: cmd=barrier_out [proxy:0:0@l3] PMI response: cmd=barrier_out [proxy:0:0@l3] PMI response: cmd=barrier_out [proxy:0:0@l3] PMI response: cmd=barrier_out [proxy:0:0@l3] PMI response: cmd=barrier_out [proxy:0:0@l3] PMI response: cmd=barrier_out [proxy:0:0@l3] PMI response: cmd=barrier_out [proxy:0:0@l3] PMI response: cmd=barrier_out [proxy:0:0@l3] PMI response: cmd=barrier_out [proxy:0:0@l3] PMI response: cmd=barrier_out [proxy:0:0@l3] PMI response: cmd=barrier_out [proxy:0:0@l3] PMI response: cmd=barrier_out [proxy:0:0@l3] PMI response: cmd=barrier_out [proxy:0:0@l3] PMI response: cmd=barrier_out [proxy:0:0@l3] PMI response: cmd=barrier_out [proxy:0:0@l3] PMI response: cmd=barrier_out [proxy:0:0@l3] PMI response: cmd=barrier_out [proxy:0:0@l3] PMI response: cmd=barrier_out [proxy:0:0@l3] PMI response: cmd=barrier_out [proxy:0:0@l3] PMI response: cmd=barrier_out [proxy:0:0@l3] got pmi command (from 5): get kvsname=kvs_32745_0 key=sharedFilename[0] [proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[0]) upstream [proxy:0:0@l3] got pmi command (from 6): get kvsname=kvs_32745_0 key=sharedFilename[0] [proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[0]) upstream [proxy:0:0@l3] got pmi command (from 11): get kvsname=kvs_32745_0 key=sharedFilename[0] [proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[0]) upstream [proxy:0:0@l3] got pmi command (from 14): get kvsname=kvs_32745_0 key=sharedFilename[0] [proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[0]) upstream [proxy:0:0@l3] got pmi command (from 17): get kvsname=kvs_32745_0 key=sharedFilename[0] [proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[0]) upstream [proxy:0:0@l3] got pmi command (from 20): get kvsname=kvs_32745_0 key=sharedFilename[0] [proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[0]) upstream [proxy:0:0@l3] got pmi command (from 23): get kvsname=kvs_32745_0 key=sharedFilename[0] [proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[0]) upstream [proxy:0:0@l3] got pmi command (from 26): get kvsname=kvs_32745_0 key=sharedFilename[0] [proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[0]) upstream [proxy:0:0@l3] got pmi command (from 29): get kvsname=kvs_32745_0 key=sharedFilename[0] [proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[0]) upstream [proxy:0:0@l3] got pmi command (from 32): get kvsname=kvs_32745_0 key=sharedFilename[0] [proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[0]) upstream [proxy:0:0@l3] got pmi command (from 35): get kvsname=kvs_32745_0 key=sharedFilename[0] [proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[0]) upstream [proxy:0:1@l4] PMI response: cmd=barrier_out [proxy:0:1@l4] got pmi command (from 6): get kvsname=kvs_32745_0 key=sharedFilename[1] [proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[1]) upstream [proxy:0:1@l4] got pmi command (from 11): get kvsname=kvs_32745_0 key=sharedFilename[1] [proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[1]) upstream [proxy:0:1@l4] got pmi command (from 16): get kvsname=kvs_32745_0 key=sharedFilename[1] [proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[1]) upstream [proxy:0:1@l4] got pmi command (from 19): get kvsname=kvs_32745_0 key=sharedFilename[1] [proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[1]) upstream [proxy:0:1@l4] got pmi command (from 25): get kvsname=kvs_32745_0 key=sharedFilename[1] [proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[1]) upstream [proxy:0:1@l4] got pmi command (from 28): get kvsname=kvs_32745_0 key=sharedFilename[1] [proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[1]) upstream [proxy:0:1@l4] got pmi command (from 31): get kvsname=kvs_32745_0 key=sharedFilename[1] [proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[1]) upstream [proxy:0:1@l4] got pmi command (from 34): get kvsname=kvs_32745_0 key=sharedFilename[1] [proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[1]) upstream [proxy:0:1@l4] got pmi command (from 43): get kvsname=kvs_32745_0 key=sharedFilename[1] [proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[1]) upstream [proxy:0:1@l4] got pmi command (from 46): get kvsname=kvs_32745_0 key=sharedFilename[1] [proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[1]) upstream [proxy:0:1@l4] got pmi command (from 49): get kvsname=kvs_32745_0 key=sharedFilename[1] [proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[1]) upstream [proxy:0:1@l4] got pmi command (from 52): get kvsname=kvs_32745_0 key=sharedFilename[1] [proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[1]) upstream [proxy:0:1@l4] got pmi command (from 55): get kvsname=kvs_32745_0 key=sharedFilename[1] [proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[1]) upstream [proxy:0:1@l4] got pmi command (from 22): get kvsname=kvs_32745_0 key=sharedFilename[1] [proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[1]) upstream [proxy:0:1@l4] got pmi command (from 37): get kvsname=kvs_32745_0 key=sharedFilename[1] [proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[1]) upstream [proxy:0:1@l4] got pmi command (from 40): get kvsname=kvs_32745_0 key=sharedFilename[1] [proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[1]) upstream [proxy:0:1@l4] got pmi command (from 58): get kvsname=kvs_32745_0 key=sharedFilename[1] [proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[1]) upstream [proxy:0:1@l4] got pmi command (from 61): get kvsname=kvs_32745_0 key=sharedFilename[1] [proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[1]) upstream [proxy:0:1@l4] got pmi command (from 64): get kvsname=kvs_32745_0 key=sharedFilename[1] [proxy:0:1@l4] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[1]) upstream [mpiexec@l4] [pgid: 0] got PMI command: cmd=get kvsname=kvs_32745_0 key=sharedFilename[1] [mpiexec@l4] PMI response to fd 6 pid 6: cmd=get_result rc=0 msg=success value=/dev/shm/mpich_shar_tmp42EbvZ [mpiexec@l4] [pgid: 0] got PMI command: cmd=get kvsname=kvs_32745_0 key=sharedFilename[0] [mpiexec@l4] PMI response to fd 9 pid 6: cmd=get_result rc=0 msg=success value=/dev/shm/mpich_shar_tmphAojti [proxy:0:0@l3] got pmi command (from 38): get kvsname=kvs_32745_0 key=sharedFilename[0] [proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[0]) upstream [proxy:0:0@l3] got pmi command (from 41): get kvsname=kvs_32745_0 key=sharedFilename[0] [proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[0]) upstream [proxy:0:0@l3] got pmi command (from 44): get kvsname=kvs_32745_0 key=sharedFilename[0] [proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[0]) upstream [proxy:0:0@l3] got pmi command (from 47): get kvsname=kvs_32745_0 key=sharedFilename[0] [proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[0]) upstream [proxy:0:0@l3] got pmi command (from 50): get kvsname=kvs_32745_0 key=sharedFilename[0] [proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[0]) upstream [proxy:0:0@l3] got pmi command (from 53): get kvsname=kvs_32745_0 key=sharedFilename[0] [proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[0]) upstream [proxy:0:0@l3] we don't understand the response get_result; forwarding downstream [proxy:0:0@l3] got pmi command (from 56): get kvsname=kvs_32745_0 key=sharedFilename[0] [proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[0]) upstream [proxy:0:0@l3] got pmi command (from 59): get kvsname=kvs_32745_0 key=sharedFilename[0] [proxy:0:0@l3] forwarding command (cmd=get kvsname=kvs_32745_0 key=sharedFilename[0]) upstream [proxy:0:1@l4] we don't understand the response get_result; forwarding downstream [mpiexec@l4] [pgid: 0] got PMI command: cmd=get kvsname=kvs_32745_0 key=sharedFilename[1] [mpiexec@l4] PMI response to fd 6 pid 11: cmd=get_result rc=0 msg=success value=/dev _______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss