Another:
joni@mpi1:/mpi3/S3/mpich-3.0.4$ mpiexec -np 6 hostname
host: 192.168.0.41
host: 192.168.0.42
host: 192.168.0.43
==================================================================================================
mpiexec options:
----------------
Base path: /mpi3/C3/mpich-3.0.4/bin/
Launcher: (null)
Debug level: 1
Enable X: -1
Global environment:
-------------------
MUMPS=/mpi3/S3/MUMPS_4.10.0
LC_PAPER=fi_FI.UTF-8
LC_ADDRESS=fi_FI.UTF-8
SSH_AGENT_PID=12144
LC_MONETARY=fi_FI.UTF-8
MUMPS_I=/mpi3/C3/MUMPS_4.10.0
HYDRA_DEMUX=select
GPG_AGENT_INFO=/tmp/keyring-kJwpJQ/gpg:0:1
JPK_LMETISDIR_S5=/mpi3/S3/parmetis-4.0.2
TERM=xterm
SHELL=/bin/bash
XDG_SESSION_COOKIE=6d6390cb56a32b6678c10da600000412-1377606907.629665-1922379047
FFT=/mpi3/C3/fftw2
HYDRA_ENV=all
JPK_NETGEN=/mpi3/C3/netgen_668
JPK_VER_S=S3
HYDRA_CKPOINTLIB=blcr
HYDRA_CKPOINT_INTERVAL=10800
WINDOWID=54602522
LC_NUMERIC=fi_FI.UTF-8
HYDRA_CKPOINT_PREFIX=/mpi3/chekpoint/default.chk
GNOME_KEYRING_CONTROL=/tmp/keyring-kJwpJQ
JPK_ELMER=/mpi3/C3/elmer_6283
PARDISO_LIC_PATH=/mpi3/C3/pardiso
METIS_INCLUDE_DIR=/mpi3/C3/ParMetis-3.2.0
JPK_NETGEN_S=/mpi3/S3/netgen_668
USER=joni
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:
JPK_TOGL=/mpi3/C3/Togl-1.7
LD_LIBRARY_PATH=/mpi3/C3/mpich-3.0.4/lib:/mpi3/C3/mpich-3.0.4/bin:/mpi3/C3/blcr-0.8.5/lib:/mpi3/C3/ParMetis-3.2.0:/mpi3/C3/acml5.3.1/gfortran64_mp/lib:/lib:/mpi3/C3/hypre-2.8.0b/lib:/mpi3/C3/scotch_6.0.0/lib:/mpi3/S3/MUMPS_4.10.0/lib:/mpi3/C3/acml5.3.1/gfortran64_mp/lib:/mpi3/C3/scalapack-2.0.2/lib:/mpi3/C3/hypre-2.8.0b/lib:/mpi3/C3/pardiso:/mpi3/C3/ParMetis-3.2.0:/mpi3/C3/ARPACK:/mpi3/C3/hdf5-1.8.10-patch1/lib:/mpi3/C3/VTK-5.8.0/lib/vtk-5.8:/mpi3/C3/elmer_6283/lib:/mpi3/C3/Togl-1.7:/mpi3/C3/netgen_668/lib:/usr/lib/:/usr/local/lib://mpi3/C3/vrpn/lib://mpi3/C3/hidapi/lib:/usr/include/libusb-1.0
LC_TELEPHONE=fi_FI.UTF-8
XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0
JPK_OCC=/usr/include/oce
XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0
HYDRA_HOST_FILE=/mpi4/hosts
SSH_AUTH_SOCK=/tmp/ssh-NnhxNTH12143/agent.12143
SCOTCHDIR=/mpi3/C3/scotch_6.0.0
HYDRA_LAUNCHER=rsh
JPK_VER_B=B3
SESSION_MANAGER=local/mpi1:@/tmp/.ICE-unix/4284,unix/mpi1:/tmp/.ICE-unix/4284
DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path
ELMER_HOME=/mpi3/C3/elmer_6283
BLACS=/mpi3/C3/scalapack-2.0.2
BLAS32=/mpi3/C3/acml5.3.1/gfortran64_mp
METIS_DIR=Exec list: hostname (2 processes);MPI_LIBS=-L/mpi3/C3/mpich-3.0.4/lib -lmpich -lmpichf90 -lmpl -lopa -lmpichcxx
XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg
JPK_MPI_DIR=/mpi3
JPK_HDF5_S=/mpi3/S3/hdf5-1.8.10-patch1
MPIEXEC_PORT_RANGE=7000:7500
PATH=/mpi3/C3/cmake-2.8.10.2/bin:/mpi3/C3/blcr-0.8.5/bin:/mpi3/C3/mpich-3.0.4/bin:/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/mpi3/C3/elmer_6283/bin:/mpi3/C3/elmer_6283/lib:/mpi3/C3/ParaView3
DESKTOP_SESSION=ubuntu
BLAS=/mpi3/C3/acml5.3.1/gfortran64_mp
METIS_LIBDIR=/mpi3/C3/ParMetis-3.2.0
CMAKE_COMMAND=/mpi3/C3/cmake-2.8.10.2/bin
QT_QMAKE_EXECUTABLE=/usr/bin/qmake-qt4
LC_IDENTIFICATION=fi_FI.UTF-8
JPK_SCOTCHDIR_S=/mpi3/S3/scotch_6.0.0_esmumps
JPK_LMETISDIR_S=/mpi3/S3/ParMetis-3.2.0
PWD=/mpi3/S3/mpich-3.0.4
NETGENDIR=/mpi3/C3/netgen_668/bin
EDITOR=nano
JPK_LMETISDIR=/mpi3/C3/ParMetis-3.2.0
GNOME_KEYRING_PID=4273
LANG=fi_FI.UTF-8
MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path
OMPI_77=//mpi3/C3/mpich-3.0.4/bin/mpif77
LC_MEASUREMENT=fi_FI.UTF-8
JPK_HDF5=/mpi3/C3/hdf5-1.8.10-patch1
UBUNTU_MENUPROXY=libappmenu.so
COMPIZ_CONFIG_PROFILE=ubuntu
ELMER_POST_HOME=/mpi3/C3/elmer_6283/bin
JPK_INS=/mpi3/C3
ELMER_LIB=/mpi3/C3/elmer_6283/share/elmersolver/lib
HYDRA_PROXY_RETRY_COUNT=3
GDMSESSION=ubuntu
JPK_ELMER_S=/mpi3/S3/elmer_6283
JPK_LMETISDIR5=/mpi3/C3/parmetis-4.0.2
JPK_LMETISDIR32=/mpi3/C3/ParMetis-3.2.0
HYDRA_DEBUG=1
JPK_BUI=/mpi3/S3
VTK_INCLUDEPATH=/mpi3/C3/VTK-5.8.0/include
SHLVL=1
HOME=/home/joni
OMPI_CC=//mpi3/C3/mpich-3.0.4/bin/mpicc
LANGUAGE=fi:en
OMPI_90=//mpi3/C3/mpich-3.0.4/bin/mpif90
ELMERGUI_HOME=/mpi3/C3/elmer_6283/bin
GNOME_DESKTOP_SESSION_ID=this-is-deprecated
MPI_IMPLEMENTATION=mpich
MKL_SERIAL=YES
LOGNAME=joni
HYPRE=/mpi3/C3/hypre-2.8.0b
JPK_ARPACK_S=/mpi3/S3/ARPACK
JPK_JOBS=7
JPK_VTK_DIR=/mpi3/C3/VTK-5.8.0
SCALAPACK=/mpi3/C3/scalapack-2.0.2
XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/
DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-gAYmjGaklf,guid=26c3f15a7a5ee55e8782415700000034
JPK_ARPACK=/mpi3/C3/ARPACK
MPI_HOME=/mpi3/C3/mpich-3.0.4
LESSOPEN=| /usr/bin/lesspipe %s
LACPACK=/mpi3/C3/acml5.3.1/gfortran64_mp
OMPI_CXX=//mpi3/C3/mpich-3.0.4/bin/mpicxx
OMP_NUM_THREADS=6
JPK_TOGL_S=/mpi3/S3/Togl-1.7
HYDRA_LAUNCHER_EXEC=/usr/bin/rsh-redone-rsh
JPK_MPICH2=/mpi3/C3/mpich-3.0.4
PARDISO=/mpi3/C3/pardiso
PARDISOLICMESSAGE=1
JPK_VER=C3
XDG_CURRENT_DESKTOP=Unity
LESSCLOSE=/usr/bin/lesspipe %s %s
LC_TIME=fi_FI.UTF-8
JPK_HYPRE_S=/mpi3/S3/hypre-2.8.0b
JPK_MPICH2_S=/mpi3/S3/mpich-3.0.4
COLORTERM=gnome-terminal
XAUTHORITY=/home/joni/.Xauthority
LC_NAME=fi_FI.UTF-8
_=/mpi3/C3/mpich-3.0.4/bin/mpiexec
OLDPWD=/home/joni
Hydra internal environment:
---------------------------
MPICH_ENABLE_CKPOINT=1
GFORTRAN_UNBUFFERED_PRECONNECTED=y
Proxy information:
*********************
[1] proxy: 192.168.0.41 (2 cores)Exec list: hostname (2 processes);
[2] proxy: 192.168.0.42 (2 cores)
[3] proxy: 192.168.0.43 (2 cores)
Exec list: hostname (2 processes);--version 3.0.4 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME --hostname 192.168.0.41 --global-core-map 0,2,6 --pmi-id-map 0,0 --global-process-count 6 --auto-cleanup 1 --pmi-kvsname kvs_12243_0 --pmi-process-mapping (vector,(0,3,2)) --ckpoint-prefix /mpi3/chekpoint/default.chk --ckpoint-num -1 --global-inherited-env 121 'MUMPS=/mpi3/S3/MUMPS_4.10.0' 'LC_PAPER=fi_FI.UTF-8' 'LC_ADDRESS=fi_FI.UTF-8' 'SSH_AGENT_PID=12144' 'LC_MONETARY=fi_FI.UTF-8' 'MUMPS_I=/mpi3/C3/MUMPS_4.10.0' 'HYDRA_DEMUX=select' 'GPG_AGENT_INFO=/tmp/keyring-kJwpJQ/gpg:0:1' 'JPK_LMETISDIR_S5=/mpi3/S3/parmetis-4.0.2' 'TERM=xterm' 'SHELL=/bin/bash' 'XDG_SESSION_COOKIE=6d6390cb56a32b6678c10da600000412-1377606907.629665-1922379047' 'FFT=/mpi3/C3/fftw2' 'HYDRA_ENV=all' 'JPK_NETGEN=/mpi3/C3/netgen_668' 'JPK_VER_S=S3' 'HYDRA_CKPOINTLIB=blcr' 'HYDRA_CKPOINT_INTERVAL=10800' 'WINDOWID=54602522' 'LC_NUMERIC=fi_FI.UTF-8' 'HYDRA_CKPOINT_PREFIX=/mpi3/chekpoint/default.chk' 'GNOME_KEYRING_CONTROL=/tmp/keyring-kJwpJQ' 'JPK_ELMER=/mpi3/C3/elmer_6283' 'PARDISO_LIC_PATH=/mpi3/C3/pardiso' 'METIS_INCLUDE_DIR=/mpi3/C3/ParMetis-3.2.0' 'JPK_NETGEN_S=/mpi3/S3/netgen_668' 'USER=joni' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:' 'JPK_TOGL=/mpi3/C3/Togl-1.7' 'LD_LIBRARY_PATH=/mpi3/C3/mpich-3.0.4/lib:/mpi3/C3/mpich-3.0.4/bin:/mpi3/C3/blcr-0.8.5/lib:/mpi3/C3/ParMetis-3.2.0:/mpi3/C3/acml5.3.1/gfortran64_mp/lib:/lib:/mpi3/C3/hypre-2.8.0b/lib:/mpi3/C3/scotch_6.0.0/lib:/mpi3/S3/MUMPS_4.10.0/lib:/mpi3/C3/acml5.3.1/gfortran64_mp/lib:/mpi3/C3/scalapack-2.0.2/lib:/mpi3/C3/hypre-2.8.0b/lib:/mpi3/C3/pardiso:/mpi3/C3/ParMetis-3.2.0:/mpi3/C3/ARPACK:/mpi3/C3/hdf5-1.8.10-patch1/lib:/mpi3/C3/VTK-5.8.0/lib/vtk-5.8:/mpi3/C3/elmer_6283/lib:/mpi3/C3/Togl-1.7:/mpi3/C3/netgen_668/lib:/usr/lib/:/usr/local/lib://mpi3/C3/vrpn/lib://mpi3/C3/hidapi/lib:/usr/include/libusb-1.0' 'LC_TELEPHONE=fi_FI.UTF-8' 'XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0' 'JPK_OCC=/usr/include/oce' 'XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0' 'HYDRA_HOST_FILE=/mpi4/hosts' 'SSH_AUTH_SOCK=/tmp/ssh-NnhxNTH12143/agent.12143' 'SCOTCHDIR=/mpi3/C3/scotch_6.0.0' 'HYDRA_LAUNCHER=rsh' 'JPK_VER_B=B3' 'SESSION_MANAGER=local/mpi1:@/tmp/.ICE-unix/4284,unix/mpi1:/tmp/.ICE-unix/4284' 'DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path' 'ELMER_HOME=/mpi3/C3/elmer_6283' 'BLACS=/mpi3/C3/scalapack-2.0.2' 'BLAS32=/mpi3/C3/acml5.3.1/gfortran64_mp' 'METIS_DIR=' 'MPI_LIBS=-L/mpi3/C3/mpich-3.0.4/lib -lmpich -lmpichf90 -lmpl -lopa -lmpichcxx' 'XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg' 'JPK_MPI_DIR=/mpi3' 'JPK_HDF5_S=/mpi3/S3/hdf5-1.8.10-patch1' 'MPIEXEC_PORT_RANGE=7000:7500' 'PATH=/mpi3/C3/cmake-2.8.10.2/bin:/mpi3/C3/blcr-0.8.5/bin:/mpi3/C3/mpich-3.0.4/bin:/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/mpi3/C3/elmer_6283/bin:/mpi3/C3/elmer_6283/lib:/mpi3/C3/ParaView3' 'DESKTOP_SESSION=ubuntu' 'BLAS=/mpi3/C3/acml5.3.1/gfortran64_mp' 'METIS_LIBDIR=/mpi3/C3/ParMetis-3.2.0' 'CMAKE_COMMAND=/mpi3/C3/cmake-2.8.10.2/bin' 'QT_QMAKE_EXECUTABLE=/usr/bin/qmake-qt4' 'LC_IDENTIFICATION=fi_FI.UTF-8' 'JPK_SCOTCHDIR_S=/mpi3/S3/scotch_6.0.0_esmumps' 'JPK_LMETISDIR_S=/mpi3/S3/ParMetis-3.2.0' 'PWD=/mpi3/S3/mpich-3.0.4' 'NETGENDIR=/mpi3/C3/netgen_668/bin' 'EDITOR=nano' 'JPK_LMETISDIR=/mpi3/C3/ParMetis-3.2.0' 'GNOME_KEYRING_PID=4273' 'LANG=fi_FI.UTF-8' 'MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path' 'OMPI_77=//mpi3/C3/mpich-3.0.4/bin/mpif77' 'LC_MEASUREMENT=fi_FI.UTF-8' 'JPK_HDF5=/mpi3/C3/hdf5-1.8.10-patch1' 'UBUNTU_MENUPROXY=libappmenu.so' 'COMPIZ_CONFIG_PROFILE=ubuntu' 'ELMER_POST_HOME=/mpi3/C3/elmer_6283/bin' 'JPK_INS=/mpi3/C3' 'ELMER_LIB=/mpi3/C3/elmer_6283/share/elmersolver/lib' 'HYDRA_PROXY_RETRY_COUNT=3' 'GDMSESSION=ubuntu' 'JPK_ELMER_S=/mpi3/S3/elmer_6283' 'JPK_LMETISDIR5=/mpi3/C3/parmetis-4.0.2' 'JPK_LMETISDIR32=/mpi3/C3/ParMetis-3.2.0' 'HYDRA_DEBUG=1' 'JPK_BUI=/mpi3/S3' 'VTK_INCLUDEPATH=/mpi3/C3/VTK-5.8.0/include' 'SHLVL=1' 'HOME=/home/joni' 'OMPI_CC=//mpi3/C3/mpich-3.0.4/bin/mpicc' 'LANGUAGE=fi:en' 'OMPI_90=//mpi3/C3/mpich-3.0.4/bin/mpif90' 'ELMERGUI_HOME=/mpi3/C3/elmer_6283/bin' 'GNOME_DESKTOP_SESSION_ID=this-is-deprecated' 'MPI_IMPLEMENTATION=mpich' 'MKL_SERIAL=YES' 'LOGNAME=joni' 'HYPRE=/mpi3/C3/hypre-2.8.0b' 'JPK_ARPACK_S=/mpi3/S3/ARPACK' 'JPK_JOBS=7' 'JPK_VTK_DIR=/mpi3/C3/VTK-5.8.0' 'SCALAPACK=/mpi3/C3/scalapack-2.0.2' 'XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/' 'DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-gAYmjGaklf,guid=26c3f15a7a5ee55e8782415700000034' 'JPK_ARPACK=/mpi3/C3/ARPACK' 'MPI_HOME=/mpi3/C3/mpich-3.0.4' 'LESSOPEN=| /usr/bin/lesspipe %s' 'LACPACK=/mpi3/C3/acml5.3.1/gfortran64_mp' 'OMPI_CXX=//mpi3/C3/mpich-3.0.4/bin/mpicxx' 'OMP_NUM_THREADS=6' 'JPK_TOGL_S=/mpi3/S3/Togl-1.7' 'HYDRA_LAUNCHER_EXEC=/usr/bin/rsh-redone-rsh' 'JPK_MPICH2=/mpi3/C3/mpich-3.0.4' 'PARDISO=/mpi3/C3/pardiso' 'PARDISOLICMESSAGE=1' 'JPK_VER=C3' 'XDG_CURRENT_DESKTOP=Unity' 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'LC_TIME=fi_FI.UTF-8' 'JPK_HYPRE_S=/mpi3/S3/hypre-2.8.0b' 'JPK_MPICH2_S=/mpi3/S3/mpich-3.0.4' 'COLORTERM=gnome-terminal' 'XAUTHORITY=/home/joni/.Xauthority' 'LC_NAME=fi_FI.UTF-8' '_=/mpi3/C3/mpich-3.0.4/bin/mpiexec' 'OLDPWD=/home/joni' --global-user-env 0 --global-system-env 2 'MPICH_ENABLE_CKPOINT=1' 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --genv-prop all --proxy-core-count 2 --exec --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir /mpi3/S3/mpich-3.0.4 --exec-args 1 hostname
==================================================================================================
[mpiexec@mpi1] Timeout set to -1 (-1 means infinite)
[mpiexec@mpi1] Got a control port string of 192.168.0.41:7001
Proxy launch args: /mpi3/C3/mpich-3.0.4/bin/hydra_pmi_proxy --control-port 192.168.0.41:7001 --debug --rmk user --launcher rsh --launcher-exec /usr/bin/rsh-redone-rsh --demux poll --pgid 0 --retries 3 --usize -2 --proxy-id
Arguments being passed to proxy 0:
--version 3.0.4 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME --hostname 192.168.0.42 --global-core-map 0,2,6 --pmi-id-map 0,2 --global-process-count 6 --auto-cleanup 1 --pmi-kvsname kvs_12243_0 --pmi-process-mapping (vector,(0,3,2)) --ckpoint-prefix /mpi3/chekpoint/default.chk --ckpoint-num -1 --global-inherited-env 121 'MUMPS=/mpi3/S3/MUMPS_4.10.0' 'LC_PAPER=fi_FI.UTF-8' 'LC_ADDRESS=fi_FI.UTF-8' 'SSH_AGENT_PID=12144' 'LC_MONETARY=fi_FI.UTF-8' 'MUMPS_I=/mpi3/C3/MUMPS_4.10.0' 'HYDRA_DEMUX=select' 'GPG_AGENT_INFO=/tmp/keyring-kJwpJQ/gpg:0:1' 'JPK_LMETISDIR_S5=/mpi3/S3/parmetis-4.0.2' 'TERM=xterm' 'SHELL=/bin/bash' 'XDG_SESSION_COOKIE=6d6390cb56a32b6678c10da600000412-1377606907.629665-1922379047' 'FFT=/mpi3/C3/fftw2' 'HYDRA_ENV=all' 'JPK_NETGEN=/mpi3/C3/netgen_668' 'JPK_VER_S=S3' 'HYDRA_CKPOINTLIB=blcr' 'HYDRA_CKPOINT_INTERVAL=10800' 'WINDOWID=54602522' 'LC_NUMERIC=fi_FI.UTF-8' 'HYDRA_CKPOINT_PREFIX=/mpi3/chekpoint/default.chk' 'GNOME_KEYRING_CONTROL=/tmp/keyring-kJwpJQ' 'JPK_ELMER=/mpi3/C3/elmer_6283' 'PARDISO_LIC_PATH=/mpi3/C3/pardiso' 'METIS_INCLUDE_DIR=/mpi3/C3/ParMetis-3.2.0' 'JPK_NETGEN_S=/mpi3/S3/netgen_668' 'USER=joni' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:' 'JPK_TOGL=/mpi3/C3/Togl-1.7' 'LD_LIBRARY_PATH=/mpi3/C3/mpich-3.0.4/lib:/mpi3/C3/mpich-3.0.4/bin:/mpi3/C3/blcr-0.8.5/lib:/mpi3/C3/ParMetis-3.2.0:/mpi3/C3/acml5.3.1/gfortran64_mp/lib:/lib:/mpi3/C3/hypre-2.8.0b/lib:/mpi3/C3/scotch_6.0.0/lib:/mpi3/S3/MUMPS_4.10.0/lib:/mpi3/C3/acml5.3.1/gfortran64_mp/lib:/mpi3/C3/scalapack-2.0.2/lib:/mpi3/C3/hypre-2.8.0b/lib:/mpi3/C3/pardiso:/mpi3/C3/ParMetis-3.2.0:/mpi3/C3/ARPACK:/mpi3/C3/hdf5-1.8.10-patch1/lib:/mpi3/C3/VTK-5.8.0/lib/vtk-5.8:/mpi3/C3/elmer_6283/lib:/mpi3/C3/Togl-1.7:/mpi3/C3/netgen_668/lib:/usr/lib/:/usr/local/lib://mpi3/C3/vrpn/lib://mpi3/C3/hidapi/lib:/usr/include/libusb-1.0' 'LC_TELEPHONE=fi_FI.UTF-8' 'XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0' 'JPK_OCC=/usr/include/oce' 'XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0' 'HYDRA_HOST_FILE=/mpi4/hosts' 'SSH_AUTH_SOCK=/tmp/ssh-NnhxNTH12143/agent.12143' 'SCOTCHDIR=/mpi3/C3/scotch_6.0.0' 'HYDRA_LAUNCHER=rsh' 'JPK_VER_B=B3' 'SESSION_MANAGER=local/mpi1:@/tmp/.ICE-unix/4284,unix/mpi1:/tmp/.ICE-unix/4284' 'DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path' 'ELMER_HOME=/mpi3/C3/elmer_6283' 'BLACS=/mpi3/C3/scalapack-2.0.2' 'BLAS32=/mpi3/C3/acml5.3.1/gfortran64_mp' 'METIS_DIR=' 'MPI_LIBS=-L/mpi3/C3/mpich-3.0.4/lib -lmpich -lmpichf90 -lmpl -lopa -lmpichcxx' 'XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg' 'JPK_MPI_DIR=/mpi3' 'JPK_HDF5_S=/mpi3/S3/hdf5-1.8.10-patch1' 'MPIEXEC_PORT_RANGE=7000:7500' 'PATH=/mpi3/C3/cmake-2.8.10.2/bin:/mpi3/C3/blcr-0.8.5/bin:/mpi3/C3/mpich-3.0.4/bin:/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/mpi3/C3/elmer_6283/bin:/mpi3/C3/elmer_6283/lib:/mpi3/C3/ParaView3' 'DESKTOP_SESSION=ubuntu' 'BLAS=/mpi3/C3/acml5.3.1/gfortran64_mp' 'METIS_LIBDIR=/mpi3/C3/ParMetis-3.2.0' 'CMAKE_COMMAND=/mpi3/C3/cmake-2.8.10.2/bin' 'QT_QMAKE_EXECUTABLE=/usr/bin/qmake-qt4' 'LC_IDENTIFICATION=fi_FI.UTF-8' 'JPK_SCOTCHDIR_S=/mpi3/S3/scotch_6.0.0_esmumps' 'JPK_LMETISDIR_S=/mpi3/S3/ParMetis-3.2.0' 'PWD=/mpi3/S3/mpich-3.0.4' 'NETGENDIR=/mpi3/C3/netgen_668/bin' 'EDITOR=nano' 'JPK_LMETISDIR=/mpi3/C3/ParMetis-3.2.0' 'GNOME_KEYRING_PID=4273' 'LANG=fi_FI.UTF-8' 'MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path' 'OMPI_77=//mpi3/C3/mpich-3.0.4/bin/mpif77' 'LC_MEASUREMENT=fi_FI.UTF-8' 'JPK_HDF5=/mpi3/C3/hdf5-1.8.10-patch1' 'UBUNTU_MENUPROXY=libappmenu.so' 'COMPIZ_CONFIG_PROFILE=ubuntu' 'ELMER_POST_HOME=/mpi3/C3/elmer_6283/bin' 'JPK_INS=/mpi3/C3' 'ELMER_LIB=/mpi3/C3/elmer_6283/share/elmersolver/lib' 'HYDRA_PROXY_RETRY_COUNT=3' 'GDMSESSION=ubuntu' 'JPK_ELMER_S=/mpi3/S3/elmer_6283' 'JPK_LMETISDIR5=/mpi3/C3/parmetis-4.0.2' 'JPK_LMETISDIR32=/mpi3/C3/ParMetis-3.2.0' 'HYDRA_DEBUG=1' 'JPK_BUI=/mpi3/S3' 'VTK_INCLUDEPATH=/mpi3/C3/VTK-5.8.0/include' 'SHLVL=1' 'HOME=/home/joni' 'OMPI_CC=//mpi3/C3/mpich-3.0.4/bin/mpicc' 'LANGUAGE=fi:en' 'OMPI_90=//mpi3/C3/mpich-3.0.4/bin/mpif90' 'ELMERGUI_HOME=/mpi3/C3/elmer_6283/bin' 'GNOME_DESKTOP_SESSION_ID=this-is-deprecated' 'MPI_IMPLEMENTATION=mpich' 'MKL_SERIAL=YES' 'LOGNAME=joni' 'HYPRE=/mpi3/C3/hypre-2.8.0b' 'JPK_ARPACK_S=/mpi3/S3/ARPACK' 'JPK_JOBS=7' 'JPK_VTK_DIR=/mpi3/C3/VTK-5.8.0' 'SCALAPACK=/mpi3/C3/scalapack-2.0.2' 'XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/' 'DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-gAYmjGaklf,guid=26c3f15a7a5ee55e8782415700000034' 'JPK_ARPACK=/mpi3/C3/ARPACK' 'MPI_HOME=/mpi3/C3/mpich-3.0.4' 'LESSOPEN=| /usr/bin/lesspipe %s' 'LACPACK=/mpi3/C3/acml5.3.1/gfortran64_mp' 'OMPI_CXX=//mpi3/C3/mpich-3.0.4/bin/mpicxx' 'OMP_NUM_THREADS=6' 'JPK_TOGL_S=/mpi3/S3/Togl-1.7' 'HYDRA_LAUNCHER_EXEC=/usr/bin/rsh-redone-rsh' 'JPK_MPICH2=/mpi3/C3/mpich-3.0.4' 'PARDISO=/mpi3/C3/pardiso' 'PARDISOLICMESSAGE=1' 'JPK_VER=C3' 'XDG_CURRENT_DESKTOP=Unity' 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'LC_TIME=fi_FI.UTF-8' 'JPK_HYPRE_S=/mpi3/S3/hypre-2.8.0b' 'JPK_MPICH2_S=/mpi3/S3/mpich-3.0.4' 'COLORTERM=gnome-terminal' 'XAUTHORITY=/home/joni/.Xauthority' 'LC_NAME=fi_FI.UTF-8' '_=/mpi3/C3/mpich-3.0.4/bin/mpiexec' 'OLDPWD=/home/joni' --global-user-env 0 --global-system-env 2 'MPICH_ENABLE_CKPOINT=1' 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --genv-prop all --proxy-core-count 2 --exec --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir /mpi3/S3/mpich-3.0.4 --exec-args 1 hostname
Arguments being passed to proxy 1:
--version 3.0.4 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME --hostname 192.168.0.43 --global-core-map 0,2,6 --pmi-id-map 0,4 --global-process-count 6 --auto-cleanup 1 --pmi-kvsname kvs_12243_0 --pmi-process-mapping (vector,(0,3,2)) --ckpoint-prefix /mpi3/chekpoint/default.chk --ckpoint-num -1 --global-inherited-env 121 'MUMPS=/mpi3/S3/MUMPS_4.10.0' 'LC_PAPER=fi_FI.UTF-8' 'LC_ADDRESS=fi_FI.UTF-8' 'SSH_AGENT_PID=12144' 'LC_MONETARY=fi_FI.UTF-8' 'MUMPS_I=/mpi3/C3/MUMPS_4.10.0' 'HYDRA_DEMUX=select' 'GPG_AGENT_INFO=/tmp/keyring-kJwpJQ/gpg:0:1' 'JPK_LMETISDIR_S5=/mpi3/S3/parmetis-4.0.2' 'TERM=xterm' 'SHELL=/bin/bash' 'XDG_SESSION_COOKIE=6d6390cb56a32b6678c10da600000412-1377606907.629665-1922379047' 'FFT=/mpi3/C3/fftw2' 'HYDRA_ENV=all' 'JPK_NETGEN=/mpi3/C3/netgen_668' 'JPK_VER_S=S3' 'HYDRA_CKPOINTLIB=blcr' 'HYDRA_CKPOINT_INTERVAL=10800' 'WINDOWID=54602522' 'LC_NUMERIC=fi_FI.UTF-8' 'HYDRA_CKPOINT_PREFIX=/mpi3/chekpoint/default.chk' 'GNOME_KEYRING_CONTROL=/tmp/keyring-kJwpJQ' 'JPK_ELMER=/mpi3/C3/elmer_6283' 'PARDISO_LIC_PATH=/mpi3/C3/pardiso' 'METIS_INCLUDE_DIR=/mpi3/C3/ParMetis-3.2.0' 'JPK_NETGEN_S=/mpi3/S3/netgen_668' 'USER=joni' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:' 'JPK_TOGL=/mpi3/C3/Togl-1.7' 'LD_LIBRARY_PATH=/mpi3/C3/mpich-3.0.4/lib:/mpi3/C3/mpich-3.0.4/bin:/mpi3/C3/blcr-0.8.5/lib:/mpi3/C3/ParMetis-3.2.0:/mpi3/C3/acml5.3.1/gfortran64_mp/lib:/lib:/mpi3/C3/hypre-2.8.0b/lib:/mpi3/C3/scotch_6.0.0/lib:/mpi3/S3/MUMPS_4.10.0/lib:/mpi3/C3/acml5.3.1/gfortran64_mp/lib:/mpi3/C3/scalapack-2.0.2/lib:/mpi3/C3/hypre-2.8.0b/lib:/mpi3/C3/pardiso:/mpi3/C3/ParMetis-3.2.0:/mpi3/C3/ARPACK:/mpi3/C3/hdf5-1.8.10-patch1/lib:/mpi3/C3/VTK-5.8.0/lib/vtk-5.8:/mpi3/C3/elmer_6283/lib:/mpi3/C3/Togl-1.7:/mpi3/C3/netgen_668/lib:/usr/lib/:/usr/local/lib://mpi3/C3/vrpn/lib://mpi3/C3/hidapi/lib:/usr/include/libusb-1.0' 'LC_TELEPHONE=fi_FI.UTF-8' 'XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0' 'JPK_OCC=/usr/include/oce' 'XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0' 'HYDRA_HOST_FILE=/mpi4/hosts' 'SSH_AUTH_SOCK=/tmp/ssh-NnhxNTH12143/agent.12143' 'SCOTCHDIR=/mpi3/C3/scotch_6.0.0' 'HYDRA_LAUNCHER=rsh' 'JPK_VER_B=B3' 'SESSION_MANAGER=local/mpi1:@/tmp/.ICE-unix/4284,unix/mpi1:/tmp/.ICE-unix/4284' 'DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path' 'ELMER_HOME=/mpi3/C3/elmer_6283' 'BLACS=/mpi3/C3/scalapack-2.0.2' 'BLAS32=/mpi3/C3/acml5.3.1/gfortran64_mp' 'METIS_DIR=' 'MPI_LIBS=-L/mpi3/C3/mpich-3.0.4/lib -lmpich -lmpichf90 -lmpl -lopa -lmpichcxx' 'XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg' 'JPK_MPI_DIR=/mpi3' 'JPK_HDF5_S=/mpi3/S3/hdf5-1.8.10-patch1' 'MPIEXEC_PORT_RANGE=7000:7500' 'PATH=/mpi3/C3/cmake-2.8.10.2/bin:/mpi3/C3/blcr-0.8.5/bin:/mpi3/C3/mpich-3.0.4/bin:/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/mpi3/C3/elmer_6283/bin:/mpi3/C3/elmer_6283/lib:/mpi3/C3/ParaView3' 'DESKTOP_SESSION=ubuntu' 'BLAS=/mpi3/C3/acml5.3.1/gfortran64_mp' 'METIS_LIBDIR=/mpi3/C3/ParMetis-3.2.0' 'CMAKE_COMMAND=/mpi3/C3/cmake-2.8.10.2/bin' 'QT_QMAKE_EXECUTABLE=/usr/bin/qmake-qt4' 'LC_IDENTIFICATION=fi_FI.UTF-8' 'JPK_SCOTCHDIR_S=/mpi3/S3/scotch_6.0.0_esmumps' 'JPK_LMETISDIR_S=/mpi3/S3/ParMetis-3.2.0' 'PWD=/mpi3/S3/mpich-3.0.4' 'NETGENDIR=/mpi3/C3/netgen_668/bin' 'EDITOR=nano' 'JPK_LMETISDIR=/mpi3/C3/ParMetis-3.2.0' 'GNOME_KEYRING_PID=4273' 'LANG=fi_FI.UTF-8' 'MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path' 'OMPI_77=//mpi3/C3/mpich-3.0.4/bin/mpif77' 'LC_MEASUREMENT=fi_FI.UTF-8' 'JPK_HDF5=/mpi3/C3/hdf5-1.8.10-patch1' 'UBUNTU_MENUPROXY=libappmenu.so' 'COMPIZ_CONFIG_PROFILE=ubuntu' 'ELMER_POST_HOME=/mpi3/C3/elmer_6283/bin' 'JPK_INS=/mpi3/C3' 'ELMER_LIB=/mpi3/C3/elmer_6283/share/elmersolver/lib' 'HYDRA_PROXY_RETRY_COUNT=3' 'GDMSESSION=ubuntu' 'JPK_ELMER_S=/mpi3/S3/elmer_6283' 'JPK_LMETISDIR5=/mpi3/C3/parmetis-4.0.2' 'JPK_LMETISDIR32=/mpi3/C3/ParMetis-3.2.0' 'HYDRA_DEBUG=1' 'JPK_BUI=/mpi3/S3' 'VTK_INCLUDEPATH=/mpi3/C3/VTK-5.8.0/include' 'SHLVL=1' 'HOME=/home/joni' 'OMPI_CC=//mpi3/C3/mpich-3.0.4/bin/mpicc' 'LANGUAGE=fi:en' 'OMPI_90=//mpi3/C3/mpich-3.0.4/bin/mpif90' 'ELMERGUI_HOME=/mpi3/C3/elmer_6283/bin' 'GNOME_DESKTOP_SESSION_ID=this-is-deprecated' 'MPI_IMPLEMENTATION=mpich' 'MKL_SERIAL=YES' 'LOGNAME=joni' 'HYPRE=/mpi3/C3/hypre-2.8.0b' 'JPK_ARPACK_S=/mpi3/S3/ARPACK' 'JPK_JOBS=7' 'JPK_VTK_DIR=/mpi3/C3/VTK-5.8.0' 'SCALAPACK=/mpi3/C3/scalapack-2.0.2' 'XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/' 'DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-gAYmjGaklf,guid=26c3f15a7a5ee55e8782415700000034' 'JPK_ARPACK=/mpi3/C3/ARPACK' 'MPI_HOME=/mpi3/C3/mpich-3.0.4' 'LESSOPEN=| /usr/bin/lesspipe %s' 'LACPACK=/mpi3/C3/acml5.3.1/gfortran64_mp' 'OMPI_CXX=//mpi3/C3/mpich-3.0.4/bin/mpicxx' 'OMP_NUM_THREADS=6' 'JPK_TOGL_S=/mpi3/S3/Togl-1.7' 'HYDRA_LAUNCHER_EXEC=/usr/bin/rsh-redone-rsh' 'JPK_MPICH2=/mpi3/C3/mpich-3.0.4' 'PARDISO=/mpi3/C3/pardiso' 'PARDISOLICMESSAGE=1' 'JPK_VER=C3' 'XDG_CURRENT_DESKTOP=Unity' 'LESSCLOSE=/usr/bin/lesspipe %s %s' 'LC_TIME=fi_FI.UTF-8' 'JPK_HYPRE_S=/mpi3/S3/hypre-2.8.0b' 'JPK_MPICH2_S=/mpi3/S3/mpich-3.0.4' 'COLORTERM=gnome-terminal' 'XAUTHORITY=/home/joni/.Xauthority' 'LC_NAME=fi_FI.UTF-8' '_=/mpi3/C3/mpich-3.0.4/bin/mpiexec' 'OLDPWD=/home/joni' --global-user-env 0 --global-system-env 2 'MPICH_ENABLE_CKPOINT=1' 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --genv-prop all --proxy-core-count 2 --exec --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir /mpi3/S3/mpich-3.0.4 --exec-args 1 hostname
Arguments being passed to proxy 2:
[mpiexec@mpi1] Launch arguments: /usr/bin/rsh-redone-rsh 192.168.0.43 "/mpi3/C3/mpich-3.0.4/bin/hydra_pmi_proxy" --control-port 192.168.0.41:7001 --debug --rmk user --launcher rsh --launcher-exec /usr/bin/rsh-redone-rsh --demux poll --pgid 0 --retries 3 --usize -2 --proxy-id 2
[mpiexec@mpi1] Launch arguments: /mpi3/C3/mpich-3.0.4/bin/hydra_pmi_proxy --control-port 192.168.0.41:7001 --debug --rmk user --launcher rsh --launcher-exec /usr/bin/rsh-redone-rsh --demux poll --pgid 0 --retries 3 --usize -2 --proxy-id 0
[mpiexec@mpi1] Launch arguments: /usr/bin/rsh-redone-rsh 192.168.0.42 "/mpi3/C3/mpich-3.0.4/bin/hydra_pmi_proxy" --control-port 192.168.0.41:7001 --debug --rmk user --launcher rsh --launcher-exec /usr/bin/rsh-redone-rsh --demux poll --pgid 0 --retries 3 --usize -2 --proxy-id 1
mpi1
m--2013/8/27 Pavan Balaji <balaji@mcs.anl.gov>
Please don't drop discuss@mpich.org from the cc list.
I doubt demux, --assert-level and blcr are relevant here. Also, the output of "make testing" is not helpful for us because those tests can fail even if your machines are too slow.
Did you try my suggestion from the previous email? Could you try them and report back (just with that information)?
-- Pavan
On 08/27/2013 09:35 AM, Joni-Pekka Kurronen wrote:
192.168.0.41:2 <http://192.168.0.41:2>
hi,
I have allraedy cheked up.
This is not new install, just jumped up from mpich2 to 3 and
gcc 4.6 to gcc 4.7.
I have rsh-redone-rsh as main but tested ssh as well,...
due at crash clinet's keep running at slave's and useing harddisk,...
Following work's:
any machine alone
mpi1 and kaak or mpi1 and ugh
but not all to gether except hostname,...
This could be realated:
- demux ( have tried select and poll, whit poll have to restart slave
machine's)
- nfs4 ( some reason nfs4 must be manually mounted at moment after
restart at slave's)
- have changed --assert-level to 0 (default 2)
- blcr
ch3:socket setting:
=============
hosts file,..
192.168.0.42:2 <http://192.168.0.42:2>
#192.168.0.43:2 <http://192.168.0.43:2>192.168.0.41:2 <http://192.168.0.41:2>
=============
summary.xml errors
<MPITEST>
<NAME>spawninfo1</NAME>
<NP>1</NP>
<WORKDIR>./spawn</WORKDIR>
<STATUS>fail</STATUS>
<TESTDIFF>
[mpiexec@mpi1] APPLICATION TIMED OUT
[proxy:0:0@mpi1] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed
[proxy:0:0@mpi1] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:0@mpi1] main (./pm/pmiserv/pmip.c:206): demux engine error
waiting for event
[proxy:1:0@mpi1] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed
[proxy:1:0@mpi1] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[proxy:1:0@mpi1] main (./pm/pmiserv/pmip.c:206): demux engine error
waiting for event
[mpiexec@mpi1] HYDT_bscu_wait_for_completion
(./tools/bootstrap/utils/bscu_wait.c:76): one of the processes
terminated badly; aborting
[mpiexec@mpi1] HYDT_bsci_wait_for_completion
(./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting
for completion
[mpiexec@mpi1] HYD_pmci_wait_for_completion
(./pm/pmiserv/pmiserv_pmci.c:188): launcher returned error waiting for
completion
[mpiexec@mpi1] main (./ui/mpich/mpiexec.c:331): process manager error
waiting for completion
</TESTDIFF>
</MPITEST>
<MPITEST>
<NAME>rdwrord</NAME>
<NP>4</NP>
<WORKDIR>./io</WORKDIR>
<STATUS>fail</STATUS>
<TESTDIFF>
Fatal error in PMPI_Bcast: Other MPI error
[cli_2]: aborting job:
Fatal error in PMPI_Bcast: Other MPI error
Fatal error in PMPI_Bcast: Other MPI error
[cli_0]: aborting job:
Fatal error in PMPI_Bcast: Other MPI error
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 1
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
</TESTDIFF>
</MPITEST>
<MPITEST>
<NAME>rdwrzero</NAME>
<NP>4</NP>
<WORKDIR>./io</WORKDIR>
<STATUS>fail</STATUS>
<TESTDIFF>
Fatal error in PMPI_Bcast: Other MPI error
[cli_2]: aborting job:
Fatal error in PMPI_Bcast: Other MPI error
Fatal error in PMPI_Bcast: Other MPI error
[cli_0]: aborting job:
Fatal error in PMPI_Bcast: Other MPI error
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 1
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
</TESTDIFF>
</MPITEST>
<MPITEST>
<NAME>getextent</NAME>
<NP>2</NP>
<WORKDIR>./io</WORKDIR>
<STATUS>pass</STATUS>
</MPITEST>
<MPITEST>
<NAME>setinfo</NAME>
<NP>4</NP>
<WORKDIR>./io</WORKDIR>
<STATUS>fail</STATUS>
<TESTDIFF>
Fatal error in PMPI_Bcast: Other MPI error
[cli_2]: aborting job:
Fatal error in PMPI_Bcast: Other MPI error
Fatal error in PMPI_Bcast: Other MPI error
[cli_0]: aborting job:
Fatal error in PMPI_Bcast: Other MPI error
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 1
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
</TESTDIFF>
</MPITEST>
<MPITEST>
<NAME>setviewcur</NAME>
<NP>4</NP>
<WORKDIR>./io</WORKDIR>
<STATUS>fail</STATUS>
<TESTDIFF>
Fatal error in PMPI_Bcast: Other MPI error
[cli_2]: aborting job:
Fatal error in PMPI_Bcast: Other MPI error
Fatal error in PMPI_Bcast: Other MPI error
[cli_0]: aborting job:
Fatal error in PMPI_Bcast: Other MPI error
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 1
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
</TESTDIFF>
</MPITEST>
....
....
....
============
============
hosts file,..
192.168.0.42:2 <http://192.168.0.42:2>
192.168.0.43:2 <http://192.168.0.43:2>#export WM_HOSTS="192.168.0.41:6 <http://192.168.0.41:6> 192.168.0.42:6
============
============
When needed more than 4 process will hang,..
Unexpected output in allred3: [mpiexec@mpi1] APPLICATION TIMED OUT
Unexpected output in allred3: [proxy:0:0@mpi1]
HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:886): assert
(!closed) failed
Unexpected output in allred3: [proxy:0:0@mpi1]
HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback
returned error status
Unexpected output in allred3: [proxy:0:0@mpi1] main
(./pm/pmiserv/pmip.c:206): demux engine error waiting for event
======
if useing ch3:nemesis hardisk is not running all the time as whit
sockets's,...
================================================
This is read by rsh-redone-rsh for every process,...
================================================
#!/bin/bash
# JPK-Integration for Ubuntu 12.4 LTS
#
https://sites.google.com/site/jpsdatareviewstheboy007/ubuntu-lts-12-4-companion-whit-ltsp-mpich2-elmer-openfoam
#
# CMAKE goes loop and can not build, cmake build under development says
documentation
# gcc 4.7
# bdver1 optimization
shopt -s expand_aliases
export JPK_MPI_DIR=/mpi3 # MAIN DIRECTORY, SUBDIRCTORYES:
export JPK_VER=C3 # BINARY CODE
export JPK_VER_S=S3 # SOURCE CODE
export JPK_VER_B=B3 # BASH FILES TO COMPILE AND CONFIGURE
export JPK_INS=$JPK_MPI_DIR/$JPK_VER
export JPK_BUI=$JPK_MPI_DIR/$JPK_VER_S
export JPK_ELMER=$JPK_INS/elmer_6283 #035
export JPK_ELMER_S=$JPK_BUI/elmer_6283
export JPK_NETGEN_S=$JPK_BUI/netgen_668
export JPK_NETGEN=$JPK_INS/netgen_668
#GCC
#export JPK_FLAGS="-Wl,--no-as-needed -fPIC -DAdd_ -m64 -pthread -O3
-fopenmp -lgomp -march=bdver1 -ftree-vectorize -funroll-loops"
#export CFLAGS="-Wl,--no-as-needed -fPIC -DAdd_ -m64 -pthread -fopenmp
-lgomp"
# M A K E
export JPK_JOBS=7
# O P E N MP
export OMP_NUM_THREADS=6
# M P I C 3
# http://wiki.mcs.anl.gov/mpich2/index.php/Using_the_Hydra_Process_Manager
export JPK_MPICH2_S=$JPK_BUI/mpich-3.0.4
export JPK_MPICH2=$JPK_INS/mpich-3.0.4
export PATH=$JPK_MPICH2/bin:$PATH
export MPI_HOME=$JPK_MPICH2
export MPI_LIBS="-L$JPK_MPICH2/lib -lmpich -lmpichf90 -lmpl -lopa
-lmpichcxx"
export LD_LIBRARY_PATH=$JPK_MPICH2/lib:$JPK_MPICH2/bin # FIRST
# M P I
export MPI_IMPLEMENTATION=mpich
export OMPI_CC=/$JPK_MPI_DIR/$JPK_VER/mpich-3.0.4/bin/mpicc
export OMPI_CXX=/$JPK_MPI_DIR/$JPK_VER/mpich-3.0.4/bin/mpicxx
export OMPI_77=/$JPK_MPI_DIR/$JPK_VER/mpich-3.0.4/bin/mpif77
export OMPI_90=/$JPK_MPI_DIR/$JPK_VER/mpich-3.0.4/bin/mpif90
# http://wiki.mcs.anl.gov/mpich2/index.php/Using_the_Hydra_Process_Manager
export HYDRA_DEBUG=0
export HYDRA_HOST_FILE=/mpi4/hosts
export HYDRA_LAUNCHER=rsh
#export HYDRA_LAUNCHER=ssh
#export HYDRA_LAUNCHER_EXEC=/usr/bin/netkit-rsh
export HYDRA_LAUNCHER_EXEC=/usr/bin/rsh-redone-rsh
#export HYDRA_LAUNCHER_EXEC=/usr/bin/ssh
export HYDRA_DEMUX=select
#export HYDRA_DEMUX=select #more porseses than core's
export HYDRA_PROXY_RETRY_COUNT=3
#export HYDRA_RMK=pbs
#export HYDRA_DEFAULT_RMK=pbs
export HYDRA_ENV=all
export MPIEXEC_PORT_RANGE=7000:7500
#mpirun -launcher rsh -launcher-exec /usr/bin/netkit-rsh -demux select
-n 21 ddd ./cpi
# b l c r
export HYDRA_CKPOINTLIB=blcr
export HYDRA_CKPOINT_PREFIX=/mpi3/chekpoint/default.chk
export HYDRA_CKPOINT_INTERVAL=10800
export PATH=$JPK_INS/blcr-0.8.5/bin:$PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JPK_INS/blcr-0.8.5/lib
#-ckpoint-num 5
# C M A K E - BUILD
export PATH=$JPK_INS/cmake-2.8.10.2/bin:$PATH
export CMAKE_COMMAND=$JPK_INS/cmake-2.8.10.2/bin
# T O G L - UI netgen
export JPK_TOGL="$JPK_INS/Togl-1.7"
export JPK_TOGL_S="$JPK_BUI/Togl-1.7"
# OCC
export JPK_OCC=/usr/include/oce
# M A T H
export JPK_ARPACK_S=$JPK_BUI/ARPACK
export JPK_ARPACK=$JPK_INS/ARPACK
export BLAS=$JPK_INS/acml5.3.1/gfortran64_mp
export BLAS32=$JPK_INS/acml5.3.1/gfortran64_mp
#export BLAS=$JPK_INS/clAmdBlas-1.10.321/lib64
#export BLAS32=$JPK_INS/clAmdBlas-1.10.321/include
export FFT=$JPK_INS/fftw2
export LACPACK=$BLAS
export SCALAPACK=$JPK_INS/scalapack-2.0.2
export BLACS=$SCALAPACK
export JPK_LMETISDIR_S=$JPK_BUI/ParMetis-3.2.0
export JPK_LMETISDIR=$JPK_INS/ParMetis-3.2.0
export JPK_LMETISDIR32=$JPK_LMETISDIR
export JPK_LMETISDIR_S5=$JPK_BUI/parmetis-4.0.2
export JPK_LMETISDIR5=$JPK_INS/parmetis-4.0.2
export METIS_DIR="" #$JPK_LMETISDIR MUST BE EMPTY
export METIS_INCLUDE_DIR=$JPK_LMETISDIR
export METIS_LIBDIR=$JPK_LMETISDIR
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JPK_LMETISDIR:$BLAS/lib:$FTT/lib
#/mpi4/S/metis-5.0.2/GKlib
export SCOTCHDIR=$JPK_INS/scotch_6.0.0
export JPK_SCOTCHDIR_S=$JPK_BUI/scotch_6.0.0_esmumps
export MUMPS_I=$JPK_INS/MUMPS_4.10.0
export MUMPS=$JPK_BUI/MUMPS_4.10.0
export HYPRE=$JPK_INS/hypre-2.8.0b
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HYPRE/lib
export JPK_HYPRE_S=$JPK_BUI/hypre-2.8.0b
export PARDISOLICMESSAGE=1
export PARDISO=$JPK_INS/pardiso
export PARDISO_LIC_PATH=$PARDISO
export MKL_SERIAL=YES
export
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$SCOTCHDIR/lib:$MUMPS/lib:$BLAS/lib:$SCALAPACK/lib:$HYPRE/lib:$PARDISO:$METIS_LIBDIR:$JPK_ARPACK
#HDF5
#export JPK_HDF5_S=$JPK_BUI/hdf5-1.8.10-patch1 for vtk testing
#export JPK_HDF5=$JPK_INS/hdf5-1.8.10-patch1
export JPK_HDF5_S=$JPK_BUI/hdf5-1.8.10-patch1
export JPK_HDF5=$JPK_INS/hdf5-1.8.10-patch1
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JPK_HDF5/lib
# V T K
export JPK_VTK_DIR=$JPK_INS/VTK-5.8.0
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JPK_VTK_DIR/lib/vtk-5.8
export VTK_INCLUDEPATH=$JPK_VTK_DIR/include
# Q T
export QT_QMAKE_EXECUTABLE=/usr/bin/qmake-qt4
# O P E N F O A M
# http://www.openfoam.org/download/source.php
#export WM_SCHEDULER=wmakeScheduler
<http://192.168.0.42:6> 192.168.0.43:6 <http://192.168.0.43:6>"2013/8/27 Pavan Balaji <balaji@mcs.anl.gov <mailto:balaji@mcs.anl.gov>>
#export WM_NCOMPPROCS=$($WM_SCHEDULER -count)
#export WM_COLOURS="black blue green cyan red magenta yellow"
#export FOAM_INST_DIR=/mpi2/OpenFOAM
#foamDotFile=$FOAM_INST_DIR/OpenFOAM-2.1.x/etc/bashrc
#[ -f $foamDotFile ] && . $foamDotFile
#source /mpi3/OpenFOAM/OpenFOAM-2.1.x/etc/bashrc
#export FOAM_RUN=/mpi2/om
#export OpenBIN=/mpi2/OpenFOAM/OpenFOAM-2.1.x/bin/tools
#export PATH=OpenBIN$:$PATH
#export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/apps/OpenFOAM/tools/lib64
#export
ParaView_DIR=/mpi2/OpenFOAM/ThirdParty-2.1.x/platforms/linux64Gcc/paraview-3.12.0
#export PATH=$ParaView_DIR/bin:$PATH
#export PV_PLUGIN_PATH=$FOAM_LIBBIN/paraview-3.12
# E L M E R
export ELMER_HOME=$JPK_ELMER
export ELMER_LIB=$JPK_ELMER/share/elmersolver/lib
export PATH=$PATH:$ELMER_HOME/bin:$ELMER_HOME/lib
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ELMER_HOME/lib
export ELMERGUI_HOME=$ELMER_HOME/bin
export ELMER_POST_HOME=$ELMER_HOME/bin
# S a l o m é
#cd /mpi2/salome-meca/SALOME-MECA-2012.2-LGPL ; source envSalomeMeca.sh
#cd ~/
# Paraview
#export PATH=$PATH:$JPK_INS/ParaView-3.14.1-Linux-64bit
export PATH=$PATH:$JPK_INS/ParaView3
# N E T G E N P A R A L L E L $JPK_TCL/lib:$JPK_TK/lib:
#export
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JPK_TOGL:$JPK_NETGEN\par/lib:/usr/lib/
#export NETGENDIR=$JPK_NETGEN\par/bin
# NETGEN
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JPK_TOGL:$JPK_NETGEN/lib:/usr/lib/
export NETGENDIR=$JPK_NETGEN/bin
#crontab, ext editor
export EDITOR=nano
#space ball
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
#vrpn & hidapi
export
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/$JPK_MPI_DIR/$JPK_VER/vrpn/lib:/$JPK_MPI_DIR/$JPK_VER/hidapi/lib:/usr/include/libusb-1.0
(./tools/demux/demux_poll.c:__77): callback returned error status
This is almost certainly a network issue with your third machine
(kaak, I presume?).
Thanks for making sure "hostname" works fine on all machines. That
means that your ssh connections are setup correctly. But a non-MPI
program, such as hostname, does not check the connection from kaak
back to mpi1.
Can you try a simple program like "examples/cpi" in the build
directory on all machines? Try it on 2 machines (mpiexec -np 4) and
3 machines (mpiexec -np 6).
If the third machine is in fact having problems running the application:
1. Make sure there's no firewall on the third machines.
2. Make sure the /etc/hosts file is consistent on both the machines
(mpi1 and kaak).
-- Pavan
On 08/27/2013 06:46 AM, Joni-Pekka Kurronen wrote:
I have:
-Ubuntu 12.4
-rsh-redo-rsh
-three machines
-mpich3
-have tried export HYDRA_DEMUX=select / poll
-have tried ssh/rsh
-have added to LIBS: event_core event_pthreads
I can run test at on to two machines whitout error but
when I take third machine to cluster demux engine goes mad,...
there is connection hanging,... and nothing happens,...
<MPITEST>
<NAME>uoplong</NAME>
<NP>11</NP>
<WORKDIR>./coll</WORKDIR>
<STATUS>fail</STATUS>
<TESTDIFF>
[mpiexec@mpi1] APPLICATION TIMED OUT
[proxy:0:0@mpi1] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed
[proxy:0:0@mpi1] HYDT_dmxu_poll_wait_for_event(./tools/bootstrap/utils/bscu___wait.c:76): one of the processes
[proxy:0:0@mpi1] main (./pm/pmiserv/pmip.c:206): demux engine error
waiting for event
[mpiexec@mpi1] HYDT_bscu_wait_for_completion(./tools/bootstrap/src/bsci___wait.c:23): launcher returned
terminated badly; aborting
[mpiexec@mpi1] HYDT_bsci_wait_for_completion(./pm/pmiserv/pmiserv_pmci.c:__188): launcher returned error
error waiting
for completion
[mpiexec@mpi1] HYD_pmci_wait_for_completionJoni.Kurronen@gmail.com <mailto:Joni.Kurronen@gmail.com>
waiting for
completion
[mpiexec@mpi1] main (./ui/mpich/mpiexec.c:331): process manager
error
waiting for completion
</TESTDIFF>
</MPITEST>
Also I can run
joni@mpi1:/mpi3/S3/hpcc-1.4.2$ mpiexec -np 6 hostname
mpi1
mpi1
ugh
ugh
kaak
kaak
but if I run
joni@mpi1:/mpi3/S3/hpcc-1.4.2$ mpiexec -np 6 ls
I get only one directory as output and
system will cease until I have re-started slave machines !
--
Pavan Balaji
http://www.mcs.anl.gov/~balaji
--
Joni-Pekka Kurronen
Joni-Pekka Kurronen