Re: [mpich-discuss] ./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed

28 Aug 2013

      hi,

I can get time to time 6 processs running and sometime it crash when 5
started,...
so takeing third machine, what so ever start's the problem,... it's question
communication between two slave's,... and  crash when two salves are
started to work together  mpirin -np 2 -hosts ugh,kaak ls

File rights should be  ok, ports are open, useing rsh,....

Communication, demux engine ?
Could PAM controll affect ?
Still wondering NFS4 ?

joni

hosts
======
mpi1:2
ugh:2
kaak:2

Below error's debug after crash
======================
mpiexec -np 5 hostname
....
[mpiexec@mpi1] Launch arguments: /mpi3/C3/mpich-3.0.4/bin/hydra_pmi_proxy
--control-port 192.168.0.41:7000 --debug --rmk user --launcher rsh
--launcher-exec /usr/bin/rsh-redone-rsh --demux poll --pgid 0 --retries 2
--usize -2 --proxy-id 0
[mpiexec@mpi1] Launch arguments: /usr/bin/rsh-redone-rsh 192.168.0.42
"/mpi3/C3/mpich-3.0.4/bin/hydra_pmi_proxy" --control-port
192.168.0.41:7000--debug --rmk user --launcher rsh --launcher-exec
/usr/bin/rsh-redone-rsh
--demux poll --pgid 0 --retries 2 --usize -2 --proxy-id 1
[mpiexec@mpi1] Launch arguments: /usr/bin/rsh-redone-rsh 192.168.0.43
"/mpi3/C3/mpich-3.0.4/bin/hydra_pmi_proxy" --control-port
192.168.0.41:7000--debug --rmk user --launcher rsh --launcher-exec
/usr/bin/rsh-redone-rsh
--demux poll --pgid 0 --retries 2 --usize -2 --proxy-id 2
mpi1
mpi1
^X^C[mpiexec@mpi1] Sending Ctrl-C to processes as requested
[mpiexec@mpi1] Press Ctrl-C again to force abort
[mpiexec@mpi1] HYDU_sock_write (./utils/sock/sock.c:291): write error (Bad
file descriptor)
[mpiexec@mpi1] HYD_pmcd_pmiserv_send_signal
(./pm/pmiserv/pmiserv_cb.c:170): unable to write data to proxy
[mpiexec@mpi1] ui_cmd_cb (./pm/pmiserv/pmiserv_pmci.c:79): unable to send
signal downstream
[mpiexec@mpi1] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:77): callback returned error status
[mpiexec@mpi1] HYD_pmci_wait_for_completion
(./pm/pmiserv/pmiserv_pmci.c:197): error waiting for event
[mpiexec@mpi1] main (./ui/mpich/mpiexec.c:331): process manager error
waiting for completion
joni@mpi1:~$ echo $HYDRA_DEMUX
select
joni@mpi1:~$ sudo killall rsh-redone-rsh
[sudo] password for joni:
rsh-redone-rsh: ei prosesseja
joni@mpi1:~$

Below blcr test after re-compile
======================
root@mpi1:/mpi3/S3/blcr-0.8.5# master kaak errors:

34 child 21967 completed
#ST_ALARM:60
035 child 22072 is READY (context=SIGNAL stopped=YES)
036 child 22072 is STOPped
!!! Alarm clock expired
!!! Missing final DONE
!!! Test killed unexpectedly by signal 9
FAIL: stopped.st

root@ug:/mpi3/S3/blcr-0.8.5# slave kaak errors:

root@kaak:/mpi3/S3/blcr-0.8.5# slave kaak errors:

make check

34 child 7472 completed
#ST_ALARM:60
035 child 7577 is READY (context=SIGNAL stopped=YES)
036 child 7577 is STOPped
!!! Alarm clock expired
!!! Missing final DONE
!!! Test killed unexpectedly by signal 9
FAIL: stopped.st
PASS: edeadlk.st

/mpi3/S3/blcr-0.8.5/tests/.libs/lt-filedescriptors[8051]: file
"filedescriptors.c", line 270, in check_stat_simple: File attributes
changed.  1 mismatches
/mpi3/S3/blcr-0.8.5/tests/.libs/lt-filedescriptors[8051]: file "crut.c",
line 615, in crut_main: test_restart() unexpectedly returned -1
restart/nonzeroexit (255)
FAIL: filedescriptors.ct

2013/8/27 Joni-Pekka Kurronen <[email protected]>
...
Another:
joni@mpi1:/mpi3/S3/mpich-3.0.4$ mpiexec -np 6 hostname
host: 192.168.0.41
host: 192.168.0.42
host: 192.168.0.43
==================================================================================================
mpiexec options:
----------------
  Base path: /mpi3/C3/mpich-3.0.4/bin/
  Launcher: (null)
  Debug level: 1
  Enable X: -1
Global environment:
  -------------------
MUMPS=/mpi3/S3/MUMPS_4.10.0
    LC_PAPER=fi_FI.UTF-8
    LC_ADDRESS=fi_FI.UTF-8
    SSH_AGENT_PID=12144
    LC_MONETARY=fi_FI.UTF-8
    MUMPS_I=/mpi3/C3/MUMPS_4.10.0
    HYDRA_DEMUX=select
    GPG_AGENT_INFO=/tmp/keyring-kJwpJQ/gpg:0:1
    JPK_LMETISDIR_S5=/mpi3/S3/parmetis-4.0.2
    TERM=xterm
    SHELL=/bin/bash
XDG_SESSION_COOKIE=6d6390cb56a32b6678c10da600000412-1377606907.629665-1922379047
    FFT=/mpi3/C3/fftw2
    HYDRA_ENV=all
    JPK_NETGEN=/mpi3/C3/netgen_668
    JPK_VER_S=S3
    HYDRA_CKPOINTLIB=blcr
    HYDRA_CKPOINT_INTERVAL=10800
    WINDOWID=54602522
    LC_NUMERIC=fi_FI.UTF-8
    HYDRA_CKPOINT_PREFIX=/mpi3/chekpoint/default.chk
    GNOME_KEYRING_CONTROL=/tmp/keyring-kJwpJQ
    JPK_ELMER=/mpi3/C3/elmer_6283
    PARDISO_LIC_PATH=/mpi3/C3/pardiso
    METIS_INCLUDE_DIR=/mpi3/C3/ParMetis-3.2.0
    JPK_NETGEN_S=/mpi3/S3/netgen_668
    USER=joni
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:
    JPK_TOGL=/mpi3/C3/Togl-1.7
LD_LIBRARY_PATH=/mpi3/C3/mpich-3.0.4/lib:/mpi3/C3/mpich-3.0.4/bin:/mpi3/C3/blcr-0.8.5/lib:/mpi3/C3/ParMetis-3.2.0:/mpi3/C3/acml5.3.1/gfortran64_mp/lib:/lib:/mpi3/C3/hypre-2.8.0b/lib:/mpi3/C3/scotch_6.0.0/lib:/mpi3/S3/MUMPS_4.10.0/lib:/mpi3/C3/acml5.3.1/gfortran64_mp/lib:/mpi3/C3/scalapack-2.0.2/lib:/mpi3/C3/hypre-2.8.0b/lib:/mpi3/C3/pardiso:/mpi3/C3/ParMetis-3.2.0:/mpi3/C3/ARPACK:/mpi3/C3/hdf5-1.8.10-patch1/lib:/mpi3/C3/VTK-5.8.0/lib/vtk-5.8:/mpi3/C3/elmer_6283/lib:/mpi3/C3/Togl-1.7:/mpi3/C3/netgen_668/lib:/usr/lib/:/usr/local/lib://mpi3/C3/vrpn/lib://mpi3/C3/hidapi/lib:/usr/include/libusb-1.0
    LC_TELEPHONE=fi_FI.UTF-8
    XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0
    JPK_OCC=/usr/include/oce
    XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0
    HYDRA_HOST_FILE=/mpi4/hosts
    SSH_AUTH_SOCK=/tmp/ssh-NnhxNTH12143/agent.12143
    SCOTCHDIR=/mpi3/C3/scotch_6.0.0
    HYDRA_LAUNCHER=rsh
    JPK_VER_B=B3
    SESSION_MANAGER=local/mpi1:@
/tmp/.ICE-unix/4284,unix/mpi1:/tmp/.ICE-unix/4284
    DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path
    ELMER_HOME=/mpi3/C3/elmer_6283
    BLACS=/mpi3/C3/scalapack-2.0.2
    BLAS32=/mpi3/C3/acml5.3.1/gfortran64_mp
    METIS_DIR=
    MPI_LIBS=-L/mpi3/C3/mpich-3.0.4/lib -lmpich -lmpichf90 -lmpl -lopa
-lmpichcxx
    XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg
    JPK_MPI_DIR=/mpi3
    JPK_HDF5_S=/mpi3/S3/hdf5-1.8.10-patch1
    MPIEXEC_PORT_RANGE=7000:7500
PATH=/mpi3/C3/cmake-2.8.10.2/bin:/mpi3/C3/blcr-0.8.5/bin:/mpi3/C3/mpich-3.0.4/bin:/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/mpi3/C3/elmer_6283/bin:/mpi3/C3/elmer_6283/lib:/mpi3/C3/ParaView3
    DESKTOP_SESSION=ubuntu
    BLAS=/mpi3/C3/acml5.3.1/gfortran64_mp
    METIS_LIBDIR=/mpi3/C3/ParMetis-3.2.0
    CMAKE_COMMAND=/mpi3/C3/cmake-2.8.10.2/bin
    QT_QMAKE_EXECUTABLE=/usr/bin/qmake-qt4
    LC_IDENTIFICATION=fi_FI.UTF-8
    JPK_SCOTCHDIR_S=/mpi3/S3/scotch_6.0.0_esmumps
    JPK_LMETISDIR_S=/mpi3/S3/ParMetis-3.2.0
    PWD=/mpi3/S3/mpich-3.0.4
    NETGENDIR=/mpi3/C3/netgen_668/bin
    EDITOR=nano
    JPK_LMETISDIR=/mpi3/C3/ParMetis-3.2.0
    GNOME_KEYRING_PID=4273
    LANG=fi_FI.UTF-8
    MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path
    OMPI_77=//mpi3/C3/mpich-3.0.4/bin/mpif77
    LC_MEASUREMENT=fi_FI.UTF-8
    JPK_HDF5=/mpi3/C3/hdf5-1.8.10-patch1
    UBUNTU_MENUPROXY=libappmenu.so
    COMPIZ_CONFIG_PROFILE=ubuntu
    ELMER_POST_HOME=/mpi3/C3/elmer_6283/bin
    JPK_INS=/mpi3/C3
    ELMER_LIB=/mpi3/C3/elmer_6283/share/elmersolver/lib
    HYDRA_PROXY_RETRY_COUNT=3
    GDMSESSION=ubuntu
    JPK_ELMER_S=/mpi3/S3/elmer_6283
    JPK_LMETISDIR5=/mpi3/C3/parmetis-4.0.2
    JPK_LMETISDIR32=/mpi3/C3/ParMetis-3.2.0
    HYDRA_DEBUG=1
    JPK_BUI=/mpi3/S3
    VTK_INCLUDEPATH=/mpi3/C3/VTK-5.8.0/include
    SHLVL=1
    HOME=/home/joni
    OMPI_CC=//mpi3/C3/mpich-3.0.4/bin/mpicc
    LANGUAGE=fi:en
    OMPI_90=//mpi3/C3/mpich-3.0.4/bin/mpif90
    ELMERGUI_HOME=/mpi3/C3/elmer_6283/bin
    GNOME_DESKTOP_SESSION_ID=this-is-deprecated
    MPI_IMPLEMENTATION=mpich
    MKL_SERIAL=YES
    LOGNAME=joni
    HYPRE=/mpi3/C3/hypre-2.8.0b
    JPK_ARPACK_S=/mpi3/S3/ARPACK
    JPK_JOBS=7
    JPK_VTK_DIR=/mpi3/C3/VTK-5.8.0
    SCALAPACK=/mpi3/C3/scalapack-2.0.2
XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/
DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-gAYmjGaklf,guid=26c3f15a7a5ee55e8782415700000034
    JPK_ARPACK=/mpi3/C3/ARPACK
    MPI_HOME=/mpi3/C3/mpich-3.0.4
    LESSOPEN=| /usr/bin/lesspipe %s
    LACPACK=/mpi3/C3/acml5.3.1/gfortran64_mp
    OMPI_CXX=//mpi3/C3/mpich-3.0.4/bin/mpicxx
    OMP_NUM_THREADS=6
    JPK_TOGL_S=/mpi3/S3/Togl-1.7
    HYDRA_LAUNCHER_EXEC=/usr/bin/rsh-redone-rsh
    JPK_MPICH2=/mpi3/C3/mpich-3.0.4
    PARDISO=/mpi3/C3/pardiso
    PARDISOLICMESSAGE=1
    JPK_VER=C3
    XDG_CURRENT_DESKTOP=Unity
    LESSCLOSE=/usr/bin/lesspipe %s %s
    LC_TIME=fi_FI.UTF-8
    JPK_HYPRE_S=/mpi3/S3/hypre-2.8.0b
    JPK_MPICH2_S=/mpi3/S3/mpich-3.0.4
    COLORTERM=gnome-terminal
    XAUTHORITY=/home/joni/.Xauthority
    LC_NAME=fi_FI.UTF-8
    _=/mpi3/C3/mpich-3.0.4/bin/mpiexec
    OLDPWD=/home/joni
Hydra internal environment:
  ---------------------------
    MPICH_ENABLE_CKPOINT=1
    GFORTRAN_UNBUFFERED_PRECONNECTED=y
Proxy information:
    *********************
      [1] proxy: 192.168.0.41 (2 cores)
      Exec list: hostname (2 processes);
[2] proxy: 192.168.0.42 (2 cores)
      Exec list: hostname (2 processes);
[3] proxy: 192.168.0.43 (2 cores)
      Exec list: hostname (2 processes);
==================================================================================================
[mpiexec@mpi1] Timeout set to -1 (-1 means infinite)
[mpiexec@mpi1] Got a control port string of 192.168.0.41:7001
Proxy launch args: /mpi3/C3/mpich-3.0.4/bin/hydra_pmi_proxy --control-port
192.168.0.41:7001 --debug --rmk user --launcher rsh --launcher-exec
/usr/bin/rsh-redone-rsh --demux poll --pgid 0 --retries 3 --usize -2
--proxy-id
Arguments being passed to proxy 0:
--version 3.0.4 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME --hostname
192.168.0.41 --global-core-map 0,2,6 --pmi-id-map 0,0
--global-process-count 6 --auto-cleanup 1 --pmi-kvsname kvs_12243_0
--pmi-process-mapping (vector,(0,3,2)) --ckpoint-prefix
/mpi3/chekpoint/default.chk --ckpoint-num -1 --global-inherited-env 121
'MUMPS=/mpi3/S3/MUMPS_4.10.0' 'LC_PAPER=fi_FI.UTF-8'
'LC_ADDRESS=fi_FI.UTF-8' 'SSH_AGENT_PID=12144' 'LC_MONETARY=fi_FI.UTF-8'
'MUMPS_I=/mpi3/C3/MUMPS_4.10.0' 'HYDRA_DEMUX=select'
'GPG_AGENT_INFO=/tmp/keyring-kJwpJQ/gpg:0:1'
'JPK_LMETISDIR_S5=/mpi3/S3/parmetis-4.0.2' 'TERM=xterm' 'SHELL=/bin/bash'
'XDG_SESSION_COOKIE=6d6390cb56a32b6678c10da600000412-1377606907.629665-1922379047'
'FFT=/mpi3/C3/fftw2' 'HYDRA_ENV=all' 'JPK_NETGEN=/mpi3/C3/netgen_668'
'JPK_VER_S=S3' 'HYDRA_CKPOINTLIB=blcr' 'HYDRA_CKPOINT_INTERVAL=10800'
'WINDOWID=54602522' 'LC_NUMERIC=fi_FI.UTF-8'
'HYDRA_CKPOINT_PREFIX=/mpi3/chekpoint/default.chk'
'GNOME_KEYRING_CONTROL=/tmp/keyring-kJwpJQ' 'JPK_ELMER=/mpi3/C3/elmer_6283'
'PARDISO_LIC_PATH=/mpi3/C3/pardiso'
'METIS_INCLUDE_DIR=/mpi3/C3/ParMetis-3.2.0'
'JPK_NETGEN_S=/mpi3/S3/netgen_668' 'USER=joni'
'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:'
'JPK_TOGL=/mpi3/C3/Togl-1.7'
'LD_LIBRARY_PATH=/mpi3/C3/mpich-3.0.4/lib:/mpi3/C3/mpich-3.0.4/bin:/mpi3/C3/blcr-0.8.5/lib:/mpi3/C3/ParMetis-3.2.0:/mpi3/C3/acml5.3.1/gfortran64_mp/lib:/lib:/mpi3/C3/hypre-2.8.0b/lib:/mpi3/C3/scotch_6.0.0/lib:/mpi3/S3/MUMPS_4.10.0/lib:/mpi3/C3/acml5.3.1/gfortran64_mp/lib:/mpi3/C3/scalapack-2.0.2/lib:/mpi3/C3/hypre-2.8.0b/lib:/mpi3/C3/pardiso:/mpi3/C3/ParMetis-3.2.0:/mpi3/C3/ARPACK:/mpi3/C3/hdf5-1.8.10-patch1/lib:/mpi3/C3/VTK-5.8.0/lib/vtk-5.8:/mpi3/C3/elmer_6283/lib:/mpi3/C3/Togl-1.7:/mpi3/C3/netgen_668/lib:/usr/lib/:/usr/local/lib://mpi3/C3/vrpn/lib://mpi3/C3/hidapi/lib:/usr/include/libusb-1.0'
'LC_TELEPHONE=fi_FI.UTF-8'
'XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0'
'JPK_OCC=/usr/include/oce'
'XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0'
'HYDRA_HOST_FILE=/mpi4/hosts'
'SSH_AUTH_SOCK=/tmp/ssh-NnhxNTH12143/agent.12143'
'SCOTCHDIR=/mpi3/C3/scotch_6.0.0' 'HYDRA_LAUNCHER=rsh' 'JPK_VER_B=B3'
'SESSION_MANAGER=local/mpi1:@/tmp/.ICE-unix/4284,unix/mpi1:/tmp/.ICE-unix/4284'
'DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path'
'ELMER_HOME=/mpi3/C3/elmer_6283' 'BLACS=/mpi3/C3/scalapack-2.0.2'
'BLAS32=/mpi3/C3/acml5.3.1/gfortran64_mp' 'METIS_DIR='
'MPI_LIBS=-L/mpi3/C3/mpich-3.0.4/lib -lmpich -lmpichf90 -lmpl -lopa
-lmpichcxx' 'XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg'
'JPK_MPI_DIR=/mpi3' 'JPK_HDF5_S=/mpi3/S3/hdf5-1.8.10-patch1'
'MPIEXEC_PORT_RANGE=7000:7500'
'PATH=/mpi3/C3/cmake-2.8.10.2/bin:/mpi3/C3/blcr-0.8.5/bin:/mpi3/C3/mpich-3.0.4/bin:/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/mpi3/C3/elmer_6283/bin:/mpi3/C3/elmer_6283/lib:/mpi3/C3/ParaView3'
'DESKTOP_SESSION=ubuntu' 'BLAS=/mpi3/C3/acml5.3.1/gfortran64_mp'
'METIS_LIBDIR=/mpi3/C3/ParMetis-3.2.0'
'CMAKE_COMMAND=/mpi3/C3/cmake-2.8.10.2/bin'
'QT_QMAKE_EXECUTABLE=/usr/bin/qmake-qt4' 'LC_IDENTIFICATION=fi_FI.UTF-8'
'JPK_SCOTCHDIR_S=/mpi3/S3/scotch_6.0.0_esmumps'
'JPK_LMETISDIR_S=/mpi3/S3/ParMetis-3.2.0' 'PWD=/mpi3/S3/mpich-3.0.4'
'NETGENDIR=/mpi3/C3/netgen_668/bin' 'EDITOR=nano'
'JPK_LMETISDIR=/mpi3/C3/ParMetis-3.2.0' 'GNOME_KEYRING_PID=4273'
'LANG=fi_FI.UTF-8' 'MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path'
'OMPI_77=//mpi3/C3/mpich-3.0.4/bin/mpif77' 'LC_MEASUREMENT=fi_FI.UTF-8'
'JPK_HDF5=/mpi3/C3/hdf5-1.8.10-patch1' 'UBUNTU_MENUPROXY=libappmenu.so'
'COMPIZ_CONFIG_PROFILE=ubuntu' 'ELMER_POST_HOME=/mpi3/C3/elmer_6283/bin'
'JPK_INS=/mpi3/C3' 'ELMER_LIB=/mpi3/C3/elmer_6283/share/elmersolver/lib'
'HYDRA_PROXY_RETRY_COUNT=3' 'GDMSESSION=ubuntu'
'JPK_ELMER_S=/mpi3/S3/elmer_6283' 'JPK_LMETISDIR5=/mpi3/C3/parmetis-4.0.2'
'JPK_LMETISDIR32=/mpi3/C3/ParMetis-3.2.0' 'HYDRA_DEBUG=1'
'JPK_BUI=/mpi3/S3' 'VTK_INCLUDEPATH=/mpi3/C3/VTK-5.8.0/include' 'SHLVL=1'
'HOME=/home/joni' 'OMPI_CC=//mpi3/C3/mpich-3.0.4/bin/mpicc'
'LANGUAGE=fi:en' 'OMPI_90=//mpi3/C3/mpich-3.0.4/bin/mpif90'
'ELMERGUI_HOME=/mpi3/C3/elmer_6283/bin'
'GNOME_DESKTOP_SESSION_ID=this-is-deprecated' 'MPI_IMPLEMENTATION=mpich'
'MKL_SERIAL=YES' 'LOGNAME=joni' 'HYPRE=/mpi3/C3/hypre-2.8.0b'
'JPK_ARPACK_S=/mpi3/S3/ARPACK' 'JPK_JOBS=7'
'JPK_VTK_DIR=/mpi3/C3/VTK-5.8.0' 'SCALAPACK=/mpi3/C3/scalapack-2.0.2'
'XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/'
'DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-gAYmjGaklf,guid=26c3f15a7a5ee55e8782415700000034'
'JPK_ARPACK=/mpi3/C3/ARPACK' 'MPI_HOME=/mpi3/C3/mpich-3.0.4' 'LESSOPEN=|
/usr/bin/lesspipe %s' 'LACPACK=/mpi3/C3/acml5.3.1/gfortran64_mp'
'OMPI_CXX=//mpi3/C3/mpich-3.0.4/bin/mpicxx' 'OMP_NUM_THREADS=6'
'JPK_TOGL_S=/mpi3/S3/Togl-1.7'
'HYDRA_LAUNCHER_EXEC=/usr/bin/rsh-redone-rsh'
'JPK_MPICH2=/mpi3/C3/mpich-3.0.4' 'PARDISO=/mpi3/C3/pardiso'
'PARDISOLICMESSAGE=1' 'JPK_VER=C3' 'XDG_CURRENT_DESKTOP=Unity'
'LESSCLOSE=/usr/bin/lesspipe %s %s' 'LC_TIME=fi_FI.UTF-8'
'JPK_HYPRE_S=/mpi3/S3/hypre-2.8.0b' 'JPK_MPICH2_S=/mpi3/S3/mpich-3.0.4'
'COLORTERM=gnome-terminal' 'XAUTHORITY=/home/joni/.Xauthority'
'LC_NAME=fi_FI.UTF-8' '_=/mpi3/C3/mpich-3.0.4/bin/mpiexec'
'OLDPWD=/home/joni' --global-user-env 0 --global-system-env 2
'MPICH_ENABLE_CKPOINT=1' 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --genv-prop
all --proxy-core-count 2 --exec --exec-appnum 0 --exec-proc-count 2
--exec-local-env 0 --exec-wdir /mpi3/S3/mpich-3.0.4 --exec-args 1 hostname
Arguments being passed to proxy 1:
--version 3.0.4 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME --hostname
192.168.0.42 --global-core-map 0,2,6 --pmi-id-map 0,2
--global-process-count 6 --auto-cleanup 1 --pmi-kvsname kvs_12243_0
--pmi-process-mapping (vector,(0,3,2)) --ckpoint-prefix
/mpi3/chekpoint/default.chk --ckpoint-num -1 --global-inherited-env 121
'MUMPS=/mpi3/S3/MUMPS_4.10.0' 'LC_PAPER=fi_FI.UTF-8'
'LC_ADDRESS=fi_FI.UTF-8' 'SSH_AGENT_PID=12144' 'LC_MONETARY=fi_FI.UTF-8'
'MUMPS_I=/mpi3/C3/MUMPS_4.10.0' 'HYDRA_DEMUX=select'
'GPG_AGENT_INFO=/tmp/keyring-kJwpJQ/gpg:0:1'
'JPK_LMETISDIR_S5=/mpi3/S3/parmetis-4.0.2' 'TERM=xterm' 'SHELL=/bin/bash'
'XDG_SESSION_COOKIE=6d6390cb56a32b6678c10da600000412-1377606907.629665-1922379047'
'FFT=/mpi3/C3/fftw2' 'HYDRA_ENV=all' 'JPK_NETGEN=/mpi3/C3/netgen_668'
'JPK_VER_S=S3' 'HYDRA_CKPOINTLIB=blcr' 'HYDRA_CKPOINT_INTERVAL=10800'
'WINDOWID=54602522' 'LC_NUMERIC=fi_FI.UTF-8'
'HYDRA_CKPOINT_PREFIX=/mpi3/chekpoint/default.chk'
'GNOME_KEYRING_CONTROL=/tmp/keyring-kJwpJQ' 'JPK_ELMER=/mpi3/C3/elmer_6283'
'PARDISO_LIC_PATH=/mpi3/C3/pardiso'
'METIS_INCLUDE_DIR=/mpi3/C3/ParMetis-3.2.0'
'JPK_NETGEN_S=/mpi3/S3/netgen_668' 'USER=joni'
'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:'
'JPK_TOGL=/mpi3/C3/Togl-1.7'
'LD_LIBRARY_PATH=/mpi3/C3/mpich-3.0.4/lib:/mpi3/C3/mpich-3.0.4/bin:/mpi3/C3/blcr-0.8.5/lib:/mpi3/C3/ParMetis-3.2.0:/mpi3/C3/acml5.3.1/gfortran64_mp/lib:/lib:/mpi3/C3/hypre-2.8.0b/lib:/mpi3/C3/scotch_6.0.0/lib:/mpi3/S3/MUMPS_4.10.0/lib:/mpi3/C3/acml5.3.1/gfortran64_mp/lib:/mpi3/C3/scalapack-2.0.2/lib:/mpi3/C3/hypre-2.8.0b/lib:/mpi3/C3/pardiso:/mpi3/C3/ParMetis-3.2.0:/mpi3/C3/ARPACK:/mpi3/C3/hdf5-1.8.10-patch1/lib:/mpi3/C3/VTK-5.8.0/lib/vtk-5.8:/mpi3/C3/elmer_6283/lib:/mpi3/C3/Togl-1.7:/mpi3/C3/netgen_668/lib:/usr/lib/:/usr/local/lib://mpi3/C3/vrpn/lib://mpi3/C3/hidapi/lib:/usr/include/libusb-1.0'
'LC_TELEPHONE=fi_FI.UTF-8'
'XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0'
'JPK_OCC=/usr/include/oce'
'XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0'
'HYDRA_HOST_FILE=/mpi4/hosts'
'SSH_AUTH_SOCK=/tmp/ssh-NnhxNTH12143/agent.12143'
'SCOTCHDIR=/mpi3/C3/scotch_6.0.0' 'HYDRA_LAUNCHER=rsh' 'JPK_VER_B=B3'
'SESSION_MANAGER=local/mpi1:@/tmp/.ICE-unix/4284,unix/mpi1:/tmp/.ICE-unix/4284'
'DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path'
'ELMER_HOME=/mpi3/C3/elmer_6283' 'BLACS=/mpi3/C3/scalapack-2.0.2'
'BLAS32=/mpi3/C3/acml5.3.1/gfortran64_mp' 'METIS_DIR='
'MPI_LIBS=-L/mpi3/C3/mpich-3.0.4/lib -lmpich -lmpichf90 -lmpl -lopa
-lmpichcxx' 'XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg'
'JPK_MPI_DIR=/mpi3' 'JPK_HDF5_S=/mpi3/S3/hdf5-1.8.10-patch1'
'MPIEXEC_PORT_RANGE=7000:7500'
'PATH=/mpi3/C3/cmake-2.8.10.2/bin:/mpi3/C3/blcr-0.8.5/bin:/mpi3/C3/mpich-3.0.4/bin:/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/mpi3/C3/elmer_6283/bin:/mpi3/C3/elmer_6283/lib:/mpi3/C3/ParaView3'
'DESKTOP_SESSION=ubuntu' 'BLAS=/mpi3/C3/acml5.3.1/gfortran64_mp'
'METIS_LIBDIR=/mpi3/C3/ParMetis-3.2.0'
'CMAKE_COMMAND=/mpi3/C3/cmake-2.8.10.2/bin'
'QT_QMAKE_EXECUTABLE=/usr/bin/qmake-qt4' 'LC_IDENTIFICATION=fi_FI.UTF-8'
'JPK_SCOTCHDIR_S=/mpi3/S3/scotch_6.0.0_esmumps'
'JPK_LMETISDIR_S=/mpi3/S3/ParMetis-3.2.0' 'PWD=/mpi3/S3/mpich-3.0.4'
'NETGENDIR=/mpi3/C3/netgen_668/bin' 'EDITOR=nano'
'JPK_LMETISDIR=/mpi3/C3/ParMetis-3.2.0' 'GNOME_KEYRING_PID=4273'
'LANG=fi_FI.UTF-8' 'MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path'
'OMPI_77=//mpi3/C3/mpich-3.0.4/bin/mpif77' 'LC_MEASUREMENT=fi_FI.UTF-8'
'JPK_HDF5=/mpi3/C3/hdf5-1.8.10-patch1' 'UBUNTU_MENUPROXY=libappmenu.so'
'COMPIZ_CONFIG_PROFILE=ubuntu' 'ELMER_POST_HOME=/mpi3/C3/elmer_6283/bin'
'JPK_INS=/mpi3/C3' 'ELMER_LIB=/mpi3/C3/elmer_6283/share/elmersolver/lib'
'HYDRA_PROXY_RETRY_COUNT=3' 'GDMSESSION=ubuntu'
'JPK_ELMER_S=/mpi3/S3/elmer_6283' 'JPK_LMETISDIR5=/mpi3/C3/parmetis-4.0.2'
'JPK_LMETISDIR32=/mpi3/C3/ParMetis-3.2.0' 'HYDRA_DEBUG=1'
'JPK_BUI=/mpi3/S3' 'VTK_INCLUDEPATH=/mpi3/C3/VTK-5.8.0/include' 'SHLVL=1'
'HOME=/home/joni' 'OMPI_CC=//mpi3/C3/mpich-3.0.4/bin/mpicc'
'LANGUAGE=fi:en' 'OMPI_90=//mpi3/C3/mpich-3.0.4/bin/mpif90'
'ELMERGUI_HOME=/mpi3/C3/elmer_6283/bin'
'GNOME_DESKTOP_SESSION_ID=this-is-deprecated' 'MPI_IMPLEMENTATION=mpich'
'MKL_SERIAL=YES' 'LOGNAME=joni' 'HYPRE=/mpi3/C3/hypre-2.8.0b'
'JPK_ARPACK_S=/mpi3/S3/ARPACK' 'JPK_JOBS=7'
'JPK_VTK_DIR=/mpi3/C3/VTK-5.8.0' 'SCALAPACK=/mpi3/C3/scalapack-2.0.2'
'XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/'
'DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-gAYmjGaklf,guid=26c3f15a7a5ee55e8782415700000034'
'JPK_ARPACK=/mpi3/C3/ARPACK' 'MPI_HOME=/mpi3/C3/mpich-3.0.4' 'LESSOPEN=|
/usr/bin/lesspipe %s' 'LACPACK=/mpi3/C3/acml5.3.1/gfortran64_mp'
'OMPI_CXX=//mpi3/C3/mpich-3.0.4/bin/mpicxx' 'OMP_NUM_THREADS=6'
'JPK_TOGL_S=/mpi3/S3/Togl-1.7'
'HYDRA_LAUNCHER_EXEC=/usr/bin/rsh-redone-rsh'
'JPK_MPICH2=/mpi3/C3/mpich-3.0.4' 'PARDISO=/mpi3/C3/pardiso'
'PARDISOLICMESSAGE=1' 'JPK_VER=C3' 'XDG_CURRENT_DESKTOP=Unity'
'LESSCLOSE=/usr/bin/lesspipe %s %s' 'LC_TIME=fi_FI.UTF-8'
'JPK_HYPRE_S=/mpi3/S3/hypre-2.8.0b' 'JPK_MPICH2_S=/mpi3/S3/mpich-3.0.4'
'COLORTERM=gnome-terminal' 'XAUTHORITY=/home/joni/.Xauthority'
'LC_NAME=fi_FI.UTF-8' '_=/mpi3/C3/mpich-3.0.4/bin/mpiexec'
'OLDPWD=/home/joni' --global-user-env 0 --global-system-env 2
'MPICH_ENABLE_CKPOINT=1' 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --genv-prop
all --proxy-core-count 2 --exec --exec-appnum 0 --exec-proc-count 2
--exec-local-env 0 --exec-wdir /mpi3/S3/mpich-3.0.4 --exec-args 1 hostname
Arguments being passed to proxy 2:
--version 3.0.4 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME --hostname
192.168.0.43 --global-core-map 0,2,6 --pmi-id-map 0,4
--global-process-count 6 --auto-cleanup 1 --pmi-kvsname kvs_12243_0
--pmi-process-mapping (vector,(0,3,2)) --ckpoint-prefix
/mpi3/chekpoint/default.chk --ckpoint-num -1 --global-inherited-env 121
'MUMPS=/mpi3/S3/MUMPS_4.10.0' 'LC_PAPER=fi_FI.UTF-8'
'LC_ADDRESS=fi_FI.UTF-8' 'SSH_AGENT_PID=12144' 'LC_MONETARY=fi_FI.UTF-8'
'MUMPS_I=/mpi3/C3/MUMPS_4.10.0' 'HYDRA_DEMUX=select'
'GPG_AGENT_INFO=/tmp/keyring-kJwpJQ/gpg:0:1'
'JPK_LMETISDIR_S5=/mpi3/S3/parmetis-4.0.2' 'TERM=xterm' 'SHELL=/bin/bash'
'XDG_SESSION_COOKIE=6d6390cb56a32b6678c10da600000412-1377606907.629665-1922379047'
'FFT=/mpi3/C3/fftw2' 'HYDRA_ENV=all' 'JPK_NETGEN=/mpi3/C3/netgen_668'
'JPK_VER_S=S3' 'HYDRA_CKPOINTLIB=blcr' 'HYDRA_CKPOINT_INTERVAL=10800'
'WINDOWID=54602522' 'LC_NUMERIC=fi_FI.UTF-8'
'HYDRA_CKPOINT_PREFIX=/mpi3/chekpoint/default.chk'
'GNOME_KEYRING_CONTROL=/tmp/keyring-kJwpJQ' 'JPK_ELMER=/mpi3/C3/elmer_6283'
'PARDISO_LIC_PATH=/mpi3/C3/pardiso'
'METIS_INCLUDE_DIR=/mpi3/C3/ParMetis-3.2.0'
'JPK_NETGEN_S=/mpi3/S3/netgen_668' 'USER=joni'
'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:'
'JPK_TOGL=/mpi3/C3/Togl-1.7'
'LD_LIBRARY_PATH=/mpi3/C3/mpich-3.0.4/lib:/mpi3/C3/mpich-3.0.4/bin:/mpi3/C3/blcr-0.8.5/lib:/mpi3/C3/ParMetis-3.2.0:/mpi3/C3/acml5.3.1/gfortran64_mp/lib:/lib:/mpi3/C3/hypre-2.8.0b/lib:/mpi3/C3/scotch_6.0.0/lib:/mpi3/S3/MUMPS_4.10.0/lib:/mpi3/C3/acml5.3.1/gfortran64_mp/lib:/mpi3/C3/scalapack-2.0.2/lib:/mpi3/C3/hypre-2.8.0b/lib:/mpi3/C3/pardiso:/mpi3/C3/ParMetis-3.2.0:/mpi3/C3/ARPACK:/mpi3/C3/hdf5-1.8.10-patch1/lib:/mpi3/C3/VTK-5.8.0/lib/vtk-5.8:/mpi3/C3/elmer_6283/lib:/mpi3/C3/Togl-1.7:/mpi3/C3/netgen_668/lib:/usr/lib/:/usr/local/lib://mpi3/C3/vrpn/lib://mpi3/C3/hidapi/lib:/usr/include/libusb-1.0'
'LC_TELEPHONE=fi_FI.UTF-8'
'XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0'
'JPK_OCC=/usr/include/oce'
'XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0'
'HYDRA_HOST_FILE=/mpi4/hosts'
'SSH_AUTH_SOCK=/tmp/ssh-NnhxNTH12143/agent.12143'
'SCOTCHDIR=/mpi3/C3/scotch_6.0.0' 'HYDRA_LAUNCHER=rsh' 'JPK_VER_B=B3'
'SESSION_MANAGER=local/mpi1:@/tmp/.ICE-unix/4284,unix/mpi1:/tmp/.ICE-unix/4284'
'DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path'
'ELMER_HOME=/mpi3/C3/elmer_6283' 'BLACS=/mpi3/C3/scalapack-2.0.2'
'BLAS32=/mpi3/C3/acml5.3.1/gfortran64_mp' 'METIS_DIR='
'MPI_LIBS=-L/mpi3/C3/mpich-3.0.4/lib -lmpich -lmpichf90 -lmpl -lopa
-lmpichcxx' 'XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg'
'JPK_MPI_DIR=/mpi3' 'JPK_HDF5_S=/mpi3/S3/hdf5-1.8.10-patch1'
'MPIEXEC_PORT_RANGE=7000:7500'
'PATH=/mpi3/C3/cmake-2.8.10.2/bin:/mpi3/C3/blcr-0.8.5/bin:/mpi3/C3/mpich-3.0.4/bin:/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/mpi3/C3/elmer_6283/bin:/mpi3/C3/elmer_6283/lib:/mpi3/C3/ParaView3'
'DESKTOP_SESSION=ubuntu' 'BLAS=/mpi3/C3/acml5.3.1/gfortran64_mp'
'METIS_LIBDIR=/mpi3/C3/ParMetis-3.2.0'
'CMAKE_COMMAND=/mpi3/C3/cmake-2.8.10.2/bin'
'QT_QMAKE_EXECUTABLE=/usr/bin/qmake-qt4' 'LC_IDENTIFICATION=fi_FI.UTF-8'
'JPK_SCOTCHDIR_S=/mpi3/S3/scotch_6.0.0_esmumps'
'JPK_LMETISDIR_S=/mpi3/S3/ParMetis-3.2.0' 'PWD=/mpi3/S3/mpich-3.0.4'
'NETGENDIR=/mpi3/C3/netgen_668/bin' 'EDITOR=nano'
'JPK_LMETISDIR=/mpi3/C3/ParMetis-3.2.0' 'GNOME_KEYRING_PID=4273'
'LANG=fi_FI.UTF-8' 'MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path'
'OMPI_77=//mpi3/C3/mpich-3.0.4/bin/mpif77' 'LC_MEASUREMENT=fi_FI.UTF-8'
'JPK_HDF5=/mpi3/C3/hdf5-1.8.10-patch1' 'UBUNTU_MENUPROXY=libappmenu.so'
'COMPIZ_CONFIG_PROFILE=ubuntu' 'ELMER_POST_HOME=/mpi3/C3/elmer_6283/bin'
'JPK_INS=/mpi3/C3' 'ELMER_LIB=/mpi3/C3/elmer_6283/share/elmersolver/lib'
'HYDRA_PROXY_RETRY_COUNT=3' 'GDMSESSION=ubuntu'
'JPK_ELMER_S=/mpi3/S3/elmer_6283' 'JPK_LMETISDIR5=/mpi3/C3/parmetis-4.0.2'
'JPK_LMETISDIR32=/mpi3/C3/ParMetis-3.2.0' 'HYDRA_DEBUG=1'
'JPK_BUI=/mpi3/S3' 'VTK_INCLUDEPATH=/mpi3/C3/VTK-5.8.0/include' 'SHLVL=1'
'HOME=/home/joni' 'OMPI_CC=//mpi3/C3/mpich-3.0.4/bin/mpicc'
'LANGUAGE=fi:en' 'OMPI_90=//mpi3/C3/mpich-3.0.4/bin/mpif90'
'ELMERGUI_HOME=/mpi3/C3/elmer_6283/bin'
'GNOME_DESKTOP_SESSION_ID=this-is-deprecated' 'MPI_IMPLEMENTATION=mpich'
'MKL_SERIAL=YES' 'LOGNAME=joni' 'HYPRE=/mpi3/C3/hypre-2.8.0b'
'JPK_ARPACK_S=/mpi3/S3/ARPACK' 'JPK_JOBS=7'
'JPK_VTK_DIR=/mpi3/C3/VTK-5.8.0' 'SCALAPACK=/mpi3/C3/scalapack-2.0.2'
'XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/'
'DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-gAYmjGaklf,guid=26c3f15a7a5ee55e8782415700000034'
'JPK_ARPACK=/mpi3/C3/ARPACK' 'MPI_HOME=/mpi3/C3/mpich-3.0.4' 'LESSOPEN=|
/usr/bin/lesspipe %s' 'LACPACK=/mpi3/C3/acml5.3.1/gfortran64_mp'
'OMPI_CXX=//mpi3/C3/mpich-3.0.4/bin/mpicxx' 'OMP_NUM_THREADS=6'
'JPK_TOGL_S=/mpi3/S3/Togl-1.7'
'HYDRA_LAUNCHER_EXEC=/usr/bin/rsh-redone-rsh'
'JPK_MPICH2=/mpi3/C3/mpich-3.0.4' 'PARDISO=/mpi3/C3/pardiso'
'PARDISOLICMESSAGE=1' 'JPK_VER=C3' 'XDG_CURRENT_DESKTOP=Unity'
'LESSCLOSE=/usr/bin/lesspipe %s %s' 'LC_TIME=fi_FI.UTF-8'
'JPK_HYPRE_S=/mpi3/S3/hypre-2.8.0b' 'JPK_MPICH2_S=/mpi3/S3/mpich-3.0.4'
'COLORTERM=gnome-terminal' 'XAUTHORITY=/home/joni/.Xauthority'
'LC_NAME=fi_FI.UTF-8' '_=/mpi3/C3/mpich-3.0.4/bin/mpiexec'
'OLDPWD=/home/joni' --global-user-env 0 --global-system-env 2
'MPICH_ENABLE_CKPOINT=1' 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --genv-prop
all --proxy-core-count 2 --exec --exec-appnum 0 --exec-proc-count 2
--exec-local-env 0 --exec-wdir /mpi3/S3/mpich-3.0.4 --exec-args 1 hostname
[mpiexec@mpi1] Launch arguments: /mpi3/C3/mpich-3.0.4/bin/hydra_pmi_proxy
--control-port 192.168.0.41:7001 --debug --rmk user --launcher rsh
--launcher-exec /usr/bin/rsh-redone-rsh --demux poll --pgid 0 --retries 3
--usize -2 --proxy-id 0
[mpiexec@mpi1] Launch arguments: /usr/bin/rsh-redone-rsh 192.168.0.42
"/mpi3/C3/mpich-3.0.4/bin/hydra_pmi_proxy" --control-port
192.168.0.41:7001 --debug --rmk user --launcher rsh --launcher-exec
/usr/bin/rsh-redone-rsh --demux poll --pgid 0 --retries 3 --usize -2
--proxy-id 1
[mpiexec@mpi1] Launch arguments: /usr/bin/rsh-redone-rsh 192.168.0.43
"/mpi3/C3/mpich-3.0.4/bin/hydra_pmi_proxy" --control-port
192.168.0.41:7001 --debug --rmk user --launcher rsh --launcher-exec
/usr/bin/rsh-redone-rsh --demux poll --pgid 0 --retries 3 --usize -2
--proxy-id 2
mpi1
m
2013/8/27 Pavan Balaji <[email protected]>
...
Please don't drop [email protected] from the cc list.
I doubt demux, --assert-level and blcr are relevant here.  Also, the
output of "make testing" is not helpful for us because those tests can fail
even if your machines are too slow.
Did you try my suggestion from the previous email?  Could you try them
and report back (just with that information)?
-- Pavan
On 08/27/2013 09:35 AM, Joni-Pekka Kurronen wrote:
...
hi,
I have allraedy cheked up.
This is not new install, just jumped up from mpich2 to 3 and
gcc 4.6 to gcc 4.7.
I have rsh-redone-rsh as main but tested ssh as well,...
due at crash clinet's keep running at slave's and useing harddisk,...
Following work's:
any machine alone
mpi1 and kaak  or  mpi1 and ugh
but not all to gether except hostname,...
This could be realated:
- demux  ( have tried select and poll, whit poll have to restart slave
machine's)
- nfs4 ( some reason nfs4 must be manually mounted at moment after
restart at slave's)
- have changed --assert-level to 0 (default 2)
- blcr
ch3:socket setting:
=============
hosts file,..
192.168.0.41:2 <http://192.168.0.41:2>
192.168.0.42:2 <http://192.168.0.42:2>
#192.168.0.43:2 <http://192.168.0.43:2>
=============
summary.xml errors
<MPITEST>
<NAME>spawninfo1</NAME>
<NP>1</NP>
<WORKDIR>./spawn</WORKDIR>
<STATUS>fail</STATUS>
<TESTDIFF>
[mpiexec@mpi1] APPLICATION TIMED OUT
[proxy:0:0@mpi1] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed
[proxy:0:0@mpi1] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:**77): callback returned error status
[proxy:0:0@mpi1] main (./pm/pmiserv/pmip.c:206): demux engine error
waiting for event
[proxy:1:0@mpi1] HYD_pmcd_pmip_control_cmd_cb
(./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed
[proxy:1:0@mpi1] HYDT_dmxu_poll_wait_for_event
(./tools/demux/demux_poll.c:**77): callback returned error status
[proxy:1:0@mpi1] main (./pm/pmiserv/pmip.c:206): demux engine error
waiting for event
[mpiexec@mpi1] HYDT_bscu_wait_for_completion
(./tools/bootstrap/utils/bscu_**wait.c:76): one of the processes
terminated badly; aborting
[mpiexec@mpi1] HYDT_bsci_wait_for_completion
(./tools/bootstrap/src/bsci_**wait.c:23): launcher returned error
waiting
for completion
[mpiexec@mpi1] HYD_pmci_wait_for_completion
(./pm/pmiserv/pmiserv_pmci.c:**188): launcher returned error waiting for
completion
[mpiexec@mpi1] main (./ui/mpich/mpiexec.c:331): process manager error
waiting for completion
</TESTDIFF>
</MPITEST>
<MPITEST>
<NAME>rdwrord</NAME>
<NP>4</NP>
<WORKDIR>./io</WORKDIR>
<STATUS>fail</STATUS>
<TESTDIFF>
Fatal error in PMPI_Bcast: Other MPI error
[cli_2]: aborting job:
Fatal error in PMPI_Bcast: Other MPI error
Fatal error in PMPI_Bcast: Other MPI error
[cli_0]: aborting job:
Fatal error in PMPI_Bcast: Other MPI error
==============================**==============================**
=======================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   EXIT CODE: 1
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
==============================**==============================**
=======================
</TESTDIFF>
</MPITEST>
<MPITEST>
<NAME>rdwrzero</NAME>
<NP>4</NP>
<WORKDIR>./io</WORKDIR>
<STATUS>fail</STATUS>
<TESTDIFF>
Fatal error in PMPI_Bcast: Other MPI error
[cli_2]: aborting job:
Fatal error in PMPI_Bcast: Other MPI error
Fatal error in PMPI_Bcast: Other MPI error
[cli_0]: aborting job:
Fatal error in PMPI_Bcast: Other MPI error
==============================**==============================**
=======================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   EXIT CODE: 1
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
==============================**==============================**
=======================
</TESTDIFF>
</MPITEST>
<MPITEST>
<NAME>getextent</NAME>
<NP>2</NP>
<WORKDIR>./io</WORKDIR>
<STATUS>pass</STATUS>
</MPITEST>
<MPITEST>
<NAME>setinfo</NAME>
<NP>4</NP>
<WORKDIR>./io</WORKDIR>
<STATUS>fail</STATUS>
<TESTDIFF>
Fatal error in PMPI_Bcast: Other MPI error
[cli_2]: aborting job:
Fatal error in PMPI_Bcast: Other MPI error
Fatal error in PMPI_Bcast: Other MPI error
[cli_0]: aborting job:
Fatal error in PMPI_Bcast: Other MPI error
==============================**==============================**
=======================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   EXIT CODE: 1
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
==============================**==============================**
=======================
</TESTDIFF>
</MPITEST>
<MPITEST>
<NAME>setviewcur</NAME>
<NP>4</NP>
<WORKDIR>./io</WORKDIR>
<STATUS>fail</STATUS>
<TESTDIFF>
Fatal error in PMPI_Bcast: Other MPI error
[cli_2]: aborting job:
Fatal error in PMPI_Bcast: Other MPI error
Fatal error in PMPI_Bcast: Other MPI error
[cli_0]: aborting job:
Fatal error in PMPI_Bcast: Other MPI error
==============================**==============================**
=======================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   EXIT CODE: 1
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
==============================**==============================**
=======================
</TESTDIFF>
</MPITEST>
....
....
....
============
============
hosts file,..
192.168.0.41:2 <http://192.168.0.41:2>
192.168.0.42:2 <http://192.168.0.42:2>
192.168.0.43:2 <http://192.168.0.43:2>
============
============
When needed more than 4 process will hang,..
Unexpected output in allred3: [mpiexec@mpi1] APPLICATION TIMED OUT
Unexpected output in allred3: [proxy:0:0@mpi1]
HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:886): assert
(!closed) failed
Unexpected output in allred3: [proxy:0:0@mpi1]
HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:**77):
callback
returned error status
Unexpected output in allred3: [proxy:0:0@mpi1] main
(./pm/pmiserv/pmip.c:206): demux engine error waiting for event
======
if useing ch3:nemesis hardisk is not running all the time as whit
sockets's,...
==============================**==================
This is read by rsh-redone-rsh for every process,...
==============================**==================
#!/bin/bash
# JPK-Integration for Ubuntu 12.4 LTS
#
https://sites.google.com/site/**jpsdatareviewstheboy007/**
ubuntu-lts-12-4-companion-**whit-ltsp-mpich2-elmer-**openfoam<https://sites.google.com/site/jpsdatareviewstheboy007/ubuntu-lts-12-4-companion-whit-ltsp-mpich2-elmer-openfoam>
#
# CMAKE goes loop and can not build, cmake build under development says
documentation
# gcc 4.7
# bdver1 optimization
shopt -s expand_aliases
export JPK_MPI_DIR=/mpi3         # MAIN DIRECTORY, SUBDIRCTORYES:
export JPK_VER=C3                # BINARY CODE
export JPK_VER_S=S3              # SOURCE CODE
export JPK_VER_B=B3              # BASH FILES TO COMPILE AND CONFIGURE
export JPK_INS=$JPK_MPI_DIR/$JPK_VER
export JPK_BUI=$JPK_MPI_DIR/$JPK_VER_**S
export JPK_ELMER=$JPK_INS/elmer_6283 #035
export JPK_ELMER_S=$JPK_BUI/elmer_**6283
export JPK_NETGEN_S=$JPK_BUI/netgen_**668
export JPK_NETGEN=$JPK_INS/netgen_668
#GCC
#export JPK_FLAGS="-Wl,--no-as-needed -fPIC -DAdd_ -m64 -pthread -O3
-fopenmp -lgomp -march=bdver1 -ftree-vectorize -funroll-loops"
#export CFLAGS="-Wl,--no-as-needed -fPIC -DAdd_ -m64 -pthread -fopenmp
-lgomp"
# M A K E
export JPK_JOBS=7
# O P E N  MP
export OMP_NUM_THREADS=6
# M P I C 3
# http://wiki.mcs.anl.gov/**mpich2/index.php/Using_the_**
Hydra_Process_Manager<http://wiki.mcs.anl.gov/mpich2/index.php/Using_the_Hydra_Process_Manager>
export JPK_MPICH2_S=$JPK_BUI/mpich-3.**0.4
export JPK_MPICH2=$JPK_INS/mpich-3.0.**4
export PATH=$JPK_MPICH2/bin:$PATH
export MPI_HOME=$JPK_MPICH2
export MPI_LIBS="-L$JPK_MPICH2/lib -lmpich -lmpichf90 -lmpl -lopa
-lmpichcxx"
export LD_LIBRARY_PATH=$JPK_MPICH2/**lib:$JPK_MPICH2/bin # FIRST
# M P I
export MPI_IMPLEMENTATION=mpich
export OMPI_CC=/$JPK_MPI_DIR/$JPK_**VER/mpich-3.0.4/bin/mpicc
export OMPI_CXX=/$JPK_MPI_DIR/$JPK_**VER/mpich-3.0.4/bin/mpicxx
export OMPI_77=/$JPK_MPI_DIR/$JPK_**VER/mpich-3.0.4/bin/mpif77
export OMPI_90=/$JPK_MPI_DIR/$JPK_**VER/mpich-3.0.4/bin/mpif90
# http://wiki.mcs.anl.gov/**mpich2/index.php/Using_the_**
Hydra_Process_Manager<http://wiki.mcs.anl.gov/mpich2/index.php/Using_the_Hydra_Process_Manager>
export HYDRA_DEBUG=0
export HYDRA_HOST_FILE=/mpi4/hosts
export HYDRA_LAUNCHER=rsh
#export HYDRA_LAUNCHER=ssh
#export HYDRA_LAUNCHER_EXEC=/usr/bin/**netkit-rsh
export HYDRA_LAUNCHER_EXEC=/usr/bin/**rsh-redone-rsh
#export HYDRA_LAUNCHER_EXEC=/usr/bin/**ssh
export HYDRA_DEMUX=select
#export HYDRA_DEMUX=select #more porseses than core's
export HYDRA_PROXY_RETRY_COUNT=3
#export HYDRA_RMK=pbs
#export HYDRA_DEFAULT_RMK=pbs
export HYDRA_ENV=all
export MPIEXEC_PORT_RANGE=7000:7500
#mpirun -launcher rsh -launcher-exec /usr/bin/netkit-rsh -demux select
-n 21 ddd ./cpi
# b l c r
export HYDRA_CKPOINTLIB=blcr
export HYDRA_CKPOINT_PREFIX=/mpi3/**chekpoint/default.chk
export HYDRA_CKPOINT_INTERVAL=10800
export PATH=$JPK_INS/blcr-0.8.5/bin:$**PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_**PATH:$JPK_INS/blcr-0.8.5/lib
#-ckpoint-num 5
# C M A K E - BUILD
export PATH=$JPK_INS/cmake-2.8.10.2/**bin:$PATH
export CMAKE_COMMAND=$JPK_INS/cmake-**2.8.10.2/bin
# T O G L - UI netgen
export JPK_TOGL="$JPK_INS/Togl-1.7"
export JPK_TOGL_S="$JPK_BUI/Togl-1.7"
# OCC
export JPK_OCC=/usr/include/oce
# M A T H
export JPK_ARPACK_S=$JPK_BUI/ARPACK
export JPK_ARPACK=$JPK_INS/ARPACK
export BLAS=$JPK_INS/acml5.3.1/**gfortran64_mp
export BLAS32=$JPK_INS/acml5.3.1/**gfortran64_mp
#export BLAS=$JPK_INS/clAmdBlas-1.10.**321/lib64
#export BLAS32=$JPK_INS/clAmdBlas-1.**10.321/include
export FFT=$JPK_INS/fftw2
export LACPACK=$BLAS
export SCALAPACK=$JPK_INS/scalapack-**2.0.2
export BLACS=$SCALAPACK
export JPK_LMETISDIR_S=$JPK_BUI/**ParMetis-3.2.0
export JPK_LMETISDIR=$JPK_INS/**ParMetis-3.2.0
export JPK_LMETISDIR32=$JPK_LMETISDIR
export JPK_LMETISDIR_S5=$JPK_BUI/**parmetis-4.0.2
export JPK_LMETISDIR5=$JPK_INS/**parmetis-4.0.2
export METIS_DIR="" #$JPK_LMETISDIR MUST BE EMPTY
export METIS_INCLUDE_DIR=$JPK_**LMETISDIR
export METIS_LIBDIR=$JPK_LMETISDIR
export LD_LIBRARY_PATH=$LD_LIBRARY_**PATH:$JPK_LMETISDIR:$BLAS/lib:**
$FTT/lib
#/mpi4/S/metis-5.0.2/GKlib
export SCOTCHDIR=$JPK_INS/scotch_6.0.**0
export JPK_SCOTCHDIR_S=$JPK_BUI/**scotch_6.0.0_esmumps
export MUMPS_I=$JPK_INS/MUMPS_4.10.0
export MUMPS=$JPK_BUI/MUMPS_4.10.0
export HYPRE=$JPK_INS/hypre-2.8.0b
export LD_LIBRARY_PATH=$LD_LIBRARY_**PATH:$HYPRE/lib
export JPK_HYPRE_S=$JPK_BUI/hypre-2.**8.0b
export PARDISOLICMESSAGE=1
export PARDISO=$JPK_INS/pardiso
export PARDISO_LIC_PATH=$PARDISO
export MKL_SERIAL=YES
export
LD_LIBRARY_PATH=$LD_LIBRARY_**PATH:$SCOTCHDIR/lib:$MUMPS/**
lib:$BLAS/lib:$SCALAPACK/lib:$**HYPRE/lib:$PARDISO:$METIS_**
LIBDIR:$JPK_ARPACK
#HDF5
#export JPK_HDF5_S=$JPK_BUI/hdf5-1.8.**10-patch1 for vtk testing
#export JPK_HDF5=$JPK_INS/hdf5-1.8.10-**patch1
export JPK_HDF5_S=$JPK_BUI/hdf5-1.8.**10-patch1
export JPK_HDF5=$JPK_INS/hdf5-1.8.10-**patch1
export LD_LIBRARY_PATH=$LD_LIBRARY_**PATH:$JPK_HDF5/lib
# V T K
export JPK_VTK_DIR=$JPK_INS/VTK-5.8.0
export LD_LIBRARY_PATH=$LD_LIBRARY_**PATH:$JPK_VTK_DIR/lib/vtk-5.8
export VTK_INCLUDEPATH=$JPK_VTK_DIR/**include
# Q T
export QT_QMAKE_EXECUTABLE=/usr/bin/**qmake-qt4
# O P E N    F O A M
# http://www.openfoam.org/**download/source.php<http://www.openfoam.org/download/source.php>
#export WM_SCHEDULER=wmakeScheduler
#export WM_HOSTS="192.168.0.41:6 <http://192.168.0.41:6> 192.168.0.42:6
<http://192.168.0.42:6> 192.168.0.43:6 <http://192.168.0.43:6>"
#export WM_NCOMPPROCS=$($WM_SCHEDULER -count)
#export WM_COLOURS="black blue green cyan red magenta yellow"
#export FOAM_INST_DIR=/mpi2/OpenFOAM
#foamDotFile=$FOAM_INST_DIR/**OpenFOAM-2.1.x/etc/bashrc
#[ -f $foamDotFile ] && . $foamDotFile
#source /mpi3/OpenFOAM/OpenFOAM-2.1.x/**etc/bashrc
#export FOAM_RUN=/mpi2/om
#export OpenBIN=/mpi2/OpenFOAM/**OpenFOAM-2.1.x/bin/tools
#export PATH=OpenBIN$:$PATH
#export LD_LIBRARY_PATH=$LD_LIBRARY_**PATH:/apps/OpenFOAM/tools/**lib64
#export
ParaView_DIR=/mpi2/OpenFOAM/**ThirdParty-2.1.x/platforms/**
linux64Gcc/paraview-3.12.0
#export PATH=$ParaView_DIR/bin:$PATH
#export PV_PLUGIN_PATH=$FOAM_LIBBIN/**paraview-3.12
# E L M E R
export ELMER_HOME=$JPK_ELMER
export ELMER_LIB=$JPK_ELMER/share/**elmersolver/lib
export PATH=$PATH:$ELMER_HOME/bin:$**ELMER_HOME/lib
export LD_LIBRARY_PATH=$LD_LIBRARY_**PATH:$ELMER_HOME/lib
export ELMERGUI_HOME=$ELMER_HOME/bin
export ELMER_POST_HOME=$ELMER_HOME/**bin
# S a l o m é
#cd /mpi2/salome-meca/SALOME-MECA-**2012.2-LGPL ; source
envSalomeMeca.sh
#cd ~/
# Paraview
#export PATH=$PATH:$JPK_INS/ParaView-**3.14.1-Linux-64bit
export PATH=$PATH:$JPK_INS/ParaView3
# N E T G E N   P A R A L L E L $JPK_TCL/lib:$JPK_TK/lib:
#export
LD_LIBRARY_PATH=$LD_LIBRARY_**PATH:$JPK_TOGL:$JPK_NETGEN\**
par/lib:/usr/lib/
#export NETGENDIR=$JPK_NETGEN\par/bin
# NETGEN
export LD_LIBRARY_PATH=$LD_LIBRARY_**PATH:$JPK_TOGL:$JPK_NETGEN/**
lib:/usr/lib/
export NETGENDIR=$JPK_NETGEN/bin
#crontab, ext editor
export EDITOR=nano
#space ball
export LD_LIBRARY_PATH=$LD_LIBRARY_**PATH:/usr/local/lib
#vrpn & hidapi
export
LD_LIBRARY_PATH=$LD_LIBRARY_**PATH:/$JPK_MPI_DIR/$JPK_VER/**
vrpn/lib:/$JPK_MPI_DIR/$JPK_**VER/hidapi/lib:/usr/include/**libusb-1.0
2013/8/27 Pavan Balaji <[email protected] <mailto:[email protected]>>
This is almost certainly a network issue with your third machine
    (kaak, I presume?).
Thanks for making sure "hostname" works fine on all machines.  That
    means that your ssh connections are setup correctly.  But a non-MPI
    program, such as hostname, does not check the connection from kaak
    back to mpi1.
Can you try a simple program like "examples/cpi" in the build
    directory on all machines?  Try it on 2 machines (mpiexec -np 4) and
    3 machines (mpiexec -np 6).
If the third machine is in fact having problems running the
application:
1. Make sure there's no firewall on the third machines.
2. Make sure the /etc/hosts file is consistent on both the machines
    (mpi1 and kaak).
-- Pavan
On 08/27/2013 06:46 AM, Joni-Pekka Kurronen wrote:
I have:
        -Ubuntu 12.4
        -rsh-redo-rsh
        -three machines
        -mpich3
        -have tried export HYDRA_DEMUX=select / poll
        -have tried ssh/rsh
        -have added to LIBS: event_core event_pthreads
I can run test at on to two machines whitout error but
        when I take third machine to cluster demux engine goes mad,...
           there is connection hanging,... and nothing happens,...
<MPITEST>
        <NAME>uoplong</NAME>
        <NP>11</NP>
        <WORKDIR>./coll</WORKDIR>
        <STATUS>fail</STATUS>
        <TESTDIFF>
        [mpiexec@mpi1] APPLICATION TIMED OUT
        [proxy:0:0@mpi1] HYD_pmcd_pmip_control_cmd_cb
        (./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed
        [proxy:0:0@mpi1] HYDT_dmxu_poll_wait_for_event
        (./tools/demux/demux_poll.c:__**77): callback returned error
status
[proxy:0:0@mpi1] main (./pm/pmiserv/pmip.c:206): demux engine
error
        waiting for event
        [mpiexec@mpi1] HYDT_bscu_wait_for_completion
        (./tools/bootstrap/utils/bscu_**__wait.c:76): one of the
processes
terminated badly; aborting
        [mpiexec@mpi1] HYDT_bsci_wait_for_completion
        (./tools/bootstrap/src/bsci___**wait.c:23): launcher returned
error waiting
        for completion
        [mpiexec@mpi1] HYD_pmci_wait_for_completion
        (./pm/pmiserv/pmiserv_pmci.c:_**_188): launcher returned error
waiting for
        completion
        [mpiexec@mpi1] main (./ui/mpich/mpiexec.c:331): process manager
        error
        waiting for completion
        </TESTDIFF>
        </MPITEST>
Also I can run
        joni@mpi1:/mpi3/S3/hpcc-1.4.2$ mpiexec -np 6 hostname
        mpi1
        mpi1
        ugh
        ugh
        kaak
        kaak
but if I run
        joni@mpi1:/mpi3/S3/hpcc-1.4.2$ mpiexec -np 6 ls
        I get only one directory as output and
        system will cease until I have re-started slave machines !
--
    Pavan Balaji
    http://www.mcs.anl.gov/~balaji
--
Joni-Pekka Kurronen
[email protected] <mailto:Joni.Kurronen@gmail.**com<[email protected]>
...
gsm. +358 50 521 2279
--
Pavan Balaji
http://www.mcs.anl.gov/~balaji
--
Joni-Pekka Kurronen
[email protected]
gsm. +358 50 521 2279
-- 
Joni-Pekka Kurronen
[email protected]
gsm. +358 50 521 2279

Joni-Pekka Kurronen

tags

participants (1)