July 2014 - discuss - mailman3.cels.anl.gov

problems with installing mpich: library missed
by Kirill Voronin 24 Jul '14

24 Jul '14

Hello everyone! I am trying to install mpich in order to use PARDISO project 5.0.0 (cluster MPI version for distributed memory machines), and since it fails during the compilation with the errors ld: warning: libmpich.so.3, needed by /ifs/home/icmmg/voronin/pardiso_proekt/pardiso_project/new/libpardiso500-MPI-INTEL1301-X86-64.so, not found (try using -rpath or -rpath-link) ld: warning: libpmi.so.0, needed by /ifs/home/icmmg/voronin/pardiso_proekt/pardiso_project/new/libpardiso500-MPI-INTEL1301-X86-64.so, not found (try using -rpath or -rpath-link) ld: warning: liblimic2.so.0, needed by /ifs/home/icmmg/voronin/pardiso_proekt/pardiso_project/new/libpardiso500-MPI-INTEL1301-X86-64.so, not found (try using -rpath or -rpath-link) I thought something related to mpich is missing (I found mpich mvapich2-x-1.9-6 installed on the main node but the required libraries were missing). Following the MPICH Installer's guide I tried to install the latest version of mpich into my home directory (with --enable-shared option) on the cluster, but only libmpich.so appears (and some other stuff which looks very far from the names of the libraries I need). No traces of libpmi.s0.0 or liblimic2.so.0 and the wrong version of libimpich.so. Can you give an idea why this happened or how can I deal with this problem? Thank you in advance! -- Kind regards, Kirill Voronin

1 0

Benefits of using intel compiler over gcc when building mpich
by Lukasz Zalewski 23 Jul '14

23 Jul '14

Hi all, Are there any benefits of building mpich with intel compiler over gcc? https://software.intel.com/sites/default/files/article/476380/intel-linux-c… states that it is compatible with gcc (4.1 or later). We would like to reduce number of different builds for the same mpich version. Regards L

1 0

Re: [mpich-discuss] Optimal Firewall Settings for MPICH/HYDRA
by Capehart, William J 22 Jul '14

22 Jul '14

Hi Pavan: I had already done that. With the firewall down, all is fine. With the firewall up, with the port files allowing the range of ports as requested in MPIEXEC_PORT_RANGE the program starts but once data begins to be tossed about in the cpi or fpi program, things go goes badly, I’m adding the full verbose dump of the cpi.c (below) Bill [me@localhost:/home/me]% mpiexec -v -n 2 -f nodesshort cpi.exe host: {local.machine.ip.address} host: {remote.machine.ip.address} =========================================================================== ======================= mpiexec options: ---------------- Base path: /usr/local/mpich/bin/ Launcher: (null) Debug level: 1 Enable X: -1 Global environment: ------------------- USER=me LOGNAME=me HOME=/home/me PATH=./:/bin:/usr/bin:/usr/local/bin:/usr/local/lib:/usr/local/netcdf/bin:/ usr/local/mpich/bin/openmpi:/usr/local/mpich/bin:/usr/local/mpich/include:/ usr/local/mpich/lib:/usr/local/ncarg:/usr/local/netcdf:/projects/WRF_UTIL/W PSV3:/usr/local/netcdf/lib:/usr/local/netcdf/include:/usr/local/include:/us r/local/lib:/home/me/bin:/usr/local/ncarg/bin:/usr/lib64/qt-3.3/bin:/usr/lo cal/bin:/bin:/usr/bin:/usr/local/pgi/linux86-64/2014/bin:/usr/local/pgi/lin ux86-64/2014/lib MAIL=/var/spool/mail/me SHELL=/bin/tcsh SSH_CLIENT={local.machine.ip.address} 41583 22 SSH_CONNECTION={local.machine.ip.address} 41583 {local.machine.ip.address} 22 SSH_TTY=/dev/pts/3 TERM=xterm-color SELINUX_ROLE_REQUESTED= SELINUX_LEVEL_REQUESTED= SELINUX_USE_CURRENT_RANGE= HOSTTYPE=x86_64-linux VENDOR=unknown OSTYPE=linux MACHTYPE=x86_64 SHLVL=1 PWD=/home/me GROUP=iasusers HOST=local.host.name REMOTEHOST=local.host.name HOSTNAME=local.host.name LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33; 01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30; 42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01; 31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:* .Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;3 1:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*. rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg= 01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01 ;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;3 5:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35: *.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*. mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm= 01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01; 35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:* .axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.fla c=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg= 01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01; 36: CVS_RSH=ssh GDL_PATH=+/usr/share/gnudatalanguage G_BROKEN_FILENAMES=1 SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass LANG=en_US.UTF-8 LESSOPEN=|/usr/bin/lesspipe.sh %s QTDIR=/usr/lib64/qt-3.3 QTINC=/usr/lib64/qt-3.3/include QTLIB=/usr/lib64/qt-3.3/lib COMPILER_OPTION=PGI LINUX_MPIHOME=/usr/local/mpich MPICH=/usr/local/mpich LD_LIBRARY_PATH=/usr/local/mpich/lib:/usr/local/mpich/lib:/usr/local/lib:/u sr/local/netcdf/lib:/usr/local/pgi/linux86-64/2014/libso/ LD_RUN_PATH=/usr/local/mpich/include/openmpi:/usr/local/mpich/include:/usr/ local/netcdf/include:/usr/local/include:/usr/local/lib NODES=/home/me/nodes HYDRA_HOST_FILE=/home/me/nodes MPIEXEC_PORT_RANGE=10000:10100 MPIR_CVAR_CH3_PORT_RANGE=10000:10100 NCARG_ROOT=/usr/local/ncarg NCARG_BIN=/usr/local/ncarg/bin NCARG_LIB=/usr/local/ncarg/lib NCARG_INCLUDE=/usr/local/ncarg/include NCL_COMMAND=/usr/local/ncarg/bin/ncl NCARG_RANGS=/data/NCAR/RANGS ITT=/usr/local/exelis IDL_DIR=/usr/local/exelis/idl83 ENVI_DIR=/usr/local/exelis/envi51 EXELIS_DIR=/usr/local/exelis IDL_PATH=+/home/me/tools:+/usr/local/exelis/idl83/lib:+/usr/local/exelis/id l83/examples:/projects/idl_coyote NETCDF=/usr/local/netcdf NETCDFLIB=/usr/local/netcdf/lib NETCDFINC=/usr/local/netcdf/include NETCDF4=1 PNETINC=-I/usr/local/parallel_netcdf_hdf/include PNETLIB=-L/usr/local/parallel_netcdf_hdf/lib -lnetcdf -lnetcdff -ldl -lhdf5 -lhdf5_hl -lz -lsz HDF5=/usr/local HDFLIB=/usr/local/lib HDFINC=/usr/local/include PGI=/usr/local/pgi PGIVERSION=/usr/local/pgi/linux86-64/2014 LM_LICENSE_FILE=/usr/local/pgi/license.dat CC=pgcc FC=pgfortran F90=pgfortran F77=pgfortran CXX=pgcpp MPIFC=mpif90 MPIF90=mpif90 MPIF77=mpif90 MPICC=mpicc MPICXX=mpicxx CPP=pgCC -E CFLAGS= -Msignextend -fPIC CPPFLAGS= -DNDEBUG -DpgiFortran -fPIC CXXFLAGS= -fPIC F90FLAGS= -fPIC FFLAGS= -w -fPIC LDFLAGS= RSHCOMMAND=ssh MP_STACK_SIZE=80000000 OMP_NUM_THREADS=16 JASPERLIB=/usr/local/lib JASPERINC=/usr/local/include LFC=-lgfortran LDSO=/lib64/ld-linux-x86-64.so.2 GCCDIR=/usr/lib/gcc/x86_64-redhat-linux/4.4.7 GCCINC=/usr/lib/gcc/x86_64-redhat-linux/4.4.7/include G77DIR=/usr/lib/gcc/x86_64-redhat-linux/4.4.7 HDF5_DISABLE_VERSION_CHECK=1 WRFIO_NCD_LARGE_FILE_SUPPORT=1 ESMF_DIR=/usr/local/esmfinstall ESMF_OS=Linux ESMF_BOPT=O ESMF_OPTLEVEL=0 ESMF_ABI=64 ESMF_COMM=mpich ESMF_COMPILER=pgi ESMF_INSTALL_PREFIX=/usr/local/esmf Hydra internal environment: --------------------------- GFORTRAN_UNBUFFERED_PRECONNECTED=y Proxy information: ********************* [1] proxy: {local.machine.ip.address} (1 cores) Exec list: cpi.exe (1 processes); [2] proxy: {remote.machine.ip.address} (1 cores) Exec list: cpi.exe (1 processes); =========================================================================== ======================= [mpiexec(a)local.host.name] Timeout set to -1 (-1 means infinite) [mpiexec(a)local.host.name] Got a control port string of {local.machine.ip.address}:10000 Proxy launch args: /usr/local/mpich/bin/hydra_pmi_proxy --control-port {local.machine.ip.address}:10000 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id Arguments being passed to proxy 0: --version 3.0.4 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME --hostname {local.machine.ip.address} --global-core-map 0,1,2 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_6142_0 --pmi-process-mapping (vector,(0,2,1)) --ckpoint-num -1 --global-inherited-env 102 'USER=me' 'LOGNAME=me' 'HOME=/home/me' 'PATH=./:/bin:/usr/bin:/usr/local/bin:/usr/local/lib:/usr/local/netcdf/bin: /usr/local/mpich/bin/openmpi:/usr/local/mpich/bin:/usr/local/mpich/include: /usr/local/mpich/lib:/usr/local/ncarg:/usr/local/netcdf:/projects/WRF_UTIL/ WPSV3:/usr/local/netcdf/lib:/usr/local/netcdf/include:/usr/local/include:/u sr/local/lib:/home/me/bin:/usr/local/ncarg/bin:/usr/lib64/qt-3.3/bin:/usr/l ocal/bin:/bin:/usr/bin:/usr/local/pgi/linux86-64/2014/bin:/usr/local/pgi/li nux86-64/2014/lib' 'MAIL=/var/spool/mail/me' 'SHELL=/bin/tcsh' 'SSH_CLIENT={local.machine.ip.address} 41583 22' 'SSH_CONNECTION={local.machine.ip.address} 41583 {local.machine.ip.address} 22' 'SSH_TTY=/dev/pts/3' 'TERM=xterm-color' 'SELINUX_ROLE_REQUESTED=' 'SELINUX_LEVEL_REQUESTED=' 'SELINUX_USE_CURRENT_RANGE=' 'HOSTTYPE=x86_64-linux' 'VENDOR=unknown' 'OSTYPE=linux' 'MACHTYPE=x86_64' 'SHLVL=1' 'PWD=/home/me' 'GROUP=iasusers' 'HOST=local.host.name' 'REMOTEHOST=local.host.name' 'HOSTNAME=local.host.name' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33 ;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30 ;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01 ;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31: *.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01; 31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:* .rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg =01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=0 1;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01; 35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35 :*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:* .mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm =01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01 ;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35: *.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.fl ac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg =01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01 ;36:' 'CVS_RSH=ssh' 'GDL_PATH=+/usr/share/gnudatalanguage' 'G_BROKEN_FILENAMES=1' 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass' 'LANG=en_US.UTF-8' 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'QTDIR=/usr/lib64/qt-3.3' 'QTINC=/usr/lib64/qt-3.3/include' 'QTLIB=/usr/lib64/qt-3.3/lib' 'COMPILER_OPTION=PGI' 'LINUX_MPIHOME=/usr/local/mpich' 'MPICH=/usr/local/mpich' 'LD_LIBRARY_PATH=/usr/local/mpich/lib:/usr/local/mpich/lib:/usr/local/lib:/ usr/local/netcdf/lib:/usr/local/pgi/linux86-64/2014/libso/' 'LD_RUN_PATH=/usr/local/mpich/include/openmpi:/usr/local/mpich/include:/usr /local/netcdf/include:/usr/local/include:/usr/local/lib' 'NODES=/home/me/nodes' 'HYDRA_HOST_FILE=/home/me/nodes' 'MPIEXEC_PORT_RANGE=10000:10100' 'MPIR_CVAR_CH3_PORT_RANGE=10000:10100' 'NCARG_ROOT=/usr/local/ncarg' 'NCARG_BIN=/usr/local/ncarg/bin' 'NCARG_LIB=/usr/local/ncarg/lib' 'NCARG_INCLUDE=/usr/local/ncarg/include' 'NCL_COMMAND=/usr/local/ncarg/bin/ncl' 'NCARG_RANGS=/data/NCAR/RANGS' 'ITT=/usr/local/exelis' 'IDL_DIR=/usr/local/exelis/idl83' 'ENVI_DIR=/usr/local/exelis/envi51' 'EXELIS_DIR=/usr/local/exelis' 'IDL_PATH=+/home/me/tools:+/usr/local/exelis/idl83/lib:+/usr/local/exelis/i dl83/examples:/projects/idl_coyote' 'NETCDF=/usr/local/netcdf' 'NETCDFLIB=/usr/local/netcdf/lib' 'NETCDFINC=/usr/local/netcdf/include' 'NETCDF4=1' 'PNETINC=-I/usr/local/parallel_netcdf_hdf/include' 'PNETLIB=-L/usr/local/parallel_netcdf_hdf/lib -lnetcdf -lnetcdff -ldl -lhdf5 -lhdf5_hl -lz -lsz ' 'HDF5=/usr/local' 'HDFLIB=/usr/local/lib' 'HDFINC=/usr/local/include' 'PGI=/usr/local/pgi' 'PGIVERSION=/usr/local/pgi/linux86-64/2014' 'LM_LICENSE_FILE=/usr/local/pgi/license.dat' 'CC=pgcc' 'FC=pgfortran' 'F90=pgfortran' 'F77=pgfortran' 'CXX=pgcpp' 'MPIFC=mpif90' 'MPIF90=mpif90' 'MPIF77=mpif90' 'MPICC=mpicc' 'MPICXX=mpicxx' 'CPP=pgCC -E' 'CFLAGS= -Msignextend -fPIC ' 'CPPFLAGS= -DNDEBUG -DpgiFortran -fPIC ' 'CXXFLAGS= -fPIC ' 'F90FLAGS= -fPIC ' 'FFLAGS= -w -fPIC ' 'LDFLAGS= ' 'RSHCOMMAND=ssh' 'MP_STACK_SIZE=80000000' 'OMP_NUM_THREADS=16' 'JASPERLIB=/usr/local/lib' 'JASPERINC=/usr/local/include' 'LFC=-lgfortran' 'LDSO=/lib64/ld-linux-x86-64.so.2' 'GCCDIR=/usr/lib/gcc/x86_64-redhat-linux/4.4.7' 'GCCINC=/usr/lib/gcc/x86_64-redhat-linux/4.4.7/include' 'G77DIR=/usr/lib/gcc/x86_64-redhat-linux/4.4.7' 'HDF5_DISABLE_VERSION_CHECK=1' 'WRFIO_NCD_LARGE_FILE_SUPPORT=1' 'ESMF_DIR=/usr/local/esmfinstall' 'ESMF_OS=Linux' 'ESMF_BOPT=O' 'ESMF_OPTLEVEL=0' 'ESMF_ABI=64' 'ESMF_COMM=mpich' 'ESMF_COMPILER=pgi' 'ESMF_INSTALL_PREFIX=/usr/local/esmf' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/me --exec-args 1 cpi.exe Arguments being passed to proxy 1: --version 3.0.4 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME --hostname {remote.machine.ip.address} --global-core-map 0,1,2 --pmi-id-map 0,1 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_6142_0 --pmi-process-mapping (vector,(0,2,1)) --ckpoint-num -1 --global-inherited-env 102 'USER=me' 'LOGNAME=me' 'HOME=/home/me' 'PATH=./:/bin:/usr/bin:/usr/local/bin:/usr/local/lib:/usr/local/netcdf/bin: /usr/local/mpich/bin/openmpi:/usr/local/mpich/bin:/usr/local/mpich/include: /usr/local/mpich/lib:/usr/local/ncarg:/usr/local/netcdf:/projects/WRF_UTIL/ WPSV3:/usr/local/netcdf/lib:/usr/local/netcdf/include:/usr/local/include:/u sr/local/lib:/home/me/bin:/usr/local/ncarg/bin:/usr/lib64/qt-3.3/bin:/usr/l ocal/bin:/bin:/usr/bin:/usr/local/pgi/linux86-64/2014/bin:/usr/local/pgi/li nux86-64/2014/lib' 'MAIL=/var/spool/mail/me' 'SHELL=/bin/tcsh' 'SSH_CLIENT={local.machine.ip.address} 41583 22' 'SSH_CONNECTION={local.machine.ip.address} 41583 {local.machine.ip.address} 22' 'SSH_TTY=/dev/pts/3' 'TERM=xterm-color' 'SELINUX_ROLE_REQUESTED=' 'SELINUX_LEVEL_REQUESTED=' 'SELINUX_USE_CURRENT_RANGE=' 'HOSTTYPE=x86_64-linux' 'VENDOR=unknown' 'OSTYPE=linux' 'MACHTYPE=x86_64' 'SHLVL=1' 'PWD=/home/me' 'GROUP=iasusers' 'HOST=local.host.name' 'REMOTEHOST=local.host.name' 'HOSTNAME=local.host.name' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33 ;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30 ;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01 ;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31: *.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01; 31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:* .rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg =01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=0 1;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01; 35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35 :*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:* .mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm =01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01 ;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35: *.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.fl ac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg =01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01 ;36:' 'CVS_RSH=ssh' 'GDL_PATH=+/usr/share/gnudatalanguage' 'G_BROKEN_FILENAMES=1' 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass' 'LANG=en_US.UTF-8' 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'QTDIR=/usr/lib64/qt-3.3' 'QTINC=/usr/lib64/qt-3.3/include' 'QTLIB=/usr/lib64/qt-3.3/lib' 'COMPILER_OPTION=PGI' 'LINUX_MPIHOME=/usr/local/mpich' 'MPICH=/usr/local/mpich' 'LD_LIBRARY_PATH=/usr/local/mpich/lib:/usr/local/mpich/lib:/usr/local/lib:/ usr/local/netcdf/lib:/usr/local/pgi/linux86-64/2014/libso/' 'LD_RUN_PATH=/usr/local/mpich/include/openmpi:/usr/local/mpich/include:/usr /local/netcdf/include:/usr/local/include:/usr/local/lib' 'NODES=/home/me/nodes' 'HYDRA_HOST_FILE=/home/me/nodes' 'MPIEXEC_PORT_RANGE=10000:10100' 'MPIR_CVAR_CH3_PORT_RANGE=10000:10100' 'NCARG_ROOT=/usr/local/ncarg' 'NCARG_BIN=/usr/local/ncarg/bin' 'NCARG_LIB=/usr/local/ncarg/lib' 'NCARG_INCLUDE=/usr/local/ncarg/include' 'NCL_COMMAND=/usr/local/ncarg/bin/ncl' 'NCARG_RANGS=/data/NCAR/RANGS' 'ITT=/usr/local/exelis' 'IDL_DIR=/usr/local/exelis/idl83' 'ENVI_DIR=/usr/local/exelis/envi51' 'EXELIS_DIR=/usr/local/exelis' 'IDL_PATH=+/home/me/tools:+/usr/local/exelis/idl83/lib:+/usr/local/exelis/i dl83/examples:/projects/idl_coyote' 'NETCDF=/usr/local/netcdf' 'NETCDFLIB=/usr/local/netcdf/lib' 'NETCDFINC=/usr/local/netcdf/include' 'NETCDF4=1' 'PNETINC=-I/usr/local/parallel_netcdf_hdf/include' 'PNETLIB=-L/usr/local/parallel_netcdf_hdf/lib -lnetcdf -lnetcdff -ldl -lhdf5 -lhdf5_hl -lz -lsz ' 'HDF5=/usr/local' 'HDFLIB=/usr/local/lib' 'HDFINC=/usr/local/include' 'PGI=/usr/local/pgi' 'PGIVERSION=/usr/local/pgi/linux86-64/2014' 'LM_LICENSE_FILE=/usr/local/pgi/license.dat' 'CC=pgcc' 'FC=pgfortran' 'F90=pgfortran' 'F77=pgfortran' 'CXX=pgcpp' 'MPIFC=mpif90' 'MPIF90=mpif90' 'MPIF77=mpif90' 'MPICC=mpicc' 'MPICXX=mpicxx' 'CPP=pgCC -E' 'CFLAGS= -Msignextend -fPIC ' 'CPPFLAGS= -DNDEBUG -DpgiFortran -fPIC ' 'CXXFLAGS= -fPIC ' 'F90FLAGS= -fPIC ' 'FFLAGS= -w -fPIC ' 'LDFLAGS= ' 'RSHCOMMAND=ssh' 'MP_STACK_SIZE=80000000' 'OMP_NUM_THREADS=16' 'JASPERLIB=/usr/local/lib' 'JASPERINC=/usr/local/include' 'LFC=-lgfortran' 'LDSO=/lib64/ld-linux-x86-64.so.2' 'GCCDIR=/usr/lib/gcc/x86_64-redhat-linux/4.4.7' 'GCCINC=/usr/lib/gcc/x86_64-redhat-linux/4.4.7/include' 'G77DIR=/usr/lib/gcc/x86_64-redhat-linux/4.4.7' 'HDF5_DISABLE_VERSION_CHECK=1' 'WRFIO_NCD_LARGE_FILE_SUPPORT=1' 'ESMF_DIR=/usr/local/esmfinstall' 'ESMF_OS=Linux' 'ESMF_BOPT=O' 'ESMF_OPTLEVEL=0' 'ESMF_ABI=64' 'ESMF_COMM=mpich' 'ESMF_COMPILER=pgi' 'ESMF_INSTALL_PREFIX=/usr/local/esmf' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/me --exec-args 1 cpi.exe [mpiexec(a)local.host.name] Launch arguments: /usr/local/mpich/bin/hydra_pmi_proxy --control-port {local.machine.ip.address}:10000 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 0 [mpiexec(a)local.host.name] Launch arguments: /usr/bin/ssh -x {remote.machine.ip.address} "/usr/local/mpich/bin/hydra_pmi_proxy" --control-port {local.machine.ip.address}:10000 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 1 [proxy:0:[email protected]] got pmi command (from 0): init pmi_version=1 pmi_subversion=1 [proxy:0:[email protected]] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:[email protected]] got pmi command (from 0): get_maxes [proxy:0:[email protected]] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:[email protected]] got pmi command (from 0): get_appnum [proxy:0:[email protected]] PMI response: cmd=appnum appnum=0 [proxy:0:[email protected]] got pmi command (from 0): get_my_kvsname [proxy:0:[email protected]] PMI response: cmd=my_kvsname kvsname=kvs_6142_0 [proxy:0:[email protected]] got pmi command (from 0): get_my_kvsname [proxy:0:[email protected]] PMI response: cmd=my_kvsname kvsname=kvs_6142_0 [proxy:0:[email protected]] got pmi command (from 0): get kvsname=kvs_6142_0 key=PMI_process_mapping [proxy:0:[email protected]] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:[email protected]] got pmi command (from 0): barrier_in [proxy:0:[email protected]] forwarding command (cmd=barrier_in) upstream [mpiexec(a)local.host.name] [pgid: 0] got PMI command: cmd=barrier_in [proxy:0:[email protected]] got pmi command (from 4): init pmi_version=1 pmi_subversion=1 [proxy:0:[email protected]] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:[email protected]] got pmi command (from 4): get_maxes [proxy:0:[email protected]] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:[email protected]] got pmi command (from 4): get_appnum [proxy:0:[email protected]] PMI response: cmd=appnum appnum=0 [proxy:0:[email protected]] got pmi command (from 4): get_my_kvsname [proxy:0:[email protected]] PMI response: cmd=my_kvsname kvsname=kvs_6142_0 [proxy:0:[email protected]] got pmi command (from 4): get_my_kvsname [proxy:0:[email protected]] PMI response: cmd=my_kvsname kvsname=kvs_6142_0 [mpiexec(a)local.host.name] [pgid: 0] got PMI command: cmd=barrier_in [mpiexec(a)local.host.name] PMI response to fd 6 pid 4: cmd=barrier_out [mpiexec(a)local.host.name] PMI response to fd 7 pid 4: cmd=barrier_out [proxy:0:[email protected]] got pmi command (from 4): get kvsname=kvs_6142_0 key=PMI_process_mapping [proxy:0:[email protected]] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:[email protected]] got pmi command (from 4): barrier_in [proxy:0:[email protected]] forwarding command (cmd=barrier_in) upstream [proxy:0:[email protected]] PMI response: cmd=barrier_out [proxy:0:[email protected]] got pmi command (from 0): put kvsname=kvs_6142_0 key=P0-businesscard value=description#{local.machine.ip.address}$port#33774$ifname#{local.machi ne.ip.address}$ [proxy:0:[email protected]] cached command: P0-businesscard=description#{local.machine.ip.address}$port#33774$ifname#{l ocal.machine.ip.address}$ [proxy:0:[email protected]] PMI response: cmd=put_result rc=0 msg=success [proxy:0:[email protected]] got pmi command (from 0): barrier_in [proxy:0:[email protected]] flushing 1 put command(s) out [proxy:0:[email protected]] forwarding command (cmd=put P0-businesscard=description#{local.machine.ip.address}$port#33774$ifname#{l ocal.machine.ip.address}$) upstream [mpiexec(a)local.host.name] [pgid: 0] got PMI command: cmd=put P0-businesscard=description#{local.machine.ip.address}$port#33774$ifname#{l ocal.machine.ip.address}$ [proxy:0:[email protected]] forwarding command (cmd=barrier_in) upstream [mpiexec(a)local.host.name] [pgid: 0] got PMI command: cmd=barrier_in [proxy:0:[email protected]] PMI response: cmd=barrier_out [proxy:0:[email protected]] got pmi command (from 4): put kvsname=kvs_6142_0 key=P1-businesscard value=description#{remote.machine.ip.address}$port#44324$ifname#{remote.mac hine.ip.address}$ [proxy:0:[email protected]] cached command: P1-businesscard=description#{remote.machine.ip.address}$port#44324$ifname#{ remote.machine.ip.address}$ [proxy:0:[email protected]] PMI response: cmd=put_result rc=0 msg=success [mpiexec(a)local.host.name] [pgid: 0] got PMI command: cmd=put P1-businesscard=description#{remote.machine.ip.address}$port#44324$ifname#{ remote.machine.ip.address}$ [proxy:0:[email protected]] got pmi command (from 4): barrier_in [proxy:0:[email protected]] flushing 1 put command(s) out [proxy:0:[email protected]] forwarding command (cmd=put P1-businesscard=description#{remote.machine.ip.address}$port#44324$ifname#{ remote.machine.ip.address}$) upstream [proxy:0:[email protected]] forwarding command (cmd=barrier_in) upstream [mpiexec(a)local.host.name] [pgid: 0] got PMI command: cmd=barrier_in [mpiexec(a)local.host.name] PMI response to fd 6 pid 4: cmd=keyval_cache P0-businesscard=description#{local.machine.ip.address}$port#33774$ifname#{l ocal.machine.ip.address}$ P1-businesscard=description#{remote.machine.ip.address}$port#44324$ifname#{ remote.machine.ip.address}$ [mpiexec(a)local.host.name] PMI response to fd 7 pid 4: cmd=keyval_cache P0-businesscard=description#{local.machine.ip.address}$port#33774$ifname#{l ocal.machine.ip.address}$ P1-businesscard=description#{remote.machine.ip.address}$port#44324$ifname#{ remote.machine.ip.address}$ [mpiexec(a)local.host.name] PMI response to fd 6 pid 4: cmd=barrier_out [mpiexec(a)local.host.name] PMI response to fd 7 pid 4: cmd=barrier_out [proxy:0:[email protected]] PMI response: cmd=barrier_out Process 0 of 2 is on local.host.name [proxy:0:[email protected]] got pmi command (from 0): get kvsname=kvs_6142_0 key=P1-businesscard [proxy:0:[email protected]] PMI response: cmd=get_result rc=0 msg=success value=description#{remote.machine.ip.address}$port#44324$ifname#{remote.mac hine.ip.address}$ [proxy:0:[email protected]] PMI response: cmd=barrier_out Process 1 of 2 is on remote.host.name Fatal error in PMPI_Reduce: A process has failed, error stack: PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff91d58820, rbuf=0x7fff91d58828, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(835)..........: MPIR_Reduce_binomial(144).......: MPIDI_CH3U_Recvq_FDU_or_AEP(667): Communication error with rank 1 =========================================================================== ======== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =========================================================================== ======== [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status [proxy:0:[email protected]] main (./pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec(a)local.host.name] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec(a)local.host.name] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec(a)local.host.name] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for completion [mpiexec(a)local.host.name] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion On 7/22/14, 13:44 MDT, "Balaji, Pavan" <balaji(a)anl.gov> wrote: >Bill, > >Just to make sure this is a firewall problem, can you try disabling the >firewall for a short time to try out MPICH and see if it works correctly? > Remember to turn off the firewall on all machines, not just the head >node. > > — Pavan > >On Jul 22, 2014, at 2:18 PM, Capehart, William J ><William.Capehart(a)sdsmt.edu> wrote: > >> That would be the one that comes with PGI 14.6 (MPICH 3.0.4) >> >> Bill >> >> >> On 7/22/14, 11:52 MDT, "Kenneth Raffenetti" <raffenet(a)mcs.anl.gov> >>wrote: >> >>> What version of MPICH/Hydra is this? >>> >>> On 07/22/2014 12:48 PM, Capehart, William J wrote: >>>> Hi All >>>> >>>> We¹re running MPICH on a couple machines with a brand new UNIX distro >>>> (SL 6.5) and that are on a vulnerable network and rather than leave >>>>the >>>> firewalls dropped we would like to run it through the firewall. >>>> >>>> We have included the MPIEXEC_PORT_RANGE and MPIR_CVAR_CH3_PORT_RANGE >>>> fields and >>>> have adjusted our iptables accordingly and in line with the ³FAQ² >>>> guidance. >>>> >>>> Our passwordless SSH works fine between the machines. >>>> >>>> But all of this gives us momentary success with the cpi and fpi MPICH >>>> test programs. But they crash with the firewall up. (but of course >>>>run >>>> happily with the firewall down). >>>> >>>> An example of the basic output is below (node short sends one process >>>>to >>>> ³this.machine² and one to remote ³that.machine² >>>> >>>> >>>> [this.machine]% mpiexec -n 2 -f nodesshort cpi.exe >>>> >>>> Process 0 of 2 is on this.machine >>>> >>>> Process 1 of 2 is on that.machine >>>> >>>> Fatal error in PMPI_Reduce: A process has failed, error stack: >>>> >>>> PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff466a94d0, >>>> rbuf=0x7fff466a94d8, count=1, MPI_DOUBLE, MPI_SUM, root=0, >>>> MPI_COMM_WORLD) failed >>>> >>>> MPIR_Reduce_impl(1029)..........: >>>> >>>> MPIR_Reduce_intra(835)..........: >>>> >>>> MPIR_Reduce_binomial(144).......: >>>> >>>> MPIDI_CH3U_Recvq_FDU_or_AEP(667): Communication error with rank 1 >>>> >>>> >>>> >>>> >>>>======================================================================= >>>>== >>>> ========== >>>> >>>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>>> >>>> = EXIT CODE: 1 >>>> >>>> = CLEANING UP REMAINING PROCESSES >>>> >>>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>>> >>>> >>>> >>>>======================================================================= >>>>== >>>> ========== >>>> >>>> [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb >>>> (./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed >>>> >>>> [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event >>>> (./tools/demux/demux_poll.c:77): callback returned error status >>>> >>>> [proxy:0:[email protected]] main (./pm/pmiserv/pmip.c:206): demux engine >>>> error waiting for event >>>> >>>> [mpiexec(a)this.machine] HYDT_bscu_wait_for_completion >>>> (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes >>>> terminated badly; aborting >>>> >>>> [mpiexec(a)this.machine] HYDT_bsci_wait_for_completion >>>> (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error >>>>waiting >>>> for completion >>>> >>>> [mpiexec(a)this.machine] HYD_pmci_wait_for_completion >>>> (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for >>>> completion >>>> >>>> [mpiexec(a)this.machine] main (./ui/mpich/mpiexec.c:331): process >>>>manager >>>> error waiting for completion >>>> >>>> >>>> >>>> In debug mode it affirms that it is at least *starting with the first >>>> available port as listed in MPIEXEC_PORT_RANGE >>>> >>>> But later we get output like this: >>>> >>>> [mpiexec(a)this.machine] PMI response to fd 6 pid 4: cmd=keyval_cache >>>> >>>> >>>>P0-businesscard=description#{this.machine¹s.ip.address}$port#54105$ifna >>>>me >>>> #{this.machine¹s.ip.address}$ >>>> >>>> >>>>P1-businesscard=description#{that.machine¹s.ip.address}$port#47302$ifna >>>>me >>>> #{that.machine¹s.ip.address}$ >>>> >>>> >>>> >>>> Does this mean that we have missed a firewall setting either in the >>>> environment variables or in the ip tables themselves? >>>> >>>> >>>> Ideas? >>>> >>>> >>>> >>>> Thanks Much >>>> >>>> Bill >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > >-- >Pavan Balaji ✉️ >http://www.mcs.anl.gov/~balaji > >_______________________________________________ >discuss mailing list discuss(a)mpich.org >To manage subscription options or unsubscribe: >https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Optimal Firewall Settings for MPICH/HYDRA
by Balaji, Pavan 22 Jul '14

22 Jul '14

Bill, Just to make sure this is a firewall problem, can you try disabling the firewall for a short time to try out MPICH and see if it works correctly? Remember to turn off the firewall on all machines, not just the head node. — Pavan On Jul 22, 2014, at 2:18 PM, Capehart, William J <William.Capehart(a)sdsmt.edu> wrote: > That would be the one that comes with PGI 14.6 (MPICH 3.0.4) > > Bill > > > On 7/22/14, 11:52 MDT, "Kenneth Raffenetti" <raffenet(a)mcs.anl.gov> wrote: > >> What version of MPICH/Hydra is this? >> >> On 07/22/2014 12:48 PM, Capehart, William J wrote: >>> Hi All >>> >>> We¹re running MPICH on a couple machines with a brand new UNIX distro >>> (SL 6.5) and that are on a vulnerable network and rather than leave the >>> firewalls dropped we would like to run it through the firewall. >>> >>> We have included the MPIEXEC_PORT_RANGE and MPIR_CVAR_CH3_PORT_RANGE >>> fields and >>> have adjusted our iptables accordingly and in line with the ³FAQ² >>> guidance. >>> >>> Our passwordless SSH works fine between the machines. >>> >>> But all of this gives us momentary success with the cpi and fpi MPICH >>> test programs. But they crash with the firewall up. (but of course run >>> happily with the firewall down). >>> >>> An example of the basic output is below (node short sends one process to >>> ³this.machine² and one to remote ³that.machine² >>> >>> >>> [this.machine]% mpiexec -n 2 -f nodesshort cpi.exe >>> >>> Process 0 of 2 is on this.machine >>> >>> Process 1 of 2 is on that.machine >>> >>> Fatal error in PMPI_Reduce: A process has failed, error stack: >>> >>> PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff466a94d0, >>> rbuf=0x7fff466a94d8, count=1, MPI_DOUBLE, MPI_SUM, root=0, >>> MPI_COMM_WORLD) failed >>> >>> MPIR_Reduce_impl(1029)..........: >>> >>> MPIR_Reduce_intra(835)..........: >>> >>> MPIR_Reduce_binomial(144).......: >>> >>> MPIDI_CH3U_Recvq_FDU_or_AEP(667): Communication error with rank 1 >>> >>> >>> >>> ========================================================================= >>> ========== >>> >>> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >>> >>> = EXIT CODE: 1 >>> >>> = CLEANING UP REMAINING PROCESSES >>> >>> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >>> >>> >>> ========================================================================= >>> ========== >>> >>> [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb >>> (./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed >>> >>> [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event >>> (./tools/demux/demux_poll.c:77): callback returned error status >>> >>> [proxy:0:[email protected]] main (./pm/pmiserv/pmip.c:206): demux engine >>> error waiting for event >>> >>> [mpiexec(a)this.machine] HYDT_bscu_wait_for_completion >>> (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes >>> terminated badly; aborting >>> >>> [mpiexec(a)this.machine] HYDT_bsci_wait_for_completion >>> (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting >>> for completion >>> >>> [mpiexec(a)this.machine] HYD_pmci_wait_for_completion >>> (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for >>> completion >>> >>> [mpiexec(a)this.machine] main (./ui/mpich/mpiexec.c:331): process manager >>> error waiting for completion >>> >>> >>> >>> In debug mode it affirms that it is at least *starting with the first >>> available port as listed in MPIEXEC_PORT_RANGE >>> >>> But later we get output like this: >>> >>> [mpiexec(a)this.machine] PMI response to fd 6 pid 4: cmd=keyval_cache >>> >>> P0-businesscard=description#{this.machine¹s.ip.address}$port#54105$ifname >>> #{this.machine¹s.ip.address}$ >>> >>> P1-businesscard=description#{that.machine¹s.ip.address}$port#47302$ifname >>> #{that.machine¹s.ip.address}$ >>> >>> >>> >>> Does this mean that we have missed a firewall setting either in the >>> environment variables or in the ip tables themselves? >>> >>> >>> Ideas? >>> >>> >>> >>> Thanks Much >>> >>> Bill >>> >>> >>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji ✉️ http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Optimal Firewall Settings for MPICH/HYDRA
by Capehart, William J 22 Jul '14

22 Jul '14

That would be the one that comes with PGI 14.6 (MPICH 3.0.4) Bill On 7/22/14, 11:52 MDT, "Kenneth Raffenetti" <raffenet(a)mcs.anl.gov> wrote: >What version of MPICH/Hydra is this? > >On 07/22/2014 12:48 PM, Capehart, William J wrote: >> Hi All >> >> We¹re running MPICH on a couple machines with a brand new UNIX distro >> (SL 6.5) and that are on a vulnerable network and rather than leave the >> firewalls dropped we would like to run it through the firewall. >> >> We have included the MPIEXEC_PORT_RANGE and MPIR_CVAR_CH3_PORT_RANGE >> fields and >> have adjusted our iptables accordingly and in line with the ³FAQ² >>guidance. >> >> Our passwordless SSH works fine between the machines. >> >> But all of this gives us momentary success with the cpi and fpi MPICH >> test programs. But they crash with the firewall up. (but of course run >> happily with the firewall down). >> >> An example of the basic output is below (node short sends one process to >> ³this.machine² and one to remote ³that.machine² >> >> >> [this.machine]% mpiexec -n 2 -f nodesshort cpi.exe >> >> Process 0 of 2 is on this.machine >> >> Process 1 of 2 is on that.machine >> >> Fatal error in PMPI_Reduce: A process has failed, error stack: >> >> PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff466a94d0, >> rbuf=0x7fff466a94d8, count=1, MPI_DOUBLE, MPI_SUM, root=0, >> MPI_COMM_WORLD) failed >> >> MPIR_Reduce_impl(1029)..........: >> >> MPIR_Reduce_intra(835)..........: >> >> MPIR_Reduce_binomial(144).......: >> >> MPIDI_CH3U_Recvq_FDU_or_AEP(667): Communication error with rank 1 >> >> >> >>========================================================================= >>========== >> >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> >> = EXIT CODE: 1 >> >> = CLEANING UP REMAINING PROCESSES >> >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> >> >>========================================================================= >>========== >> >> [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb >> (./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed >> >> [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event >> (./tools/demux/demux_poll.c:77): callback returned error status >> >> [proxy:0:[email protected]] main (./pm/pmiserv/pmip.c:206): demux engine >> error waiting for event >> >> [mpiexec(a)this.machine] HYDT_bscu_wait_for_completion >> (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes >> terminated badly; aborting >> >> [mpiexec(a)this.machine] HYDT_bsci_wait_for_completion >> (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting >> for completion >> >> [mpiexec(a)this.machine] HYD_pmci_wait_for_completion >> (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for >> completion >> >> [mpiexec(a)this.machine] main (./ui/mpich/mpiexec.c:331): process manager >> error waiting for completion >> >> >> >> In debug mode it affirms that it is at least *starting with the first >> available port as listed in MPIEXEC_PORT_RANGE >> >> But later we get output like this: >> >> [mpiexec(a)this.machine] PMI response to fd 6 pid 4: cmd=keyval_cache >> >>P0-businesscard=description#{this.machine¹s.ip.address}$port#54105$ifname >>#{this.machine¹s.ip.address}$ >> >>P1-businesscard=description#{that.machine¹s.ip.address}$port#47302$ifname >>#{that.machine¹s.ip.address}$ >> >> >> >> Does this mean that we have missed a firewall setting either in the >> environment variables or in the ip tables themselves? >> >> >> Ideas? >> >> >> >> Thanks Much >> >> Bill >> >> >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >_______________________________________________ >discuss mailing list discuss(a)mpich.org >To manage subscription options or unsubscribe: >https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Optimal Firewall Settings for MPICH/HYDRA
by Kenneth Raffenetti 22 Jul '14

22 Jul '14

What version of MPICH/Hydra is this? On 07/22/2014 12:48 PM, Capehart, William J wrote: > Hi All > > We’re running MPICH on a couple machines with a brand new UNIX distro > (SL 6.5) and that are on a vulnerable network and rather than leave the > firewalls dropped we would like to run it through the firewall. > > We have included the MPIEXEC_PORT_RANGE and MPIR_CVAR_CH3_PORT_RANGE > fields and > have adjusted our iptables accordingly and in line with the “FAQ” guidance. > > Our passwordless SSH works fine between the machines. > > But all of this gives us momentary success with the cpi and fpi MPICH > test programs. But they crash with the firewall up. (but of course run > happily with the firewall down). > > An example of the basic output is below (node short sends one process to > “this.machine” and one to remote “that.machine” > > > [this.machine]% mpiexec -n 2 -f nodesshort cpi.exe > > Process 0 of 2 is on this.machine > > Process 1 of 2 is on that.machine > > Fatal error in PMPI_Reduce: A process has failed, error stack: > > PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff466a94d0, > rbuf=0x7fff466a94d8, count=1, MPI_DOUBLE, MPI_SUM, root=0, > MPI_COMM_WORLD) failed > > MPIR_Reduce_impl(1029)..........: > > MPIR_Reduce_intra(835)..........: > > MPIR_Reduce_binomial(144).......: > > MPIDI_CH3U_Recvq_FDU_or_AEP(667): Communication error with rank 1 > > > =================================================================================== > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > = EXIT CODE: 1 > > = CLEANING UP REMAINING PROCESSES > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > =================================================================================== > > [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb > (./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed > > [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event > (./tools/demux/demux_poll.c:77): callback returned error status > > [proxy:0:[email protected]] main (./pm/pmiserv/pmip.c:206): demux engine > error waiting for event > > [mpiexec(a)this.machine] HYDT_bscu_wait_for_completion > (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes > terminated badly; aborting > > [mpiexec(a)this.machine] HYDT_bsci_wait_for_completion > (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting > for completion > > [mpiexec(a)this.machine] HYD_pmci_wait_for_completion > (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for > completion > > [mpiexec(a)this.machine] main (./ui/mpich/mpiexec.c:331): process manager > error waiting for completion > > > > In debug mode it affirms that it is at least *starting with the first > available port as listed in MPIEXEC_PORT_RANGE > > But later we get output like this: > > [mpiexec(a)this.machine] PMI response to fd 6 pid 4: cmd=keyval_cache > P0-businesscard=description#{this.machine’s.ip.address}$port#54105$ifname#{this.machine’s.ip.address}$ > P1-businesscard=description#{that.machine’s.ip.address}$port#47302$ifname#{that.machine’s.ip.address}$ > > > > Does this mean that we have missed a firewall setting either in the > environment variables or in the ip tables themselves? > > > Ideas? > > > > Thanks Much > > Bill > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Optimal Firewall Settings for MPICH/HYDRA
by Capehart, William J 22 Jul '14

22 Jul '14

Hi All We're running MPICH on a couple machines with a brand new UNIX distro (SL 6.5) and that are on a vulnerable network and rather than leave the firewalls dropped we would like to run it through the firewall. We have included the MPIEXEC_PORT_RANGE and MPIR_CVAR_CH3_PORT_RANGE fields and have adjusted our iptables accordingly and in line with the "FAQ" guidance. Our passwordless SSH works fine between the machines. But all of this gives us momentary success with the cpi and fpi MPICH test programs. But they crash with the firewall up. (but of course run happily with the firewall down). An example of the basic output is below (node short sends one process to "this.machine" and one to remote "that.machine" [this.machine]% mpiexec -n 2 -f nodesshort cpi.exe Process 0 of 2 is on this.machine Process 1 of 2 is on that.machine Fatal error in PMPI_Reduce: A process has failed, error stack: PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff466a94d0, rbuf=0x7fff466a94d8, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(835)..........: MPIR_Reduce_binomial(144).......: MPIDI_CH3U_Recvq_FDU_or_AEP(667): Communication error with rank 1 =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status [proxy:0:[email protected]] main (./pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec(a)this.machine] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec(a)this.machine] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec(a)this.machine] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for completion [mpiexec(a)this.machine] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion In debug mode it affirms that it is at least *starting with the first available port as listed in MPIEXEC_PORT_RANGE But later we get output like this: [mpiexec(a)this.machine] PMI response to fd 6 pid 4: cmd=keyval_cache P0-businesscard=description#{this.machine's.ip.address}$port#54105$ifname#{this.machine's.ip.address}$ P1-businesscard=description#{that.machine's.ip.address}$port#47302$ifname#{that.machine's.ip.address}$ Does this mean that we have missed a firewall setting either in the environment variables or in the ip tables themselves? Ideas? Thanks Much Bill

1 0

Re: [mpich-discuss] Deadlock when using MPICH 3.1.1 and per-object critical sections on BG/Q
by Michael Blocksome 18 Jul '14

18 Jul '14

Halim, Thanks for creating the ticket. This likely will not be fixed in time for mpich 3.1.2, but I hope to get time to investigate this problem in the 3.1.3 timerframe .... whenever that is scheduled. Michael Blocksome Parallel Environment MPI Middleware Team Lead, TCEM POWER, x86, and Blue Gene HPC Messaging blocksom(a)us.ibm.com From: Halim <halim.amer(a)gmail.com> To: discuss(a)mpich.org Date: 07/17/2014 02:54 PM Subject: Re: [mpich-discuss] Deadlock when using MPICH 3.1.1 and per-object critical sections on BG/Q Sent by: discuss-bounces(a)mpich.org Hi Michael, Thanks. I created a ticket (#2132) on trac.mpich.org to track the problem. Regards, --Halim On 2014年06月26日 17:41, Michael Blocksome wrote: > Halim, > > This problem sounds similar to another issue we are debugging related to > cancel and multiple endpoints in per-object locking mode. I'll try a few > things and post status. > > Thanks, > > Michael Blocksome > Parallel Environment MPI Middleware Team Lead, TCEM > POWER, x86, and Blue Gene HPC Messaging > blocksom(a)us.ibm.com > > > > > From: Halim <halim.amer(a)gmail.com> > To: discuss(a)mpich.org, > Date: 06/26/2014 02:31 PM > Subject: [mpich-discuss] Deadlock when using MPICH 3.1.1 and > per-object critical sections on BG/Q > Sent by: discuss-bounces(a)mpich.org > > > > Hi, > > I have a specific issue that arises with MPICH (I use 3.1.1 built with > gcc) + MPI_THREAD_MULTIPLE + per-object critical sections on BG/Q. > > A deadlock happens in the attached hybrid MPI+OpenMP example code with 2 > processes and more than one thread per process. > > Debugging shows that one process is stuck in MPI_Allreduce while the > other is blocked in MPI_Finalize. > > A similar communication pattern happens in my application, but in this > case both processes are stuck in MPI_Allreduce. > > Note that the problem disappears when removing MPI_Allreduce, or > avoiding request cancellation (cancel+wait+test_cancelled). Both > Allreduce and cancellation operations can be avoided in this test while > ensuring a correct result. But in my application, both operations are > necessary. > > In addition, using a global critical section (default) results in a > correct execution. > > My configure line is as follows: > > ./configure --prefix=/home/aamer/usr --host=powerpc64-bgq-linux > --with-device=pamid --with-file-system=gpfs:BGQ > --with-file-system=bg+bglockless --with-atomic-primitives > --enable-handle-allocation=tls --enable-refcount=lock-free > -disable-predefined-refcount --disable-error-checking --without-timing > --without-mpit-pvars --enable-fast=O3,ndebug --enable-thread-cs=per-object > > I appreciate any advice to solve this issue? > > Regards, > --Halim > [attachment "allred_cancel.c" deleted by Michael Blocksome/Rochester/IBM] > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Deadlock when using MPICH 3.1.1 and per-object critical sections on BG/Q
by Halim 17 Jul '14

17 Jul '14

Hi Michael, Thanks. I created a ticket (#2132) on trac.mpich.org to track the problem. Regards, --Halim On 2014年06月26日 17:41, Michael Blocksome wrote: > Halim, > > This problem sounds similar to another issue we are debugging related to > cancel and multiple endpoints in per-object locking mode. I'll try a few > things and post status. > > Thanks, > > Michael Blocksome > Parallel Environment MPI Middleware Team Lead, TCEM > POWER, x86, and Blue Gene HPC Messaging > blocksom(a)us.ibm.com > > > > > From: Halim <halim.amer(a)gmail.com> > To: discuss(a)mpich.org, > Date: 06/26/2014 02:31 PM > Subject: [mpich-discuss] Deadlock when using MPICH 3.1.1 and > per-object critical sections on BG/Q > Sent by: discuss-bounces(a)mpich.org > > > > Hi, > > I have a specific issue that arises with MPICH (I use 3.1.1 built with > gcc) + MPI_THREAD_MULTIPLE + per-object critical sections on BG/Q. > > A deadlock happens in the attached hybrid MPI+OpenMP example code with 2 > processes and more than one thread per process. > > Debugging shows that one process is stuck in MPI_Allreduce while the > other is blocked in MPI_Finalize. > > A similar communication pattern happens in my application, but in this > case both processes are stuck in MPI_Allreduce. > > Note that the problem disappears when removing MPI_Allreduce, or > avoiding request cancellation (cancel+wait+test_cancelled). Both > Allreduce and cancellation operations can be avoided in this test while > ensuring a correct result. But in my application, both operations are > necessary. > > In addition, using a global critical section (default) results in a > correct execution. > > My configure line is as follows: > > ./configure --prefix=/home/aamer/usr --host=powerpc64-bgq-linux > --with-device=pamid --with-file-system=gpfs:BGQ > --with-file-system=bg+bglockless --with-atomic-primitives > --enable-handle-allocation=tls --enable-refcount=lock-free > -disable-predefined-refcount --disable-error-checking --without-timing > --without-mpit-pvars --enable-fast=O3,ndebug --enable-thread-cs=per-object > > I appreciate any advice to solve this issue? > > Regards, > --Halim > [attachment "allred_cancel.c" deleted by Michael Blocksome/Rochester/IBM] > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] MPICH 2.1.5 checkpoint
by Bland, Wesley B. 17 Jul '14

17 Jul '14

I’m not sure I understand your question here, but I believe the way the checkpointing works (or worked, I don’t think it’s functional in recent versions) is that it would periodically freeze the execution to write a checkpoint of all of the data. That checkpointing would take place periodically based on a parameter given by the user. If that interval is shorter than the interval it takes to make a checkpoint, then it would behave poorly (possible freezing the checkpoint to write another checkpoint, then freeze that checkpoint, etc.). Honestly, the person who wrote this code in MPICH is gone and it’s not working in the current version so I can’t give you a lot more information than that. It’s on the roadmap to fix this for future versions, but for now, all I can tell you is to extend the time between checkpoints because the checkpointing system isn’t robust enough to do much more than the simple thing. Thanks, Wesley On Jul 17, 2014, at 9:52 AM, Marcela Castro León <mcastrol(a)gmail.com<mailto:[email protected]>> wrote: But Do you know why after the first checkpoint the execution is frezzed during a time similar to the interval and, after that, a new checkpoint is triggered? Thank you. 2014-07-17 16:43 GMT+02:00 Bland, Wesley B. <wbland(a)anl.gov<mailto:[email protected]>>: You’ll probably need to somehow get your application to run longer then. Unfortunately, MPICH doesn’t support manually starting a checkpoint at this time. On Jul 17, 2014, at 9:29 AM, Marcela Castro León <mcastrol(a)gmail.com<mailto:[email protected]>> wrote: Hi I just want to make a one checkpoint during the execution, at the middle. We'are trying to observe the I/O to reduce the time using a parallel file system. Thanks. 2014-07-17 13:30 GMT+02:00 Bland, Wesley B. <wbland(a)anl.gov<mailto:[email protected]>>: Is there a reason you can't just take the checkpoints less frequently? On Jul 17, 2014, at 4:41 AM, "Marcela Castro León" <mcastrol(a)gmail.com<mailto:[email protected]>> wrote: Hi, I'm using mpich 2.1.5 compiled to use blcr checkpoint. I'm having problems with the checkpoint interval. When I execute: mpiexec -ckpointlib blcr -ckpoint-prefix /partnfs/mpichchk -ckpoint-interval 120 -f maquinas -n 16 ./bt.C.16 In fact, at the second 120, the execution is interrupted for checkpointing, but, as the checkpoint last more than 120 seconds, another checkpoint is immediately triggered instead of resuming the application. I only achieve to get a checkpoint by setting a checkpoint interval almost at the end of the execution but it is not useful. How can I solve it? Besides, Is there a way to know how long is the checkpoint in time? Thank you very much. Marcela mpiexec -info HYDRA build details: Version: 1.5 Release Date: Mon Oct 8 14:00:48 CDT 2012 CC: gcc CXX: c++ F77: gfortran F90: gfortran Configure options: '--disable-option-checking' '--prefix=/soft/mpich2/mpich2-1.5-blcr8.3' '--with-hydra-ckpointlib=blcr' '--with-blcr=/soft/blcr' '--enable-checkpointing' '--cache-file=/dev/null' '--srcdir=.' 'CC=gcc' 'CFLAGS= -O2' 'LDFLAGS= -L/soft/blcr/lib64 -L/soft/blcr/lib' 'LIBS=-lrt -lcr -lpthread ' 'CPPFLAGS= -I/SRC/mpi/mpich2-1.5/src/mpl/include -I/SRC/mpi/mpich2-1.5/src/mpl/include -I/SRC/mpi/mpich2-1.5/src/openpa/src -I/SRC/mpi/mpich2-1.5/src/openpa/src -I/SRC/mpi/mpich2-1.5/src/mpi/romio/include -I/soft/blcr/include' Process Manager: pmi Launchers available: ssh rsh fork slurm ll lsf sge manual persist Topology libraries available: hwloc Resource management kernels available: user slurm ll lsf sge pbs Checkpointing libraries available: blcr Demux engines available: poll select _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0