Dear MPICH users,

I'm having an issue with, apparently, communication between nodes of the local cluster we're using. To fix ideas, the cluster is made of 8 different nodes and it was set-up with now outdated versions of the Intel compilers and MPI libraries. I did an mpbboot/mpdringtest and it seems to work fine.

Now, I would like to use my application code with MPICH 3.3 installed from PETSc and with gcc-8 installed from linuxbrew on the different nodes.

I tested a simple Fortran program:

program test

  use mpi

  use ISO_fortran_env, only: output_unit

  

  implicit none

  integer :: irank, nproc, ierr

  character(len=80) :: hostname

  call MPI_INIT(ierr)

  call MPI_COMM_SIZE(MPI_COMM_WORLD,nproc,ierr)

  call MPI_COMM_RANK(MPI_COMM_WORLD,irank,ierr)

  if (irank == 0) THEN

    WRITE(output_unit,'(1X,A,I3,A)') 'Started test program with', nproc, ' MPI processes'

  end if

  call MPI_BARRIER(MPI_COMM_WORLD,ierr) 

  call HOSTNM(hostname,ierr)

  WRITE(output_unit,'(1X,A,I3,A)') 'I am processor #', irank, ' running on '//hostname

  call MPI_FINALIZE(ierr)

end program test

It works fine on the local host. But when I try to launch it on all the nodes with "mpiexec -f path_to_hostfile -n 8 path_to_my_program" I get:

 Started test program with  8 MPI processes

Fatal error in PMPI_Barrier: Unknown error class, error stack:

PMPI_Barrier(287)...........................: MPI_Barrier(MPI_COMM_WORLD) failed

PMPI_Barrier(273)...........................: 

MPIR_Barrier_impl(173)......................: 

MPIR_Barrier_intra_auto(108)................: 

MPIR_Barrier_intra_recursive_doubling(47)...: 

MPIC_Sendrecv(347)..........................: 

MPIC_Wait(73)...............................: 

MPIDI_CH3i_Progress_wait(242)...............: an error occurred while handling an event returned by MPIDI_CH3I_Sock_Wait()

MPIDI_CH3I_Progress_handle_sock_event(698)..: 

MPIDI_CH3_Sockconn_handle_connect_event(597): [ch3:sock] failed to connnect to remote process

MPIDI_CH3I_Sock_post_connect_ifaddr(1774)...: unexpected operating system error (set=0,sock=5,errno=101:Network is unreachable)

Note that when I comment the call to MPI_BARRIER, it works fine as well. Therefore, communication between nodes seems to be an issue. All the nodes see and have access to the same mpiexec/mpifort executables: I checked $PATH and 'which mpiexec'/'which mpifort'.

 

Could you give me ways for an efficient diagnostic of what's possibly wrong?

mpiexec --version gives

HYDRA build details:

    Version:                                 3.3b1

    Release Date:                            Mon Feb  5 10:16:15 CST 2018

    CC:                              gcc-8  -fPIC -fstack-protector -O3 -march=native  

    CXX:                             g++-8  -fstack-protector -O3 -march=native -fPIC  

    F77:                             gfortran-8 -fPIC -ffree-line-length-0 -O3 -march=native  

    F90:                             gfortran-8 -fPIC -ffree-line-length-0 -O3 -march=native  

    Configure options:                       '--disable-option-checking' '--prefix=/home/petsc/icm_cplx' 'MAKE=/usr/bin/make' '--libdir=/home/petsc/icm_cplx/lib' 'CC=gcc-8' 'CFLAGS=-fPIC -fstack-protector -O3 -march=native -O2' 'AR=/usr/bin/ar' 'ARFLAGS=cr' 'CXX=g++-8' 'CXXFLAGS=-fstack-protector -O3 -march=native -fPIC -O2' 'F77=gfortran-8' 'FFLAGS=-fPIC -ffree-line-length-0 -O3 -march=native -O2' 'FC=gfortran-8' 'FCFLAGS=-fPIC -ffree-line-length-0 -O3 -march=native -O2' '--enable-shared' '--with-device=ch3:sock' '--with-pm=hydra' '--enable-g=meminit' '--cache-file=/dev/null' '--srcdir=.' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= -I/home/petsc/icm_cplx/externalpackages/mpich-3.3b1/src/mpl/include -I/home/petsc/icm_cplx/externalpackages/mpich-3.3b1/src/mpl/include -I/home/petsc/icm_cplx/externalpackages/mpich-3.3b1/src/openpa/src -I/home/petsc/icm_cplx/externalpackages/mpich-3.3b1/src/openpa/src -D_REENTRANT -I/home/petsc/icm_cplx/externalpackages/mpich-3.3b1/src/mpi/romio/include' 'MPLLIBNAME=mpl'

    Process Manager:                         pmi

    Launchers available:                     ssh rsh fork slurm ll lsf sge manual persist

    Topology libraries available:            hwloc

    Resource management kernels available:   user slurm ll lsf sge pbs cobalt

    Checkpointing libraries available:       

    Demux engines available:                 poll select

Thank you,

Thibaut

_______________________________________________
discuss mailing list     discuss@mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss