Good Afternoon,

    I am working on an MPI application which needs to listen to a UDP socket connection in the background. I caught this behavior and I was curious if this is expected or if this is a potential issue.

Essentially, until you call MPI_Init_thread, the Linux socket command will return 0 for all ranks except rank 0. This seems to me like a problem. It is easy enough to fix if MPI_Init is called first, however it does not lend itself well if you want to do any initialization work before MPI is started.

Thanks for your input
Marvin Smith

Given the following code sample...

// Libraries
#include <mpi.h>
#include <arpa/inet.h>
#include <iostream>
#include <sys/socket.h>
#include <unistd.h>

/**
* Main Program
*/
int main( int argc, char* argv[] )
{

    // Test Socket Command
    std::cout << "Pre Socket: " << socket( AF_INET, SOCK_DGRAM, IPPROTO_UDP ) << std::endl;

    // Initialize MPI
    int mpi_thread_act;
    std::cout << "MPI_Init" << std::endl;
    MPI_Init_thread( &argc, &argv, MPI_THREAD_MULTIPLE, &mpi_thread_act );

    // Test another socket
    std::cout << "Post Socket: " << socket( AF_INET, SOCK_DGRAM, IPPROTO_UDP ) << std::endl;

    // Finalize MPI
    MPI_Finalize();

    // Exit
    return 0;
}

The application returns output this with mpirun.

$ mpirun -np 2 ./test-socket
Pre Socket: 10
MPI_Init
Pre Socket: 0
MPI_Init
Post Socket: 14
Post Socket: 10

$ mpirun -np 3 ./test-socket
Pre Socket: 10
MPI_Init
Pre Socket: 0
MPI_Init
Pre Socket: 0
MPI_Init
Post Socket: 14
Post Socket: 10
Post Socket: 8

It appears that all ranks except 0 will not create valid sockets until MPI_Init_thread is called.

I am running this code sample using Red-Hat Enterprise Linux 7.1 with mpich

Below is my output from mpirun
$ mpirun --version
HYDRA build details:
    Version:                                 3.0.4
    Release Date:                            Wed Apr 24 10:08:10 CDT 2013
    CC:                              cc -m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -fPIC -Wl,-z,noexecstack
    CXX:                             c++ -m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -fPIC -Wl,-z,noexecstack
    F77:                             gfortran -m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -fPIC -Wl,-z,noexecstack
    F90:                             gfortran -m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -fPIC -Wl,-z,noexecstack
    Configure options:                       '--disable-option-checking' '--prefix=/usr' '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--enable-sharedlibs=gcc' '--enable-shared' '--enable-lib-depend' '--disable-rpath' '--enable-fc' '--with-device=ch3:nemesis' '--with-pm=hydra:gforker' '--sysconfdir=/etc/mpich-x86_64' '--includedir=/usr/include/mpich-x86_64' '--bindir=/usr/lib64/mpich/bin' '--libdir=/usr/lib64/mpich/lib' '--datadir=/usr/share/mpich' '--mandir=/usr/share/man/mpich' '--docdir=/usr/share/mpich/doc' '--htmldir=/usr/share/mpich/doc' '--with-hwloc-prefix=system' 'FC=gfortran' 'F77=gfortran' 'CFLAGS=-m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC -O2' 'CXXFLAGS=-m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC -O2' 'FCFLAGS=-m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC -O2' 'FFLAGS=-m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC -O2' 'LDFLAGS=-Wl,-z,noexecstack ' 'MPICH2LIB_CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' 'MPICH2LIB_CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' 'MPICH2LIB_FCFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' 'MPICH2LIB_FFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' '--cache-file=/dev/null' '--srcdir=.' 'CC=cc' 'LIBS=-lrt -lpthread ' 'CPPFLAGS= -I/builddir/build/BUILD/mpich-3.0.4/src/mpl/include -I/builddir/build/BUILD/mpich-3.0.4/src/mpl/include -I/builddir/build/BUILD/mpich-3.0.4/src/openpa/src -I/builddir/build/BUILD/mpich-3.0.4/src/openpa/src -I/builddir/build/BUILD/mpich-3.0.4/src/mpi/romio/include'
    Process Manager:                         pmi
    Launchers available:                     ssh rsh fork slurm ll lsf sge manual persist
    Topology libraries available:
    Resource management kernels available:   user slurm ll lsf sge pbs cobalt
    Checkpointing libraries available:
    Demux engines available:                 poll select

CONFIDENTIALITY NOTICE - SNC EMAIL: This email and any attachments are confidential, may contain proprietary, protected, or export controlled information, and are intended for the use of the intended recipients only. Any review, reliance, distribution, disclosure, or forwarding of this email and/or attachments outside of Sierra Nevada Corporation (SNC) without express written approval of the sender, except to the extent required to further properly approved SNC business purposes, is strictly prohibited. If you are not the intended recipient of this email, please notify the sender immediately, and delete all copies without reading, printing, or saving in any manner. --- Thank You.