On Thu, Mar 22, 2018 at 3:52 PM, Martin Cuma <martin.cuma@utah.edu> wrote:
I finally discovered that MPICH 3.2.1 includes ofi netmod which I am trying to build to provide unified library for tcp and InfiniBand. That's awesome since I am looking forward to ofi as a replacement for the old mxm netmod.

So far it's working great, though it seems like the performance is lagging a bit behind the old mxm netmod on a few IB clusters I tried but I figure that's due to the ofi infancy.

However, I am having trouble building 3.2.1 with ofi using PGI compilers (18.1), where they complain:
libtool: compile:  pgcc -DHAVE_CONFIG_H -I. -I../../../srcdir/mpich/3.2.1 -I./src/include -I../../../srcdir/mpich/3.2.1/src/include -I../../../srcdir/mpich/3.2.1/src/mpi/datatype -I../../../srcdir/mpich/3.2.1/src/mpi/spawn -I../../../srcdir/mpich/3.2.1/src/util/dbg -Isrc/util/logging/common -I../../../srcdir/mpich/3.2.1/src/util/mem -I../../../srcdir/mpich/3.2.1/src/util/thread -I../../../srcdir/mpich/3.2.1/src/util/type -I../../../srcdir/mpich/3.2.1/src/util/wrappers -I./src/util/wrappers -I../../../srcdir/mpich/3.2.1/src/util/assert -I./src/util/timers -I../../../srcdir/mpich/3.2.1/src/util/pointer -I./src/binding/cxx -I/uufs/chpc.utah.edu/sys/srcdir/mpich/3.2.1/src/binding/fortran/mpif_h -Isrc/binding/fortran/use_mpi -I../../../srcdir/mpich/3.2.1/src/mpid/ch3/include -I./src/mpid/ch3/include -I../../../srcdir/mpich/3.2.1/src/mpid/ch3/channels/nemesis/include -I./src/mpid/ch3/channels/nemesis/include -I../../../srcdir/mpich/3.2.1/src/mpid/ch3/channels/nemesis/utils/monitor -I../../../srcdir/mpich/3.2.1/src/mpid/common/datatype -I../../../srcdir/mpich/3.2.1/src/mpid/common/datatype -I../../../srcdir/mpich/3.2.1/src/mpid/common/sched -I../../../srcdir/mpich/3.2.1/src/mpid/common/thread -I../../../srcdir/mpich/3.2.1/src/mpid/common/timers -I../../../srcdir/mpich/3.2.1/src/pmi/simple -I/uufs/chpc.utah.edu/sys/builddir/mpich/3.2.1p/src/mpl/include -I/uufs/chpc.utah.edu/sys/srcdir/mpich/3.2.1/src/mpl/include -I/uufs/chpc.utah.edu/sys/srcdir/mpich/3.2.1/src/openpa/src -I/uufs/chpc.utah.edu/sys/builddir/mpich/3.2.1p/src/openpa/src -D_REENTRANT -I/uufs/chpc.utah.edu/sys/builddir/mpich/3.2.1p/src/mpi/romio/include -I/uufs/chpc.utah.edu/sys/installdir/libfabric/1.6.0/include -tp=nehalem -fPIC -DNDEBUG -DNVALGRIND -O2 -c ../../../srcdir/mpich/3.2.1/src/mpid/ch3/channels/nemesis/netmod/ofi/ofi_tagged.c -MD  -fpic -DPIC -o src/mpid/ch3/channels/nemesis/netmod/ofi/.libs/lib_libmpi_la-ofi_tagged.o
PGC-S-0209-EOF in macro call to FI_RC_RETRY (../../../srcdir/mpich/3.2.1/src/mpid/ch3/channels/nemesis/netmod/ofi/ofi_tagged.c: 143)
PGC-S-0218-Missing #endif (../../../srcdir/mpich/3.2.1/src/mpid/ch3/channels/nemesis/netmod/ofi/ofi_tagged.c: 143)
PGC/x86-64 Linux 18.1-1: compilation completed with severe errors

Any thought on what can be the problem? The same build went fine with GNU and Intel compilers.

If GCC and ICC are fine, my guess is that this is a bug in the PGI C compiler.  This wouldn't be the first time there were issues related to PGI in MPICH...

Given that the C ABI is defined by the system rather than the compiler, why do you need to use PGI?  You need to use PGI Fortran to get the Fortran stuff to work, but GCC, Clang, or ICC should be fine (except for a possible issue with GCC+PGF builds related to memset changes I made years ago, which I assert is a PGI bug).
 
The suggestion concerns verbose make output. The user's guide says to run "make VERBOSE=1" to see what I pasted above. However, that did not work for me. What did work is "make V=1". If that is indeed the current magic word to make make verbose, can you please update the user's guide to save us (including me again) time to search through the Makefile to figure this out? 

The V=1 option is documented at line 125 of README.vin, which seems like a reasonable place to document it.

I don't see "verbose" anywhere in https://www.mpich.org/static/downloads/3.2.1/mpich-3.2.1-userguide.pdf.  Frankly, I've never read that document anyways.  I rely on README* or "git grep" for everything.

Jeff


--