June 2023 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after
by Raffenetti, Ken 08 Jun '23

08 Jun '23

Hi, I believe this bug was recently fixed in https://github.com/pmodels/mpich/pull/6543. The fix is part of the MPICH 4.1.2 release just posted to our website and Github. I confirmed that your test program works as expected now vs. an older 4.1 release. Ken From: "Solomonik, Edgar via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, June 8, 2023 at 3:37 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Solomonik, Edgar" <solomon2(a)illinois.edu> Subject: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after Hello, Our library's autobuild (CTF, which uses MPI extensively and in relatively sophisticated ways) started failing on multiple architectures after github workflows moved to later OS versions (and so later MPI versions). I believe I have narrowed the issue to an MPI bug associated with very basic usage of MPI Reduce. The following test code runs into a segmentation fault inside MPI when running with 2 MPI processes with the latest Ubuntu MPI build and MPI 4.0. It works for smaller values of message size (n) or if the root is rank 0. The usage of MPI_IN_PLACE adheres with the MPI standard. Best, Edgar Solomonik #include <mpi.h> #include <iostream> int main(int argc, char ** argv){ int64_t n = 257; MPI_Init(&argc, &argv); int rank; MPI_Comm_rank(MPI_COMM_WORLD, &rank); double * A = (double*)malloc(sizeof(double)*n); for (int i=0; i<n; i++){ A[i] = (double)i; } if (rank == 1){ MPI_Reduce(MPI_IN_PLACE, A, n, MPI_DOUBLE, MPI_SUM, 1, MPI_COMM_WORLD); } else { MPI_Reduce(A, NULL, n, MPI_DOUBLE, MPI_SUM, 1, MPI_COMM_WORLD); } free(A); MPI_Finalize(); return 0; }

1 0

MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after
by Solomonik, Edgar 08 Jun '23

08 Jun '23

Hello, Our library's autobuild (CTF, which uses MPI extensively and in relatively sophisticated ways) started failing on multiple architectures after github workflows moved to later OS versions (and so later MPI versions). I believe I have narrowed the issue to an MPI bug associated with very basic usage of MPI Reduce. The following test code runs into a segmentation fault inside MPI when running with 2 MPI processes with the latest Ubuntu MPI build and MPI 4.0. It works for smaller values of message size (n) or if the root is rank 0. The usage of MPI_IN_PLACE adheres with the MPI standard. Best, Edgar Solomonik #include <mpi.h> #include <iostream> int main(int argc, char ** argv){ int64_t n = 257; MPI_Init(&argc, &argv); int rank; MPI_Comm_rank(MPI_COMM_WORLD, &rank); double * A = (double*)malloc(sizeof(double)*n); for (int i=0; i<n; i++){ A[i] = (double)i; } if (rank == 1){ MPI_Reduce(MPI_IN_PLACE, A, n, MPI_DOUBLE, MPI_SUM, 1, MPI_COMM_WORLD); } else { MPI_Reduce(A, NULL, n, MPI_DOUBLE, MPI_SUM, 1, MPI_COMM_WORLD); } free(A); MPI_Finalize(); return 0; }

1 0

Re: [mpich-discuss] Building 4.1.1
by Raffenetti, Ken 31 May '23

31 May '23

Try adding –disable-opx to the configure line to disable the OPX provider in the embedded libfabric. There must be some incompatibility with the compiler that is not detected by the configure script. Ken From: "Palmer, Bruce J via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Wednesday, May 31, 2023 at 6:47 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Subject: [mpich-discuss] Building 4.1.1 I’m trying to build 4.1.1 on our local cluster. I’m configuring with ./configure --prefix=/people/d3g293/mpich/mpich-4.1.1/install --with-device=ch4:ofi --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ (I also tried --with-device=ch4:ofi:sockets) and my environment is Currently Loaded Modulefiles: 1) gcc/8.1.0 3) cmake/3.22.1 5) mkl/2019u5 2) python/3.7.2 4) git/2.11.0(default) 6) texlive/2017 I keep getting errors that look like In file included from ./prov/opx/include/rdma/opx/fi_opx_reliability.h:36, from ./prov/opx/include/rdma/opx/fi_opx_domain.h:44, from prov/opx/src/fi_opx_cntr.c:35: ./prov/opx/include/rdma/opx/fi_opx_hfi1_transport.h: At top level: ./prov/opx/include/rdma/opx/fi_opx.h:209:44: error: expected identifier or ‘(’ before ‘if’ #define OPX_COMPILE_TIME_ASSERT(cond, msg) if(0){switch(0){case 0:case cond:;}} Any suggestions on how to cure this? I’ve had pretty good luck building earlier versions of MPICH (4.0.2) using this formula. Bruce Palmer Computer Scientist Pacific Northwest National Laboratory (509) 375-3899

1 0