Re: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after

8 Jun 2023

      Thanks, indeed the env variable fixes the issue and glad to hear its fixed in the latest version.

Best,
Edgar
________________________________
From: Raffenetti, Ken via discuss <[email protected]>
Sent: Thursday, June 8, 2023 3:50 PM
To: [email protected] <[email protected]>
Cc: Raffenetti, Ken <[email protected]>
Subject: Re: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after

FWIW, you can workaround the bug in older versions by setting MPIR_CVAR_DEVICE_COLLECTIVES=none in your environment.

Ken

From: "Raffenetti, Ken via discuss" <[email protected]>
Reply-To: "[email protected]" <[email protected]>
Date: Thursday, June 8, 2023 at 3:45 PM
To: "[email protected]" <[email protected]>
Cc: "Raffenetti, Ken" <[email protected]>
Subject: Re: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after

Hi,

I believe this bug was recently fixed in https://github.com/pmodels/mpich/pull/6543<https://urldefense.com/v3/__https://github.com/pmodels/mpich/pull/6543__;!!DZ3fjg!8AzKEU73uMciZu4bNdH1-uJ_sMFhjhYR4z6BeYLyhi-KiaPNXdrn62dnQZ4iz3VzxMVG5mCaYmCzlj5lcno$>. The fix is part of the MPICH 4.1.2 release just posted to our website and Github. I confirmed that your test program works as expected now vs. an older 4.1 release.

Ken

From: "Solomonik, Edgar via discuss" <[email protected]>
Reply-To: "[email protected]" <[email protected]>
Date: Thursday, June 8, 2023 at 3:37 PM
To: "[email protected]" <[email protected]>
Cc: "Solomonik, Edgar" <[email protected]>
Subject: [mpich-discuss] MPI Reduce with MPI_IN_PLACE fails with non-0 root rank for message sizes over 256 with MPI version 4 and after

Hello,

Our library's autobuild (CTF, which uses MPI extensively and in relatively sophisticated ways) started failing on multiple architectures after github workflows moved to later OS versions (and so later MPI versions). I believe I have narrowed the issue to an MPI bug associated with very basic usage of MPI Reduce. The following test code runs into a segmentation fault inside MPI when running with 2 MPI processes with the latest Ubuntu MPI build and MPI 4.0. It works for smaller values of message size (n) or if the root is rank 0. The usage of MPI_IN_PLACE adheres with the MPI standard.

Best,

Edgar Solomonik

#include <mpi.h>

#include <iostream>

int main(int argc, char ** argv){

  int64_t n = 257;

  MPI_Init(&argc, &argv);

  int rank;

  MPI_Comm_rank(MPI_COMM_WORLD, &rank);

  double * A = (double*)malloc(sizeof(double)*n);

  for (int i=0; i<n; i++){

    A[i] = (double)i;

  }

  if (rank == 1){

    MPI_Reduce(MPI_IN_PLACE, A, n, MPI_DOUBLE, MPI_SUM, 1, MPI_COMM_WORLD);

  } else {

    MPI_Reduce(A, NULL, n, MPI_DOUBLE, MPI_SUM, 1, MPI_COMM_WORLD);

  }

  free(A);

  MPI_Finalize();

  return 0;

}