[mpich-discuss] Questions about MPICH multi-thread support

26 Jan 2021

      Dear MPICH community,

I am currently developing an event system built upon MPI and I have a few
questions about the current state of multi-thread support from MPICH in the
*MPI_THREAD_MULTIPLE* mode.

*Question 1*. I am aware of the message envelope tuples (source,
destination, tag, communicator) that may be used to identify and filter
different messages. While using MPICH, can I rely on such information in
order to guarantee that the correct messages will reach/be matched with
their destinations in a multiprocess multithreaded program? I know this is
quite a broad question, but I am more interested in the analysis of the
following scenarios.

   - Two threads in a process A want to send different messages to two
   different processes, B and C (each with one thread), while using the
   *same* *communicator* *and* *tag*. Code example:

  // One process with N threads to N process with one thread (same tag)
...
if (rank == 0) {
    std::thread t1([&]() {
      // Generate data ...
      MPI_Send(data, 1, MPI_CHAR, 1, 0, MPI_COMM_WORLD);
    });
    std::thread t2([&]() {
      // Generate data ...
      MPI_Send(data, 1, MPI_CHAR, 2, 0, MPI_COMM_WORLD);
    });
    t1.join(); t2.join();
  } else {
    MPI_Recv(data, 1, MPI_CHAR, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
  }
- The same scenario as above, but now each thread uses a *different tag*
   to communicate to their respective process while only sharing the *same*
   *communicator*. Code example:

  // One process with N threads to N process with one thread (different
...
tags)
  if (rank == 0) {
    std::thread t1([&]() {
      // Generate data ...
      MPI_Send(data, 1, MPI_CHAR, 1, 1, MPI_COMM_WORLD);
    });
    std::thread t2([&]() {
      // Generate data ...
      MPI_Send(data, 1, MPI_CHAR, 2, 2, MPI_COMM_WORLD);
    });
    t1.join(); t2.join();
  } else {
    MPI_Recv(data, 1, MPI_CHAR, 0, rank, MPI_COMM_WORLD,
MPI_STATUS_IGNORE);
  }
- Multiple threads from a process A want each to send a message to the
   same thread of another process B using the *same communicator and*
*tag*. Code
   example:

  // One process with N threads to one process with one thread (same tags)
...
if (rank == 0) {
    std::thread t1([&]() {
      // Generate data ...
      MPI_Send(data, 1, MPI_CHAR, 1, 0, MPI_COMM_WORLD);
    });
    std::thread t2([&]() {
      // Generate data ...
      MPI_Send(data, 1, MPI_CHAR, 1, 0, MPI_COMM_WORLD);
    });
    t1.join(); t2.join();
  } else if (rank == 1) {
    MPI_Recv(data, 1, MPI_CHAR, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
    // Process data ...
    MPI_Recv(data, 1, MPI_CHAR, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
    // Process data ...
  }
- One thread from a process A wants to send a message to one of multiple
   threads from a process B using the *same communicator and tag*. Code
   example:

  // One process with one thread to one process with N threads (same tags)
...
if (rank == 0) {
    MPI_Request requests[2];
    // Generate data ...
    MPI_Isend(data, 1, MPI_CHAR, 1, 0, MPI_COMM_WORLD, &requests[0]);
    // Generate data ...
    MPI_Isend(data, 1, MPI_CHAR, 1, 0, MPI_COMM_WORLD, &requests[1]);
MPI_Waitall(2, requests, MPI_STATUSES_IGNORE);
  } else if (rank == 1) {
    std::thread t1([&]() {
      MPI_Recv(data, 1, MPI_CHAR, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
      // Process data ...
    });
    std::thread t2([&]() {
      MPI_Recv(data, 1, MPI_CHAR, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
      // Process data ...
    });
    t1.join(); t2.join();
  }
- Two threads from a process A want each to send a message to two
   threads from a process B using different tags for each pair of threads
   (e.g. the pair A.1/B.1 uses a different tag from pair A.2/B.2). Code
   example:

  // One process with N threads to one process with N thread (different
...
tags)
  if (rank == 0) {
    std::thread t1([&]() {
      // Generate data ...
      MPI_Send(data, 1, MPI_CHAR, 1, 1, MPI_COMM_WORLD);
    });
    std::thread t2([&]() {
      // Generate data ...
      MPI_Send(data, 1, MPI_CHAR, 1, 2, MPI_COMM_WORLD);
    });
    t1.join(); t2.join();
  } else if (rank == 1) {
    std::thread t1([&]() {
      MPI_Recv(data, 1, MPI_CHAR, 0, 1, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
      // Process data ...
    });
    std::thread t2([&]() {
      MPI_Recv(data, 1, MPI_CHAR, 0, 2, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
      // Process data ...
    });
    t1.join(); t2.join();
  }
*Question 2*. Is there a problem when mixing blocking and non-blocking
calls on opposite sides of a message? Even in the multithreaded scenarios
described previously? (e.g. matching a *MPI_Isend* with a *MPI_Recv*, and
vice-versa.)

Thank you.

Regards,
Guilherme Valarini