Dear MPICH community,

I am currently developing an event system built upon MPI and I have a few questions about the current state of multi-thread support from MPICH in the MPI_THREAD_MULTIPLE mode.

Question 1. I am aware of the message envelope tuples (source, destination, tag, communicator) that may be used to identify and filter different messages. While using MPICH, can I rely on such information in order to guarantee that the correct messages will reach/be matched with their destinations in a multiprocess multithreaded program? I know this is quite a broad question, but I am more interested in the analysis of the following scenarios.

Two threads in a process A want to send different messages to two different processes, B and C (each with one thread), while using the same communicator and tag. Code example:

// One process with N threads to N process with one thread (same tag)
if (rank == 0) {
std::thread t1([&]() {
// Generate data ...
MPI_Send(data, 1, MPI_CHAR, 1, 0, MPI_COMM_WORLD);
});
std::thread t2([&]() {
// Generate data ...
MPI_Send(data, 1, MPI_CHAR, 2, 0, MPI_COMM_WORLD);
});
t1.join(); t2.join();
} else {
MPI_Recv(data, 1, MPI_CHAR, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
}

The same scenario as above, but now each thread uses a different tag to communicate to their respective process while only sharing the same communicator. Code example:

// One process with N threads to N process with one thread (different tags)
if (rank == 0) {
std::thread t1([&]() {
// Generate data ...
MPI_Send(data, 1, MPI_CHAR, 1, 1, MPI_COMM_WORLD);
});
std::thread t2([&]() {
// Generate data ...
MPI_Send(data, 1, MPI_CHAR, 2, 2, MPI_COMM_WORLD);
});
t1.join(); t2.join();
} else {
MPI_Recv(data, 1, MPI_CHAR, 0, rank, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
}

Multiple threads from a process A want each to send a message to the same thread of another process B using the same communicator and tag. Code example:

// One process with N threads to one process with one thread (same tags)
if (rank == 0) {
std::thread t1([&]() {
// Generate data ...
MPI_Send(data, 1, MPI_CHAR, 1, 0, MPI_COMM_WORLD);
});
std::thread t2([&]() {
// Generate data ...
MPI_Send(data, 1, MPI_CHAR, 1, 0, MPI_COMM_WORLD);
});
t1.join(); t2.join();
} else if (rank == 1) {
MPI_Recv(data, 1, MPI_CHAR, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
// Process data ...
MPI_Recv(data, 1, MPI_CHAR, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
// Process data ...
}

One thread from a process A wants to send a message to one of multiple threads from a process B using the same communicator and tag. Code example:

// One process with one thread to one process with N threads (same tags)
if (rank == 0) {
MPI_Request requests[2];
// Generate data ...
MPI_Isend(data, 1, MPI_CHAR, 1, 0, MPI_COMM_WORLD, &requests[0]);
// Generate data ...
MPI_Isend(data, 1, MPI_CHAR, 1, 0, MPI_COMM_WORLD, &requests[1]);

MPI_Waitall(2, requests, MPI_STATUSES_IGNORE);
} else if (rank == 1) {
std::thread t1([&]() {
MPI_Recv(data, 1, MPI_CHAR, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
// Process data ...
});
std::thread t2([&]() {
MPI_Recv(data, 1, MPI_CHAR, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
// Process data ...
});
t1.join(); t2.join();
}

Two threads from a process A want each to send a message to two threads from a process B using different tags for each pair of threads (e.g. the pair A.1/B.1 uses a different tag from pair A.2/B.2). Code example:

// One process with N threads to one process with N thread (different tags)
if (rank == 0) {
std::thread t1([&]() {
// Generate data ...
MPI_Send(data, 1, MPI_CHAR, 1, 1, MPI_COMM_WORLD);
});
std::thread t2([&]() {
// Generate data ...
MPI_Send(data, 1, MPI_CHAR, 1, 2, MPI_COMM_WORLD);
});
t1.join(); t2.join();
} else if (rank == 1) {
std::thread t1([&]() {
MPI_Recv(data, 1, MPI_CHAR, 0, 1, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
// Process data ...
});
std::thread t2([&]() {
MPI_Recv(data, 1, MPI_CHAR, 0, 2, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
// Process data ...
});
t1.join(); t2.join();
}

Question 2. Is there a problem when mixing blocking and non-blocking calls on opposite sides of a message? Even in the multithreaded scenarios described previously? (e.g. matching a MPI_Isend with a MPI_Recv, and vice-versa.)

Thank you.

Regards,

Guilherme Valarini