Thank you for the answer! Second solution just suits me. I wanted to implement simple release-acquire semantics through atomic flag. For example, two processes include two threads, first of which waits for the flag by MPI_Get_accumulate and the other one remotely sets flag by MPI_Accumulate on the other process. In order to keep atomicity of operations, I have to use one window. Now all works fine: MPI_Win win; int flag = 0; int nproc = 0; int myrank = 0; void *thread(void *arg) { int flag_get = 0; do { MPI_Get_accumulate(NULL, 0, MPI_INT, &flag_get, 1, MPI_INT, myrank, 0, 1, MPI_INT, MPI_NO_OP, win); MPI_Win_flush(myrank, win); } while (flag_get == 0); printf("%d \t I got a flag!\n", myrank); return NULL; } int main(int argc, char *argv[]) { MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, NULL); MPI_Comm_size(MPI_COMM_WORLD, &nproc); MPI_Comm_rank(MPI_COMM_WORLD, &myrank); assert(nproc == 2); int *buf = NULL; const int bufsize = 1; MPI_Alloc_mem(bufsize, MPI_INFO_NULL, &buf); MPI_Win_create(buf, 1, 1, MPI_INFO_NULL, MPI_COMM_WORLD, &win); MPI_Win_lock_all(0, win); pthread_t tid; pthread_create(&tid, NULL, thread, NULL); int peer = 1 - myrank; int flag_put = 1; MPI_Accumulate(&flag_put, 1, MPI_INT, peer, 0, 1, MPI_INT, MPI_REPLACE, win); MPI_Win_flush(peer, win); pthread_join(tid, NULL); MPI_Win_unlock_all(win); MPI_Win_free(&win); MPI_Finalize(); return 0; } On Thu, Jun 6, 2019 at 23:18 Jeff Hammond wrote:

Your threads are trying to lock a window that is already locked or unlock a window that is not locked. That's because you are using one window with multiple threads.

If you want to use multiple threads like this, do one of two things: 1) use a window per thread 2) synchronize with flush(_all). only lock and unlock the window once (immediately after construction and immediately before destruction, respectively)

Jeff

On Thu, Jun 6, 2019 at 12:30 PM Alexey Paznikov via discuss < [email protected]> wrote:

...

Hi,

I have multithreaded MPI program (MPI_THREAD_MULTIPLE mode) with RMA calls.This is a simplified example:

#include <stdio.h> #include <mpi.h> #include <pthread.h>

MPI_Win win; pthread_mutex_t lock;

void *thread(void *arg) { MPI_Win_lock_all(0, win); MPI_Win_unlock_all(win); return NULL; }

int main(int argc, char *argv[]) { MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, NULL); pthread_mutex_init(&lock, NULL);

int *buf = NULL; const int bufsize = 1; MPI_Alloc_mem(bufsize, MPI_INFO_NULL, &buf); MPI_Win_create(buf, 1, 1, MPI_INFO_NULL, MPI_COMM_WORLD, &win);

pthread_t tid; pthread_create(&tid, NULL, thread, NULL);

MPI_Win_lock_all(0, win); MPI_Win_unlock_all(win);

pthread_join(tid, NULL);

MPI_Win_free(&win);

MPI_Finalize();

return 0; }

If I run such program, it crashes with an error message:

Fatal error in PMPI_Win_lock_all: Wrong synchronization of RMA calls , error stack: PMPI_Win_lock_all(149).: MPI_Win_lock_all(assert=0, win=0xa0000000) failed MPID_Win_lock_all(1522): Wrong synchronization of RMA calls

If I replace MPI_Win_lock_all with MPI_Win_lock the problem remains:

Fatal error in PMPI_Win_lock: Wrong synchronization of RMA calls , error stack: PMPI_Win_lock(157).: MPI_Win_lock(lock_type=234, rank=0, assert=0, win=0xa0000000) failed MPID_Win_lock(1163): Wrong synchronization of RMA calls

(this message is repeated many times)

If I protect RMA operations with a mutex, the problem disappears.

MPICH version 3.3. Similar problem is also in MVAPICH2 2.3.1.

Are the RMA operations not thread safe at the moment? Could you tell me how to deal with this problem?

_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

-- Jeff Hammond [email protected] http://jeffhammond.github.io/

-- Alexey Paznikov