I don't use RMA from threads much but I do not expect MPICH to have internal concurrency for bandwidth-limited RMA operations in shared-memory, so it doesn't surprise me that the bandwidth is not improved by multiple threads. Jeff On Mon, Dec 31, 2018 at 2:30 PM Kun Feng <[email protected]> wrote:

Good advice. Now I find out that the performance drops to sequential again when I move all the sources of MPI_Put into threads in one MPI rank even using MPI_Win_allocate. Do you have similar experience before?

Thanks Kun

On Sat, Dec 29, 2018 at 9:36 PM Jeff Hammond <[email protected]> wrote:

...

I don’t know why you are timing win_allocate. I’d only time lock-put-unlock or put-flush.

Jeff

On Sat, Dec 29, 2018 at 9:11 AM Kun Feng <[email protected]> wrote:

...

Thank you for the replies. MPI_Win_allocate gives me much better performance. It is even faster than what I got from pure memory bandwidth test. I'm putting the same memory block from the source rank to the same memory address on the destination rank followed by MPI_Win_flush to synchronize. Do I do it correctly? The source code is attached.

Thanks Kun

On Fri, Dec 21, 2018 at 11:15 AM Jeff Hammond <[email protected]> wrote:

...

Use MPI_Win_allocate instead of MPI_Win_create. MPI_Win_create cannot allocate shared memory so you will not get good performance within a node.

Jeff

On Fri, Dec 21, 2018 at 8:18 AM Kun Feng via discuss <[email protected]> wrote:

...

Hi all,

I'm working on a project in which one half of the processes need to send data to the other half in each node. I'm using passive target mode of one-sided communication in which the receivers expose memory using MPI_Win_create, wait on MPI_Win_free and the senders send the data using MPI_Put. The code works. However, I get weird performance using this concurrent MPI_Put communication. The peak aggregate bandwidth is only around 5GB/s. It does not make sense as an aggregate performance in one single node. I thought the node-local communication is implemented as local memcpy. But concurrent memcpy on the same testbed has 4x to 5x higher aggregate bandwidth. Even concurrent memcpy using Linux shared memory across processes is 3x faster than my code. I'm using CH3 in MPICH 3.2.1. CH4 in MPICH 3.3 is even 2x slower. Does the performance make sense? Does MPICH has some queue for all one-sided communication in one node? Or do I understand it incorrectly?

Thanks Kun _______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

-- Jeff Hammond [email protected] http://jeffhammond.github.io/

-- Jeff Hammond [email protected] http://jeffhammond.github.io/

-- Jeff Hammond [email protected] http://jeffhammond.github.io/