I have a use-case where I need to transmit an Array of Structures (AoS) among a group of processors in a circular ("ring") fashion.
I do that currently by using a non-blocking send and receive, and using two auxiliary buffers on the senders and receivers side respectively.
Since the data I am transferring is extremely huge (~60 gb before partitioning), allocating two buffers seems inefficient to me. My question is -- Is there a better way to do it by avoiding "two buffers"? (Perhaps just using one buffer in some way?).
//Pack AoS into send buffer (which is a vector<char>)
char *msg_ptr = &(msg_send_buf_.front());
for (auto& param : local_primal_params_) {
param.pack_to(msg_ptr);
msg_ptr += PARAM_BYTENUM;
}
//Perform non-blocking send receive
{
MPI_Request send_request;
MPI_Status send_stat, recv_stat;
MPI_Isend(&(msg_send_buf_.front()), local_primal_params_.size() * PARAM_BYTENUM, MPI_CHAR,
send_to, 0,
MPI_COMM_WORLD, &send_request);
MPI_Recv(&(msg_recv_buf_.front()), local_primal_params_.size() * PARAM_BYTENUM, MPI_CHAR,
recv_from, 0,
MPI_COMM_WORLD, &recv_stat);
MPI_Wait(&send_request, &send_stat);
}
//Unpack receive buffer back into AoS on receiver side
{
char *msg_ptr = &(msg_recv_buf_.front());
for (auto& param : local_primal_params_) {
param.unpack_from(msg_ptr);
msg_ptr += PARAM_BYTENUM;
}
}