Hi Ron,

Depending on how the algorithm is structured, I may well be that the faster computers are generating messages faster than the slow computer is able to process them. As the problem size increases, the amount of data in unprocessed messages may get too high for that computer. In a very simple example (but note that there are many other possible situations):

Hosts A and B:
for(int i=0; i<n; i++) {
    MPI_Send(buf, count, MPI_INT, host_c, 0, MPI_COMM_WORLD);
    buf += count;
}

Host C:
for(int i=0; i<2*n; i++) {
    MPI_Recv(buf, count, MPI_INT, MPI_ANY_SOURCE, 0, MPI_COMM_WORLD, &status);
    process_data(buf);
}

In that case the senders keep sending data to host_c faster than it's calling MPI_Recv, and get received by MPICH as unexpected messages (MPICH doesn't know where to place that data, and uses temporary internal buffers). Note that this situation is not necessarily caused by different processing speeds, since the culprit may well be the used algorithm itself. A very simple solution to avoid the prior potential problem could be:

Host A and B:
MPI_Barrier(MPI_COMM_WORLD);
for(int i=0; i<n; i++) {
    MPI_Send(buf, count, MPI_INT, host_c, 0, MPI_COMM_WORLD);
    buf += n;
}

Host C:
for(int i=0; i<2*n; i++) {
    MPI_Irecv(buf, count, MPI_INT, MPI_ANY_SOURCE, 0, MPI_COMM_WORLD, &requests[i]);
    buf += count;
}
MPI_Barrier(MPI_COMM_WORLD);
for(int i=0; i<2*n; i++) {
    MPI_Waitany(2*n, requests, &idx, &status);
    process_data(buf+idx*count);
}

Now the data is placed on its final destination, since it's guaranteed that the MPI implementation knows its destination before receiving it. That solution would assume that there is enough memory for preallocating all receiving buffers, which may not be true and may require, for example, implementing a high-level protocol (such as based on credits) to synchronize the sends and receives to make sure that there is room on the receiver for them.

I hope this helps.

Best,
Antonio

On 05/24/2014 02:00 AM, Ron Palmer wrote: