Hello all,

In 2013 I had problems regarding the allocation of unexpected messages in MPI.

After your kind assistance, I implemented a "buffer" matrix in the receiver process, using MPI_IRECV, MPI_WAITANY and MPI_TESTANY functions (the code snippet is attached).

It has been working nicely since than until recently, when I faced the same problems again:

Fatal error in MPI_Recv: Other MPI error, error stack:
MPI_Recv(186)......................: MPI_Recv(buf=0x7fffe8dd5974, count=1, MPI_INTEGER, src=0, tag=MPI_ANY_TAG, MPI_COMM_WORLD, status=0xd213d0) failed
MPIDI_CH3I_Progress(402)...........:
MPID_nem_mpich2_blocking_recv(905).:
MPID_nem_tcp_connpoll(1838)........:
state_commrdy_handler(1676)........:
MPID_nem_tcp_recv_handler(1564)....:
MPID_nem_handle_pkt(636)...........:
MPIDI_CH3_PktHandler_EagerSend(606): Failed to allocate memory for an unexpected message. 261895 unexpected messages queued.
Fatal error in MPI_Recv: Other MPI error, error stack:
MPI_Recv(186).............: MPI_Recv(buf=0x7fffd052b9f4, count=1, MPI_INTEGER, src=0, tag=MPI_ANY_TAG, MPI_COMM_WORLD, status=0xd213d0) failed
dequeue_and_set_error(596): Communication error with rank 0
Fatal error in MPI_Recv: Other MPI error, error stack:
MPI_Recv(186).............: MPI_Recv(buf=0x7fff58fe5b74, count=1, MPI_INTEGER, src=0, tag=MPI_ANY_TAG, MPI_COMM_WORLD, status=0xd213d0) failed
dequeue_and_set_error(596): Communication error with rank 0
Fatal error in MPI_Recv: Other MPI error, error stack:
MPI_Recv(186).............: MPI_Recv(buf=0x7fff6fae19f4, count=1, MPI_INTEGER, src=0, tag=MPI_ANY_TAG, MPI_COMM_WORLD, status=0xd213d0) failed
dequeue_and_set_error(596): Communication error with rank 0
Fatal error in MPI_Recv: Other MPI error, error stack:
MPI_Recv(186).............: MPI_Recv(buf=0x7fff55bc8e74, count=1, MPI_INTEGER, src=0, tag=MPI_ANY_TAG, MPI_COMM_WORLD, status=0xd213d0) failed
dequeue_and_set_error(596): Communication error with rank 0

I'm using the MPICH2 1.4.1p1 on a Linux x64 machine (AWS EC2 instance). The last execution with this error had 63 working processes sending all the output to just one receiver/writer process.

The program and number of messages sent/received are pretty much the same. The only thing I can imagine is that, probably, the processor is proportionally faster than the network/IO speed today when compared to 2013 AWS EC2 instance. In this way, probably the writer process gets "flooded" with messages earlier. Does it make sense?

Could you please give some advice on how to solve this issue?

Best regards,

Luiz

On 13 March 2014 at 16:01, Luiz Carlos da Costa Junior <lcjunior@ufrj.br> wrote:

Thanks again Kenneth, I could solve using MPI_TESTANY.
Regards, Luiz

On 13 March 2014 15:35, Kenneth Raffenetti <raffenet@mcs.anl.gov> wrote:

On 03/13/2014 12:35 PM, Luiz Carlos da Costa Junior wrote:

Does anyone have any clue about this?

Thanks in advance.

On 12 March 2014 14:40, Luiz Carlos da Costa Junior <lcjunior@ufrj.br
<mailto:lcjunior@ufrj.br>> wrote:

Dear Kenneth,

Thanks for your quick reply.
I tested your suggestion and, unfortunately, this approach didn't work.

Question: when I call MPI_IPROBE it accounts also for the messages
that were already received asynchronously?

That should not be the case. If a message has been matched by a recv/irecv, MPI_Probe should not match it again.

Is there any way to know, for my list of mpi_requests (from my
MPI_IRECV's), which ones are "opened" and which ones have messages?

MPI_Test will take a request as an argument and tell you whether or not that requested operation has been completed.

Ken

_______________________________________________
discuss mailing list discuss@mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss