Begin forwarded message:
From: "Steven C. Pieper" <[email protected]> Subject: suggestion for mpich diagnostic Date: June 4, 2014 at 3:14:09 PM CDT To: Kenneth Raffenetti <[email protected]>, Rusty Lusk <[email protected]> Reply-To: <[email protected]>
I have just found a bug in my mpi code. On blues the bug resulted in the following single stderr line
Fatal error in MPI_Recv: Message truncated
with no clue as to where it happened. Fortunately I had noticed that in the latest production mpich that I installed (with Ken's help) on my cluster I get more info. Indeed there I got the following:
Fatal error in MPI_Recv: Message truncated, error stack: MPI_Recv(184).......................: MPI_Recv(buf=0x4036510, count=1, dtype=USER<contig>, src=MPI_ANY_SOURCE, tag=101, comm=0x84000002, status=0xb3b920) failed MPIDI_CH3U_Request_unpack_uebuf(605): Message truncated; 29292 bytes received but buffer size is 488
The tag=101 was enough in this case for me to locate the error. But a little more info would be useful:
1) The rank of the process with the failing receive. 2) The actual rank of the sending process. 3) If at all possible, trigger a traceback and coredump of the rank with the failing receive.
Steve -- Steven C. Pieper: [email protected] Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Morrison, -4100