On 11/06/2015 07:13 PM, Balaji, Pavan wrote:
Adam,
Not yet. We were working heads-down on the mpich-3.2 release.
I'm adding the mpich discuss list to cc, and poking @robl as well. Do you have a test program that reproduces the error? (sorry if you already sent it, I'm still catching up on the pending issues).
adam, thanks for reporting. Since you're the first person I've ever met who cared about external32 support, let me give you a bit of background. Our External32 support came from Intel, and we know about one bug : if the externa32 representation differs from the memory size (MPI_LONG and MPI_AINT are perhaps the two most prominent examples) - http://trac.mpich.org/projects/mpich/ticket/1754 (but look for 'goodell' and 'robl'. 'jhammond' comments are less relevant for this discussion) - http://trac.mpich.org/projects/mpich/ticket/1755 We do have an external32 test case, but it's pretty simple, and independent i/o only. OpenMPI found a bug in our external32 code, which sounds related to your problem: http://git.mpich.org/mpich.git/commitdiff/53f11fd73b2f9aa7b994e78ed05e9ae742... So you can see why Pavan was hoping you had a test case: it's taken a few tries to get it right. ==rob
-- Pavan
On Nov 6, 2015, at 2:10 PM, Adam T. Moody <[email protected]> wrote:
Hi Pavan, Did you have a chance to look into this in MPICH?
I'm pretty certain there is a lurking ROMIO bug here. -Adam
Adam T. Moody wrote:
Hi Pavan and Howard, FYI, it looks this same bug is in MPICH-3.2rc1 and Open MPI-1.10.0.
I guess no one uses collective I/O with external32 :-) -Adam
------------------------------------------------------------------------
Subject: MPI_File_read_at_all external32 bug in MV2-2.1 From: "Adam T. Moody" <[email protected]> Date: Fri, 30 Oct 2015 11:48:35 -0700 To: "[email protected]" <[email protected]>
To: "[email protected]" <[email protected]>
Hello MVAPICH team, I've hit a bug in MPI_File_read_at_all in MVAPICH2-2.1. I have an application that reads and writes files in external32 format. It writes the file just fine, but it throws the following error when reading the file back:
internal ABORT - process 0 srun: error: rzmerl2: task 0: Exited with exit code 1 Assertion failed in file src/mpid/common/datatype/mpid_ext32_segment.c at line 277: FALSE memcpy argument memory ranges overlap, dst_=0x2aaab5c160a8 src_=0x2aaab5c160a8 len_=176
Above, MPI is trying to do a memcpy where the source and destination buffer are the same address. Looking through the code for MVAPICH2-2.1, the problem seems to be at line 132 in src/mpi/romio/mpi-io/read_all.c:
if (e32_buf != NULL) { error_code = MPIU_read_external32_conversion_fn(xbuf, datatype, count, e32_buf); ADIOI_Free(e32_buf); }
I think the fix is to change "xbuf" above to "buf" as it is in read.c below:
if (e32_buf != NULL) { error_code = MPIU_read_external32_conversion_fn(buf, datatype, count, e32_buf); ADIOI_Free(e32_buf); }
When in external32 mode, xbuf == e32_buf, which acts as a temporary buffer in which to read the data. The code is then meant to unpack and convert the data from the temporary buffer into the user buffer at buf.
It's probably worth checking the other external32 code paths to look for similar bugs. Thanks, -Adam
-- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA