So should we ignore IBM's changes and plan to fix it the other way? Given the amount of effort involved, it probably won't happen until the MPI Forum formalizes a fix at the interface level. Rajeev _____ From: [email protected] [mailto:[email protected]] On Behalf Of William Gropp Sent: Wednesday, May 07, 2008 7:14 PM To: [email protected] Subject: Re: [mpich2-core] BG/P patches I think that Rob's solution needs to be part of the overall solution - rather than rely on interfaces in the MPI spec that we know are broken, we should simply do the right thing in terms of using the internal spec. For MPI implementations that don't want to expose a lower layer, and that don't want to face this real issue, we can set it up so they can continue to use the MPI routines as ROMIO does now. We'll still need the option of making the dataloop code use a type that is large enough for both Aint and Offset. And we should provide an option to detect and report truncation (MPI_Address did this for Fortran, where a similar problem existed). Bill On May 7, 2008, at 6:23 PM, Rajeev Thakur wrote: Let's see if I have understood this right: 1. IBM promotes Aint to 64 bits and does the appropriate type casts. 2. Pavan suggests using MPI_Offset internally for datatypes and I/O instead of Aint. Return MPI_Aint to the user, as defined by MPI, possibly causing truncation. 3. Bill suggests using an MPIR_Size_t internally to represent message and I/O lengths in the datatype code. I think Pavan and Bill's suggestions are similar, except that Bill wants to use the bigger size in fewer places -- only where calculations occur. I am also attaching Rob's original proposal to IBM, where he outlined two approaches. The first one suggests using the dataloop code for I/O and using MPI_Offset to hold integer values. This is similar in spirit to Pavan and Bill's. The second one is to bump up MPI_Aint size. We originally recommended to IBM that they do the first solution, but then changed our mind after a month thinking it would be too much work, and recommended the Aint option. IBM also independently evaluated the two approaches and was in favor of Aint. What would be the least amount of work and also do the job right? Suppose we went with the MPI_Aint approach, but not made it the default. Instead it has to be selected explicitly at configure time by those who need it. (These are not those who would use MPI_Aint i instead of int *i in their attribute code.) Most users will not even see the change, and on 64-bit systems it is never an issue. Those users on 32-bit systems needing large files can be told to configure it. (Last I heard, IBM was also thinking of providing two versions of their library.) Would that work? Rajeev ________________________________ From: [email protected] [mailto:[email protected]] On Behalf Of William Gropp Sent: Wednesday, May 07, 2008 11:41 AM To: [email protected] Subject: Re: [mpich2-core] BG/P patches We're starting to go around in the same circle. Here's a summary: MPI Spec issues: The MPI spec defines type extents as MPI_Aint ; some other fields are ints. File offsets are in terms of MPI_Offset , which may be bigger (or smaller) than an MPI_Aint It is possible to create MPI datatypes bigger than what can be represented in an address-sized integer; this can be relevant to datatypes used with I/O. Users expect the size of a pointer to be exactly the size of an MPI_Aint. If this is not true, some operations with attributes will fail (cause memory/stack overwrites) MPICH2 Implementation issues: Internal implementation of datatypes uses MPI_Aint; there may be performance reasons for that on some platforms (instead of always using MPI_Offset) Blue Gene issues: MPI_Offset is larger than a pointer. Some I/O libraries (HDF5, as Rob L confirms) may need extents that are greater than 32 bits. The problem is that there is no solution that simultaneously satisfies every one of these requirements. This is key; *any* solution is going to have some warts. So, what can we do, and what principles should we follow? One principle that should have priority is the integrity of the test suites. For example, I would not make any changes to the attribute tests, unless they're clearly wrong (not just ambiguous). This limits the ability to replace MPI_Aint with an integer that is longer than a pointer in MPI (this is not the same as changing the use of MPI_Aint internally to MPICH2). Another principle is to avoid mandating something that might significantly impact performance. This is why I'm leery of mandating MPI_Offset in the datatype code, at least for now. Not all processors support 64 bit integer operations with the same speed as 32 bit operations, though that may change in the next few years. One thing that can be done for the datatype code *internals* is to use an MPIR_Size_t type; this can be typedef'ed appropriately. This would permit us to distinguish between what MPI defines as an Aint, what MPI defines as an Offset, and what MPICH2 wants to use internally for message lengths and datatype operations. The tradeoff might be that for some systems with 32 bit Aints and 64bit Offsets, some file operations could not be expressed with datatypes. For the HDF5 problem, the cleanest solution might be to propose an MPI 2.2 extension and then implement it. Bill On May 7, 2008, at 9:41 AM, Pavan Balaji wrote: I understand the problem. What I'm suggesting is that whatever has to be returned to the user can be an overflown value. But internally we can store the correct value. If the user needs this returned value for allocating a buffer, etc., then that's a problem, but that part is really a problem in the MPI standard, and not in the MPICH2 implementation. -- Pavan On Wed, 7 May 2008, Rob Ross wrote: Right. We need to be able to return the correct extent. I/O libraries/ applications need this capability, as Bill says. -- Rob On May 7, 2008, at 9:27 AM, William Gropp wrote: This is exactly the problem - the application might need an extent that won't fit in an MPI_Aint . Right now, we don't have a good solution for those applications. Bill On May 7, 2008, at 8:58 AM, Pavan Balaji wrote: I agree with Rajeev's plan. One thing to bear in mind (its in the mails but might not jump out) is that some I/O applications may need an MPI_Offset sized datatype extent (HDF5 might be in this group). Just to clarify my original suggestion -- I think all internal extents and sizes should be stored as MPI_Offset. If the user requests for the extent, we can always return an MPI_Aint (ignoring the overflow). -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign -- Pavan Balaji http://www.mcs.anl.gov/~balaji William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign <bgp-mpi-io-plan.txt> William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign