On Tue, Dec 3, 2013 at 2:10 PM, Rob Latham <[email protected]> wrote:
On Tue, Dec 03, 2013 at 12:58:53PM -0800, Geoffrey Irving wrote:
Mea culpa. MPI_File_read_ordered is implemented in "stupid mode" right now, and no one ever complained -- we don't pay much attention to shared file pointers. Tell me more about how you are relying on ordered mode, please.
I'm dividing a large file into nearly equal size chunks and slurping the entire file into RAM. Then I do a bunch of rearrangement in memory. I could just as easily use MPI_File_read_at_all, but had assumed that ordered was better because my known intervals happen to be ordered and contiguous. Should I be using MPI_File_read_at_all instead, and treating the ordered routine as vestigial?
Yeah, the ordered mode routines, to borrow the "self-consistent MPI" model, are more strict than MPI_File_read_at_all -- not only are all processes participating, but they are participating in I/O that must end up in the file in rank-order.
Thanks, I'll switch to ..._read_at_all.
This is probably not a significant portion of your runtime, but have you considered combining mpi memory datatypes with mpi_file_read_at_all to get the read and the memory rearrangement in one go?
The rearrangement step is 2 orders of magnitude faster than the I/O, so it doesn't matter in this case, but datatypes do seem cleaner for future versions of similar code.
Okay, I was basically assuming that if I used a collective routine, it would be smart about accessing the file system as few times as possible. If this isn't true I should unqueue my job now and switch to MPI_File_read_at_all. Is this right? Should I also use MPI_File_write_at_all instead of the shared fp collective write routine?
Yeah, MPI_File_write_at_all and MPI_File_read_at_all will be the smart clever routines you are looking for -- though on the Cray they may need a bit of tuning. I *think* "good mode" is now the default, but I don't have a lot of experience with Cray's modifications to ROMIO.
Good to know. Thanks! Geoffrey