On Tue, Dec 03, 2013 at 12:58:53PM -0800, Geoffrey Irving wrote:
Mea culpa. MPI_File_read_ordered is implemented in "stupid mode" right now, and no one ever complained -- we don't pay much attention to shared file pointers. Tell me more about how you are relying on ordered mode, please.
I'm dividing a large file into nearly equal size chunks and slurping the entire file into RAM. Then I do a bunch of rearrangement in memory. I could just as easily use MPI_File_read_at_all, but had assumed that ordered was better because my known intervals happen to be ordered and contiguous. Should I be using MPI_File_read_at_all instead, and treating the ordered routine as vestigial?
Yeah, the ordered mode routines, to borrow the "self-consistent MPI" model, are more strict than MPI_File_read_at_all -- not only are all processes participating, but they are participating in I/O that must end up in the file in rank-order. This is probably not a significant portion of your runtime, but have you considered combining mpi memory datatypes with mpi_file_read_at_all to get the read and the memory rearrangement in one go?
Okay, I was basically assuming that if I used a collective routine, it would be smart about accessing the file system as few times as possible. If this isn't true I should unqueue my job now and switch to MPI_File_read_at_all. Is this right? Should I also use MPI_File_write_at_all instead of the shared fp collective write routine?
Yeah, MPI_File_write_at_all and MPI_File_read_at_all will be the smart clever routines you are looking for -- though on the Cray they may need a bit of tuning. I *think* "good mode" is now the default, but I don't have a lot of experience with Cray's modifications to ROMIO. ==rob
I wrote a little ordered mode shim you can link into your program. Looks like I last seriously did anything with it back in 2008 ? I cleaned up some of the warnings and made it use MPI_Count (remove HAVE_MPI_TYPE_SIZE_X from the makefile if you're on an old system). Caveat: I think the last time I ran this code was on Argonne's Blue Gene /L. Bear in mind the adage about tested code and broken code.
Thanks, though in my case switching away from the shared file pointer routine is easy, so I don't need the shim.
Geoffrey
-- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA