On Mon, 2022-01-24 at 14:16 +0100, Ahmad Tarraf via discuss wrote:
It is hard to find any literature or documentation about the asynchronous implementation ROMIO (or ADIO) use. I have few questions and It would be great if you can answer them: 1. How does ROMIO implement asynchronous I/O? I mean do you use POSIX-AIO, native Linux kernel AIO (io_submit,..), io_uring, or anything else? If I debug my code at our Clusters with gdb, I can see that POSIX-AIO is used. But is this on all systems so?
ROMIO will support async I/O if two conditions are met: 1: POSIX async I/O routines (aio_write, aio_suspend, etc) 2: "extended generalized requests" -- part of MPICH, but not OpenMPI (Some of the specialized file systems also have async routines: same approach applies though -- rely on the underlying file system or storage library to make progress see below)
2. When and how does asynchronous I/O progress occur? Do you span multiple I/O threads for a rank? Is this related to variable MPICH_ASYNC_PROGRESS? Again with gdb, I can see that several asynchronous write requests by the same MPI rank are handled successively by a single thread.
The posix aio library will make background progress (typically libc implements aio by spawning a thread -- I think kernel native aio is only used by databases?). Extended generalized requests allow ROMIO to query the state of pending AIO operations. when we designed those generalized request extensions the intent was to allow for "status checking" for those extensions. MPICH_ASYNC_PROGRESS can help speed things along, I think, in that it will kick the MPICH "progress engine" more frequently. Without that set, only TEST and WAIT routines invoke the callbacks that check the state of the async i/o routine. I have not benchmarked this and would love to hear your experiences.
3. Is there any related recent publication is can use to cite? I didn't find anything related to the implementation of asynchronous MPI-IO, except this: https://web.cels.anl.gov/~thakur/papers/grequest-redesign.pdf. But is it not outdated?
There has not been much recent work on extended generalized requests. You might find _Implementation and Evaluation of MPI Nonblocking Collective I/O_ interesting? https://dl.acm.org/doi/10.1109/CCGrid.2015.81
Best regards, Ahmad _______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss