Dear all,
While profiling my application with Darshan, I found there was a large
imbalance in an MPI I/O completion.
Summary question: If every MPI process writes a 4 MB block using
collective MPI_File_write_all, why would there be 15x difference between
fastest and slowest ranks? (On BG/Q GPFS with 4 MB block size)
Here are more details:
My program is a pure-MPI application (no OpenMP, Pthreads, etc.) that
creates a single file and all processes write collectively to the file
through the MPI_File_write_all primitive. What I see with further
investigation with HPCToolkit is that there are some processes that
spend a really long time inside the MPI I/O collective call (more than
10 times compared to other processes, see attached screenshots).
After simplifying my application as much as possible, this is the
configuration I'm running:
- BG/Q system (tried on MIRA, Juqueen and CSCS: absolute execution times
vary, but imbalance stays the same)
- GPFS filesystem, 4 MB block size
- 1024 ranks in total
- 8 ranks / node, 128 nodes in total (also tried using full mid-plane
i.e. 512 nodes with no change in behavior)
- Each rank writes exactly a single, contiguous block of 4194304 bytes
(4 MB)
- Rank 0 writes its block at the beginning of the file (offset 0), then,
rank 1, then rank 2, etc.
- File offsets are set through MPI_File_set_view
- Environment variable set: BGLOCKLESSMPIO_F_TYPE=0x47504653
- Able to reproduce similar behavior with IOR benchmark as well
Observations:
- More than half of the ranks (> 512) spend less than 0.05 seconds
inside MPI_File_write_all
- The rest of the ranks spend more than 0.05 seconds inside
MPI_File_write_all, and the maximum time inside this call goes up to 1.2
seconds for some of the ranks
I would be very thankful if someone could help me understand this
strange (or expected ?) behavior.
Tracing details are provided in the attached pictures (screenshots of
HPCToolkit):
- 1-MPI_wr_all_imbalance.png:
General trace view of all the 1024 ranks. The MPI_File_write_all time is
shown in green. One can quickly see the imbalance between ranks.
- 2-MPI_wr_all_imbalance_zoom.png:
Zoom of the previous image, from rank 122 to 368. We can see here in
more detail the imbalance.
- 3-MPI_wr_all_call_stack_zoom.png:
Same zoom as the previous image, different color scheme to illustrate
the call stack of MPI_File_write_all and spot the functions that take
more time.
Could also someone explain when the __lseek_nocancel and
__write_nocancel functions are used and what for?
Please, let me know if there is any further information that might be
helpful. I can also provide the simplified source code.
Thanks in advance,
Judit
