Dear Kevin, Thanks for your quick reply. Is there a way to find out which ranks are the aggregators? Your answer makes a lot of sense and this would explain the imbalance. However, in some cases I see the imbalance happens for every second rank, so it seems that there is an aggregator for every pair of ranks. Should I expect aggregators to be equally distributed among all the ranks? What should be the ratio between aggregators and total ranks? Thanks! Judit On 22/04/16 16:44, Harms, Kevin wrote:
In collective I/O, only a subset of ranks (aggregators) perform the actual I/O. The other ranks just transmit data to an aggregator rank. So having a "load imbalance" would be expected as the communication time of the data is likely much less than the time for a rank to write the buffered data out. Especially on Blue Genes which have high performance interconnects.
I did not look at all the details you sent, but hopefully this explains what you are seeing.
kevin
Dear all,
While profiling my application with Darshan, I found there was a large imbalance in an MPI I/O completion.
Summary question: If every MPI process writes a 4 MB block using collective MPI_File_write_all, why would there be 15x difference between fastest and slowest ranks? (On BG/Q GPFS with 4 MB block size)
Here are more details:
My program is a pure-MPI application (no OpenMP, Pthreads, etc.) that creates a single file and all processes write collectively to the file through the MPI_File_write_all primitive. What I see with further investigation with HPCToolkit is that there are some processes that spend a really long time inside the MPI I/O collective call (more than 10 times compared to other processes, see attached screenshots).
After simplifying my application as much as possible, this is the configuration I'm running: - BG/Q system (tried on MIRA, Juqueen and CSCS: absolute execution times vary, but imbalance stays the same) - GPFS filesystem, 4 MB block size - 1024 ranks in total - 8 ranks / node, 128 nodes in total (also tried using full mid-plane i.e. 512 nodes with no change in behavior) - Each rank writes exactly a single, contiguous block of 4194304 bytes (4 MB) - Rank 0 writes its block at the beginning of the file (offset 0), then, rank 1, then rank 2, etc. - File offsets are set through MPI_File_set_view - Environment variable set: BGLOCKLESSMPIO_F_TYPE=0x47504653 - Able to reproduce similar behavior with IOR benchmark as well
Observations: - More than half of the ranks (> 512) spend less than 0.05 seconds inside MPI_File_write_all - The rest of the ranks spend more than 0.05 seconds inside MPI_File_write_all, and the maximum time inside this call goes up to 1.2 seconds for some of the ranks
I would be very thankful if someone could help me understand this strange (or expected ?) behavior.
Tracing details are provided in the attached pictures (screenshots of HPCToolkit): - 1-MPI_wr_all_imbalance.png: General trace view of all the 1024 ranks. The MPI_File_write_all time is shown in green. One can quickly see the imbalance between ranks.
- 2-MPI_wr_all_imbalance_zoom.png: Zoom of the previous image, from rank 122 to 368. We can see here in more detail the imbalance.
- 3-MPI_wr_all_call_stack_zoom.png: Same zoom as the previous image, different color scheme to illustrate the call stack of MPI_File_write_all and spot the functions that take more time.
Could also someone explain when the __lseek_nocancel and __write_nocancel functions are used and what for?
Please, let me know if there is any further information that might be helpful. I can also provide the simplified source code.
Thanks in advance, Judit