On Fri, 2017-04-21 at 16:13 +0200, pramod kumbhar wrote:

Dear All,

I would like to understand some details about MPI I/O hints on bg-q and out of memory error while doing collective i/o.

Are you the same Pramod Kumbhar that works at EPFL? I only ask because it does not seem you have asked ALCF support (who run Mira) about your problem. We're going to quickly get off-topic for this general MPICH mailing list, but let me give some on-topic answers before we go discuss this on a more machine-specific venue. 1: collective I/O does consume some memory. not only is there an internal "collective buffer" maintained by MPI-IO itself, but the data exchange copies data as well before calling ALLTOALL. 1a: your darshan summary looks like it's doing the right stuff. Your tiny MPI-IO operations are getting transformed into more GPFS-friendly multi-megabyte requests. Good. Paul Coffman has done a one-sided based two-phase implementation that should be lower memory overhead. But here we should take the discussion off-list. ==rob

Quick Summary :

1. On bg-q I see cb_buffer_size as 16MB when we query on file handle using MPI_File_get_info. An application that we are looking at has code section like: 

…. MPI_File_set_view( fh, position_to_write, MPI_FLOAT, mappingType, _native_, MPI_INFO_NULL ); max_mb_on_any_rank_using_Kernel_GetMemorySize () => 275 MB MPI_File_write_all( fh, mappingBuffer, .................... MPI_FLOAT, &status); max_mb_on_any_rank_using_Kernel_GetMemorySize () => 373 MB ……

Why we see that spike in memory usage?  (see Detail section for size information)

I have seen “Kernel_GetMemorySize(KERNEL_MEMSIZE_HEAP….)” not returning accurate memory footprint but I am not sure if that is the case here. Darshan screenshot attached shows the access sizes while running on 4 rack.

2. Is romio_cb_alltoall ignored on bg-q? Even if I disable it, I see “automatic” in the output.

(I am looking at srcV1R2M4/comm/lib/dev/mpich2/src/mpi/romio/adio/ad_bg/ad_bg_hints.c and see the code section is commented.)

More Details :

We are debugging an application on MIRA which runs on 1,2,4 racks but fails at 8 racks while dumping a custom checkpoint. These are strong scaling runs and the size of checkpoint remains same (~172GB). 32 ranks per mode. Max memory usage before start of checkpoint (i.e. before single write_all call) for 8 rack is ~ 300 MB. The checkpoint size from each rank is between Kbs to few MBs (as shown by darshan). Once application call checkpoint, we see below error :

  Out of memory in file /bgsys/source/srcV1R2M2.15270/comm/lib/dev/mpich2/src/mpi/romio/adio/ ad_bg/ad_bg_wrcoll.c,     line 500

And hence I am confused about behaviour mentioned in question 1. If someone has any insight, it will be great help!

Regards, Pramod

p.s. 

Default values of all hints 

cb_buffer_size, value = 16777216 romio_cb_read, value = enable romio_cb_write, value = enable cb_nodes, value = 8320             (change based on partition size) romio_no_indep_rw, value = false romio_cb_pfr, value = disable romio_cb_fr_types, value = aar romio_cb_fr_alignment, value = 1 romio_cb_ds_threshold, value = 0 romio_cb_alltoall, value = automatic ind_rd_buffer_size, value = 4194304 romio_ds_read, value = automatic romio_ds_write, value = disable _______________________________________________ discuss mailing list     [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss