Are you the same Pramod Kumbhar that works at EPFL?

1: collective I/O does consume some memory.  not only is there an
internal "collective buffer" maintained by MPI-IO itself, but the data
exchange copies data as well before calling ALLTOALL.

Paul Coffman has done a one-sided based two-phase implementation that
should be lower memory overhead.  But here we should take the
discussion off-list.

==rob

>
> Quick Summary :
>
> 1. On bg-q I see cb_buffer_size as 16MB when we query on file handle
> using MPI_File_get_info.
> An application that we are looking at has code section like: 
>
> ….
> MPI_File_set_view( fh, position_to_write, MPI_FLOAT, mappingType,
> _native_, MPI_INFO_NULL );
> max_mb_on_any_rank_using_Kernel_GetMemorySize () => 275 MB
> MPI_File_write_all( fh, mappingBuffer, ....................
> MPI_FLOAT, &status);
> max_mb_on_any_rank_using_Kernel_GetMemorySize () => 373 MB
> ……
>
> Why we see that spike in memory usage?  (see Detail section for size
> information)
>
> I have seen “Kernel_GetMemorySize(KERNEL_MEMSIZE_HEAP….)” not
> returning accurate memory footprint but I am not sure if that is the
> case here.
> Darshan screenshot attached shows the access sizes while running on 4
> rack.
>
> 2. Is romio_cb_alltoall ignored on bg-q? Even if I disable it, I see
> “automatic” in the output.
>
> (I am looking at
> srcV1R2M4/comm/lib/dev/mpich2/src/mpi/romio/adio/ad_bg/ad_bg_hints.c
> and see the code section is commented.)
>
> More Details :
>
> We are debugging an application on MIRA which runs on 1,2,4 racks but
> fails at 8 racks while dumping a custom checkpoint. These are strong
> scaling runs and the size of checkpoint remains same (~172GB). 32
> ranks per mode. Max memory usage before start of checkpoint (i.e.
> before single write_all call)
> for 8 rack is ~ 300 MB. The checkpoint size from each rank is between
> Kbs to few MBs (as shown by darshan). Once application call
> checkpoint, we see below error :
>
>   Out of memory in file
> /bgsys/source/srcV1R2M2.15270/comm/lib/dev/mpich2/src/mpi/romio/adio/
> ad_bg/ad_bg_wrcoll.c,     line 500
>
> And hence I am confused about behaviour mentioned in question 1.
> If someone has any insight, it will be great help!
>
> Regards,
> Pramod
>
> p.s. 
>
> Default values of all hints 
>
> cb_buffer_size, value = 16777216
> romio_cb_read, value = enable
> romio_cb_write, value = enable
> cb_nodes, value = 8320             (change based on partition size)
> romio_no_indep_rw, value = false
> romio_cb_pfr, value = disable
> romio_cb_fr_types, value = aar
> romio_cb_fr_alignment, value = 1
> romio_cb_ds_threshold, value = 0
> romio_cb_alltoall, value = automatic
> ind_rd_buffer_size, value = 4194304
> romio_ds_read, value = automatic
> romio_ds_write, value = disable

> _______________________________________________
> discuss mailing list     discuss@mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss
_______________________________________________
discuss mailing list     discuss@mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss