February 2016 - Darshan-users - mailman3.cels.anl.gov

Estimating the overlap of MPI I/O wait time
by Jean-Thomas Acquaviva 10 Feb '16

10 Feb '16

Hi, I'm using Darshan to analyze the behavior of a parallel MPI Fortran application which writes to a shared file. Among 1024 MPI ranks only 48 are effectively doing I/O, this "geometry" is successfully reported by Darshan. However, for the remaining 48 ranks Darshan reports the MPI I/O time spent writing to the shared file on a per rank basis, but I did not find a way to estimate the overlap between these different ranks. At one end of the spectrum, if write accesses are serialized then the total I/O wait time for the application is the sum of the individual wait time of each rank. On the other end, if write accesses are perfectly parallel the total app wait time is the max among the 48 writers. Probably the exact number lies between these two values, is there a way to estimate it ? Best regards, jean-thomas

2 3

Re: [Darshan-users] Profiling arbitrary parts of an application with Darshan
by Phil Carns 01 Feb '16

01 Feb '16

Hi Judit, Is there any chance that we could have a look at your log to help debug it? You can send it off-list if you prefer, and we can also tell you how to anonymize the information (file names, command line, etc.) in it first if needed. As a side note, Florin Isaila implemented some Darshan extensions (which can be found in the darshan-florin-extensions branch in git) that include explicit hooks you can call to enable and disable Darshan at runtime. It will create a separate log file for each period that you have enabled Darshan. That is another option if we can't get the summary-per-file tool working quickly, but you should consider that branch to be experimental for now. thanks, -Phil On 02/01/2016 03:58 AM, Judit Planas wrote: > Hi Rob, > > Thanks for your reply. > > Unfortunately, I already tried darshan-summary-per-file.sh, but it > didn't work for me: some of the charts and tables in the generated PDF > are blank or filled with 0's. > > For example, 'I/O sizes' and 'I/O pattern' charts are completely blank. > And the 'Average I/O per process' and 'Data Transfer Per Filesystem' > tables are filled with either 0's or 'N/A'. However, I know the file was > created by my application and around 16 MB of data were written (that is > the size of the file at the end of the execution). > > When running the darshan-summary-per-file.sh script, I get the following > messages for the file I'm interested in (I guess the PDF charts are > blank because of the warnings...): > ======================================================= > Slowest unique file time: 0 > Slowest shared file time: 65.503991 > Total bytes read and written by app (may be incorrect): 0 > Total absolute I/O time: 65.503991 > Warning: empty y range [0:0], adjusting to [-1:1] > Warning: empty y range [0:0], adjusting to [-1:1] > Status: Generating summary for file 81 of 177: allcurrents.bbp > ======================================================= > > Any idea? > > Thanks in advance, > Judit > > > > On 01/28/2016 05:05 AM, Judit Planas wrote: > >>/Hello, />>//>>/I am using Darshan to profile my application and I wonder if it is />>/possible to enable/disable Darshan profiling for arbitrary parts of the/ > >>/application. />>//>>/The thing is that my application does a lot of I/O work, but I only want />>/to profile a small part. If I profile the whole application, the reports />>/have a lot of noise from the I/O part that I do not want to profile. />>//>>/Darshan version is 2.3.1 (but I can update it to the latest one if />>/needed) and I am running on a BG/Q system. /> > > There are two things that might help you: > > > > - darshan-summary-per-file.sh might do the trick? you can generate > > profiles on a per-file basis when you want to, for example, exclude > > reading in the input deck and focus only on output. This approach works > > well if there are particular files you want to study, but not so well if > > there's a particular phase of I/O you are most concerned about: > > > > - I don't know if anyone has tried using MPI_PCONTROL with darshan. I > > don't think Darshan knows what to do with that routine, anyway. > > > > ==rob > > > > > >>//>/> Thanks in advance, />/> Judit />/> _______________________________________________ />/> Darshan-users mailing list />/> Darshan-users at lists.mcs.anl.gov > <https://lists.mcs.anl.gov/mailman/listinfo/darshan-users> />/> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users /

1 0

Profiling arbitrary parts of an application with Darshan
by Judit Planas 01 Feb '16

01 Feb '16

Hello, I am using Darshan to profile my application and I wonder if it is possible to enable/disable Darshan profiling for arbitrary parts of the application. The thing is that my application does a lot of I/O work, but I only want to profile a small part. If I profile the whole application, the reports have a lot of noise from the I/O part that I do not want to profile. Darshan version is 2.3.1 (but I can update it to the latest one if needed) and I am running on a BG/Q system. Thanks in advance, Judit

2 2