April 2014 - Darshan-users - mailman3.cels.anl.gov

Re: [Darshan-users] Darshan pre-release: 2.2.9-pre1
by Phil Carns 14 Apr '14

14 Apr '14

I also forgot to mention that there are some documentation updates in this release. The updated documentation is not available on the web site, but you can find it in the darshan-util/doc/ and darshan-runtime/doc/ subdirectories in the source code. thanks, -Phil On 04/14/2014 04:34 PM, Carns, Philip H. wrote: > There is a new experimental version of Darshan available for download > today. It includes fixes for several bugs as well as support for > versions 2.0 and newer of the Cray Programming Environment. The full > change log is listed below. > > Feedback is always welcome! We will do more thorough cross-platform > testing before the official 2.2.9 release; please just use this version > for testing and evaluation. > > thanks! > -Darshan team > > darshan-2.2.9-pre1 > ============= > * Bug fixes: > - Fix mnt table overflow if a large number of file systems are mounted, > reported by David Shrader. > - Fix argument parsing for darshan-convert, reported by Mouhamed Gueye. > - Fix metadata annotation overflow in darshan-convert, reported by > Mouhamed Gueye. > - Fix const-correctness in dynamic library when built against MPI 3.x > libraries. > - Fix "undefined symbol: dlsym" error when using preloaded dynamic > library > on some platforms, reported by Florin Isaila. > - Normalize timestamps to always be relative to MPI_Init(). > - Better library name matching in compiler wrappers to handle more MPICH > variations on Blue Gene systems. > - Fixed header dependency problems when using darshan-logutils.h in > external > programs (Shane Snyder). > * Enhancements: > - Add support (both in documentation and in provided module files) for > Cray PE 2.x. > - Honor CC variable to allow darshan-util to be built with other > compilers > besides gcc. > - Update documentation to clarify the interaction between rpath and > LD_PRELOAD when instrumenting dynamic libraries, issue reported and > investigated by Davide Del Vento. > - Ability to disable shared-file reduction by setting the > DARSHAN_DISABLE_SHARED_REDUCTION environment variable. > - More thorough output from darshan-parser --perf, suggested by Huong > Luu. > - Increased metadata annotation room from 64 bytes to 1KiB in header. > - CP_F_{FASTEST/SLOWEST}_RANK_TIME counters now take MPI-IO time into > account, not just POSIX time, issue reported by Huong Luu. > - Better handling of systems with many mounted file systems (after which > point Darshan will assume file resides on / file system), issue > reported > and investigated by David Shrader: > - Track up to 64 rather than 32 mounted file systems at runtime. > - Increase header space available for storing mount point > information in > log file from approximately 1 KiB to approximately 3 KiB. > - Prioritize storing information about non-NFS volumes over NFS volumes > if too many file systems are mounted to record them all. > - Added darshan-util pkgconfig file (Shane Snyder). > > _______________________________________________ > Darshan-users mailing list > Darshan-users(a)lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/darshan-users

1 0

Darshan pre-release: 2.2.9-pre1
by Phil Carns 14 Apr '14

14 Apr '14

There is a new experimental version of Darshan available for download today. It includes fixes for several bugs as well as support for versions 2.0 and newer of the Cray Programming Environment. The full change log is listed below. Feedback is always welcome! We will do more thorough cross-platform testing before the official 2.2.9 release; please just use this version for testing and evaluation. thanks! -Darshan team darshan-2.2.9-pre1 ============= * Bug fixes: - Fix mnt table overflow if a large number of file systems are mounted, reported by David Shrader. - Fix argument parsing for darshan-convert, reported by Mouhamed Gueye. - Fix metadata annotation overflow in darshan-convert, reported by Mouhamed Gueye. - Fix const-correctness in dynamic library when built against MPI 3.x libraries. - Fix "undefined symbol: dlsym" error when using preloaded dynamic library on some platforms, reported by Florin Isaila. - Normalize timestamps to always be relative to MPI_Init(). - Better library name matching in compiler wrappers to handle more MPICH variations on Blue Gene systems. - Fixed header dependency problems when using darshan-logutils.h in external programs (Shane Snyder). * Enhancements: - Add support (both in documentation and in provided module files) for Cray PE 2.x. - Honor CC variable to allow darshan-util to be built with other compilers besides gcc. - Update documentation to clarify the interaction between rpath and LD_PRELOAD when instrumenting dynamic libraries, issue reported and investigated by Davide Del Vento. - Ability to disable shared-file reduction by setting the DARSHAN_DISABLE_SHARED_REDUCTION environment variable. - More thorough output from darshan-parser --perf, suggested by Huong Luu. - Increased metadata annotation room from 64 bytes to 1KiB in header. - CP_F_{FASTEST/SLOWEST}_RANK_TIME counters now take MPI-IO time into account, not just POSIX time, issue reported by Huong Luu. - Better handling of systems with many mounted file systems (after which point Darshan will assume file resides on / file system), issue reported and investigated by David Shrader: - Track up to 64 rather than 32 mounted file systems at runtime. - Increase header space available for storing mount point information in log file from approximately 1 KiB to approximately 3 KiB. - Prioritize storing information about non-NFS volumes over NFS volumes if too many file systems are mounted to record them all. - Added darshan-util pkgconfig file (Shane Snyder).

1 0

Re: [Darshan-users] Minor issues
by Phil Carns 11 Apr '14

11 Apr '14

This thread is pretty old at this point, but I wanted to provide one more update: in addition to the bug fixes Kevin described below, the trunk version of Darshan now also adds an extra 3 KiB of room to the metadata field (which is used for annotations in darshan-convert) so that you can generally store more information there now. The upcoming 2.2.9 release is going to increment the log format version anyway so it was a good time to make this change. thanks, -Phil On 12/17/2013 03:04 PM, Harms, Kevin N. wrote: > Thank you for the bug reports. > >> 1 - darshan-convert --annotate option : nb of character is limited to 6 for the metadata name >> >> When I tried to configure a metadata with annotate option, if the number of character of metadata name is superior to 6, the metadata is malformated. > This one is partially fixed in 2.2.8. The outcome might not be exactly desirable but when darshan-convert runs it appends the previous log version in the metadata. When adding this, there is a check if the metadata field would be exceeded and if so, it drops all other metadata and inserts only the prev_ver=x.yz data. Given that the current size of the metadata is limited to 64 characters, when we try to add prev_ver=2.02 to the existing metadata, the size will always be exceeded so no existing or new metadata will be inserted and only the prev_ver data will be present. > > I created a RT ticket for this: https://trac.mcs.anl.gov/projects/darshan/ticket/121 > > I also committed a new code change that explicitly checks for overflow when appending user metadata. > > https://trac.mcs.anl.gov/projects/darshan/changeset/808 > >> 2 - darshan-convert with bad argument -> segmentation fault >> >> When I used a bad argument for darshan-convert command, a segmentation occurred. >> >> Note : for the other Darshan commands, the help is displayed. >> >> [petit-v@valx13 darshan]$ darshan-convert --anotate >> Segmentation fault (core dumped) > I opened an RT ticket and fixed this in revision 808 as well. > > https://trac.mcs.anl.gov/projects/darshan/ticket/120 > https://trac.mcs.anl.gov/projects/darshan/changeset/808 > > kevin >

1 0

fputs and internal glibc i/o calls
by Chuck Cranor 03 Apr '14

03 Apr '14

hi- I downloaded and tried 2.2.8 to better understand how it works and what it can do. The test app I used uses MPI to simulate checkpoints and then proc 0 writes the performance results to a log file. I used the LD_PRELOAD mechanism to load darshan. Looking at the darshan log file from this app, I noticed that darshan captured proc 0 creating the log file, but didn't show any writes to it (even though new data was added). This is because the app does all its writes with fputs and darshan doesn't intercept calls to fputs (or calls to fprintf either, for that matter). I'm not sure if omitting fputs/fprintf is intentional or not, but I didn't see anything in the documentation about it. Also, it is worth noting in the documentation somewhere that internal glibc I/O system calls may not be captured in the darshan log. For example, fputs() is a libc function that uses write() internally, but the internal write() will not appear in the darshan log. I believe this is because glibc does not allow internally generated system calls to be overridden. See the discussion of hidden prototypes here: https://sourceware.org/glibc/wiki/Testing/Check-localplt An easy way to demo this is with the ctime(3) function. Internally, ctime(3) uses the POSIX open(2) I/O call to load local timezone information from /etc/localtime. So you might think a simple program like: #include <stdio.h> #include <time.h> main(int argc, char **argv ) { time_t now; MPI_Init(&argc, &argv); now = time(0); printf("time=%s", ctime(&now)); MPI_Finalize(); } would show the I/O to /etc/localtime in the darshan log, but it doesn't: h0:/tmp/ctime# ./a.out time=Thu Apr 3 11:05:56 2014 h0:/tmp/ctime# h0:/tmp/ctime# strace ./a.out |& fgrep localtime open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 4 h0:/tmp/ctime# h0:/tmp/ctime# env LD_PRELOAD=/usr/local/lib/libdarshan.so mpirun.mpich2 ./a.out time=Thu Apr 3 11:06:31 2014 h0:/tmp/ctime# darshan-parser /m/pvfs/tmp/2014/4/3/root_a.out_id22281_4-3-39991-1756671920181884080_1.darshan.gz # darshan log version: 2.02 # size of file statistics: 1328 bytes # size of job statistics: 120 bytes # exe: ./a.out # uid: 0 # jobid: 22281 # start_time: 1396544791 # start_time_asci: Thu Apr 3 11:06:31 2014 # end_time: 1396544791 # end_time_asci: Thu Apr 3 11:06:31 2014 # nprocs: 1 # run time: 1 # metadata: lib_ver = 2.2.8 # metadata: h = romio_no_indep_rw=true;cb_nodes=4 # mounted file systems (device, mount point, and fs type) # ------------------------------------------------------- # mount entry: 8879946760550363901 /m/pvfs fuse # mount entry: -6525176383006974612 /l0 ext2 # mount entry: 1378004016996341907 /users/garth nfs # mount entry: -6395289464566466203 /users/qingzhen nfs # mount entry: 3967925232003708285 /proj/TableFS nfs # mount entry: -5933552100018119666 /users/chuck nfs # mount entry: -9071479661283960111 /share nfs # mount entry: 361011049498856045 /users/kair nfs # mount entry: 3179183617035706161 /dev devtmpfs # mount entry: -648807988769344735 / ext3 # no files opened. h0:/tmp/ctime# The only internal functions that glibc seems to let you override are malloc-related: h0:/l0/glibc-2.14# cat scripts/data/localplt-generic.data libc.so: calloc libc.so: free libc.so: malloc libc.so: memalign libc.so: realloc libm.so: matherr h0:/l0/glibc-2.14# that's prob for malloc debuggers, otherwise they would have trouble with functions like strdup(3) that call malloc internally for an application [and expect the app to free(3) the memory later]. I suspect the behavior might be different using a static-linked program, but haven't tried it... chuck

2 2

Re: [Darshan-users] fputs and internal glibc i/o calls
by Phil Carns 03 Apr '14

03 Apr '14

Hi Chuck, Thanks for the detailed information. Darshan definitely does not currently provide wrappers for the stream functions, and you are right that we can't catch the underlying I/O calls either. As far as whether it is intentional or not, it is at least a long-standing known problem :) We haven't added the wrappers and tested them. There are quite a few variations on the stream functions; we would need a separate wrapper for each one. Chris Daley of NERSC has run into this recently as well. I think this is actually going to become a more frequent problem, especially with applications that access genomics data in text format. We won't be able to address this in the next point release (2.2.9), but I can keep you in the loop if you would be interested in helping to test an experimental version of this feature in the medium term. We have an old ticket open on this issue. I just clarified the description and added some comments to help keep track of what's going on: http://trac.mcs.anl.gov/projects/darshan/ticket/38 thanks, -Phil On 04/03/2014 01:17 PM, Chuck Cranor wrote: > hi- > > I downloaded and tried 2.2.8 to better understand how it works > and what it can do. The test app I used uses MPI to simulate > checkpoints and then proc 0 writes the performance results to a log > file. I used the LD_PRELOAD mechanism to load darshan. > > Looking at the darshan log file from this app, I noticed that > darshan captured proc 0 creating the log file, but didn't show > any writes to it (even though new data was added). This is because > the app does all its writes with fputs and darshan doesn't intercept > calls to fputs (or calls to fprintf either, for that matter). > > I'm not sure if omitting fputs/fprintf is intentional or not, but > I didn't see anything in the documentation about it. > > > Also, it is worth noting in the documentation somewhere that > internal glibc I/O system calls may not be captured in the darshan > log. For example, fputs() is a libc function that uses write() > internally, but the internal write() will not appear in the darshan > log. I believe this is because glibc does not allow internally > generated system calls to be overridden. See the discussion of > hidden prototypes here: > > https://sourceware.org/glibc/wiki/Testing/Check-localplt > > An easy way to demo this is with the ctime(3) function. Internally, > ctime(3) uses the POSIX open(2) I/O call to load local timezone > information from /etc/localtime. So you might think a simple program like: > > #include <stdio.h> > #include <time.h> > > main(int argc, char **argv ) { > time_t now; > MPI_Init(&argc, &argv); > now = time(0); > printf("time=%s", ctime(&now)); > MPI_Finalize(); > } > > would show the I/O to /etc/localtime in the darshan log, but it > doesn't: > > h0:/tmp/ctime# ./a.out > time=Thu Apr 3 11:05:56 2014 > h0:/tmp/ctime# > > h0:/tmp/ctime# strace ./a.out |& fgrep localtime > open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 4 > h0:/tmp/ctime# > > h0:/tmp/ctime# env LD_PRELOAD=/usr/local/lib/libdarshan.so mpirun.mpich2 ./a.out > time=Thu Apr 3 11:06:31 2014 > h0:/tmp/ctime# darshan-parser /m/pvfs/tmp/2014/4/3/root_a.out_id22281_4-3-39991-1756671920181884080_1.darshan.gz > # darshan log version: 2.02 > # size of file statistics: 1328 bytes > # size of job statistics: 120 bytes > # exe: ./a.out > # uid: 0 > # jobid: 22281 > # start_time: 1396544791 > # start_time_asci: Thu Apr 3 11:06:31 2014 > # end_time: 1396544791 > # end_time_asci: Thu Apr 3 11:06:31 2014 > # nprocs: 1 > # run time: 1 > # metadata: lib_ver = 2.2.8 > # metadata: h = romio_no_indep_rw=true;cb_nodes=4 > > # mounted file systems (device, mount point, and fs type) > # ------------------------------------------------------- > # mount entry: 8879946760550363901 /m/pvfs fuse > # mount entry: -6525176383006974612 /l0 ext2 > # mount entry: 1378004016996341907 /users/garth nfs > # mount entry: -6395289464566466203 /users/qingzhen nfs > # mount entry: 3967925232003708285 /proj/TableFS nfs > # mount entry: -5933552100018119666 /users/chuck nfs > # mount entry: -9071479661283960111 /share nfs > # mount entry: 361011049498856045 /users/kair nfs > # mount entry: 3179183617035706161 /dev devtmpfs > # mount entry: -648807988769344735 / ext3 > # no files opened. > h0:/tmp/ctime# > > > The only internal functions that glibc seems to let you override > are malloc-related: > > h0:/l0/glibc-2.14# cat scripts/data/localplt-generic.data > libc.so: calloc > libc.so: free > libc.so: malloc > libc.so: memalign > libc.so: realloc > libm.so: matherr > h0:/l0/glibc-2.14# > > that's prob for malloc debuggers, otherwise they would have > trouble with functions like strdup(3) that call malloc internally > for an application [and expect the app to free(3) the memory later]. > > > I suspect the behavior might be different using a static-linked > program, but haven't tried it... > > > chuck > _______________________________________________ > Darshan-users mailing list > Darshan-users(a)lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/darshan-users

1 0

Re: [Darshan-users] darshan doesn't generate the login
by Phil Carns 03 Apr '14

03 Apr '14

Hi Nicola, I can think of a couple of things to try: a) maybe confirm that the executable has dynamically linked the mpi libraries (run "ldd lmp_fornax" to see a list of dynamic libraries) just as a safety check. b) I don't regularly use mpirun from OpenMPI, but does it automatically propagate environment variables to the executable or do you need to specify something on the command line to get that behavior? If your executable can run correctly with a single rank then you can take mpirun out of the loop temporarily for testing purposes and just try running "/scratch/director100/nicola/lammps_pistolare/src/lmp_fornax < in.friction" directly. If the instrumentation works in that case but not when using mpirun then that would indicate that something isn't being translated into the mpirun environment. thanks, -Phil On 04/02/2014 11:49 PM, nicola varini wrote: > Hi, I am new to darshan. Sorry if my question is trivial but I've tried > to search a bit through the mailing list archive and I did not find much > help. > I am try to understand how to use darshan and I've build the runtime with > ./configure --with-mem-align=8 > --with-log-path=/scratch/director100/nicola/darshan-2.2.8 CC=mpicc > --with-jobid-env=PBS_JOBID > --prefix=/scratch/director100/nicola/darshan-2.2.8 > and the util with > ./configure --prefix=/scratch/director100/nicola/darshan-2.2.8 > then I'd like to profile LAMMPS IO with darshan. Here I use openmpi > 1.6.5, so I did > export > LD_PRELOAD=/scratch/director100/nicola/darshan-2.2.8/lib/libdarshan.so > > then I make lammps > and I tried to execute a simple example with > > mpirun -np 12 > /scratch/director100/nicola/lammps_pistolare/src/lmp_fornax < in.friction > > However no darshan.gz is generated. > Any idea why? > > Thanks, > > Nicola > > > _______________________________________________ > Darshan-users mailing list > Darshan-users(a)lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/darshan-users

1 0

darshan doesn't generate the login
by nicola varini 02 Apr '14

02 Apr '14

Hi, I am new to darshan. Sorry if my question is trivial but I've tried to search a bit through the mailing list archive and I did not find much help. I am try to understand how to use darshan and I've build the runtime with ./configure --with-mem-align=8 --with-log-path=/scratch/director100/nicola/darshan-2.2.8 CC=mpicc --with-jobid-env=PBS_JOBID --prefix=/scratch/director100/nicola/darshan-2.2.8 and the util with ./configure --prefix=/scratch/director100/nicola/darshan-2.2.8 then I'd like to profile LAMMPS IO with darshan. Here I use openmpi 1.6.5, so I did export LD_PRELOAD=/scratch/director100/nicola/darshan-2.2.8/lib/libdarshan.so then I make lammps and I tried to execute a simple example with mpirun -np 12 /scratch/director100/nicola/lammps_pistolare/src/lmp_fornax < in.friction However no darshan.gz is generated. Any idea why? Thanks, Nicola

1 0