April 2013 - Darshan-users - mailman3.cels.anl.gov

cray perftools
by jgp 26 Apr '13

26 Apr '13

Hi, I would like to put darshan in production by having its modulefile loaded by default for all users. There may be an issue when a user wants to compile his code with another tool (CRAY perftools for instance) : FATAL: The program '/scratch/santis/piccinal/CRAY+orig' directly or indirectly uses the MPI profiling mechanism. Typically, you would avoid this by adding "conflict perftools" in the modulefile of darshan which will stop the loading of darshan if perftools is already loaded. But i am wondering if it would be possible to add a variable in the src code to activate/deactivate darshan on demand. Something like : While darshan is always loaded, if a user do : module load perftools # will set DARSHAN_STATUS=OFF and user can compile without darshan module rm perftools # will set DARSHAN_STATUS=ON and user can compile with darshan I'd like to know the pros/cons. Thanks for your support. jg.

2 1

Darshan 2.2.6
by Phil Carns 23 Apr '13

23 Apr '13

The official Darshan 2.2.6 release is now available on the Darshan web site (http://www.mcs.anl.gov/darshan) and the online documentation has been updated accordingly. This release includes performance enhancements as well as various minor bug fixes and compatibility improvements. darshan-2.2.6 ============= * Bug fixes: - make sure to honor user-specified hints passed in at runtime via the DARSHAN_LOGHINTS env variable. - include fread and fwrite in darshan-job-summary.pl read and write counts - fix segmentation fault on invalid arguments to darshan-parser * Performance enhancements: - collect mount point information at rank 0 and broadcast to all processes to avoid excess file system traffic on startup - change default MPI-IO hints for writing log file to romio_no_indep_rw=true and cb_nodes=4 to improve log creation performance * Install libdarshan-util and headers during installation process for darshan-util; patch contributed by Matthieu Dorier * Detect PMPI support at link time when using compiler scripts produced by the darshan-gen-* utilities. This avoids link problems when Darshan compiler wrappers are used with the ADIOS dummy MPI library; reported by Jingqing Mu * Support MPICH_{CC/CXX/F77} environment variables in compiler scripts produced by the darshan-gen-* utilities * Add instrumentation of startup time, enabled via DARSHAN_INTERNAL_TIMING environment variable * Rename cp-shutdown-bench test utility to darshan-shutdown-bench and enable benchmarking hooks in library by default so that darshan-shutdown-bench can be used with any Darshan installation * Remove deprecated --enable-st-dev-workaround configure option * Misc. documentation updates thanks, -Darshan development team

1 0

Darshan 2.2.6-pre2 available for testing
by Phil Carns 18 Apr '13

18 Apr '13

Hi everyone, We have just posted a new pre-release version of Darshan: ftp://ftp.mcs.anl.gov/pub/darshan/releases/darshan-2.2.6-pre2.tar.gz This should become an official 2.2.6 release in the next few days unless something unexpected comes up during testing. The change log for this release is listed below. thanks, -Darshan development team darshan-2.2.6-pre2 ============= * Bug fixes: - make sure to honor user-specified hints passed in at runtime via the DARSHAN_LOGHINTS env variable. - include fread and fwrite in darshan-job-summary.pl read and write counts - fix segmentation fault on invalid arguments to darshan-parser * Performance enhancements: - collect mount point information at rank 0 and broadcast to all processes to avoid excess file system traffic on startup - change default MPI-IO hints for writing log file to romio_no_indep_rw=true and cb_nodes=4 to improve log creation performance * Install libdarshan-util and headers during installation process for darshan-util; patch contributed by Matthieu Dorier * Detect PMPI support at link time when using compiler scripts produced by the darshan-gen-* utilities. This avoids link problems when Darshan compiler wrappers are used with the ADIOS dummy MPI library; reported by Jingqing Mu * Support MPICH_{CC/CXX/F77} environment variables in compiler scripts produced by the darshan-gen-* utilities * Add instrumentation of startup time, enabled via DARSHAN_INTERNAL_TIMING environment variable * Rename cp-shutdown-bench test utility to darshan-shutdown-bench and enable benchmarking hooks in library by default so that darshan-shutdown-bench can be used with any Darshan installation * Remove deprecated --enable-st-dev-workaround configure option * Misc. documentation updates

1 0

documentation for darshan-logutils
by Matthieu Dorier 16 Apr '13

16 Apr '13

Hi, I'd like to use the C API of Darshan, yet the only mention of this API is at the end of the darshan-util documentation (http://www.mcs.anl.gov/research/projects/darshan/docs/darshan-util.html) advising to take a look at darshan-logutils.h and I don't find anywhere the explanation on how to use these functions. I can guess how most of them work, but there are at least two that I don't understand: darshan_log_getmounts and darshan_log_setmounts. Could you give an example for them? Also a remark for later versions of Darshan (here I'm using 2.2.5): building a libdarshan-util.a library and installing it, along with the necessary include files, in the --prefix directory provided at configure time, would ease the use of this C API. I have myself modified the Makefile.in in darshan-util to have this done, I can share these modifications if you wish. Matthieu Dorier PhD student at ENS Cachan Brittany and IRISA http://people.irisa.fr/Matthieu.Dorier

2 2

A Ruby extension for Darshan
by Matthieu Dorier 15 Apr '13

15 Apr '13

Hi Darshan users/developers As I intended to do data-mining on a large number of Darshan-generated traces, I quickly realized that producing a text output using darshan-parser then parse this output using high-level languages (ruby or python), was extremely inefficient. I turned to the C API, but implementing data-mining algorithms in pure C is also very hard. Thus I developed a Ruby extension for Darshan. This extension is available here: http://darshan-ruby.gforge.inria.fr/ All functionalities currently available in Darshan 2.2.5 for reading binary (bz2-compressed) traces, are now available in Ruby, and can help analyzing traces more easily and faster. I hope this tool can be useful to others, feel free to give me some feedbacks :-) Phil: you'll find in this package the Makefile.in I was talking about in one of my previous mails. Best, Matthieu Dorier PhD student at ENS Cachan Brittany and IRISA http://people.irisa.fr/Matthieu.Dorier

2 2

Problem with darshan_log_getmounts and darshan_log_getexe
by Matthieu Dorier 14 Apr '13

14 Apr '13

Hi, I'm using the C API of Darshan and I have some problems with the two functions mentioned in the title of this mail. I'm testing them using 10024744407537054797.bz2 (in http://ftp.mcs.anl.gov/pub/darshan/data/intrepid/2012/9/1/) This trace is supposed to show 4 mount points, which I see when using the darshan-parse command line tool, yet darshan_log_getmounts gives me 0 mount points. The "exe" field is supposed to contain "3979759305", but when calling darshan_log_getexe I get the string "2.02". These functions do not return any error code. All other functions are working properly. I'm using version 2.2.5 of Darshan. Just to check, here is how I use these functions: int64_t** devs; char*** mnt_pts; char*** fs_types; int count; int err = darshan_log_getmounts(fd,devs,mnt_pts,fs_types,&count); // here c = 0 free(devs); free(mnt_pts); free(fs_types); char buffer[CP_EXE_LEN+2]; darshan_log_getexe(fd, buffer); Is there a bug somewhere or am I misusing these functions? Thanks! Matthieu Dorier PhD student at ENS Cachan Brittany and IRISA http://people.irisa.fr/Matthieu.Dorier

2 3

Alternatives to Darshan for non-MPI applications ?
by Maxime Boissonneault 08 Apr '13

08 Apr '13

Hi, We are interested in monitoring the IO on our cluster (Colosse @ Laval University, in Quebec). We have been made aware of Darshan last year and started to investigate further. I understand that Darshan is limited to MPI applications only. I suspect the reason for this limitation is the presence of hooks at the beginning and end of each MPI applications (MPI_Init/Finalize). Am I correct with that ? If that is the case, is anyone aware of a darshan-like utility that would allow monitoring of non-MPI applications for posix calls ? Thank you, -- --------------------------------- Maxime Boissonneault Analyste de calcul - Calcul Quï¿½bec, Universitï¿½ Laval Ph. D. en physique

2 2