November 2012 - Darshan-users - mailman3.cels.anl.gov

Re: [Darshan-users] Instrumenting statically-linked applications
by Dragos Constantin 12 Dec '12

12 Dec '12

Hi Phil, Thank you for taking time to explain what is going on. Everything is clear now. I thought darshan uses the same procedures for static and dynamic libraries. So, I have introduced the LD_PRELOAD environment variable but I have issues when I launch the job. I have to work with TACC sysadmins because there is something in the launch script that hangs the launch procedure. I will also try to see if TACC sysadmins can generate static libs. I will keep you posted. Thanks, Dragos Dragos Constantin, PhD Research Associate Department of Radiology Stanford University Lucas MRS Center 1201 Welch Rd., PS-055 Stanford CA 94305 Office: (650) 736-9961 Fax: (650) 723-5795 On 11/29/2012 8:48 AM, Phil Carns wrote: > On 11/29/2012 11:35 AM, Dragos Constantin wrote: >> Hi Phil, >> I have to recognize I am a bit confused. Let me tell you what I was >> thinking. >> >> 1. The shared and static darshan libraries were generated from the >> same code, i.e. they contain the same functions. >> 2. The only difference between them is how these functions are called >> at run time. >> 3. Darshan I/O monitoring functions catch any I/O event generated by >> the application regardless the way the application is compiled. > > No. The darshan.so library can only capture I/O events that are > handled by shared libraries. The darshan.a library (in conjunction > with a modified mpicc script) can only capture I/O events that are > handled by static libraries. You can only use one or the other, not > both, unfortunately. > >> 4. In my mind both methods should generate the same output regardless >> the way the program is linked. >> >> Now, the Geant4 toolkit have static libraries, hence my application >> is going to be a mix of static and shared libraries. From what you >> are saying is that my application should use 100% static libraries or >> 100% shared libraries. In other words there is no solution for my case. > > No, not at all! Keep doing what you are doing. The solution for the > way that you are building right now is to use darshan with the > LD_PRELOAD approach that I described. Darshan does not mind if you > have a mix of static and shared libaries. It's oblivious to that. The > only two libraries it cares about (in general) are libc and libmpich. > As long as those two are dynamic, then you must preload the > libdarshan.so library to capture I/O events. How you handle the rest > of your libraries makes no difference at all. > >> >> How hard it is to modify darshan to work with mixed cases like mine? > > No modification is needed; see above. Darshan has no interest in > libraries that don't define the symbols that it intends to instrument. > >> >> I will also try to see if TACC sysadmins can generate static libraries. > > That's up to you. You can use LD_PRELOAD and Darshan with what you > have now. It basically breaks down like this: > > libc and libmpich are statically linked: use the modified mpicc script > to inject instrumentation at compile time > libc and libmpich are dynamically linked: use the LD_PRELOAD method to > inject instrumentation at run time > > Otherwise the rest is totally up to you as to whether you use a mix of > static and dynamic libraries or not. > > Does that help? > > I was confused initially because I generally assume that an > application will either be purely static or purely dynamic, but that > was just my assumption going in. It isn't a limitation of Darshan. > > thanks! > -Phil >> >> Thanks, >> Dragos >> >> >> Dragos Constantin, PhD >> >> Research Associate >> Department of Radiology >> Stanford University >> Lucas MRS Center >> 1201 Welch Rd., PS-055 >> Stanford CA 94305 >> >> Office: (650) 736-9961 >> Fax: (650) 723-5795 >> >> On 11/29/2012 6:45 AM, Phil Carns wrote: >>> On 11/29/2012 09:42 AM, Phil Carns wrote: >>>> On 11/28/2012 11:10 PM, Dragos Constantin wrote: >>>>> Hi Phil, >>>>> Here is the ldd output: >>>>> >>>> <snip> >>>>> libmpich.so.1.2 => >>>>> /opt/apps/gcc4_4/mvapich2/1.6/lib/libmpich.so.1.2 >>>>> (0x00002b574274a000) >>>> <snip> >>>> >>>>> libc.so.6 => /lib64/libc.so.6 (0x000000358d400000) >>>> >>>> Hi Dragos, sorry about the wild goose chase, but because these two >>>> libraries are shared, you will need to use the LD_PRELOAD method to >>>> capture your I/O calls. The mechanism that you have been using >>>> (with a modified mpicc script) only works when those libraries are >>>> statically linked. >>>> >>>>> >>>>> I believe LD_PRELOAD is used by the dynamic library loader (ld.so) >>>>> to find the darshan dynamic library 'libdarshan.so'. I might be >>>>> wrong so please do not hesitate to correct me. In any case, it >>>>> does not hurt to export this environment variable but if I am >>>>> right this will not change a thing as 'ld.so' does not touch >>>>> 'libdarshan.so'. >>>> >>>> The general idea with this approach is that you link your >>>> application normally at compile time and then set LD_PRELOAD to >>>> point to libdarshan.so at run time. When your application >>>> launches, ld.so will link in this LD_PRELOAD library before >>>> anything else, which gives Darshan a chance to intercept your I/O >>>> function symbols before they get to libc or libmpich. LD_PRELOAD >>>> doesn't just set a library path, it actually specifies a specific >>>> .so file (previously unknown to the application) that the loader >>>> will link in. >>>> >>>> So the steps that you need to try are as follows: >>>> >>>> 1) recompile and relink your application using the stock mpicc >>>> (with no darshan modifications). You should not see any darshan >>>> symbols in the resulting executable. >>>> 2) add a line like this near the job of your job script: >>>> >>>> #$ -v >>>> LD_PRELOAD=/full/path/to/your/darshan/installation/lib/libdarshan.so >>> >>> Just a quick note to anyone else that may run across this in the >>> list archives; the format of that line is specific to the scheduler >>> used on Ranger. In general you might just do "export >>> LD_PRELOAD=..." to set the environment variable or else use whatever >>> is appropriate for your scheduler or job launcher. >>> >>> -Phil >>> >>>> >>>> 3) run your application as usual >>>> >>>> I think that should work in your case. As Bill mentioned you do >>>> need to call MPI_Init() and MPI_Finalize(), but it sounds like you >>>> are already doing that (otherwise you would not get the empty log >>>> that you are seeing now), so you should be all set. >>>> >>>> Sorry about the confusion. It is pretty annoying that there isn't >>>> one method that works for both static and dynamic linking. You >>>> have to go one way or the other depending on how those libraries >>>> are used. >>>> >>>> -Phil >>>> >>>> _______________________________________________ >>>> Darshan-users mailing list >>>> Darshan-users(a)lists.mcs.anl.gov >>>> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users >>> >>> _______________________________________________ >>> Darshan-users mailing list >>> Darshan-users(a)lists.mcs.anl.gov >>> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users >>> >> > >

3 2

Instrumenting statically-linked applications
by Dragos Constantin 29 Nov '12

29 Nov '12

Hi, I've installed and configured darshan-2.2.3 on TACC Ranger in my user space. I have used gcc-4.4.5 (and mvapich-1.0.1). When I try to generate the MPI compiler scripts for statically-linked applications I get the following error: login4$ ./darshan-gen-cc.pl /opt/apps/gcc4_4/mvapich/1.0.1/bin/mpicc --output mpicc.darshan Error: cannot find matching CC from: gcc -c foo.c -I/opt/apps/gcc4_4/mvapich/1.0.1/include and: gcc -Wl,-rpath,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared -Wl,-rpath-link -Wl,/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared -L/opt/apps/gcc4_4/mvapich/1.0.1/lib/shared -L/opt/apps/gcc4_4/mvapich/1.0.1/lib foo.o -o foo -lmpich -L/opt/ofed//lib64/ -libverbs -libumad -lpthread -lpthread -lrt I am not quite sure what triggered this. Any ideas how to quickly fix the issue? I will look at the perl script to see what is going on there. Thanks, Dragos Dragos Constantin, PhD Research Associate Department of Radiology Stanford University Lucas MRS Center 1201 Welch Rd., PS-055 Stanford CA 94305 Office: (650) 736-9961 Fax: (650) 723-5795

4 14

broken link
by jgp 27 Nov '12

27 Nov '12

Hi, Can you update this page ? : http://www.mcs.anl.gov/research/projects/darshan/data/ It says : "The following wiki describes the data fields that are included in each Darshan log: http://wiki.mcs.anl.gov/Darshan/index.php/Guide_to_darshan-parser_output" but the wiki seems inactive now ("This page has been deleted"). The documentation is here : http://www.mcs.anl.gov/research/projects/darshan/docs/darshan-util.html#_gu… Thanks, JG.

2 1

Darshan pre-release 2.2.4-pre4
by Phil Carns 20 Nov '12

20 Nov '12

We've posted a pre-release snapshot for Darshan 2.2.4 on the ftp site: ftp://ftp.mcs.anl.gov/pub/darshan/releases/darshan-2.2.4-pre4.tar.gz The biggest change in this pre-release is that, by default, Darshan will no longer issue a stat() call on each newly opened file. We've found that this can cause significant overhead on some systems when opening shared files at large scale. We'll be doing more follow up testing before the final 2.2.4 release, but for now our suggestion is to only use the --enable-stat-at-open option (to enable the old behavior) on Blue Gene systems for now. Changelog: darshan-2.2.4-pre4 ============= * Disable extra stat() of newly opened files by default in order to improve performance on shared files. Reported by Yushu Yao. - prevents Darshan from reporting SIZE_AT_OPEN, FILE_ALIGNENT, and DARSHAN_FILE_NOT_ALIGNED values - features can still be enabled via --enable-stat-at-open option * Fix missing -lz in post ld flags reported by Yushu Yao. Fixes a link-time error for some corner-case applications. Thanks, -Darshan development team

2 2

Darshan 2.2.3 release announcement
by Phil Carns 20 Nov '12

20 Nov '12

Darshan version 2.2.3 is now available on the download page. This release features official support for the Cray XE6 and IBM BG/Q platforms. Special thanks to Yushu Yao and Katie Antypas of NERSC for their contributions in Cray integration and testing. Change log: darshan-2.2.3 ============= * improved Cray XE6 support - support for GNU, PGI, Cray, Pathscale, and Intel compilers - patch adding Darshan capability to system compiler scripts - software module, including testing and features contributed by Yushu Yao and Katie Antypas of The National Energy Research Scientific Computing Center (NERSC) - improved documentation * minor bug fixes: - properly detect cxx library name when generating BG/Q compiler wrappers - improve timing mechanism for cp-shutdown-bench benchmark - improve hashing to avoid log file name collisions thanks, - Darshan development team

2 2

is darshan going to work with serial applications?
by Dragos Constantin 08 Nov '12

08 Nov '12

Dear Colleagues, I try to benchmark the I/O load of a serial application based on Geant4. Someone suggested I use darshan but I am not so convinced it is going to work. In order to reach statistical significance and to perform a 4D parameter sweep I have divided one huge task into tens of thousands of small tasks so each core of the cluster is solving a part of the problem. There is no message passing between tasks but I believe I can compile my application with MPI if that is needed by darshan to centralize data collection. So, is it possible to use darshan with a serial application? Thank you in advance for your help. Dragos Constantin, PhD Research Associate Department of Radiology Stanford University Lucas MRS Center 1201 Welch Rd., PS-055 Stanford CA 94305 Office: (650) 736-9961 Fax: (650) 723-5795

2 1