December 2012 - Darshan-users - mailman3.cels.anl.gov

LD_PRELOAD fails with OpenMPI
by myriam.botalla＠bull.net 10 Jan '13

10 Jan '13

Hi, When I use LD_PRELOAD to get an application instrumented with Darshan at run time, a SEGMENTATION FAULT raises. Also, the behaviour is the same when simply running mpicc to get the version - and the generated coredump displays the same stack - which seems to point the MPI wrappers as being the potential suspect. # which mpicc /home_nfs/papaureg/workspace/openmpi-1.6.2/current_AE2__2/x86_64_bullxlinux6.1.1/bin/mpicc # mpicc --showme:version Erreur de segmentation (core dumped) # gdb mpicc core.14109 GNU gdb (GDB) bullx Linux (7.2-50.bl6.Bull.1.20120306) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later < http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. Reading symbols from /home_nfs/papaureg/workspace/openmpi-1.6.2/current_AE2__2/x86_64_bullxlinux6.1.1/bin/mpicc...done. [New Thread 14109] Missing separate debuginfo for /home_nfs/botallam/install/darshan.4/lib/libdarshan.so .... Reading symbols from /home_nfs/papaureg/workspace/openmpi-1.6.2//current_AE2__2/x86_64_bullxlinux6.1.1/lib/libmpi.so.1...done. Loaded symbols for /home_nfs/papaureg/workspace/openmpi-1.6.2//current_AE2__2/x86_64_bullxlinux6.1.1/lib/libmpi.so.1 Core was generated by `/home_nfs/papaureg/workspace/openmpi-1.6.2/current_AE2__2/x86_64_bullxlinux6.1.'. Program terminated with signal 11, Segmentation fault. #0 0x00000036b7879446 in calloc () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.47.bl6_2.9.x86_64 libgcc-4.4.5-6.bl6.x86_64 numactl-2.0.3-9.bl6.x86_64 zlib-1.2.3-25.bl6.x86_64 (gdb) where #0 0x00000036b7879446 in calloc () from /lib64/libc.so.6 #1 0x00000036b7c01310 in _dlerror_run () from /lib64/libdl.so.2 #2 0x00000036b7c0107a in dlsym () from /lib64/libdl.so.2 #3 0x00007f50f98c3487 in __xstat (vers=1, path=0x7f50f966ffac "/dev/ummunotify", buf=0x7fff6b6f08c0) at lib/darshan-posix.c:711 #4 0x00007f50f9661a64 in opal_memory_linux_malloc_init_hook () at hooks.c:756 #5 0x00000036b7875b63 in ptmalloc_init () from /lib64/libc.so.6 #6 0x00000036b7879987 in malloc_hook_ini () from /lib64/libc.so.6 #7 0x00000036b78a6da1 in __alloc_dir () from /lib64/libc.so.6 #8 0x00000036b94053cd in ?? () from /usr/lib64/libnuma.so.1 #9 0x00000036b740e515 in _dl_init_internal () from /lib64/ld-linux-x86-64.so.2 #10 0x00000036b7400b3a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2 #11 0x0000000000000002 in ?? () #12 0x00007fff6b6f1a43 in ?? () #13 0x00007fff6b6f1a9e in ?? () #14 0x0000000000000000 in ?? () (gdb) Can someone help me to understand the issue? Thanks, Myriam. HERE IS the environment: LD_PRELOAD=/home_nfs/botallam/install/darshan.4/lib/libdarshan.so The Darshan library was configured and generated with environment variable OMPI_CC=gcc using mpicc: Open MPI 1.6.2 (Language: C) # which mpicc /home_nfs/papaureg/workspace/openmpi-1.6.2/current_AE2__2/x86_64_bullxlinux6.1.1/bin/mpicc # CC=mpicc CFLAGS=-g ./configure --prefix=/home_nfs/botallam/install/darshan.4 --with-mem-align=16 --with-log-path-by-env=DARSHAN_LOGPATH --with-jobid-env=SLURM_JOBID # ldd /home_nfs/botallam/install/darshan.4/lib/libdarshan.so linux-vdso.so.1 => (0x00007fffe9e93000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f8a753bd000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8a751a0000) librt.so.1 => /lib64/librt.so.1 (0x00007f8a74f98000) libz.so.1 => /lib64/libz.so.1 (0x00007f8a74d83000) libmpi.so.1 => /home_nfs/papaureg/workspace/openmpi-1.6.2//current_AE2__2/x86_64_bullxlinux6.1.1/lib/libmpi.so.1 (0x00007f8a74967000) libm.so.6 => /lib64/libm.so.6 (0x00007f8a746e3000) libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x00007f8a744db000) libnsl.so.1 => /lib64/libnsl.so.1 (0x00007f8a742c1000) libutil.so.1 => /lib64/libutil.so.1 (0x00007f8a740be000) libc.so.6 => /lib64/libc.so.6 (0x00007f8a73d2e000) /lib64/ld-linux-x86-64.so.2 (0x00000036b7400000) libimf.so => /opt/intel/composer_xe_2013.1.117/compiler/lib/intel64/libimf.so (0x00007f8a73871000) libsvml.so => /opt/intel/composer_xe_2013.1.117/compiler/lib/intel64/libsvml.so (0x00007f8a72fa3000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f8a72d8d000) libintlc.so.5 => /opt/intel/composer_xe_2013.1.117/compiler/lib/intel64/libintlc.so.5 (0x00007f8a72b3e000) # # # nm /home_nfs/botallam/install/darshan.4/lib/libdarshan.so|grep mpi 0000000000227020 B __real_ncmpi_close 0000000000227010 B __real_ncmpi_create 0000000000227018 B __real_ncmpi_open 00000000000048cc T darshan_mpi_initialize 00000000000179d2 T ncmpi_close 000000000001763c T ncmpi_create 0000000000017807 T ncmpi_open U ompi_mpi_byte U ompi_mpi_char U ompi_mpi_comm_world U ompi_mpi_double U ompi_mpi_info_null U ompi_mpi_int U ompi_mpi_long U ompi_mpi_op_land U ompi_mpi_op_lor U ompi_mpi_op_max U ompi_mpi_op_null U ompi_mpi_op_sum 0000000000016444 T resolve_mpi_symbols #

3 5

Darshan 2.2.4 release
by Phil Carns 13 Dec '12

13 Dec '12

Darshan 2.2.4 is now available on the download page. This is primarily a bug fix release, but there is also a performance optimization for large-scale shared file access on Lustre file systems. If you are upgrading Darshan on a Cray platform, please re-patch your compiler scripts. If you are upgrading Darshan on a Blue Gene platform (P or Q), please check the BG/P recipe in the installation documentation and note that the ï¿½enable-stat-at-open option is now recommended on those platforms. Change log: darshan-2.2.4 ============= * Disable extra stat() of newly opened files by default. This improves performance on shared files for some platforms. Reported by Yushu Yao. - prevents Darshan from reporting SIZE_AT_OPEN, FILE_ALIGNENT, and DARSHAN_FILE_NOT_ALIGNED values - features can still be enabled via --enable-stat-at-open option * Fix missing -lz in post ld flags reported by Yushu Yao. Fixes a link-time error for some corner-case applications. * Fix bug in Cray compiler script patches that was setting compiler flags incorrectly. Reported by Yushu Yao. WARNING: if you are using a previous Darshan release (2.2.3 through 2.2.4-pre4) on a Cray platform, please re-patch your compiler scripts. * Update darshan-gen-* scripts to support the potential for additonal LDFLAGS or CFLAGS that appear before the object or source file in the compile or link commands. This fixes compatibility with some mvapich2 installations, reported by Dragos Constantin. thanks, -Darshan development team

1 0

Re: [Darshan-users] Instrumenting statically-linked applications
by Dragos Constantin 12 Dec '12

12 Dec '12

Hi Phil, Thank you for taking time to explain what is going on. Everything is clear now. I thought darshan uses the same procedures for static and dynamic libraries. So, I have introduced the LD_PRELOAD environment variable but I have issues when I launch the job. I have to work with TACC sysadmins because there is something in the launch script that hangs the launch procedure. I will also try to see if TACC sysadmins can generate static libs. I will keep you posted. Thanks, Dragos Dragos Constantin, PhD Research Associate Department of Radiology Stanford University Lucas MRS Center 1201 Welch Rd., PS-055 Stanford CA 94305 Office: (650) 736-9961 Fax: (650) 723-5795 On 11/29/2012 8:48 AM, Phil Carns wrote: > On 11/29/2012 11:35 AM, Dragos Constantin wrote: >> Hi Phil, >> I have to recognize I am a bit confused. Let me tell you what I was >> thinking. >> >> 1. The shared and static darshan libraries were generated from the >> same code, i.e. they contain the same functions. >> 2. The only difference between them is how these functions are called >> at run time. >> 3. Darshan I/O monitoring functions catch any I/O event generated by >> the application regardless the way the application is compiled. > > No. The darshan.so library can only capture I/O events that are > handled by shared libraries. The darshan.a library (in conjunction > with a modified mpicc script) can only capture I/O events that are > handled by static libraries. You can only use one or the other, not > both, unfortunately. > >> 4. In my mind both methods should generate the same output regardless >> the way the program is linked. >> >> Now, the Geant4 toolkit have static libraries, hence my application >> is going to be a mix of static and shared libraries. From what you >> are saying is that my application should use 100% static libraries or >> 100% shared libraries. In other words there is no solution for my case. > > No, not at all! Keep doing what you are doing. The solution for the > way that you are building right now is to use darshan with the > LD_PRELOAD approach that I described. Darshan does not mind if you > have a mix of static and shared libaries. It's oblivious to that. The > only two libraries it cares about (in general) are libc and libmpich. > As long as those two are dynamic, then you must preload the > libdarshan.so library to capture I/O events. How you handle the rest > of your libraries makes no difference at all. > >> >> How hard it is to modify darshan to work with mixed cases like mine? > > No modification is needed; see above. Darshan has no interest in > libraries that don't define the symbols that it intends to instrument. > >> >> I will also try to see if TACC sysadmins can generate static libraries. > > That's up to you. You can use LD_PRELOAD and Darshan with what you > have now. It basically breaks down like this: > > libc and libmpich are statically linked: use the modified mpicc script > to inject instrumentation at compile time > libc and libmpich are dynamically linked: use the LD_PRELOAD method to > inject instrumentation at run time > > Otherwise the rest is totally up to you as to whether you use a mix of > static and dynamic libraries or not. > > Does that help? > > I was confused initially because I generally assume that an > application will either be purely static or purely dynamic, but that > was just my assumption going in. It isn't a limitation of Darshan. > > thanks! > -Phil >> >> Thanks, >> Dragos >> >> >> Dragos Constantin, PhD >> >> Research Associate >> Department of Radiology >> Stanford University >> Lucas MRS Center >> 1201 Welch Rd., PS-055 >> Stanford CA 94305 >> >> Office: (650) 736-9961 >> Fax: (650) 723-5795 >> >> On 11/29/2012 6:45 AM, Phil Carns wrote: >>> On 11/29/2012 09:42 AM, Phil Carns wrote: >>>> On 11/28/2012 11:10 PM, Dragos Constantin wrote: >>>>> Hi Phil, >>>>> Here is the ldd output: >>>>> >>>> <snip> >>>>> libmpich.so.1.2 => >>>>> /opt/apps/gcc4_4/mvapich2/1.6/lib/libmpich.so.1.2 >>>>> (0x00002b574274a000) >>>> <snip> >>>> >>>>> libc.so.6 => /lib64/libc.so.6 (0x000000358d400000) >>>> >>>> Hi Dragos, sorry about the wild goose chase, but because these two >>>> libraries are shared, you will need to use the LD_PRELOAD method to >>>> capture your I/O calls. The mechanism that you have been using >>>> (with a modified mpicc script) only works when those libraries are >>>> statically linked. >>>> >>>>> >>>>> I believe LD_PRELOAD is used by the dynamic library loader (ld.so) >>>>> to find the darshan dynamic library 'libdarshan.so'. I might be >>>>> wrong so please do not hesitate to correct me. In any case, it >>>>> does not hurt to export this environment variable but if I am >>>>> right this will not change a thing as 'ld.so' does not touch >>>>> 'libdarshan.so'. >>>> >>>> The general idea with this approach is that you link your >>>> application normally at compile time and then set LD_PRELOAD to >>>> point to libdarshan.so at run time. When your application >>>> launches, ld.so will link in this LD_PRELOAD library before >>>> anything else, which gives Darshan a chance to intercept your I/O >>>> function symbols before they get to libc or libmpich. LD_PRELOAD >>>> doesn't just set a library path, it actually specifies a specific >>>> .so file (previously unknown to the application) that the loader >>>> will link in. >>>> >>>> So the steps that you need to try are as follows: >>>> >>>> 1) recompile and relink your application using the stock mpicc >>>> (with no darshan modifications). You should not see any darshan >>>> symbols in the resulting executable. >>>> 2) add a line like this near the job of your job script: >>>> >>>> #$ -v >>>> LD_PRELOAD=/full/path/to/your/darshan/installation/lib/libdarshan.so >>> >>> Just a quick note to anyone else that may run across this in the >>> list archives; the format of that line is specific to the scheduler >>> used on Ranger. In general you might just do "export >>> LD_PRELOAD=..." to set the environment variable or else use whatever >>> is appropriate for your scheduler or job launcher. >>> >>> -Phil >>> >>>> >>>> 3) run your application as usual >>>> >>>> I think that should work in your case. As Bill mentioned you do >>>> need to call MPI_Init() and MPI_Finalize(), but it sounds like you >>>> are already doing that (otherwise you would not get the empty log >>>> that you are seeing now), so you should be all set. >>>> >>>> Sorry about the confusion. It is pretty annoying that there isn't >>>> one method that works for both static and dynamic linking. You >>>> have to go one way or the other depending on how those libraries >>>> are used. >>>> >>>> -Phil >>>> >>>> _______________________________________________ >>>> Darshan-users mailing list >>>> Darshan-users(a)lists.mcs.anl.gov >>>> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users >>> >>> _______________________________________________ >>> Darshan-users mailing list >>> Darshan-users(a)lists.mcs.anl.gov >>> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users >>> >> > >

3 2