Hi,
When I use LD_PRELOAD to get an application instrumented with Darshan at
run time, a SEGMENTATION FAULT raises.
Also, the behaviour is the same when simply running mpicc to get the
version - and the generated coredump displays the same stack - which seems
to point the MPI wrappers as being the potential suspect.
# which mpicc
/home_nfs/papaureg/workspace/openmpi-1.6.2/current_AE2__2/x86_64_bullxlinux6.1.1/bin/mpicc
# mpicc --showme:version
Erreur de segmentation (core dumped)
# gdb mpicc core.14109
GNU gdb (GDB) bullx Linux (7.2-50.bl6.Bull.1.20120306)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <
http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
Reading symbols from
/home_nfs/papaureg/workspace/openmpi-1.6.2/current_AE2__2/x86_64_bullxlinux6.1.1/bin/mpicc...done.
[New Thread 14109]
Missing separate debuginfo for
/home_nfs/botallam/install/darshan.4/lib/libdarshan.so
....
Reading symbols from
/home_nfs/papaureg/workspace/openmpi-1.6.2//current_AE2__2/x86_64_bullxlinux6.1.1/lib/libmpi.so.1...done.
Loaded symbols for
/home_nfs/papaureg/workspace/openmpi-1.6.2//current_AE2__2/x86_64_bullxlinux6.1.1/lib/libmpi.so.1
Core was generated by
`/home_nfs/papaureg/workspace/openmpi-1.6.2/current_AE2__2/x86_64_bullxlinux6.1.'.
Program terminated with signal 11, Segmentation fault.
#0 0x00000036b7879446 in calloc () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install
glibc-2.12-1.47.bl6_2.9.x86_64 libgcc-4.4.5-6.bl6.x86_64
numactl-2.0.3-9.bl6.x86_64 zlib-1.2.3-25.bl6.x86_64
(gdb) where
#0 0x00000036b7879446 in calloc () from /lib64/libc.so.6
#1 0x00000036b7c01310 in _dlerror_run () from /lib64/libdl.so.2
#2 0x00000036b7c0107a in dlsym () from /lib64/libdl.so.2
#3 0x00007f50f98c3487 in __xstat (vers=1, path=0x7f50f966ffac
"/dev/ummunotify", buf=0x7fff6b6f08c0) at lib/darshan-posix.c:711
#4 0x00007f50f9661a64 in opal_memory_linux_malloc_init_hook () at
hooks.c:756
#5 0x00000036b7875b63 in ptmalloc_init () from /lib64/libc.so.6
#6 0x00000036b7879987 in malloc_hook_ini () from /lib64/libc.so.6
#7 0x00000036b78a6da1 in __alloc_dir () from /lib64/libc.so.6
#8 0x00000036b94053cd in ?? () from /usr/lib64/libnuma.so.1
#9 0x00000036b740e515 in _dl_init_internal () from
/lib64/ld-linux-x86-64.so.2
#10 0x00000036b7400b3a in _dl_start_user () from
/lib64/ld-linux-x86-64.so.2
#11 0x0000000000000002 in ?? ()
#12 0x00007fff6b6f1a43 in ?? ()
#13 0x00007fff6b6f1a9e in ?? ()
#14 0x0000000000000000 in ?? ()
(gdb)
Can someone help me to understand the issue?
Thanks,
Myriam.
HERE IS the environment:
LD_PRELOAD=/home_nfs/botallam/install/darshan.4/lib/libdarshan.so
The Darshan library was configured and generated with environment variable
OMPI_CC=gcc
using mpicc: Open MPI 1.6.2 (Language: C)
# which mpicc
/home_nfs/papaureg/workspace/openmpi-1.6.2/current_AE2__2/x86_64_bullxlinux6.1.1/bin/mpicc
# CC=mpicc CFLAGS=-g ./configure
--prefix=/home_nfs/botallam/install/darshan.4 --with-mem-align=16
--with-log-path-by-env=DARSHAN_LOGPATH --with-jobid-env=SLURM_JOBID
# ldd /home_nfs/botallam/install/darshan.4/lib/libdarshan.so
linux-vdso.so.1 => (0x00007fffe9e93000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f8a753bd000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8a751a0000)
librt.so.1 => /lib64/librt.so.1 (0x00007f8a74f98000)
libz.so.1 => /lib64/libz.so.1 (0x00007f8a74d83000)
libmpi.so.1 =>
/home_nfs/papaureg/workspace/openmpi-1.6.2//current_AE2__2/x86_64_bullxlinux6.1.1/lib/libmpi.so.1
(0x00007f8a74967000)
libm.so.6 => /lib64/libm.so.6 (0x00007f8a746e3000)
libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x00007f8a744db000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00007f8a742c1000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007f8a740be000)
libc.so.6 => /lib64/libc.so.6 (0x00007f8a73d2e000)
/lib64/ld-linux-x86-64.so.2 (0x00000036b7400000)
libimf.so =>
/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64/libimf.so
(0x00007f8a73871000)
libsvml.so =>
/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64/libsvml.so
(0x00007f8a72fa3000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f8a72d8d000)
libintlc.so.5 =>
/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64/libintlc.so.5
(0x00007f8a72b3e000)
#
#
# nm /home_nfs/botallam/install/darshan.4/lib/libdarshan.so|grep mpi
0000000000227020 B __real_ncmpi_close
0000000000227010 B __real_ncmpi_create
0000000000227018 B __real_ncmpi_open
00000000000048cc T darshan_mpi_initialize
00000000000179d2 T ncmpi_close
000000000001763c T ncmpi_create
0000000000017807 T ncmpi_open
U ompi_mpi_byte
U ompi_mpi_char
U ompi_mpi_comm_world
U ompi_mpi_double
U ompi_mpi_info_null
U ompi_mpi_int
U ompi_mpi_long
U ompi_mpi_op_land
U ompi_mpi_op_lor
U ompi_mpi_op_max
U ompi_mpi_op_null
U ompi_mpi_op_sum
0000000000016444 T resolve_mpi_symbols
#
