October 2020 - Darshan-users - mailman3.cels.anl.gov

Darshan v3.2.1 hangs with mvapich2 2.3.3 (IOR)
by Cormac Garvey 27 Oct '20

27 Oct '20

Hi All, I am unable to get darshan v3.2.1 to work with IOR v3.2.1 built with mvapich2 v2.3.3. I am using CentOS 7.7 module load mpi/mvapich2 spack load darshan-runtime^mvapich2 export DARSHAN_LOG_DIR_PATH=/share/home/hpcuser/darshan_logs export LD_PRELOAD=${DARSHAN_RUNTIME_DIR}/lib/libdarshan.so TRANSFER_SIZE=32m SIZE=2G IO_API=POSIX IO_API_ARG="-F" TYPE_IO="direct_io" TYPE_IO_ARG="-B" MPI_OPTS=""-np 4 --hostfile $PBS_NODEFILE" mpirun -genvlist LD_PRELOAD,DARSHAN_LOG_DIR_PATH -bind-to hwthread $MPI_OPTS $IOR_BIN/ior -a $IO_API -v -i 1 $TYPE_IO_ARG -m -d 1 $IO_API_ARG -w -t $TRANSFER_SIZE -b $SIZE -o ${FILESYSTEM}/test -O summaryFormat=$SUMMARY_FORMAT -O summaryFile=ior_${IO_API}_${TYPE_IO}_${TRANSFER_SIZE}_${SIZE}_${HOST}_${NUMPROCS}.out_$$ The IOR job hangs at the end if I export the darshan LD_PRELOAD (Runs correctly if I remove the LD_PRELOAD) When I kill the job, i get the following information in PBS stderr file. "darshan_library_warning: unable to write header to file /share/home/hpcuser/darshan_logs/hpcuser_ior_id26_9-18-65156-2174583250033636718.darshan_partial." The IOR benchmark completes (i.e can see the I/O stats), but does not appear to exit correctly (remains in a hung state until I kill it) Running a similar job with IOR+mpich or IOR+OpenMPI works fine with darshan. Any ideas what I am missing? Thanks for your support. Regards, Cormac.

3 7

ioctl(LL_IOC_LOV_GETSTRIPE) crashes
by Ed Karrels 27 Oct '20

27 Oct '20

Hello Darshan team, I just wanted to add a data point to the ioctl(LL_IOC_LOV_GETSTRIPE) issue (which I see was discussed on this mailing list last October ("Help needed, Darshan keeps crashing in Lustre filesystem"), and in this issue: https://xgitlab.cels.anl.gov/darshan/darshan/-/issues/270. I was trying to use Darshan 3.2.1 on Frontera at TACC, and my test program kept locking up. After a few failures the system admin let me know my failures were crashing each node I tested on. This is one of error messages they found: [187544.836144] Lustre: test_darshan_lo: using old ioctl( LL_IOC_LOV_GETSTRIPE) on [0x200022388:0x1776:0x0], use llapi_layout_get_by_path()[187544.848624] usercopy: kernel memory exposure attempt detected from ffff8d46b11910c0 (kmalloc-64) (48032 bytes) I'll disable the Lustre module for now and keep an eye out for new releases.

2 2

profiling DASK application using darshan
by Razvan Stefanescu 21 Oct '20

21 Oct '20

Hello All, Following the documentation note saying that Darshan instrumentation of non-MPI applications is only possible with dynamically-linked applications, it seems I have to convert the multiprocessing and multithreading DASK code to a dynamically-linked code. I know some folks were able to profile Spark and TensorFlow codes with Darshan, so I wonder if you could provide some suggestions about creating such dynamically-linked code. Thank you, Razvan -- *RAZVAN STEFANESCU * Head of Statistics and Machine Learning Branch Senior Data Assimilation and Data Scientist *Spire Global, Inc.* 1050 Walnut Street, Suite 402, Boulder, CO 80302 USA +1-720- <7206432245>*643-2231* +1-850-443-1718 <4178491948>

3 5

DASK and darshan
by Razvan Stefanescu 14 Oct '20

14 Oct '20

Hello All, We need to profile some DASK parallel I/O application, and I wonder if you can provide an example on how to use darshan to perform such analysis. Your help is much appreciated. Thank you, Razvan -- *RAZVAN STEFANESCU * Head of Statistics and Machine Learning Branch Senior Data Assimilation and Data Scientist *Spire Global, Inc.* 1050 Walnut Street, Suite 402, Boulder, CO 80302 USA +1-720- <7206432245>*643-2231* +1-850-443-1718 <4178491948>

3 4

dxt_analyzer.py: 'invalid continuation byte' error
by Omlin Samuel 13 Oct '20

13 Oct '20

Hi all, I have tried the following command both on Ubuntu 16.04 (python 3.5) and on SUSE Linux Enterprise Server 15 SP1 (python 3.6) and got the same error: ``` $ python3 /usr/local/bin/dxt_analyzer.py -i omlins_a.out_id1174947_10-9-40448-1694713884344482404_1602235659.darshan Traceback (most recent call last): File "/usr/local/bin/dxt_analyzer.py", line 264, in <module> for line in infile: File "/usr/lib/python3.5/codecs.py", line 321, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe6 in position 24: invalid continuation byte ``` With python 2.7.12 (on Ubuntu), I did not get the error, but the generated plot was empty. Note that the input file does contain values generated by DXT: ``` $ darshan-dxt-parser omlins_a.out_id1174947_10-9-40448-1694713884344482404_1602235659.darshan | tail -3 X_POSIX 7 read 197 211527180650 1073741824 804.8381 806.1340 [ 0] X_POSIX 7 read 198 212600922683 1073741824 806.4404 807.7277 [ 0] X_POSIX 7 read 199 213674664716 1073741824 808.0719 809.3472 [ 0] ``` Can you fix this / help me to get this working? Thanks!! Sam

4 5