June 2021 - Darshan-users - mailman3.cels.anl.gov

Darshan 3.3.1 point release now available
by Snyder, Shane 30 Jun '21

30 Jun '21

Darshan users, A new 3.3.1 point release of Darshan is now available for download HERE<ftp://ftp.mcs.anl.gov/pub/darshan/releases/darshan-3.3.1.tar.gz>. This release contains an important bug fix that resolves dynamic linker errors when using Darshan's Lustre instrumentation module in conjunction with Darshan's LD_PRELOAD mechanism for instrumenting applications. Additionally, this release is the first Darshan release that supports instrumenting non-MPI applications that call fork(), with log files now accurately generated for both the parent process and the child process. Please report any issues, comments, or questions to us using this mailing list or our GitHub page<https://github.com/darshan-hpc/darshan>. Thanks! --Darshan team

1 0

Re: [Darshan-users] Logs not generated inspite throwing warning message
by Phil Carns 21 Jun '21

21 Jun '21

Ah, Ok. This is the problem then. The log directory has to be on a shared file system (e.g. nfs, lustre, etc.) that is visible to all nodes. The log is created and written using collective I/O operations, so all of the ranks need to be able to access the same file simultaneously. I'll open a ticket to investigate if there is a way to produce a more helpful error message in this case. thanks, -Phil On 6/21/21 5:50 PM, Hassan Asghar wrote: > it is not shared directory, but i have install the same version of > darshan on the other nodes as well. > > > -- > Regards, > *Hassan Asghar* > MS Computer Science > Mobile: +92-321-1741755 > Personal Website: http://bit.ly/2OQ0egs <http://bit.ly/2OQ0egs> > Google Scholar: https://bit.ly/3lVYuxF <https://bit.ly/3lVYuxF> > > > On Mon, Jun 21, 2021 at 10:30 PM Phil Carns <carns(a)mcs.anl.gov > <mailto:[email protected]>> wrote: > > Hi Hassan, > > Is /darshan-logs/ a shared directory? > > thanks, > > -Phil > > On 6/21/21 1:04 PM, Hassan Asghar wrote: >> Yes, I have set the log path and all permission are given when i >> run program on local machine it execute successfully but when i >> exceute on three nodes it show error. >> >> On Tue, 22 Jun 2021, 1:18 am Harms, Kevin, <harms(a)alcf.anl.gov >> <mailto:[email protected]>> wrote: >> >> Hassan, >> >> did you create the /darshan-logs directory? Looks like it >> might not be writeable. >> >> try: /darshan-logs/2021/6/21 >> >> if you want to try a quick test, set DARSHAN_LOGFILE to the >> path/name of the log, like: >> >> DARSHAN_LOGFILE=/path/I/can/write/to/logfile.darshan >> >> kevin >> >> ________________________________________ >> From: Darshan-users <darshan-users-bounces(a)lists.mcs.anl.gov >> <mailto:[email protected]>> on behalf >> of Hassan Asghar <haxxanasghar(a)gmail.com >> <mailto:[email protected]>> >> Sent: Monday, June 21, 2021 12:35 AM >> To: darshan-users(a)lists.mcs.anl.gov >> <mailto:[email protected]> >> Subject: [Darshan-users] Logs not generated inspite throwing >> warning message >> >> I am running darshan with mpich on a three-node cluster and >> my job executed successfully but not generated logs. error >> warning is given below at the end of the execution. >> >> [haxxanasghar@gpuserver2 ~]$ mpiexec -n 64 -f machinefile >> -env LD_PRELOAD /usr/l >> ocal/lib/libdarshan.so ./MADbench2 640 80 4 8 8 4 4 >> >> MADbench 2.0 IO-mode >> no_pe = 64 no_pix = 640 no_bin = 80 no_gang = 4 sblocksize >> = 8 fblocksize = 8 r_mod = 4 w_mod = 4 >> IOMETHOD = POSIX IOMODE = SYNC FILETYPE = UNIQUE REMAP = CUSTOM >> >> S_cc 0.02 [ 0.02: 0.03] >> S_bw 0.01 [ 0.01: 0.01] >> S_w 0.03 [ 0.03: 0.04] >> ------- >> S_total 0.07 [ 0.07: 0.07] >> >> W_cc 0.04 [ 0.03: 0.05] >> W_bw 0.22 [ 0.20: 0.23] >> W_r 0.02 [ 0.02: 0.02] >> W_w 0.01 [ 0.01: 0.01] >> ------- >> W_total 0.28 [ 0.28: 0.29] >> >> C_cc 0.01 [ 0.01: 0.01] >> C_bw 0.00 [ 0.00: 0.00] >> C_r 0.01 [ 0.01: 0.01] >> ------- >> C_total 0.02 [ 0.02: 0.02] >> >> >> dC[0] = 0.00000e+00 >> darshan_library_warning: unable to create log file >> /darshan-logs/2021/6/21/haxxa >> nas_MADbench2_id29484-29484_6-21-52165-1341319014666854881.darshan_partial. >> [haxxanasghar@gpuserver2 ~]$ >> >> >> -- >> Regards, >> >> Hassan Asghar >> MS Computer Science >> Mobile: +92-321-1741755 >> Personal Website: http://bit.ly/2OQ0egs <http://bit.ly/2OQ0egs> >> Google Scholar: https://bit.ly/3lVYuxF <https://bit.ly/3lVYuxF> >> >> >> _______________________________________________ >> Darshan-users mailing list >> Darshan-users(a)lists.mcs.anl.gov <mailto:[email protected]> >> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users <https://lists.mcs.anl.gov/mailman/listinfo/darshan-users> > _______________________________________________ > Darshan-users mailing list > Darshan-users(a)lists.mcs.anl.gov > <mailto:[email protected]> > https://lists.mcs.anl.gov/mailman/listinfo/darshan-users > <https://lists.mcs.anl.gov/mailman/listinfo/darshan-users> >

2 1

Logs not generated inspite throwing warning message
by Hassan Asghar 21 Jun '21

21 Jun '21

I am running darshan with mpich on a three-node cluster and my job executed successfully but not generated logs. error warning is given below at the end of the execution. [haxxanasghar@gpuserver2 ~]$ *mpiexec -n 64 -f machinefile -env LD_PRELOAD /usr/l ocal/lib/libdarshan.so ./MADbench2 640 80 4 8 8 4 4* MADbench 2.0 IO-mode no_pe = 64 no_pix = 640 no_bin = 80 no_gang = 4 sblocksize = 8 fblocksize = 8 r_mod = 4 w_mod = 4 IOMETHOD = POSIX IOMODE = SYNC FILETYPE = UNIQUE REMAP = CUSTOM S_cc 0.02 [ 0.02: 0.03] S_bw 0.01 [ 0.01: 0.01] S_w 0.03 [ 0.03: 0.04] ------- S_total 0.07 [ 0.07: 0.07] W_cc 0.04 [ 0.03: 0.05] W_bw 0.22 [ 0.20: 0.23] W_r 0.02 [ 0.02: 0.02] W_w 0.01 [ 0.01: 0.01] ------- W_total 0.28 [ 0.28: 0.29] C_cc 0.01 [ 0.01: 0.01] C_bw 0.00 [ 0.00: 0.00] C_r 0.01 [ 0.01: 0.01] ------- C_total 0.02 [ 0.02: 0.02] dC[0] = 0.00000e+00 darshan_library_warning: unable to create log file /darshan-logs/2021/6/21/haxxa nas_MADbench2_id29484-29484_6-21-52165-1341319014666854881.darshan_partial. [haxxanasghar@gpuserver2 ~]$ -- Regards, *Hassan Asghar* MS Computer Science Mobile: +92-321-1741755 Personal Website: http://bit.ly/2OQ0egs Google Scholar: https://bit.ly/3lVYuxF

3 3

Using darshan to instrument PyTorch
by Lu Weizheng 18 Jun '21

18 Jun '21

Hi, I am using darshan to instrument PyTorch on a local machine. My workload is an image classification problem on ImageNet dataset. When the training process ended, there are a lot of logs generated. Like: u2020000_python_id4719_6-15-41351-17690910011763757569_1.darshan u2020000_python_id5012_6-15-42860-17690910011763757569_1.darshan u2020000_python_id4721_6-15-41352-17690910011763757569_1.darshan u2020000_uname_id4720_6-15-41351-17690910011763757569_1.darshan u2020000_python_id4722_6-15-41352-17690910011763757569_1.darshan u2020000_uname_id4723_6-15-41354-17690910011763757569_1.darshan u2020000_python_id4758_6-15-41830-17690910011763757569_1.darshan u2020000_uname_id4724_6-15-41354-17690910011763757569_1.darshan ... After using the darshan-util analysis tool for one of the above log file, it shows: I/O performance estimate (at the POSIX layer): transferred 7.5 MiB at 36.02 MiB/s The transferred data showed in the PDF report is far less than the whole dataset size.As PyTorch DataLoader is a multi-process program, I guess darshan generate every log for every process. My question is: how can I get the IO analysis for the whole PyTorch workload task instead of these process logs?

2 4

COMM Split Error
by Hassan Asghar 17 Jun '21

17 Jun '21

I am facing the following issue: Please help [haxxanasghar@gpuserver2 ~]$ *mpiexec -n 16 -f machinefile -env LD_PRELOAD=/home/haxxanasghar/darshan/darshan-runtime/lib/libdarshan.so ./MADbench2 640 80 4 8 8 4 4* MADbench 2.0 IO-mode no_pe = 16 no_pix = 640 no_bin = 80 no_gang = 4 sblocksize = 8 fblocksize = 8 r_mod = 4 w_mod = 4 IOMETHOD = POSIX IOMODE = SYNC FILETYPE = UNIQUE REMAP = CUSTOM Fatal error in PMPI_Comm_split: A process has failed, error stack: PMPI_Comm_split(474)......: MPI_Comm_split(MPI_COMM_WORLD, color=0, key=2, new_comm=0x6073e0) failed PMPI_Comm_split(456)......: MPIR_Comm_split_impl(143).: MPIR_Allgather_impl(807)..: MPIR_Allgather(766).......: MPIR_Allgather_intra(181).: dequeue_and_set_error(888): Communication error with rank 3 MPIR_Allgather_intra(181).: dequeue_and_set_error(888): Communication error with rank 6 Fatal error in PMPI_Comm_split: A process has failed, error stack: PMPI_Comm_split(474)......: MPI_Comm_split(MPI_COMM_WORLD, color=3, key=1, new_comm=0x6073e0) failed PMPI_Comm_split(456)......: MPIR_Comm_split_impl(143).: MPIR_Allgather_impl(807)..: MPIR_Allgather(766).......: MPIR_Allgather_intra(181).: dequeue_and_set_error(888): Communication error with rank 12 MPIR_Allgather_intra(181).: dequeue_and_set_error(888): Communication error with rank 15 MPIR_Allgather_intra(181).: dequeue_and_set_error(888): Communication error with rank 9 =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [mpiexec@gpuserver2] HYDU_sock_read (./utils/sock/sock.c:243): read error (Bad file descriptor) [mpiexec@gpuserver2] control_cb (./pm/pmiserv/pmiserv_cb.c:201): unable to read command from proxy [mpiexec@gpuserver2] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status [mpiexec@gpuserver2] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:197): error waiting for event [mpiexec@gpuserver2] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion [haxxanasghar@gpuserver2 ~]$

2 2

Error: unable to inflate darshan log data.
by Tin mwo1 15 Jun '21

15 Jun '21

Hello all, I've compiled darhan-3.3.0 on RHEL 7.8 and was able to follow along with the demo at https://github.com/kchasapis/esiwace_demo_darshan for dir1 up to the point where I able to create a darharn trace file: root_ssf_exe_id55933_6-15-1667-14933317035636391624_1.darshan with a size of 2.3k when I try running the darshan-job-summary.pl I get the following error: ~/darshan-3.3.0/darshan-util/bin/darshan-job-summary.pl root_ssf_exe_id55933_6-15-1667-14933317035636391624_1.darshan Error: unable to inflate darshan log data. Error: failed to read name hash from darshan log file. darshan-parser failure: 65280 at /root/darshan-3.3.0/darshan-util/bin/ darshan-job-summary.pl line 358. After looking at the perl script I proceed to run just the darshan-parser and I'm getting the same error below. ~/darshan-3.3.0/darshan-util/bin/darshan-parser root_ssf_exe_id55933_6-15-1667-14933317035636391624_1.darshan Error: unable to inflate darshan log data. Error: failed to read name hash from darshan log file. Here's the output of the ./configure: ./configure CC=mpicc --prefix=`pwd` checking for a BSD-compatible install... /usr/bin/install -c checking for gcc... mpicc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether mpicc accepts -g... yes checking for mpicc option to accept ISO C89... none needed checking how to run the C preprocessor... mpicc -E checking for grep that handles long lines and -e... /usr/bin/grep checking for egrep... /usr/bin/grep -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking zlib.h usability... yes checking zlib.h presence... yes checking for zlib.h... yes checking for inflateEnd in -lz... yes checking if bzlib is wanted... yes checking for BZ2_bzCompressInit in -lbz2... yes checking bzlib.h usability... yes checking bzlib.h presence... yes checking for bzlib.h... yes checking for BZ2_bzCompressInit in -lbz2... (cached) yes checking bzlib in /usr... ok checking for inttypes.h... (cached) yes checking whether the inttypes.h PRIxNN macros are broken... no checking for inttypes.h... (cached) yes checking for pdflatex... yes checking for -halt-on-error argument to pdflatex... yes checking whether byte ordering is bigendian... no checking for off64_t... yes checking for python... /usr/bin/python checking for python version... 2.7 checking for python platform... linux2 checking for python script directory... ${prefix}/lib/python2.7/site-packages checking for python extension module directory... ${exec_prefix}/lib64/python2.7/site-packages checking for strndup... yes configure: creating ./config.status config.status: creating Makefile config.status: creating darshan-job-summary/bin/darshan-job-summary.pl config.status: creating maint/darshan-util.pc config.status: creating darshan-util-config.h config.status: darshan-util-config.h is unchanged and here is the output of the make: make -j 32 && make install mpicc -I . -I .. -I . -I ./../ -DDARSHAN_CONFIG_H=\"darshan-util-config.h\" -g -O2 -I/usr/include -c darshan-logutils.c -o darshan-logutils.o mpicc -I . -I .. -I . -I ./../ -DDARSHAN_CONFIG_H=\"darshan-util-config.h\" -g -O2 -I/usr/include -c darshan-posix-logutils.c -o darshan-posix-logutils.o mpicc -I . -I .. -I . -I ./../ -DDARSHAN_CONFIG_H=\"darshan-util-config.h\" -g -O2 -I/usr/include -c darshan-mpiio-logutils.c -o darshan-mpiio-logutils.o mpicc -I . -I .. -I . -I ./../ -DDARSHAN_CONFIG_H=\"darshan-util-config.h\" -g -O2 -I/usr/include -c darshan-hdf5-logutils.c -o darshan-hdf5-logutils.o mpicc -I . -I .. -I . -I ./../ -DDARSHAN_CONFIG_H=\"darshan-util-config.h\" -g -O2 -I/usr/include -c darshan-pnetcdf-logutils.c -o darshan-pnetcdf-logutils.o mpicc -I . -I .. -I . -I ./../ -DDARSHAN_CONFIG_H=\"darshan-util-config.h\" -g -O2 -I/usr/include -c darshan-bgq-logutils.c -o darshan-bgq-logutils.o mpicc -I . -I .. -I . -I ./../ -DDARSHAN_CONFIG_H=\"darshan-util-config.h\" -g -O2 -I/usr/include -c darshan-lustre-logutils.c -o darshan-lustre-logutils.o mpicc -I . -I .. -I . -I ./../ -DDARSHAN_CONFIG_H=\"darshan-util-config.h\" -g -O2 -I/usr/include -c darshan-stdio-logutils.c -o darshan-stdio-logutils.o mpicc -I . -I .. -I . -I ./../ -DDARSHAN_CONFIG_H=\"darshan-util-config.h\" -g -O2 -I/usr/include -c darshan-dxt-logutils.c -o darshan-dxt-logutils.o mpicc -I . -I .. -I . -I ./../ -DDARSHAN_CONFIG_H=\"darshan-util-config.h\" -g -O2 -I/usr/include -c darshan-mdhim-logutils.c -o darshan-mdhim-logutils.o mpicc -I . -I .. -I . -I ./../ -DDARSHAN_CONFIG_H=\"darshan-util-config.h\" -g -O2 -I/usr/include -c darshan-null-logutils.c -o darshan-null-logutils.o mpicc -I . -I .. -I . -I ./../ -DDARSHAN_CONFIG_H=\"darshan-util-config.h\" -g -O2 -I/usr/include -c lookup3.c -o lookup3.o mpicc -I . -I .. -I . -I ./../ -DDARSHAN_CONFIG_H=\"darshan-util-config.h\" -g -O2 -I/usr/include -L/usr/lib jenkins-hash-gen.c lookup3.o -o jenkins-hash-gen -lz -lbz2 ar rcs libdarshan-util.a darshan-logutils.o darshan-posix-logutils.o darshan-mpiio-logutils.o darshan-hdf5-logutils.o darshan-pnetcdf-logutils.o darshan-bgq-logutils.o darshan-lustre-logutils.o darshan-stdio-logutils.o darshan-dxt-logutils.o darshan-mdhim-logutils.o mpicc -I . -I .. -I . -I ./../ -DDARSHAN_CONFIG_H=\"darshan-util-config.h\" -g -O2 -I/usr/include -L/usr/lib darshan-analyzer.c libdarshan-util.a -o darshan-analyzer -lz -lbz2 mpicc -I . -I .. -I . -I ./../ -DDARSHAN_CONFIG_H=\"darshan-util-config.h\" -g -O2 -I/usr/include -L/usr/lib darshan-convert.c lookup3.o libdarshan-util.a -o darshan-convert -lz -lbz2 mpicc -I . -I .. -I . -I ./../ -DDARSHAN_CONFIG_H=\"darshan-util-config.h\" -g -O2 -I/usr/include -L/usr/lib darshan-diff.c libdarshan-util.a -o darshan-diff -lz -lbz2 mpicc -I . -I .. -I . -I ./../ -DDARSHAN_CONFIG_H=\"darshan-util-config.h\" -g -O2 -I/usr/include -L/usr/lib darshan-parser.c libdarshan-util.a -o darshan-parser -lz -lbz2 mpicc -I . -I .. -I . -I ./../ -DDARSHAN_CONFIG_H=\"darshan-util-config.h\" -g -O2 -I/usr/include -L/usr/lib darshan-dxt-parser.c libdarshan-util.a -o darshan-dxt-parser -lz -lbz2 mpicc -I . -I .. -I . -I ./../ -DDARSHAN_CONFIG_H=\"darshan-util-config.h\" -g -O2 -I/usr/include -L/usr/lib darshan-merge.c libdarshan-util.a -o darshan-merge -lz -lbz2 install -d /root/darshan-3.3.0/darshan-util/bin install -d /root/darshan-3.3.0/darshan-util/lib install -d /root/darshan-3.3.0/darshan-util/include install -m 755 darshan-analyzer /root/darshan-3.3.0/darshan-util/bin install -m 755 darshan-convert /root/darshan-3.3.0/darshan-util/bin install -m 755 darshan-diff /root/darshan-3.3.0/darshan-util/bin install -m 755 darshan-parser /root/darshan-3.3.0/darshan-util/bin install -m 755 darshan-dxt-parser /root/darshan-3.3.0/darshan-util/bin install -m 755 darshan-merge /root/darshan-3.3.0/darshan-util/bin install -m 755 ./darshan-summary-per-file.sh /root/darshan-3.3.0/darshan-util/bin install -m 755 ./dxt_analyzer.py /root/darshan-3.3.0/darshan-util/bin install -m 755 libdarshan-util.a /root/darshan-3.3.0/darshan-util/lib install -m 644 ./darshan-logutils.h /root/darshan-3.3.0/darshan-util/include install -m 644 ./darshan-posix-logutils.h /root/darshan-3.3.0/darshan-util/include install -m 644 ./darshan-mpiio-logutils.h /root/darshan-3.3.0/darshan-util/include install -m 644 ./darshan-hdf5-logutils.h /root/darshan-3.3.0/darshan-util/include install -m 644 ./darshan-pnetcdf-logutils.h /root/darshan-3.3.0/darshan-util/include install -m 644 ./darshan-bgq-logutils.h /root/darshan-3.3.0/darshan-util/include install -m 644 ./darshan-null-logutils.h /root/darshan-3.3.0/darshan-util/include install -m 644 ./darshan-lustre-logutils.h /root/darshan-3.3.0/darshan-util/include install -m 644 ./darshan-stdio-logutils.h /root/darshan-3.3.0/darshan-util/include install -m 644 ./darshan-dxt-logutils.h /root/darshan-3.3.0/darshan-util/include install -m 644 ./darshan-mdhim-logutils.h /root/darshan-3.3.0/darshan-util/include install -m 644 ./../darshan-null-log-format.h /root/darshan-3.3.0/darshan-util/include install -m 644 ./../darshan-posix-log-format.h /root/darshan-3.3.0/darshan-util/include install -m 644 ./../darshan-mpiio-log-format.h /root/darshan-3.3.0/darshan-util/include install -m 644 ./../darshan-hdf5-log-format.h /root/darshan-3.3.0/darshan-util/include install -m 644 ./../darshan-pnetcdf-log-format.h /root/darshan-3.3.0/darshan-util/include install -m 644 ./../darshan-bgq-log-format.h /root/darshan-3.3.0/darshan-util/include install -m 644 ./../darshan-lustre-log-format.h /root/darshan-3.3.0/darshan-util/include install -m 644 ./../darshan-stdio-log-format.h /root/darshan-3.3.0/darshan-util/include install -m 644 ./../darshan-dxt-log-format.h /root/darshan-3.3.0/darshan-util/include install -m 644 ./../darshan-mdhim-log-format.h /root/darshan-3.3.0/darshan-util/include install -d /root/darshan-3.3.0/darshan-util/include/uthash-1.9.2 install -d /root/darshan-3.3.0/darshan-util/include/uthash-1.9.2/src install -m 644 uthash-1.9.2/src/uthash.h /root/darshan-3.3.0/darshan-util/include/uthash-1.9.2/src/ install -m 644 ./../darshan-log-format.h /root/darshan-3.3.0/darshan-util/include install -m 755 darshan-job-summary/bin/darshan-job-summary.pl /root/darshan-3.3.0/darshan-util/bin install -d /root/darshan-3.3.0/darshan-util/lib/TeX install -m 644 ./darshan-job-summary/lib/TeX/Encode.pm /root/darshan-3.3.0/darshan-util/lib/TeX/ install -d /root/darshan-3.3.0/darshan-util/lib/Number install -d /root/darshan-3.3.0/darshan-util/lib/Number/Bytes install -m 644 ./darshan-job-summary/lib/Number/Bytes/Human.pm /root/darshan-3.3.0/darshan-util/lib/Number/Bytes install -d /root/darshan-3.3.0/darshan-util/lib/Pod install -m 644 ./darshan-job-summary/lib/Pod/Constants.pm /root/darshan-3.3.0/darshan-util/lib/Pod/ install -m 644 ./darshan-job-summary/lib/Pod/LaTeX.pm /root/darshan-3.3.0/darshan-util/lib/Pod/ install -d /root/darshan-3.3.0/darshan-util/share install -m 644 ./darshan-job-summary/share/* /root/darshan-3.3.0/darshan-util/share install -d /root/darshan-3.3.0/darshan-util/lib/pkgconfig install -m 644 maint/darshan-util.pc /root/darshan-3.3.0/darshan-util/lib/pkgconfig/darshan-util.pc Is there something I'm missing or not including when I'm building the tool? Thanks! Tin

2 1

Problem with darshan-job-summary.pl in 3.3.0
by Jeffrey Layton 11 Jun '21

11 Jun '21

Good afternoon, I just built version 3.3.0 and ran a simple C code that writes some text to stdout, When I try to create a summary pdf file it hangs. I built darhan-3.3.0 using gcc on Ubuntu 20.04. The command I used to create the report is, $ /home/laytonjb/bin/darshan-3.3.0/bin/darshan-job-summary.pl laytonjb_test1_id28730_6-7-43012-2131301613401632697_1.darshan --output jeff --verbose verbose: /tmp/49OfEXSumv Use of uninitialized value $summary{"POSIX_MMAPS"} in numeric lt (<) at /home/laytonjb/bin/darshan-3.3.0/bin/darshan-job-summary.pl line 435. Warning: empty y range [0:0], adjusting to [-1:1] Warning: empty y range [0:0], adjusting to [-1:1] At this point it hangs. I took a peak at the directory in /tmp and found the following: $ ls -s total 320 4 access-hist-eps.gplt 4 file-access-read-sh.dat 8 file-access-write.pdf 24 op-counts.eps 8 pattern.pdf 0 summary.log 4 title.tex 4 access-table.tex 24 file-access-shared.eps 4 file-access-write-sh.dat 4 op-counts-eps.gplt 4 posix-access-hist.dat 4 summary.tex 4 variance-table.tex 4 file-access-eps.gplt 8 file-access-shared.pdf 4 file-count-table.tex 8 op-counts.pdf 28 posix-access-hist.eps 4 time-summary.dat 4 file-access-read.dat 4 file-access-table.tex 4 fs-data-table.tex 4 pattern.dat 8 posix-access-hist.pdf 24 time-summary.eps 24 file-access-read.eps 4 file-access-write.dat 4 job-table.tex 24 pattern.eps 4 posix-op-counts.dat 4 time-summary-eps.gplt 8 file-access-read.pdf 24 file-access-write.eps 4 latex.output 4 pattern-eps.gplt 4 stdio-op-counts.dat 8 time-summary.pdf There are some pdf files and I can read those, but I was hoping for a single pdf file that contained all of the plots like that in the documentation for darshan-utils. Any thoughts on what I'm doing wrong? Thanks! Jeff <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

2 2

*****SPAM*****darshan-job-summary.pl (ERROR)
by Hassan Asghar 10 Jun '21

10 Jun '21

Hi, I am generating the analysis report by using the command *darshan-job-summary.pl <http://darshan-job-summary.pl> /home/haxxanasghar/darshan-logs/2020/6/10/haxxanas_MADbench2_id50362-50362_6-10-57172-6816275868204404779_1.darshan* I have installed all the required things like perl, epstopdf, latex, python and gnuplot: but again facing the following error: *Base class package "Pod::Select" is empty. (Perhaps you need to 'use' the module which defines that package first, or make that module available in @INC (@INC contains: /usr/local/lib /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .). at /usr/local/lib/Pod/LaTeX.pm line 29.BEGIN failed--compilation aborted at /usr/local/lib/Pod/LaTeX.pm line 29.Compilation failed in require at /usr/local/lib/TeX/Encode.pm line 10.BEGIN failed--compilation aborted at /usr/local/lib/TeX/Encode.pm line 10.Compilation failed in require at /usr/local/bin/darshan-job-summary.pl <http://darshan-job-summary.pl> line 12.BEGIN failed--compilation aborted at /usr/local/bin/darshan-job-summary.pl <http://darshan-job-summary.pl> line 12.[haxxanasghar@gpuserver2 ~]$ Compilation failed in require at /usr/local/bin/darshan-job-summary.pl <http://darshan-job-summary.pl> line 12.bash: Compilation: command not found...* Please guide me in this regard..... -- Regards, *Hassan Asghar* Ph.D Computer Science Personal Website: http://bit.ly/2OQ0egs Google Scholar: https://bit.ly/3lVYuxF

1 0

Problems about Darshan Logs
by Jie Liu 07 Jun '21

07 Jun '21

Hi, I used Darshan to do some profiling work when training Deep Learning models on ThetaGPU (Resnet50 on ImageNet, mini-batch size is 32). When I used the following command to get the summary of darshan logs: darshan-job-summary.pl /path/to/.darshan --output /path/to/summary.pdf The received summary.pdf file contains the following Error message at the firs page: WARNING: This Darshan log contains incomplete data. This happens when a module runs out of memory to store new record data. Please run darshan-parser on the log file for more information. I also tried to use darshan-parser by the following command: darshan-parser /path/to/.darshan --output /path/to/summary.txt It also shows incomplete data error: *ERROR*: The POSIX module contains incomplete data! This happens when a module runs out of memory to store new record data. The ImageNet dataset contains about 1.3 million image files, but the darshan log only shows the number of opened files is: 14792 when I trained Resnet50 on ThetaGPU with 2 nodes, 16 GPUs. (Please check the attached file for more information about the logs obtained by darshan). Is there an efficient way to make the darshan logs contain all the I/O information of 1.3 million image files during the model training? Previously, I contacted with the Support Team, their response is “the POSIX module only tracks 1024 files, once we open 1025 files Darshan no longer tracks those files”. How to make the POSIX module track all the images files. For the model training on ThetaGPU using 2 nodes and 16 GPUs. My experimental results show that every process handles 14792/16 = 924 images files on average, actually, this number is less than 1024. How to explain it? Thanks for your help. Best regards, -- Jie Liu

4 3

Re: [Darshan-users] Problems about Darshan Logs
by Chunduri, Sudheer 07 Jun '21

07 Jun '21

Bogdan, Ok. On the part that work on in Autoperf, we have received a request to have a runtime variable to ON and OFF the modules to be loaded and we will soon work on that, but that is more in darshan-runtime. But I think you are referring to the darshan-utils code, if possible, would be good to have some sample list of these constants in the code. Probably, Shane can comment better if it is on darshan-utils. From: Nicolae, Bogdan <bnicolae(a)anl.gov> Date: Monday, June 7, 2021 at 11:12 AM To: Chunduri, Sudheer <sudheer(a)anl.gov>, Jie Liu <jliu279(a)ucmerced.edu>, darshan-users(a)lists.mcs.anl.gov <darshan-users(a)lists.mcs.anl.gov> Cc: Si, Min <msi(a)anl.gov> Subject: Re: Problems about Darshan Logs Chunduri, We finally fixed the problem, Darshan simply has a lot of hardcoded constants in the code. We compiled our own version. Based on our experience, it would be better to expose these constants as environment variables. Cheers, Bogdan ________________________________ From: Chunduri, Sudheer <sudheer(a)anl.gov> Sent: Monday, June 7, 2021 9:54 AM To: Jie Liu <jliu279(a)ucmerced.edu>; darshan-users(a)lists.mcs.anl.gov <darshan-users(a)lists.mcs.anl.gov> Cc: Nicolae, Bogdan <bnicolae(a)anl.gov>; Si, Min <msi(a)anl.gov> Subject: Re: Problems about Darshan Logs Hi Jie, I see you copying darshan-users mailing list, so, Shane should hopefully see this. Meanwhile, have you tried using “darshan-parser --show-incomplete”? From: Darshan-users <darshan-users-bounces(a)lists.mcs.anl.gov> on behalf of Jie Liu <jliu279(a)ucmerced.edu> Date: Monday, June 7, 2021 at 9:42 AM To: darshan-users(a)lists.mcs.anl.gov <darshan-users(a)lists.mcs.anl.gov> Cc: Nicolae, Bogdan <bnicolae(a)anl.gov>, Si, Min <msi(a)anl.gov> Subject: [Darshan-users] Problems about Darshan Logs Hi, I used Darshan to do some profiling work when training Deep Learning models on ThetaGPU (Resnet50 on ImageNet, mini-batch size is 32). When I used the following command to get the summary of darshan logs: darshan-job-summary.pl /path/to/.darshan --output /path/to/summary.pdf The received summary.pdf file contains the following Error message at the firs page: WARNING: This Darshan log contains incomplete data. This happens when a module runs out of memory to store new record data. Please run darshan-parser on the log file for more information. I also tried to use darshan-parser by the following command: darshan-parser /path/to/.darshan --output /path/to/summary.txt It also shows incomplete data error: *ERROR*: The POSIX module contains incomplete data! This happens when a module runs out of memory to store new record data. The ImageNet dataset contains about 1.3 million image files, but the darshan log only shows the number of opened files is: 14792 when I trained Resnet50 on ThetaGPU with 2 nodes, 16 GPUs. (Please check the attached file for more information about the logs obtained by darshan). Is there an efficient way to make the darshan logs contain all the I/O information of 1.3 million image files during the model training? Previously, I contacted with the Support Team, their response is “the POSIX module only tracks 1024 files, once we open 1025 files Darshan no longer tracks those files”. How to make the POSIX module track all the images files. For the model training on ThetaGPU using 2 nodes and 16 GPUs. My experimental results show that every process handles 14792/16 = 924 images files on average, actually, this number is less than 1024. How to explain it? Thanks for your help. Best regards, -- Jie Liu

1 0