- Darshan-users - mailman3.cels.anl.gov

Recording more than 1024 files
by George Papadimitriou 01 Nov '21

01 Nov '21

Dear darshan devs, I ran into a limit when using Darshan 3.3.1, where darshan-runtime records only the first 1024 files. Is there a way to build Darshan and increase this limit? I couldn't find something in the documentation. I increased the module memory, but it didn't affect this. Thanks, George

3 2

Consecutive vs. Sequential?
by Jeffrey Layton 10 Aug '21

10 Aug '21

Good morning, I have a question about the job summary output. There is a bar chart with bars labeled as "sequential" and "consecutive". How are these defined? Thanks! Jeff P.S. Apologies for the silly Monday morning question. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

2 2

Does Darshan uses CPU time ?
by Florian Lecomte 05 Aug '21

05 Aug '21

Good evening, I'd like to know if I have to divide metrics by the number of CPUs of the machine I use if I want to know the percentage of time spent on write operations for example. Because when I divide for example STDIO_F_META_TIME by [real runtime (time spent in "real world") * number of processes], I often get something bigger than 1, which is not supposed to happen. So to sum it up : Does Darshan give the average metric value per process per CPU, or only per process, and so it can not be compared with real spent time ? Thank you very much. Cordially, Florian, student in the HPC field.

2 5

Darshan 3.3.1 aborting while generating log file
by André R. Carneiro 29 Jul '21

29 Jul '21

Hi, I'm testing version 3.3.1 with different MPI implementations. With newer versions OpenMPI (4.X with ROMIO v3.2.1) and Intel MPI (Parallel Studio XE 2019 and 2020 with ROMIO from MPICH v3.3) everything runs smoothly. But with older versions I'm getting the erros bellow while generating the log file when using a Lustre filesystem. I'm only able to generate the log files with those older versions if I configure the environment variable DARSHAN_LOGHINTS with "". The application I'm testing is the BT-IO from NAS NPB v3.3.1. The version of the Lustre FS is 2.12.4.1_cray_139_g0763d21 ====================================================== * OpenMPI 3.X with ROMIO v3.1.4 Program received signal SIGFPE: Floating-point exception - erroneous arithmetic operation. Backtrace for this error: #0 0x2b8da12e03ef in ??? #1 0x2b8da01e8abe in ??? #2 0x2b8da01ead06 in ??? #3 0x2b8da02186c0 in ??? #4 0x2b8da0218ddb in ??? #5 0x2b8da01da6f1 in ??? #6 0x2b8da015592b in ??? #7 0x2b8d9f813c40 in MPI_File_write_at_all at lib/darshan-mpiio.c:573 #8 0x2b8d9f7f5134 in darshan_log_append at lib/darshan-core.c:1884 #9 0x2b8d9f7f84bd in darshan_log_write_name_record_hash at lib/darshan-core.c:1775 #10 0x2b8d9f7f84bd in darshan_core_shutdown at lib/darshan-core.c:604 #11 0x2b8d9f7f4917 in MPI_Finalize at lib/darshan-core-init-finalize.c:85 ====================================================== * OpenMPI 3.X with OMPIO Program received signal SIGSEGV: Segmentation fault - invalid memory reference. #0 0x2b6fc617627f in ??? #1 0x2b6fc468bcfd in darshan_core_lookup_record_name at lib/darshan-core.c:2389 #2 0x2b6fc46a5484 in darshan_stdio_lookup_record_name at lib/darshan-stdio.c:1288 #3 0x2b6fc4694c87 in fileno at lib/darshan-posix.c:768 #4 0x2b6fc7b68504 in ??? #5 0x2b6fc7b69920 in ??? #6 0x2b6fc506c817 in ??? #7 0x2b6fc500ef1a in ??? #8 0x2b6fc50b4361 in ??? #9 0x2b6fc506e5c8 in ??? #10 0x2b6fc4fbafeb in ??? #11 0x2b6fc4fe8903 in ??? #12 0x2b6fc46a798b in MPI_File_open at lib/darshan-mpiio.c:345 #13 0x2b6fc468d4a1 in darshan_log_open at lib/darshan-core.c:1604 #14 0x2b6fc468d4a1 in darshan_core_shutdown at lib/darshan-core.c:584 #15 0x2b6fc468a917 in MPI_Finalize at lib/darshan-core-init-finalize.c:85 #16 0x2b6fc4d3b798 in ??? #17 0x4025cc in ??? #18 0x402f39 in ??? #19 0x2b6fc61623d4 in ??? #20 0x401868 in ??? #21 0xffffffffffffffff in ??? ====================================================== * Intel PSXE 2018 with ROMIO from MPICH v3.2 forrtl: severe (71): integer divide by zero Image PC Routine Line Source libifcoremt.so.5 00002B4887FFE4CF for__signal_handl Unknown Unknown libpthread-2.17.s 00002B4887B6F630 Unknown Unknown Unknown libmpi_lustre.so. 00002B488EF5EFDF ADIOI_LUSTRE_Get_ Unknown Unknown libmpi_lustre.so. 00002B488EF59FD9 ADIOI_LUSTRE_Writ Unknown Unknown libmpi.so.12.0 00002B4886FBD15C Unknown Unknown Unknown libmpi.so.12 00002B4886FBE1D5 PMPI_File_write_a Unknown Unknown libdarshan.so 00002B4886500B07 MPI_File_write_at Unknown Unknown libdarshan.so 00002B48864E088D Unknown Unknown Unknown libdarshan.so 00002B48864E3BC3 darshan_core_shut Unknown Unknown libdarshan.so 00002B48864E00A8 MPI_Finalize Unknown Unknown libmpifort.so.12. 00002B48867B24DA pmpi_finalize__ Unknown Unknown bt.C.36.mpi_io_fu 0000000000402A35 Unknown Unknown Unknown bt.C.36.mpi_io_fu 0000000000401D92 Unknown Unknown Unknown libc-2.17.so 00002B488A76F545 __libc_start_main Unknown Unknown bt.C.36.mpi_io_fu 0000000000401C99 Unknown Unknown Unknown ====================================================== -- Abraços³, André Ramos Carneiro.

2 1

Re: [Darshan-users] Error in job_summary
by Jeffrey Layton 28 Jul '21

28 Jul '21

Oops. DId a reply instead of a reply-all. One more question. When I specify a directory to exclude, does it include all subdirectories as well? Thanks! Jeff On Tue, Jul 27, 2021 at 12:01 PM Jeffrey Layton <laytonjb(a)gmail.com> wrote: > Shane, > > Thanks for the reply. I'm glad you're doing the changes to Darshan. This > will have a big impact on profiling DL workloads (I realize that wasn't a > focus of Darshan originally, but Darshan has become a victim of it's own > success. When people mention 'IO profiling' then immediately say 'Darshan'). > > DL is a tough workload. Let me give you an example. I just ran a pretty > simple model with 1.2M parameters. I used the CIFAR-10 data set (collection > of images) and ran PyTorch or 100 epochs (not even close to being fully > trained). During those 100 epochs, PyTorch opened 167,076 files. It closed > 1,274,000 files (I measured this using the strace output). It also used > 1,206 threads. The training code was all Python. > > Tensorflow is better in regard to IO than PyTorch for CIFAR-10. The model > only used 555K parameters. For 100 epochs it opened 4,099 files. It closed > 3.617 files. It used 350 threads during this time. > > You can see that DL frameworks do a lot of stuff in the name of IO! Being > able to track over 100K files is probably not a bad idea (I might go as far > as 1M files). > > In the meantime, is there a limit to the number of items you can include > using excludes? > > Thanks! > > Jeff > > > > On Mon, Jul 26, 2021 at 9:43 PM Snyder, Shane <ssnyder(a)mcs.anl.gov> wrote: > >> Hi Jeff, >> >> Existing Darshan releases do have some hard coded limits that have been >> increasingly problematic for our users, it seems. The limit you are likely >> hitting is just that Darshan instrumentation modules do not track more than >> 1,024 file records currently. This isn't really tunable in any way, >> unfortunately. >> >> You can get a list of files that Darshan did instrument by running >> darshan-parser with the '--file-list' option. That might give you some more >> ideas on directories you could potentially exclude to force Darshan to >> reserve instrumentation resources for other files, but that may not even be >> sufficient depending on your workload. >> >> We do have some functionality we are hoping to have merged in for our >> next release to help address this issue. In fact, it's available to try out >> in a branch in our repo if you're really motivated to get this working >> soon. There are more details here in a PR on our GitHub: >> https://github.com/darshan-hpc/darshan/pull/405 >> >> Essentially, you can use a config file to control a number of different >> Darshan settings, including the ability to change the hard coded file >> maximum from above and to provide regular expressions (rather than just >> directory names) for files Darshan should exclude from instrumentation. If >> you have more specific questions or feedback about this functionality, >> please let us know. >> >> Thanks! >> --Shane >> ------------------------------ >> *From:* Darshan-users <darshan-users-bounces(a)lists.mcs.anl.gov> on >> behalf of Jeffrey Layton <laytonjb(a)gmail.com> >> *Sent:* Monday, July 26, 2021 9:15 AM >> *To:* darshan-users(a)lists.mcs.anl.gov <darshan-users(a)lists.mcs.anl.gov> >> *Subject:* [Darshan-users] Error in job_summary >> >> Good morning, >> >> I'm post-processing a darshan file for a Tensorflow training of a simple >> model (CIFAR-10). The post-processing completes just fine, but I see an >> error on the first page: >> >> >> WARNING: This Darshan log contains incomplete data. This happens when a >> module runs out of memory to store >> new record data. Please run darshan-parser on the log file for more >> information. >> >> So I ran darshan-parser on the file and I see the following at the end. >> >> >> # ******************************************************* >> # POSIX module data >> # ******************************************************* >> >> # *ERROR*: The POSIX module contains incomplete data! >> # This happens when a module runs out of >> # memory to store new record data. >> >> # To avoid this error, consult the darshan-runtime >> # documentation and consider setting the >> # DARSHAN_EXCLUDE_DIRS environment variable to prevent >> # Darshan from instrumenting unecessary files. >> >> # You can display the (incomplete) data that is >> # present in this log using the --show-incomplete >> # option to darshan-parser. >> >> >> I have a bunch of file systems excluded: /proc,/etc,/dev,/sys,/snap,/run >> . >> >> How can I get a list of files that Darshan tracked? Is there a way to >> increase the amount of memory? >> >> Thanks! >> >> Jeff >> >> >> >>

3 2

MPI I/O and POSIX I/O missing in final pdf
by Hassan Asghar 28 Jul '21

28 Jul '21

I have successfully created the GUI pdf of darshan log. but it only shows stdio. MPI I/O and POSIX I/O is missed in graph. Anyone please help me out with this.

2 1

Error in job_summary
by Jeffrey Layton 26 Jul '21

26 Jul '21

Good morning, I'm post-processing a darshan file for a Tensorflow training of a simple model (CIFAR-10). The post-processing completes just fine, but I see an error on the first page: WARNING: This Darshan log contains incomplete data. This happens when a module runs out of memory to store new record data. Please run darshan-parser on the log file for more information. So I ran darshan-parser on the file and I see the following at the end. # ******************************************************* # POSIX module data # ******************************************************* # *ERROR*: The POSIX module contains incomplete data! # This happens when a module runs out of # memory to store new record data. # To avoid this error, consult the darshan-runtime # documentation and consider setting the # DARSHAN_EXCLUDE_DIRS environment variable to prevent # Darshan from instrumenting unecessary files. # You can display the (incomplete) data that is # present in this log using the --show-incomplete # option to darshan-parser. I have a bunch of file systems excluded: /proc,/etc,/dev,/sys,/snap,/run . How can I get a list of files that Darshan tracked? Is there a way to increase the amount of memory? Thanks! Jeff

2 1

Unable to create log:
by Hassan Asghar 19 Jul '21

19 Jul '21

I have three gpus. darshan and darshan-logs are installed and shared via nfs. all permissions are set. I don't know why logs are not created: [haxxanasghar@gpuserver2 ~]$ mpiexec -n 64 -f machinefile -env LD_PRELOAD=/home/haxxanasghar/darshan/darshan-runtime/lib/libdarshan.so ./MADbench2 640 80 4 1 8 4 4 MADbench 2.0 IO-mode no_pe = 64 no_pix = 640 no_bin = 80 no_gang = 4 sblocksize = 1 fblocksize = 8 r_mod = 4 w_mod = 4 IOMETHOD = POSIX IOMODE = SYNC FILETYPE = UNIQUE REMAP = CUSTOM S_cc 0.03 [ 0.03: 0.03] S_bw 0.02 [ 0.02: 0.02] S_w 0.03 [ 0.03: 0.03] ------- S_total 0.08 [ 0.08: 0.08] W_cc 0.12 [ 0.07: 0.14] W_bw 0.77 [ 0.72: 0.85] W_r 0.02 [ 0.02: 0.02] W_w 0.01 [ 0.01: 0.01] ------- W_total 0.92 [ 0.89: 0.95] C_cc 0.01 [ 0.01: 0.01] C_bw 0.00 [ 0.00: 0.00] C_r 0.01 [ 0.00: 0.01] ------- C_total 0.02 [ 0.02: 0.02] dC[0] = 0.00000e+00 darshan_library_warning: unable to create log file /home/haxxanasghar/darshan-logs/2021/7/15/haxxanas_MADbench2_id17444-17444_7-15-65476-1341319014666854881.darshan_partial. -- Regards, *Hassan Asghar* Ph.D Scholar Mobile: +82-010-9686-1912 Personal Website: http://bit.ly/2OQ0egs Google Scholar: https://bit.ly/3lVYuxF

3 3

Hang on post-process
by Jeffrey Layton 14 Jul '21

14 Jul '21

Good afternoon, Apologies for posting yet another problem :) I'm trying to use Darshan on a Tensorflow/Keras script. It's a simple model operating on the CIFAR-10 data set (fairly small). Darshan produces the output files but when I try to post-process one using darshan-job-summary.pl, it hangs and I end up having to kill the process (I waited about an hour - just to be sure). I run the script using the following: export DARSHAN_EXCLUDE_DIRS=/proc,/etc,/dev,/sys env LD_PRELOAD=/home/laytonjb/bin/darshan-3.3.1/lib/libdarshan.so python3 cifar10-4-checkpoint.py (I can provide the script if needed). It produces four files: $ ls -s total 72 4 laytonjb_ptxas_id6210-6210_7-13-47480-2131301613401632697_1.darshan 60 laytonjb_python3_id6041-6041_7-13-47475-2131301613401632697_1.darshan 4 laytonjb_ptxas_id6211-6211_7-13-47480-2131301613401632697_1.darshan 4 laytonjb_uname_id6056-6056_7-13-47475-2131301613401632697_1.darshan I chose to post-process the "python3" output but this is where it hangs. I'm attaching the darshan output file if that is of any help. Thanks for any help. Jeff

4 8

PyDarshan issues
by Jeffrey Layton 14 Jul '21

14 Jul '21

Good morning, Is this is a good place to bring up PyDarshan issues or is there a better one? Thanks! Jeff

2 3