- Darshan-users - mailman3.cels.anl.gov

darshan execl issue
by Adrian Jackson 12 Jul '22

12 Jul '22

Hi, I've encountered an issue using Darshan (3.3.1) with a code which calls execl from one MPI process. Using with Darshan the MPI run just hangs. Is spawning processes from a subset of MPI processes an issue for Darshan? I would say that I can still spawn processes (i.e. using fork) and it seems to work, but using execl doesn't. cheers adrianj -- Tel: +44 131 6506470 skype: remoteadrianj The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

2 5

Darshan BoF at ECP Community Days
by Snyder, Shane 13 May '22

13 May '22

Darshan users, We wanted to make sure the community is aware of a Darshan BoF session we will be putting on at the ECP Community BoF Days: Darshan: Enabling Insights into HPC I/O Behavior May 12th, 3:00 PM eastern time We will provide general updates on new Darshan instrumentation modules and features, and will also have guest speakers talk in detail about recent Darshan activities that may be of interest to the community: * AutoPerf modules for instrumenting MPI communication and Cray network counters (Sudheer Chunduri, ALCF) * PyDarshan developments and a new PyDarshan-based job summary tool (Nik Awtrey, LANL) * Interactive tools for visualizing DXT trace data (Jean Luca Bez, LBL) * To register, follow this link, expand the Darshan BoF description, and click "Register" -- this should provide you with a Zoom link to attend the BoF: https://www.exascaleproject.org/event/ecp-community-bof-days-2022/ Thanks! --Darshan team

1 2

Module contains incomplete data
by Jiří Nádvorník 10 May '22

10 May '22

Hi All, I just tried out Darshan and the potential output seems perfect for my HDF5 MPI application! Although I'm not able to get there :(. I have a log that has a big stamp "This darshan log contains incomplete data". When I run: darshan-parser --show-incomplete mylog.darshan |grep incomplete Output is: # *WARNING*: The POSIX module contains incomplete data! # *WARNING*: The MPI-IO module contains incomplete data! # *WARNING*: The STDIO module contains incomplete data! Would you be able to point me to some setting that would improve the measurements? Can I actually rely on the profiling results if it says the data is incomplete in some of the categories? Thank you very much for your help! Cheers, Jiri

3 11

darshan-3.4.0-pre1 release is now available
by Snyder, Shane 09 May '22

09 May '22

Darshan users, We are pleased to announce a pre-release version of Darshan 3.4.0 (3.4.0-pre1) is now available HERE<https://ftp.mcs.anl.gov/pub/darshan/releases/darshan-3.4.0-pre1.tar.gz>. As always, please be aware that Darshan pre-releases are experimental and not recommended for full-time use in production yet. An official 3.4.0 release will be made available soon. This release contains a number of exciting new features and enhancements to Darshan: * Added new heatmap module to record per-process histograms of I/O activity over time for POSIX, MPI-IO, and STDIO modules * Added comprehensive darshan-runtime library configuration support, via environment variables and/or configuration file * Allows user to control how much memory Darshan modules use at runtime, restricts instrumentation to specific file name patterns, etc. * See the following link for more details: https://www.mcs.anl.gov/research/projects/darshan/docs/darshan-runtime.html… * Implemented performance optimizations to Darshan's wrappers, locking mechanisms, and timing mechanisms * Includes optional RDTSCP-based timers via '--enable-rdtscp' configure option * Removed deprecated performance estimates from darshan-parser and added 2 new derived metrics when using '--perf' : * agg_time_by_slowest (total elapsed time performing I/O by the slowest rank) * slowest_rank_rw_only_time (total elapsed time performing read/write operations by the slowest rank) * Adopted automake/libtool support for Darshan build (contributed by Wei-Keng Liao) * Increased default record name memory to 1 MiB per-process to avoid recent user reports of exceeding old limit (256 KiB) This release also marks our first stable release of the PyDarshan log analysis module, including a new PyDarshan-based job summary tool (ultimately will replace darshan-job-summary script). Users can get PyDarshan directly from PyPI, e.g., using 'pip install darshan'. Documentation can be found here: https://www.mcs.anl.gov/research/projects/darshan/documentation/ Please report any questions, issues, or concerns with this pre-relase using this mailing list, or by opening an issue on our GitHub: https://github.com/darshan-hpc/darshan. Thanks! --Darshan team

1 0

HDF5 version support
by Jiří Nádvorník 29 Apr '22

29 Apr '22

Hi all, I was able to successfully run the darshan runtime without any warnings withou HDF5. Now when I compile with HDF5 (the results above are without it): ./configure --with-log-path=/gpfs/raid/darshan-logs --with-jobid-env=PBS_JOBID CC=mpicc --enable-hdf5-mod --with-hdf5=/gpfs/raid/SDSSCube/ext_lib//hdf5-1.12.0/hdf5/ It messes up my runtime and causes python to crash: mpirun -x DARSHAN_CONFIG_PATH=/gpfs/raid/SDSSCube/darshan.conf -x LD_PRELOAD=/gpfs/raid/shared_libs/darshan/darshan-runtime/lib/.libs/libdarshan.so:/gpfs/raid/SDSSCube/ext_lib/hdf5-1.12.0/hdf5/lib/libhdf5.so -np 65 --hostfile hosts --map-by node /gpfs/raid/SDSSCube/venv_par/bin/python hisscube.py --truncate ../sdss_data/ results/SDSS_cube_c_par.h5 Resulting in: INFO:rank[0]:Rank 0 pid: 137058 Darshan HDF5 module error: runtime library version (1.12) incompatible with Darshan module (1.10-). Traceback (most recent call last): File "hisscube.py", line 74, in <module> writer.ingest(fits_image_path, fits_spectra_path, truncate_file=args.truncate) File "/gpfs/raid/SDSSCube/hisscube/ParallelWriterMWMR.py", line 45, in ingest self.process_metadata(image_path, image_pattern, spectra_path, spectra_pattern, truncate_file) File "/gpfs/raid/SDSSCube/hisscube/CWriter.py", line 150, in process_metadata h5_file = self.open_h5_file_serial(truncate_file) File "/gpfs/raid/SDSSCube/hisscube/CWriter.py", line 170, in open_h5_file_serial return h5py.File(self.h5_path, 'w', fs_strategy="page", fs_page_size=4096, libver="latest") File "/gpfs/raid/SDSSCube/venv_par/lib/python3.8/site-packages/h5py-3.6.0-py3.8-linux-x86_64.egg/h5py/_hl/files.py", line 533, in __init__ fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr) File "/gpfs/raid/SDSSCube/venv_par/lib/python3.8/site-packages/h5py-3.6.0-py3.8-linux-x86_64.egg/h5py/_hl/files.py", line 232, in make_fid fid = h5f.create(name, h5f.ACC_TRUNC, fapl=fapl, fcpl=fcpl) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5f.pyx", line 126, in h5py.h5f.create File "h5py/defs.pyx", line 693, in h5py.defs.H5Fcreate RuntimeError: Unspecified error in H5Fcreate (return value <0) You are saying that darshan should be compatible with HDF5 > 1.8, which 1.12 should be, right? I checked the source and there is hardcoded if minor version > 10: not supported. Thanks for explaining what's supported. Cheers, Jiri Nadvornik

2 3

unsub
by Hassan Asghar 27 Apr '22

27 Apr '22

1 0

Darshan 3.3.1 on XC40 CLE7.0 UP03 warning about memory
by Bilel Hadri 01 Feb '22

01 Feb '22

Hi Darshan colleagues, On Shaheen, Cray XC40, we just upgraded to CLE7 UP03 and using Darshan 3.3.1 and we are facing some warning with some packages like VASP, Quantum Espresso and in house seismic code : WARNING: unexpected condition in Darshan, possibly triggered by memory corruption. Darshan log may be incorrect. What would be the reason for such warning? On the TDS, with the same OS and same Darshan version and same applications, we didn’t get such warning… Is it related to Darshan, or to a system memory issue? Beside unloading darshan module, is there a way to remove the warning? Regards, Bilel ------- Bilel Hadri, PhD Computational Scientist KAUST Supercomputing Lab Al Khawarizmi Bldg. (1) Office 126 4700 King Abdullah University of Science and Technology Thuwal 23955-6900 Kingdom of Saudi Arabia Office Phone: +966 12 808 0654 Cell Phone: + 966 544 700 893

2 1

DARSHAN 3.3.1 PATH_MAX Undefined
by francesco iannone 18 Jan '22

18 Jan '22

Dear Support I’m compiling darshan 3.3.1 cloned by github. I’ve built with IntelMPI 2020 and hdf5 1.12.1. I had errors when configured in darshan-runtime, the first of them is as follow: In file included from darshan-core-init-finalize.c(20): darshan.h(124): error: identifier "PATH_MAX" is undefined char mmap_log_name[PATH_MAX]; can you help on it? Thanks in advance Francesco Iannone ENEA Italian National Agency ICT Division - HPC Lab (Head) ENEA Frascati Research Centre Via E. Fermi 45 00044 Frascati (Roma) Italy phone 00-39-06-9400-5124 mailto:francesco.iannone<at>enea.it http://www.afs.enea.it/iannone ================================================== Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate e la casella di posta elettronica da cui e' stata inviata e' da qualificarsi quale strumento aziendale. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate (art. 616 c.p, D.Lgs. n. 196/2003 s.m.i. e GDPR Regolamento - UE 2016/679). Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione. Grazie. This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised (art. 616 c.p, D.Lgs. n. 196/2003 and subsequent amendments and GDPR UE 2016/679). If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail. Thanks. ==================================================

3 3

Darshan log parser: incompatible version
by Saisha Kamat 14 Jan '22

14 Jan '22

Hello, I installed Darshan 3.3.1 i.e. that latest version available on my ubuntu machine. I have been trying to parse the Darshan logs. I get an error stating: Error: incompatible darshan file. Error: expected version 3.21, but got "RECID", Error: failed to read darshan log file header. I also could not find Darshan version 3.21. Any help will be really appreciated. Thank you Saisha

2 2

Darshan & IPM results not the same
by Vineet Soni 10 Nov '21

10 Nov '21

Hello, I am trying to analyze the IO behavior of our codes with Darshan. Darshan: 3.3.0 Compilers: Intel 2018 MPI: Intel MPI 2018 FS: Lustre (lustre-module disabled in Darshan configuration) Darshan profiling: LD_PRELOAD I observe a big difference in IO results from Darshan and IPM (v2.0.5) for one of our codes. I guess that both profilers are not profiling the same POSIX calls? The POSIXIO calls profiled in IPM are: fopen, fdopen, freopen, open, open64 fclose, close fflush fread, read fwrite, write fseek, lseek, lseek64 ftell rewind fgetpos, fsetpos, fgetc, getc, ungetc creat truncate, ftruncate, truncate64, ftruncate64 While the ones profiled by Darshan are: https://github.com/darshan-hpc/darshan/blob/main/darshan-runtime/lib/darsha… ? However, the huge difference is observed in the “read” call, which exists in both the profilers. +-------------------+------------+-----------+ | | IPM | Darshan | +-------------------+------------+-----------+ | Read (s) | 324.57 | 6.02 | +-------------------+------------+-----------+ | Agg. Read (count) | 34 766 456 | 2 946 271 | +-------------------+------------+-----------+ I tested Darshan and IPM with other codes (reading NOT the same files) to check if this issue is faced in them as well. But, I got the same results. So, I don't understand what could be the reason that this application is not giving the same results. Do you have any idea of why this could happen? Thanks in advance. PS: The application does a lot of IO, and is expected to spend a significant time in read operations. Best regards, Vineet

3 7