July 2020 - Darshan-users - mailman3.cels.anl.gov

Issues with Fio benchmark tool being profiled with Darshan
by Neeraj Rajesh 07 Aug '20

07 Aug '20

Hello I am trying to profile an Fio command, where I do roughly a gigabyte of reads or writes or mixed at a fixed block size. eg. ``` fio --filename=./test --size=1024MB --direct=1 --rw=randrw --bs=4k --ioengine=sync --iodepth=256 --numjobs=1 --name=test-job --minimal --stats=0 ``` I have tested it with the following 'ioengines': - sync - psync - posixaio - pvsync I do get a darshan file, but all of the POSIX counters are 0, except for POSIX_F_META_TIME, POSIX_STATS, POSIX_FILE_ALIGNMENT and POSIX_MMAPS. I have ensured the operation is not simultaneously doing both POSIX and STDIO. As per my configuration it should not be getting 0s especially for the counters POSIX_SIZE_WRITE_1K_10K, POSIX_WRITES, POSIX_READS, POSIX_BYTES_WRITTEN, POSIX_BYTES_READ etc. Kindly advise what to do next. Thank you for your time, Stay safe Yours Sincerely Neeraj Rajesh

2 7

Re: [Darshan-users] [EXTERNAL] Re: Darshan error on Cray system with static compilation
by Markomanolis, George 28 Jul '20

28 Jul '20

Hi Phil, Thanks for the answer. I assume this code is there for ages, right? Because they asked me if they should try a previous version and I was not so excited about that but better ask you if it is something new but from your words, I understand that it’s not. Regards, George From: "Carns, Philip H." <carns(a)mcs.anl.gov> Date: Saturday, July 25, 2020 at 2:24 PM To: "Markomanolis, George" <markomanolig(a)ornl.gov>, "darshan-users(a)lists.mcs.anl.gov" <darshan-users(a)lists.mcs.anl.gov> Subject: [EXTERNAL] Re: Darshan error on Cray system with static compilation Hi George, We've ever seen that assertion triggered before as far as I know (it's just defensive programming, not something that is supposed to happen). It indicates that Darshan observed inconsistent results out of a binary search tree; possibly brought on by a memory corruption of some sort? Unfortunately I'm not sure what to suggest on this one; we might need more information or a reproducer. The application might have an I/O workload that triggers a buggy code path in Darshan. It's also plausible that there is a memory corruption outside of Darshan (in the application or another library) that is just impacting that Darshan data structure by chance. -Phil ________________________________ From: Darshan-users <darshan-users-bounces(a)lists.mcs.anl.gov> on behalf of Markomanolis, George <markomanolig(a)ornl.gov> Sent: Thursday, July 23, 2020 2:36 PM To: darshan-users(a)lists.mcs.anl.gov <darshan-users(a)lists.mcs.anl.gov> Subject: [Darshan-users] Darshan error on Cray system with static compilation Hi, I just send an error that we can’t reproduce, it happens sometimes and it is on a system that I don’t even have access but they informed me about this error: fms_MOM6_SIS2_compile.x: lib/darshan-common.c:262: darshan_track_common_val_counters: Assertion `found == counter' failed. forrtl: error (76): Abort trap signal Image PC Routine Line Source This is a Cray system with static compilation. This error kills the application. Do you have any idea or it is difficult with so minimal information? Regards, George

2 1

Darshan error on Cray system with static compilation
by Markomanolis, George 25 Jul '20

25 Jul '20

Hi, I just send an error that we can’t reproduce, it happens sometimes and it is on a system that I don’t even have access but they informed me about this error: fms_MOM6_SIS2_compile.x: lib/darshan-common.c:262: darshan_track_common_val_counters: Assertion `found == counter' failed. forrtl: error (76): Abort trap signal Image PC Routine Line Source This is a Cray system with static compilation. This error kills the application. Do you have any idea or it is difficult with so minimal information? Regards, George

2 1

Cause of Darshan warning on the generated report
by Liem, Radita 22 Jul '20

22 Jul '20

Dear all, I tried to run Darshan on several programs and when I tried to generate Latex report, I got warning message: This Darshan log contains incomplete data which may skew results in this document. When I checked the parsed log result, the STDIO module data is always empty. Is this the cause of the warning message on the Latex output? And if that's the case, how to fix this problem? The programs also use HDF5 called from Fortran code, the Darshan log shows that there are H5D and H5F module data but it doesn't appear on the Latex output. I also wonder what is the cause for this and how to fix it. My Darshan version is 3.2.1 with hdf5 1.8.21 Best, Radita Liem, M. Sc. IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) liem(a)itc.rwth-aachen.de <mailto:[email protected]> www.itc.rwth-aachen.de <http://www.itc.rwth-aachen.de>

2 1

Darshan not tracing with tensorflow import.
by Devarajan, Hariharan 10 Jul '20

10 Jul '20

Hello, I was able to run my test program with 3.2.1 darshan. However, when i trace an app which load tensorflow it seems darshan doesn't produce trace. I am attaching two tars one with working example and one without. The only difference between the two is in test.py where i import tensorflow on first line. Can you please assist on how I can further debug the problem? Regards Hariharan

4 11