Darshan not tracing with tensorflow import.
Hello, I was able to run my test program with 3.2.1 darshan. However, when i trace an app which load tensorflow it seems darshan doesn't produce trace. I am attaching two tars one with working example and one without. The only difference between the two is in test.py where i import tensorflow on first line. Can you please assist on how I can further debug the problem? Regards Hariharan
About 2 months ago, I tried using Darshan to trace a TensorFlow2 DL training. I could not trace the input. What I _think_ happens is the TF2 uses mmap() for reading the input files and I don't think Darshan can capture that file IO. But I'm not a Darshan expert so perhaps someone has tried this before and can help (BTW - it's possible to build TensorFlow so it doesn't use mmap() for reading files.) Jeff On Mon, Jul 6, 2020 at 1:08 PM Devarajan, Hariharan <[email protected]> wrote:
Hello,
I was able to run my test program with 3.2.1 darshan. However, when i trace an app which load tensorflow it seems darshan doesn't produce trace. I am attaching two tars one with working example and one without. The only difference between the two is in test.py where i import tensorflow on first line.
Can you please assist on how I can further debug the problem?
Regards
Hariharan
_______________________________________________ Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
The I/O is not through tensorflow. Its through h5py and numpy.load and I verified both without tensorflow import are getting traced. I already verified that import of tensorflow doesn’t change any environment variables. Hari From: Jeffrey Layton<mailto:[email protected]> Sent: Monday, July 6, 2020 9:41 AM To: Devarajan, Hariharan<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. About 2 months ago, I tried using Darshan to trace a TensorFlow2 DL training. I could not trace the input. What I _think_ happens is the TF2 uses mmap() for reading the input files and I don't think Darshan can capture that file IO. But I'm not a Darshan expert so perhaps someone has tried this before and can help (BTW - it's possible to build TensorFlow so it doesn't use mmap() for reading files.) Jeff On Mon, Jul 6, 2020 at 1:08 PM Devarajan, Hariharan <[email protected]<mailto:[email protected]>> wrote: Hello, I was able to run my test program with 3.2.1 darshan. However, when i trace an app which load tensorflow it seems darshan doesn't produce trace. I am attaching two tars one with working example and one without. The only difference between the two is in test.py where i import tensorflow on first line. Can you please assist on how I can further debug the problem? Regards Hariharan _______________________________________________ Darshan-users mailing list [email protected]<mailto:[email protected]> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
Hi Hariharan, Thanks for letting us know about this issue. I can't really think of any reason why the import of tensorflow module would result in Darshan no longer producing log files. Just to make sure I'm fully understanding, you aren't getting any logfiles at all in the case where tensorflow is imported? I ask because when using this non-MPI instrumentation, I've noticed it tends to create a lot of log files, particularly for Python modules that like to call subprocesses for using things like ls, sed, etc. I just want to make sure it's not an issue of you missing one particular log file of interest or whether you don't get any log files at all. At any rate, the environment setup looks correct in both cases to preload the Darshan library and to enable the non-MPI instrumentation support. So, that doesn't appear to be the issue. Just to verify whether Darshan is even being properly initialized/shutdown, could you try setting the DARSHAN_INTERNAL_TIMING env variable (i.e., export DARSHAN_INTERNAL_TIMING=1) before running? That should spit out some more verbose output about how long it takes Darshan to init/finalize. If you do see some additional output indicating Darshan is at least initializing, you might want to double check that there are no errors in your output logs that indicate some issue Darshan encountered. --Shane ________________________________ From: Darshan-users <[email protected]> on behalf of Devarajan, Hariharan <[email protected]> Sent: Monday, July 6, 2020 9:56 AM To: Jeffrey Layton <[email protected]> Cc: [email protected] <[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. The I/O is not through tensorflow. Its through h5py and numpy.load and I verified both without tensorflow import are getting traced. I already verified that import of tensorflow doesn’t change any environment variables. Hari From: Jeffrey Layton<mailto:[email protected]> Sent: Monday, July 6, 2020 9:41 AM To: Devarajan, Hariharan<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. About 2 months ago, I tried using Darshan to trace a TensorFlow2 DL training. I could not trace the input. What I _think_ happens is the TF2 uses mmap() for reading the input files and I don't think Darshan can capture that file IO. But I'm not a Darshan expert so perhaps someone has tried this before and can help (BTW - it's possible to build TensorFlow so it doesn't use mmap() for reading files.) Jeff On Mon, Jul 6, 2020 at 1:08 PM Devarajan, Hariharan <[email protected]<mailto:[email protected]>> wrote: Hello, I was able to run my test program with 3.2.1 darshan. However, when i trace an app which load tensorflow it seems darshan doesn't produce trace. I am attaching two tars one with working example and one without. The only difference between the two is in test.py where i import tensorflow on first line. Can you please assist on how I can further debug the problem? Regards Hariharan _______________________________________________ Darshan-users mailing list [email protected]<mailto:[email protected]> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
It produces logs but it stops tracing h5py and np.load calls. If u run the working version, u will notice we get traces from both files but when u add import of tensorflow, this stops. I have verified that darshan is initializing as we get logs just not giving tracing for those two files in the code. Hari On Jul 8, 2020, at 9:41 AM, Snyder, Shane <[email protected]> wrote: Hi Hariharan, Thanks for letting us know about this issue. I can't really think of any reason why the import of tensorflow module would result in Darshan no longer producing log files. Just to make sure I'm fully understanding, you aren't getting any logfiles at all in the case where tensorflow is imported? I ask because when using this non-MPI instrumentation, I've noticed it tends to create a lot of log files, particularly for Python modules that like to call subprocesses for using things like ls, sed, etc. I just want to make sure it's not an issue of you missing one particular log file of interest or whether you don't get any log files at all. At any rate, the environment setup looks correct in both cases to preload the Darshan library and to enable the non-MPI instrumentation support. So, that doesn't appear to be the issue. Just to verify whether Darshan is even being properly initialized/shutdown, could you try setting the DARSHAN_INTERNAL_TIMING env variable (i.e., export DARSHAN_INTERNAL_TIMING=1) before running? That should spit out some more verbose output about how long it takes Darshan to init/finalize. If you do see some additional output indicating Darshan is at least initializing, you might want to double check that there are no errors in your output logs that indicate some issue Darshan encountered. --Shane ________________________________ From: Darshan-users <[email protected]> on behalf of Devarajan, Hariharan <[email protected]> Sent: Monday, July 6, 2020 9:56 AM To: Jeffrey Layton <[email protected]> Cc: [email protected] <[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. The I/O is not through tensorflow. Its through h5py and numpy.load and I verified both without tensorflow import are getting traced. I already verified that import of tensorflow doesn’t change any environment variables. Hari From: Jeffrey Layton<mailto:[email protected]> Sent: Monday, July 6, 2020 9:41 AM To: Devarajan, Hariharan<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. About 2 months ago, I tried using Darshan to trace a TensorFlow2 DL training. I could not trace the input. What I _think_ happens is the TF2 uses mmap() for reading the input files and I don't think Darshan can capture that file IO. But I'm not a Darshan expert so perhaps someone has tried this before and can help (BTW - it's possible to build TensorFlow so it doesn't use mmap() for reading files.) Jeff On Mon, Jul 6, 2020 at 1:08 PM Devarajan, Hariharan <[email protected]<mailto:[email protected]>> wrote: Hello, I was able to run my test program with 3.2.1 darshan. However, when i trace an app which load tensorflow it seems darshan doesn't produce trace. I am attaching two tars one with working example and one without. The only difference between the two is in test.py where i import tensorflow on first line. Can you please assist on how I can further debug the problem? Regards Hariharan _______________________________________________ Darshan-users mailing list [email protected]<mailto:[email protected]> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
I see. Thanks for the clarification. The only issue I'm aware of that currently causes us to lose some log data is in the case of applications calling fork(). Maybe that or something similar is happening in the import of tensorflow, with the h5/numpy I/O then happening in the child process? I can try to reproduce the issue to see if I can get a better idea of what's happening. We'd like to make sure tensorflow use cases work, but admittedly haven't really tested it. --Shane ________________________________ From: Devarajan, Hariharan <[email protected]> Sent: Wednesday, July 8, 2020 9:45 AM To: Snyder, Shane <[email protected]> Cc: Jeffrey Layton <[email protected]>; [email protected] <[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. It produces logs but it stops tracing h5py and np.load calls. If u run the working version, u will notice we get traces from both files but when u add import of tensorflow, this stops. I have verified that darshan is initializing as we get logs just not giving tracing for those two files in the code. Hari On Jul 8, 2020, at 9:41 AM, Snyder, Shane <[email protected]> wrote: Hi Hariharan, Thanks for letting us know about this issue. I can't really think of any reason why the import of tensorflow module would result in Darshan no longer producing log files. Just to make sure I'm fully understanding, you aren't getting any logfiles at all in the case where tensorflow is imported? I ask because when using this non-MPI instrumentation, I've noticed it tends to create a lot of log files, particularly for Python modules that like to call subprocesses for using things like ls, sed, etc. I just want to make sure it's not an issue of you missing one particular log file of interest or whether you don't get any log files at all. At any rate, the environment setup looks correct in both cases to preload the Darshan library and to enable the non-MPI instrumentation support. So, that doesn't appear to be the issue. Just to verify whether Darshan is even being properly initialized/shutdown, could you try setting the DARSHAN_INTERNAL_TIMING env variable (i.e., export DARSHAN_INTERNAL_TIMING=1) before running? That should spit out some more verbose output about how long it takes Darshan to init/finalize. If you do see some additional output indicating Darshan is at least initializing, you might want to double check that there are no errors in your output logs that indicate some issue Darshan encountered. --Shane ________________________________ From: Darshan-users <[email protected]> on behalf of Devarajan, Hariharan <[email protected]> Sent: Monday, July 6, 2020 9:56 AM To: Jeffrey Layton <[email protected]> Cc: [email protected] <[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. The I/O is not through tensorflow. Its through h5py and numpy.load and I verified both without tensorflow import are getting traced. I already verified that import of tensorflow doesn’t change any environment variables. Hari From: Jeffrey Layton<mailto:[email protected]> Sent: Monday, July 6, 2020 9:41 AM To: Devarajan, Hariharan<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. About 2 months ago, I tried using Darshan to trace a TensorFlow2 DL training. I could not trace the input. What I _think_ happens is the TF2 uses mmap() for reading the input files and I don't think Darshan can capture that file IO. But I'm not a Darshan expert so perhaps someone has tried this before and can help (BTW - it's possible to build TensorFlow so it doesn't use mmap() for reading files.) Jeff On Mon, Jul 6, 2020 at 1:08 PM Devarajan, Hariharan <[email protected]<mailto:[email protected]>> wrote: Hello, I was able to run my test program with 3.2.1 darshan. However, when i trace an app which load tensorflow it seems darshan doesn't produce trace. I am attaching two tars one with working example and one without. The only difference between the two is in test.py where i import tensorflow on first line. Can you please assist on how I can further debug the problem? Regards Hariharan _______________________________________________ Darshan-users mailing list [email protected]<mailto:[email protected]> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
Here are the reproducers. The working one has the tensorflow import commented (1st line). If u uncomment that, the hdf5 file and npz file stops getting traced. Rest of the py files are traced so Darshan is working. Hari From: Snyder, Shane<mailto:[email protected]> Sent: Wednesday, July 8, 2020 10:27 AM To: Devarajan, Hariharan<mailto:[email protected]> Cc: Jeffrey Layton<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. I see. Thanks for the clarification. The only issue I'm aware of that currently causes us to lose some log data is in the case of applications calling fork(). Maybe that or something similar is happening in the import of tensorflow, with the h5/numpy I/O then happening in the child process? I can try to reproduce the issue to see if I can get a better idea of what's happening. We'd like to make sure tensorflow use cases work, but admittedly haven't really tested it. --Shane From: Devarajan, Hariharan <[email protected]> Sent: Wednesday, July 8, 2020 9:45 AM To: Snyder, Shane <[email protected]> Cc: Jeffrey Layton <[email protected]>; [email protected] <[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. It produces logs but it stops tracing h5py and np.load calls. If u run the working version, u will notice we get traces from both files but when u add import of tensorflow, this stops. I have verified that darshan is initializing as we get logs just not giving tracing for those two files in the code. Hari On Jul 8, 2020, at 9:41 AM, Snyder, Shane <[email protected]> wrote: Hi Hariharan, Thanks for letting us know about this issue. I can't really think of any reason why the import of tensorflow module would result in Darshan no longer producing log files. Just to make sure I'm fully understanding, you aren't getting any logfiles at all in the case where tensorflow is imported? I ask because when using this non-MPI instrumentation, I've noticed it tends to create a lot of log files, particularly for Python modules that like to call subprocesses for using things like ls, sed, etc. I just want to make sure it's not an issue of you missing one particular log file of interest or whether you don't get any log files at all. At any rate, the environment setup looks correct in both cases to preload the Darshan library and to enable the non-MPI instrumentation support. So, that doesn't appear to be the issue. Just to verify whether Darshan is even being properly initialized/shutdown, could you try setting the DARSHAN_INTERNAL_TIMING env variable (i.e., export DARSHAN_INTERNAL_TIMING=1) before running? That should spit out some more verbose output about how long it takes Darshan to init/finalize. If you do see some additional output indicating Darshan is at least initializing, you might want to double check that there are no errors in your output logs that indicate some issue Darshan encountered. --Shane From: Darshan-users <[email protected]> on behalf of Devarajan, Hariharan <[email protected]> Sent: Monday, July 6, 2020 9:56 AM To: Jeffrey Layton <[email protected]> Cc: [email protected] <[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. The I/O is not through tensorflow. Its through h5py and numpy.load and I verified both without tensorflow import are getting traced. I already verified that import of tensorflow doesn’t change any environment variables. Hari From: Jeffrey Layton<mailto:[email protected]> Sent: Monday, July 6, 2020 9:41 AM To: Devarajan, Hariharan<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. About 2 months ago, I tried using Darshan to trace a TensorFlow2 DL training. I could not trace the input. What I _think_ happens is the TF2 uses mmap() for reading the input files and I don't think Darshan can capture that file IO. But I'm not a Darshan expert so perhaps someone has tried this before and can help (BTW - it's possible to build TensorFlow so it doesn't use mmap() for reading files.) Jeff On Mon, Jul 6, 2020 at 1:08 PM Devarajan, Hariharan <[email protected]<mailto:[email protected]>> wrote: Hello, I was able to run my test program with 3.2.1 darshan. However, when i trace an app which load tensorflow it seems darshan doesn't produce trace. I am attaching two tars one with working example and one without. The only difference between the two is in test.py where i import tensorflow on first line. Can you please assist on how I can further debug the problem? Regards Hariharan _______________________________________________ Darshan-users mailing list [email protected]<mailto:[email protected]> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
Only a comment, Could be when you import tensorflow the number of files to trace exceeds the maximum files to trace defined by Darshan? This could be the reason that you don't see the file logs that you comment. Sandra.- On Wed, 8 Jul 2020 at 21:38, Devarajan, Hariharan <[email protected]> wrote:
Here are the reproducers. The working one has the tensorflow import commented (1st line). If u uncomment that, the hdf5 file and npz file stops getting traced. Rest of the py files are traced so Darshan is working.
Hari
*From: *Snyder, Shane <[email protected]> *Sent: *Wednesday, July 8, 2020 10:27 AM *To: *Devarajan, Hariharan <[email protected]> *Cc: *Jeffrey Layton <[email protected]>; [email protected] *Subject: *Re: [Darshan-users] Darshan not tracing with tensorflow import.
I see. Thanks for the clarification.
The only issue I'm aware of that currently causes us to lose some log data is in the case of applications calling fork(). Maybe that or something similar is happening in the import of tensorflow, with the h5/numpy I/O then happening in the child process?
I can try to reproduce the issue to see if I can get a better idea of what's happening. We'd like to make sure tensorflow use cases work, but admittedly haven't really tested it.
--Shane
*From:* Devarajan, Hariharan <[email protected]> *Sent:* Wednesday, July 8, 2020 9:45 AM *To:* Snyder, Shane <[email protected]> *Cc:* Jeffrey Layton <[email protected]>; [email protected] <[email protected]> *Subject:* Re: [Darshan-users] Darshan not tracing with tensorflow import.
It produces logs but it stops tracing h5py and np.load calls. If u run the working version, u will notice we get traces from both files but when u add import of tensorflow, this stops. I have verified that darshan is initializing as we get logs just not giving tracing for those two files in the code.
Hari
On Jul 8, 2020, at 9:41 AM, Snyder, Shane <[email protected]> wrote:
Hi Hariharan,
Thanks for letting us know about this issue.
I can't really think of any reason why the import of tensorflow module would result in Darshan no longer producing log files. Just to make sure I'm fully understanding, you aren't getting any logfiles at all in the case where tensorflow is imported? I ask because when using this non-MPI instrumentation, I've noticed it tends to create a lot of log files, particularly for Python modules that like to call subprocesses for using things like ls, sed, etc. I just want to make sure it's not an issue of you missing one particular log file of interest or whether you don't get any log files at all.
At any rate, the environment setup looks correct in both cases to preload the Darshan library and to enable the non-MPI instrumentation support. So, that doesn't appear to be the issue.
Just to verify whether Darshan is even being properly initialized/shutdown, could you try setting the DARSHAN_INTERNAL_TIMING env variable (i.e., export
DARSHAN_INTERNAL_TIMING=1) before running? That should spit out some more verbose output about how long it takes Darshan to init/finalize. If you do see some additional output indicating Darshan is at least initializing, you might want to double check that there are no errors in your output logs that indicate some issue Darshan encountered.
--Shane
*From:* Darshan-users <[email protected]> on behalf of Devarajan, Hariharan <[email protected]> *Sent:* Monday, July 6, 2020 9:56 AM *To:* Jeffrey Layton <[email protected]> *Cc:* [email protected] <[email protected]> *Subject:* Re: [Darshan-users] Darshan not tracing with tensorflow import.
The I/O is not through tensorflow. Its through h5py and numpy.load and I verified both without tensorflow import are getting traced. I already verified that import of tensorflow doesn’t change any environment variables.
Hari
*From: *Jeffrey Layton <[email protected]> *Sent: *Monday, July 6, 2020 9:41 AM *To: *Devarajan, Hariharan <[email protected]> *Cc: *[email protected] *Subject: *Re: [Darshan-users] Darshan not tracing with tensorflow import.
About 2 months ago, I tried using Darshan to trace a TensorFlow2 DL training. I could not trace the input. What I _think_ happens is the TF2 uses mmap() for reading the input files and I don't think Darshan can capture that file IO. But I'm not a Darshan expert so perhaps someone has tried this before and can help
(BTW - it's possible to build TensorFlow so it doesn't use mmap() for reading files.)
Jeff
On Mon, Jul 6, 2020 at 1:08 PM Devarajan, Hariharan <[email protected]> wrote:
Hello,
I was able to run my test program with 3.2.1 darshan. However, when i trace an app which load tensorflow it seems darshan doesn't produce trace. I am attaching two tars one with working example and one without. The only difference between the two is in test.py where i import tensorflow on first line.
Can you please assist on how I can further debug the problem?
Regards
Hariharan
_______________________________________________ Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
_______________________________________________ Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
I see. There are 1024 files exactly in the darshan trace. What is the limit of the total number of files? Can we increase it? Hari From: Sandra A. Mendez<mailto:[email protected]> Sent: Wednesday, July 8, 2020 2:49 PM To: Devarajan, Hariharan<mailto:[email protected]> Cc: Snyder, Shane<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. Only a comment, Could be when you import tensorflow the number of files to trace exceeds the maximum files to trace defined by Darshan? This could be the reason that you don't see the file logs that you comment. Sandra.- On Wed, 8 Jul 2020 at 21:38, Devarajan, Hariharan <[email protected]<mailto:[email protected]>> wrote: Here are the reproducers. The working one has the tensorflow import commented (1st line). If u uncomment that, the hdf5 file and npz file stops getting traced. Rest of the py files are traced so Darshan is working. Hari From: Snyder, Shane<mailto:[email protected]> Sent: Wednesday, July 8, 2020 10:27 AM To: Devarajan, Hariharan<mailto:[email protected]> Cc: Jeffrey Layton<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. I see. Thanks for the clarification. The only issue I'm aware of that currently causes us to lose some log data is in the case of applications calling fork(). Maybe that or something similar is happening in the import of tensorflow, with the h5/numpy I/O then happening in the child process? I can try to reproduce the issue to see if I can get a better idea of what's happening. We'd like to make sure tensorflow use cases work, but admittedly haven't really tested it. --Shane From: Devarajan, Hariharan <[email protected]<mailto:[email protected]>> Sent: Wednesday, July 8, 2020 9:45 AM To: Snyder, Shane <[email protected]<mailto:[email protected]>> Cc: Jeffrey Layton <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. It produces logs but it stops tracing h5py and np.load calls. If u run the working version, u will notice we get traces from both files but when u add import of tensorflow, this stops. I have verified that darshan is initializing as we get logs just not giving tracing for those two files in the code. Hari On Jul 8, 2020, at 9:41 AM, Snyder, Shane <[email protected]<mailto:[email protected]>> wrote: Hi Hariharan, Thanks for letting us know about this issue. I can't really think of any reason why the import of tensorflow module would result in Darshan no longer producing log files. Just to make sure I'm fully understanding, you aren't getting any logfiles at all in the case where tensorflow is imported? I ask because when using this non-MPI instrumentation, I've noticed it tends to create a lot of log files, particularly for Python modules that like to call subprocesses for using things like ls, sed, etc. I just want to make sure it's not an issue of you missing one particular log file of interest or whether you don't get any log files at all. At any rate, the environment setup looks correct in both cases to preload the Darshan library and to enable the non-MPI instrumentation support. So, that doesn't appear to be the issue. Just to verify whether Darshan is even being properly initialized/shutdown, could you try setting the DARSHAN_INTERNAL_TIMING env variable (i.e., export DARSHAN_INTERNAL_TIMING=1) before running? That should spit out some more verbose output about how long it takes Darshan to init/finalize. If you do see some additional output indicating Darshan is at least initializing, you might want to double check that there are no errors in your output logs that indicate some issue Darshan encountered. --Shane From: Darshan-users <[email protected]<mailto:[email protected]>> on behalf of Devarajan, Hariharan <[email protected]<mailto:[email protected]>> Sent: Monday, July 6, 2020 9:56 AM To: Jeffrey Layton <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. The I/O is not through tensorflow. Its through h5py and numpy.load and I verified both without tensorflow import are getting traced. I already verified that import of tensorflow doesn’t change any environment variables. Hari From: Jeffrey Layton<mailto:[email protected]> Sent: Monday, July 6, 2020 9:41 AM To: Devarajan, Hariharan<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. About 2 months ago, I tried using Darshan to trace a TensorFlow2 DL training. I could not trace the input. What I _think_ happens is the TF2 uses mmap() for reading the input files and I don't think Darshan can capture that file IO. But I'm not a Darshan expert so perhaps someone has tried this before and can help (BTW - it's possible to build TensorFlow so it doesn't use mmap() for reading files.) Jeff On Mon, Jul 6, 2020 at 1:08 PM Devarajan, Hariharan <[email protected]<mailto:[email protected]>> wrote: Hello, I was able to run my test program with 3.2.1 darshan. However, when i trace an app which load tensorflow it seems darshan doesn't produce trace. I am attaching two tars one with working example and one without. The only difference between the two is in test.py where i import tensorflow on first line. Can you please assist on how I can further debug the problem? Regards Hariharan _______________________________________________ Darshan-users mailing list [email protected]<mailto:[email protected]> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users _______________________________________________ Darshan-users mailing list [email protected]<mailto:[email protected]> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
I think you can change the value in the source code in darshan-runtime (in darshan.h there is a variable, but I suppose darshan developers can provide a more appropriate solution). I had a similar case, but I decided to filter folders by using the DARSHAN_EXCLUDE_DIRS environment variable and only trace files open by the user's application and not trace files related to the python libraries. Regards, Sandra.- On Thu, 9 Jul 2020 at 00:09, Devarajan, Hariharan <[email protected]> wrote:
I see. There are 1024 files exactly in the darshan trace. What is the limit of the total number of files? Can we increase it?
Hari
*From: *Sandra A. Mendez <[email protected]> *Sent: *Wednesday, July 8, 2020 2:49 PM *To: *Devarajan, Hariharan <[email protected]> *Cc: *Snyder, Shane <[email protected]>; [email protected] *Subject: *Re: [Darshan-users] Darshan not tracing with tensorflow import.
Only a comment, Could be when you import tensorflow the number of files to trace exceeds the maximum files to trace defined by Darshan? This could be the reason that you don't see the file logs that you comment.
Sandra.-
On Wed, 8 Jul 2020 at 21:38, Devarajan, Hariharan <[email protected]> wrote:
Here are the reproducers. The working one has the tensorflow import commented (1st line). If u uncomment that, the hdf5 file and npz file stops getting traced. Rest of the py files are traced so Darshan is working.
Hari
*From: *Snyder, Shane <[email protected]> *Sent: *Wednesday, July 8, 2020 10:27 AM *To: *Devarajan, Hariharan <[email protected]> *Cc: *Jeffrey Layton <[email protected]>; [email protected] *Subject: *Re: [Darshan-users] Darshan not tracing with tensorflow import.
I see. Thanks for the clarification.
The only issue I'm aware of that currently causes us to lose some log data is in the case of applications calling fork(). Maybe that or something similar is happening in the import of tensorflow, with the h5/numpy I/O then happening in the child process?
I can try to reproduce the issue to see if I can get a better idea of what's happening. We'd like to make sure tensorflow use cases work, but admittedly haven't really tested it.
--Shane
*From:* Devarajan, Hariharan <[email protected]> *Sent:* Wednesday, July 8, 2020 9:45 AM *To:* Snyder, Shane <[email protected]> *Cc:* Jeffrey Layton <[email protected]>; [email protected] <[email protected]> *Subject:* Re: [Darshan-users] Darshan not tracing with tensorflow import.
It produces logs but it stops tracing h5py and np.load calls. If u run the working version, u will notice we get traces from both files but when u add import of tensorflow, this stops. I have verified that darshan is initializing as we get logs just not giving tracing for those two files in the code.
Hari
On Jul 8, 2020, at 9:41 AM, Snyder, Shane <[email protected]> wrote:
Hi Hariharan,
Thanks for letting us know about this issue.
I can't really think of any reason why the import of tensorflow module would result in Darshan no longer producing log files. Just to make sure I'm fully understanding, you aren't getting any logfiles at all in the case where tensorflow is imported? I ask because when using this non-MPI instrumentation, I've noticed it tends to create a lot of log files, particularly for Python modules that like to call subprocesses for using things like ls, sed, etc. I just want to make sure it's not an issue of you missing one particular log file of interest or whether you don't get any log files at all.
At any rate, the environment setup looks correct in both cases to preload the Darshan library and to enable the non-MPI instrumentation support. So, that doesn't appear to be the issue.
Just to verify whether Darshan is even being properly initialized/shutdown, could you try setting the DARSHAN_INTERNAL_TIMING env variable (i.e., export
DARSHAN_INTERNAL_TIMING=1) before running? That should spit out some more verbose output about how long it takes Darshan to init/finalize. If you do see some additional output indicating Darshan is at least initializing, you might want to double check that there are no errors in your output logs that indicate some issue Darshan encountered.
--Shane
*From:* Darshan-users <[email protected]> on behalf of Devarajan, Hariharan <[email protected]> *Sent:* Monday, July 6, 2020 9:56 AM *To:* Jeffrey Layton <[email protected]> *Cc:* [email protected] <[email protected]> *Subject:* Re: [Darshan-users] Darshan not tracing with tensorflow import.
The I/O is not through tensorflow. Its through h5py and numpy.load and I verified both without tensorflow import are getting traced. I already verified that import of tensorflow doesn’t change any environment variables.
Hari
*From: *Jeffrey Layton <[email protected]> *Sent: *Monday, July 6, 2020 9:41 AM *To: *Devarajan, Hariharan <[email protected]> *Cc: *[email protected] *Subject: *Re: [Darshan-users] Darshan not tracing with tensorflow import.
About 2 months ago, I tried using Darshan to trace a TensorFlow2 DL training. I could not trace the input. What I _think_ happens is the TF2 uses mmap() for reading the input files and I don't think Darshan can capture that file IO. But I'm not a Darshan expert so perhaps someone has tried this before and can help
(BTW - it's possible to build TensorFlow so it doesn't use mmap() for reading files.)
Jeff
On Mon, Jul 6, 2020 at 1:08 PM Devarajan, Hariharan <[email protected]> wrote:
Hello,
I was able to run my test program with 3.2.1 darshan. However, when i trace an app which load tensorflow it seems darshan doesn't produce trace. I am attaching two tars one with working example and one without. The only difference between the two is in test.py where i import tensorflow on first line.
Can you please assist on how I can further debug the problem?
Regards
Hariharan
_______________________________________________ Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
_______________________________________________ Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
Ok i added the exclusion list and now i see that darshan is tracing. I will try the main app now and see if it solves the problem. Thanks I will keep you posted. Hariharan ________________________________ From: Sandra A. Mendez <[email protected]> Sent: Wednesday, July 8, 2020 5:22:43 PM To: Devarajan, Hariharan <[email protected]> Cc: Snyder, Shane <[email protected]>; [email protected] <[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. I think you can change the value in the source code in darshan-runtime (in darshan.h there is a variable, but I suppose darshan developers can provide a more appropriate solution). I had a similar case, but I decided to filter folders by using the DARSHAN_EXCLUDE_DIRS environment variable and only trace files open by the user's application and not trace files related to the python libraries. Regards, Sandra.- On Thu, 9 Jul 2020 at 00:09, Devarajan, Hariharan <[email protected]<mailto:[email protected]>> wrote: I see. There are 1024 files exactly in the darshan trace. What is the limit of the total number of files? Can we increase it? Hari From: Sandra A. Mendez<mailto:[email protected]> Sent: Wednesday, July 8, 2020 2:49 PM To: Devarajan, Hariharan<mailto:[email protected]> Cc: Snyder, Shane<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. Only a comment, Could be when you import tensorflow the number of files to trace exceeds the maximum files to trace defined by Darshan? This could be the reason that you don't see the file logs that you comment. Sandra.- On Wed, 8 Jul 2020 at 21:38, Devarajan, Hariharan <[email protected]<mailto:[email protected]>> wrote: Here are the reproducers. The working one has the tensorflow import commented (1st line). If u uncomment that, the hdf5 file and npz file stops getting traced. Rest of the py files are traced so Darshan is working. Hari From: Snyder, Shane<mailto:[email protected]> Sent: Wednesday, July 8, 2020 10:27 AM To: Devarajan, Hariharan<mailto:[email protected]> Cc: Jeffrey Layton<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. I see. Thanks for the clarification. The only issue I'm aware of that currently causes us to lose some log data is in the case of applications calling fork(). Maybe that or something similar is happening in the import of tensorflow, with the h5/numpy I/O then happening in the child process? I can try to reproduce the issue to see if I can get a better idea of what's happening. We'd like to make sure tensorflow use cases work, but admittedly haven't really tested it. --Shane From: Devarajan, Hariharan <[email protected]<mailto:[email protected]>> Sent: Wednesday, July 8, 2020 9:45 AM To: Snyder, Shane <[email protected]<mailto:[email protected]>> Cc: Jeffrey Layton <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. It produces logs but it stops tracing h5py and np.load calls. If u run the working version, u will notice we get traces from both files but when u add import of tensorflow, this stops. I have verified that darshan is initializing as we get logs just not giving tracing for those two files in the code. Hari On Jul 8, 2020, at 9:41 AM, Snyder, Shane <[email protected]<mailto:[email protected]>> wrote: Hi Hariharan, Thanks for letting us know about this issue. I can't really think of any reason why the import of tensorflow module would result in Darshan no longer producing log files. Just to make sure I'm fully understanding, you aren't getting any logfiles at all in the case where tensorflow is imported? I ask because when using this non-MPI instrumentation, I've noticed it tends to create a lot of log files, particularly for Python modules that like to call subprocesses for using things like ls, sed, etc. I just want to make sure it's not an issue of you missing one particular log file of interest or whether you don't get any log files at all. At any rate, the environment setup looks correct in both cases to preload the Darshan library and to enable the non-MPI instrumentation support. So, that doesn't appear to be the issue. Just to verify whether Darshan is even being properly initialized/shutdown, could you try setting the DARSHAN_INTERNAL_TIMING env variable (i.e., export DARSHAN_INTERNAL_TIMING=1) before running? That should spit out some more verbose output about how long it takes Darshan to init/finalize. If you do see some additional output indicating Darshan is at least initializing, you might want to double check that there are no errors in your output logs that indicate some issue Darshan encountered. --Shane From: Darshan-users <[email protected]<mailto:[email protected]>> on behalf of Devarajan, Hariharan <[email protected]<mailto:[email protected]>> Sent: Monday, July 6, 2020 9:56 AM To: Jeffrey Layton <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. The I/O is not through tensorflow. Its through h5py and numpy.load and I verified both without tensorflow import are getting traced. I already verified that import of tensorflow doesn’t change any environment variables. Hari From: Jeffrey Layton<mailto:[email protected]> Sent: Monday, July 6, 2020 9:41 AM To: Devarajan, Hariharan<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. About 2 months ago, I tried using Darshan to trace a TensorFlow2 DL training. I could not trace the input. What I _think_ happens is the TF2 uses mmap() for reading the input files and I don't think Darshan can capture that file IO. But I'm not a Darshan expert so perhaps someone has tried this before and can help (BTW - it's possible to build TensorFlow so it doesn't use mmap() for reading files.) Jeff On Mon, Jul 6, 2020 at 1:08 PM Devarajan, Hariharan <[email protected]<mailto:[email protected]>> wrote: Hello, I was able to run my test program with 3.2.1 darshan. However, when i trace an app which load tensorflow it seems darshan doesn't produce trace. I am attaching two tars one with working example and one without. The only difference between the two is in test.py where i import tensorflow on first line. Can you please assist on how I can further debug the problem? Regards Hariharan _______________________________________________ Darshan-users mailing list [email protected]<mailto:[email protected]> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users _______________________________________________ Darshan-users mailing list [email protected]<mailto:[email protected]> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
Thanks for the update, Hari. And thanks to Sandra for the good suggestion! I opened an issue on our GitLab (https://xgitlab.cels.anl.gov/darshan/darshan/issues/284) to keep track of this, but we will try to make sure it's more clear going forward that a log contains incomplete data. Darshan does track this information, and should print some warnings out when running darshan-parser, but there's probably more that we can do to make it clear. We should probably additionally print out a suggested workaround along the lines of Sandra's. [https://xgitlab.cels.anl.gov/assets/gitlab_logo-7ae504fe4f68fdebb3c2034e36621930cd36ea87924c11ff65dbcb8ed50dca58.png]<https://xgitlab.cels.anl.gov/darshan/darshan/issues/284> darshan-util tools should be more explicit if logs contain incomplete data (#284) · Issues · darshan / darshan<https://xgitlab.cels.anl.gov/darshan/darshan/issues/284> darshan-parser has a per-module warning message printed to stdout to indicate a log contains partial data, but things like darshan-job-summary don't effectively communicate this to users. It would be nice... xgitlab.cels.anl.gov --Shane ________________________________ From: Devarajan, Hariharan <[email protected]> Sent: Wednesday, July 8, 2020 9:37 PM To: Sandra A. Mendez <[email protected]> Cc: Snyder, Shane <[email protected]>; [email protected] <[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. Ok i added the exclusion list and now i see that darshan is tracing. I will try the main app now and see if it solves the problem. Thanks I will keep you posted. Hariharan ________________________________ From: Sandra A. Mendez <[email protected]> Sent: Wednesday, July 8, 2020 5:22:43 PM To: Devarajan, Hariharan <[email protected]> Cc: Snyder, Shane <[email protected]>; [email protected] <[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. I think you can change the value in the source code in darshan-runtime (in darshan.h there is a variable, but I suppose darshan developers can provide a more appropriate solution). I had a similar case, but I decided to filter folders by using the DARSHAN_EXCLUDE_DIRS environment variable and only trace files open by the user's application and not trace files related to the python libraries. Regards, Sandra.- On Thu, 9 Jul 2020 at 00:09, Devarajan, Hariharan <[email protected]<mailto:[email protected]>> wrote: I see. There are 1024 files exactly in the darshan trace. What is the limit of the total number of files? Can we increase it? Hari From: Sandra A. Mendez<mailto:[email protected]> Sent: Wednesday, July 8, 2020 2:49 PM To: Devarajan, Hariharan<mailto:[email protected]> Cc: Snyder, Shane<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. Only a comment, Could be when you import tensorflow the number of files to trace exceeds the maximum files to trace defined by Darshan? This could be the reason that you don't see the file logs that you comment. Sandra.- On Wed, 8 Jul 2020 at 21:38, Devarajan, Hariharan <[email protected]<mailto:[email protected]>> wrote: Here are the reproducers. The working one has the tensorflow import commented (1st line). If u uncomment that, the hdf5 file and npz file stops getting traced. Rest of the py files are traced so Darshan is working. Hari From: Snyder, Shane<mailto:[email protected]> Sent: Wednesday, July 8, 2020 10:27 AM To: Devarajan, Hariharan<mailto:[email protected]> Cc: Jeffrey Layton<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. I see. Thanks for the clarification. The only issue I'm aware of that currently causes us to lose some log data is in the case of applications calling fork(). Maybe that or something similar is happening in the import of tensorflow, with the h5/numpy I/O then happening in the child process? I can try to reproduce the issue to see if I can get a better idea of what's happening. We'd like to make sure tensorflow use cases work, but admittedly haven't really tested it. --Shane From: Devarajan, Hariharan <[email protected]<mailto:[email protected]>> Sent: Wednesday, July 8, 2020 9:45 AM To: Snyder, Shane <[email protected]<mailto:[email protected]>> Cc: Jeffrey Layton <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. It produces logs but it stops tracing h5py and np.load calls. If u run the working version, u will notice we get traces from both files but when u add import of tensorflow, this stops. I have verified that darshan is initializing as we get logs just not giving tracing for those two files in the code. Hari On Jul 8, 2020, at 9:41 AM, Snyder, Shane <[email protected]<mailto:[email protected]>> wrote: Hi Hariharan, Thanks for letting us know about this issue. I can't really think of any reason why the import of tensorflow module would result in Darshan no longer producing log files. Just to make sure I'm fully understanding, you aren't getting any logfiles at all in the case where tensorflow is imported? I ask because when using this non-MPI instrumentation, I've noticed it tends to create a lot of log files, particularly for Python modules that like to call subprocesses for using things like ls, sed, etc. I just want to make sure it's not an issue of you missing one particular log file of interest or whether you don't get any log files at all. At any rate, the environment setup looks correct in both cases to preload the Darshan library and to enable the non-MPI instrumentation support. So, that doesn't appear to be the issue. Just to verify whether Darshan is even being properly initialized/shutdown, could you try setting the DARSHAN_INTERNAL_TIMING env variable (i.e., export DARSHAN_INTERNAL_TIMING=1) before running? That should spit out some more verbose output about how long it takes Darshan to init/finalize. If you do see some additional output indicating Darshan is at least initializing, you might want to double check that there are no errors in your output logs that indicate some issue Darshan encountered. --Shane From: Darshan-users <[email protected]<mailto:[email protected]>> on behalf of Devarajan, Hariharan <[email protected]<mailto:[email protected]>> Sent: Monday, July 6, 2020 9:56 AM To: Jeffrey Layton <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. The I/O is not through tensorflow. Its through h5py and numpy.load and I verified both without tensorflow import are getting traced. I already verified that import of tensorflow doesn’t change any environment variables. Hari From: Jeffrey Layton<mailto:[email protected]> Sent: Monday, July 6, 2020 9:41 AM To: Devarajan, Hariharan<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]> Subject: Re: [Darshan-users] Darshan not tracing with tensorflow import. About 2 months ago, I tried using Darshan to trace a TensorFlow2 DL training. I could not trace the input. What I _think_ happens is the TF2 uses mmap() for reading the input files and I don't think Darshan can capture that file IO. But I'm not a Darshan expert so perhaps someone has tried this before and can help (BTW - it's possible to build TensorFlow so it doesn't use mmap() for reading files.) Jeff On Mon, Jul 6, 2020 at 1:08 PM Devarajan, Hariharan <[email protected]<mailto:[email protected]>> wrote: Hello, I was able to run my test program with 3.2.1 darshan. However, when i trace an app which load tensorflow it seems darshan doesn't produce trace. I am attaching two tars one with working example and one without. The only difference between the two is in test.py where i import tensorflow on first line. Can you please assist on how I can further debug the problem? Regards Hariharan _______________________________________________ Darshan-users mailing list [email protected]<mailto:[email protected]> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users _______________________________________________ Darshan-users mailing list [email protected]<mailto:[email protected]> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
participants (4)
-
Devarajan, Hariharan -
Jeffrey Layton -
Sandra A. Mendez -
Snyder, Shane