Darshan not creating trace file and not complaining
Hi everyone, I built Darshan 3.1.4 with OpenMPI 2.0.2 and gcc 7.2.1 using the following configure flags: ./configure --with-mem-align=8 --with-log-path-by-env=DARSHAN_LOGPATH --with-jobid-env=NONE CC=mpicc --prefix=/usr/local/darshan-3.1.4-ompi I have also attached the config.log file. When I run my application with LD_PRELOAD, Darshan does not create any trace file and neither does it complain: [miahw@bedale bin]$ export DARSHAN_LOGPATH=. [miahw@bedale bin]$ LD_PRELOAD=/usr/local/darshan-3.1.4-ompi/lib/libdarshan.so mpirun -n 4 ./bt.A.4.mpi_io_full [ ... lots of application output ... ] [miahw@bedale bin]$ ls bt.A.4 bt.A.4.mpi_io_full btio.full.out inputbt.data As you can see, there is no trace file in the current working directory. Any help will be greatly appreciated. Thanks in advance, Wadud. --------------------------------------- Dr. Wadud Miah Numerical Algorithms Group Disclaimer The Numerical Algorithms Group Ltd is a company registered in England and Wales with company number 1249803. The registered office is: Wilkinson House, Jordan Hill Road, Oxford OX2 8DR, United Kingdom. This e-mail has been scanned for all viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business.
Hi Wadud, It looks like you are trying to run one of the NAS benchmark codes, which I believe are Fortran programs, right? This is due to a known "issue" with OpenMPI2 where the Fortran bindings are calling PMPI_ routines directly, bypassing Darshan's use of the MPI profiling interface (PMPI). Basically, Darshan is unable to intercept any MPI calls, including MPI_Init/Finalize which is where Darshan initializes its data structures and does the generation of the log file, respectively. No error is generated for the same reason: Darshan never initializes itself and doesn't think it should be instrumenting anything. If interested in more details, check out this github issue: https://github.com/open-mpi/ompi/issues/3954 Ignoring the philosophical question of whether the "issue" here lies with the MPI implementation or tools that use the MPI profiling interface (this issue affects all PMPI tools, not just Darshan), we've actually spent some time developing a workaround for this issue recently, so you are in luck! I have not gone through with making a pre-release version of Darshan with this workaround publicly available, though I really have been meaning to. I will definitely make sure to do that today, and will send an update to this mailing list when its ready so you (and others) are aware. So, you have a couple of options: 1.) Wait for that pre-release to come out later this afternoon, or 2.) Check out the master branch of the Darshan git repo (https://xgitlab.cels.anl.gov/darshan/darshan) Either should resolve the issue, but be sure to let us know if not. Thanks, --Shane On 10/31/2017 10:51 AM, Wadud Miah wrote:
Hi everyone,
I built Darshan 3.1.4 with OpenMPI 2.0.2 and gcc 7.2.1 using the following configure flags:
./configure --with-mem-align=8 --with-log-path-by-env=DARSHAN_LOGPATH --with-jobid-env=NONE CC=mpicc --prefix=/usr/local/darshan-3.1.4-ompi
I have also attached the config.log file. When I run my application with LD_PRELOAD, Darshan does not create any trace file and neither does it complain:
[miahw@bedale bin]$ export DARSHAN_LOGPATH=. [miahw@bedale bin]$ LD_PRELOAD=/usr/local/darshan-3.1.4-ompi/lib/libdarshan.so mpirun -n 4 ./bt.A.4.mpi_io_full [ ... lots of application output ... ] [miahw@bedale bin]$ ls bt.A.4 bt.A.4.mpi_io_full btio.full.out inputbt.data
As you can see, there is no trace file in the current working directory. Any help will be greatly appreciated.
Thanks in advance, Wadud.
---------------------------------------
Dr. Wadud Miah
Numerical Algorithms Group
*Disclaimer*
The Numerical Algorithms Group Ltd is a company registered in England and Wales with company number 1249803. The registered office is: Wilkinson House, Jordan Hill Road, Oxford OX2 8DR, United Kingdom.
This e-mail has been scanned for all viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business.
_______________________________________________ Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
Assuming users have no issues with the pre-release, this workaround will be generally available starting in Darshan 3.1.5, which we can probably release officially within the next few weeks. --Shane On 10/31/2017 11:35 AM, Shane Snyder wrote:
Hi Wadud,
It looks like you are trying to run one of the NAS benchmark codes, which I believe are Fortran programs, right?
This is due to a known "issue" with OpenMPI2 where the Fortran bindings are calling PMPI_ routines directly, bypassing Darshan's use of the MPI profiling interface (PMPI). Basically, Darshan is unable to intercept any MPI calls, including MPI_Init/Finalize which is where Darshan initializes its data structures and does the generation of the log file, respectively. No error is generated for the same reason: Darshan never initializes itself and doesn't think it should be instrumenting anything. If interested in more details, check out this github issue: https://github.com/open-mpi/ompi/issues/3954
Ignoring the philosophical question of whether the "issue" here lies with the MPI implementation or tools that use the MPI profiling interface (this issue affects all PMPI tools, not just Darshan), we've actually spent some time developing a workaround for this issue recently, so you are in luck!
I have not gone through with making a pre-release version of Darshan with this workaround publicly available, though I really have been meaning to. I will definitely make sure to do that today, and will send an update to this mailing list when its ready so you (and others) are aware. So, you have a couple of options:
1.) Wait for that pre-release to come out later this afternoon, or 2.) Check out the master branch of the Darshan git repo (https://xgitlab.cels.anl.gov/darshan/darshan)
Either should resolve the issue, but be sure to let us know if not.
Thanks, --Shane
On 10/31/2017 10:51 AM, Wadud Miah wrote:
Hi everyone,
I built Darshan 3.1.4 with OpenMPI 2.0.2 and gcc 7.2.1 using the following configure flags:
./configure --with-mem-align=8 --with-log-path-by-env=DARSHAN_LOGPATH --with-jobid-env=NONE CC=mpicc --prefix=/usr/local/darshan-3.1.4-ompi
I have also attached the config.log file. When I run my application with LD_PRELOAD, Darshan does not create any trace file and neither does it complain:
[miahw@bedale bin]$ export DARSHAN_LOGPATH=. [miahw@bedale bin]$ LD_PRELOAD=/usr/local/darshan-3.1.4-ompi/lib/libdarshan.so mpirun -n 4 ./bt.A.4.mpi_io_full [ ... lots of application output ... ] [miahw@bedale bin]$ ls bt.A.4 bt.A.4.mpi_io_full btio.full.out inputbt.data
As you can see, there is no trace file in the current working directory. Any help will be greatly appreciated.
Thanks in advance, Wadud.
---------------------------------------
Dr. Wadud Miah
Numerical Algorithms Group
*Disclaimer*
The Numerical Algorithms Group Ltd is a company registered in England and Wales with company number 1249803. The registered office is: Wilkinson House, Jordan Hill Road, Oxford OX2 8DR, United Kingdom.
This e-mail has been scanned for all viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business.
_______________________________________________ Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
_______________________________________________ Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
Hi Shane, Yes, you are right that I am trying to profile a Fortran code. Thanks for letting me know about the pre-release. I will wait for it once it is released as I don’t have an urgent need for it yet. Thanks for the quick resolution. Cheers, Wadud. From: Darshan-users [mailto:[email protected]] On Behalf Of Shane Snyder Sent: 31 October 2017 16:36 To: [email protected] Subject: Re: [Darshan-users] Darshan not creating trace file and not complaining Hi Wadud, It looks like you are trying to run one of the NAS benchmark codes, which I believe are Fortran programs, right? This is due to a known "issue" with OpenMPI2 where the Fortran bindings are calling PMPI_ routines directly, bypassing Darshan's use of the MPI profiling interface (PMPI). Basically, Darshan is unable to intercept any MPI calls, including MPI_Init/Finalize which is where Darshan initializes its data structures and does the generation of the log file, respectively. No error is generated for the same reason: Darshan never initializes itself and doesn't think it should be instrumenting anything. If interested in more details, check out this github issue: https://github.com/open-mpi/ompi/issues/3954 Ignoring the philosophical question of whether the "issue" here lies with the MPI implementation or tools that use the MPI profiling interface (this issue affects all PMPI tools, not just Darshan), we've actually spent some time developing a workaround for this issue recently, so you are in luck! I have not gone through with making a pre-release version of Darshan with this workaround publicly available, though I really have been meaning to. I will definitely make sure to do that today, and will send an update to this mailing list when its ready so you (and others) are aware. So, you have a couple of options: 1.) Wait for that pre-release to come out later this afternoon, or 2.) Check out the master branch of the Darshan git repo (https://xgitlab.cels.anl.gov/darshan/darshan) Either should resolve the issue, but be sure to let us know if not. Thanks, --Shane On 10/31/2017 10:51 AM, Wadud Miah wrote: Hi everyone, I built Darshan 3.1.4 with OpenMPI 2.0.2 and gcc 7.2.1 using the following configure flags: ./configure --with-mem-align=8 --with-log-path-by-env=DARSHAN_LOGPATH --with-jobid-env=NONE CC=mpicc --prefix=/usr/local/darshan-3.1.4-ompi I have also attached the config.log file. When I run my application with LD_PRELOAD, Darshan does not create any trace file and neither does it complain: [miahw@bedale bin]$ export DARSHAN_LOGPATH=. [miahw@bedale bin]$ LD_PRELOAD=/usr/local/darshan-3.1.4-ompi/lib/libdarshan.so mpirun -n 4 ./bt.A.4.mpi_io_full [ ... lots of application output ... ] [miahw@bedale bin]$ ls bt.A.4 bt.A.4.mpi_io_full btio.full.out inputbt.data As you can see, there is no trace file in the current working directory. Any help will be greatly appreciated. Thanks in advance, Wadud. --------------------------------------- Dr. Wadud Miah Numerical Algorithms Group Disclaimer The Numerical Algorithms Group Ltd is a company registered in England and Wales with company number 1249803. The registered office is: Wilkinson House, Jordan Hill Road, Oxford OX2 8DR, United Kingdom. This e-mail has been scanned for all viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. _______________________________________________ Darshan-users mailing list [email protected]<mailto:[email protected]> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users Disclaimer The Numerical Algorithms Group Ltd is a company registered in England and Wales with company number 1249803. The registered office is: Wilkinson House, Jordan Hill Road, Oxford OX2 8DR, United Kingdom. This e-mail has been scanned for all viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business.
participants (2)
-
Shane Snyder -
Wadud Miah