I'm trying to get Darshan running on a RedHat/X86_64 cluster with a few different flavors of MPI available. I'm most recently trying Intel MPI and gfortran (Intel Compilers didn't seem to work) and after a fair bit of tweaking, got the executable recompiled with libdarshan.so successfully linked in. In my job run script, I verify that libdarshan is linked in and the job runs to completion; however no output is generated. I also tried setting DARSHAN_INTERNAL_TIMING=1, but still no output. Should I be able to use Intel MPI and/or is there a place that documents which MPI stacks are supported? -- _____________________ Cameron Harr | SAIC | C4IT BU Systems Analyst | FNMOC phone: 831.656.4154 mobile: 831.233.8216 email: [email protected]<mailto:[email protected]> email: [email protected]<mailto:[email protected]>
I don't have any first-hand experience with Intel MPI, but in principle it should work. Does your MPI compiler produces dynamic executables by default? If so, can you try leaving your executable/binary unmodified and load the libdarshan.so library at runtime using the LD_PRELOAD environment variable, rather than directly linking it to your application? There is some more detail about this approach on this page: http://wiki.mcs.anl.gov/Darshan/index.php/Dynamic_linking thanks, -Phil On 02/20/2012 01:28 PM, Harr, Cameron Contractor, SAIC wrote:
I'm trying to get Darshan running on a RedHat/X86_64 cluster with a few different flavors of MPI available. I'm most recently trying Intel MPI and gfortran (Intel Compilers didn't seem to work) and after a fair bit of tweaking, got the executable recompiled with libdarshan.so successfully linked in. In my job run script, I verify that libdarshan is linked in and the job runs to completion; however no output is generated. I also tried setting DARSHAN_INTERNAL_TIMING=1, but still no output. Should I be able to use Intel MPI and/or is there a place that documents which MPI stacks are supported? -- _____________________ *Cameron Harr *| SAIC | C4IT BU Systems Analyst | FNMOC phone: 831.656.4154 mobile: 831.233.8216 email: [email protected] email: [email protected]
_______________________________________________ Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
Thanks Phil. Using the PRELOAD, I can get a sample output with one of the test C programs that comes with the Darshan source; however, I still don't get any output from my executable. Let me ask a basic question: My app is all Fortran; do I need to compile in any support into Darshan for Fortran compatibility? I see that it should support Fortran, but perhaps I'm doing something wrong. On 02/20/2012 10:44 AM, Phil Carns wrote: I don't have any first-hand experience with Intel MPI, but in principle it should work. Does your MPI compiler produces dynamic executables by default? If so, can you try leaving your executable/binary unmodified and load the libdarshan.so library at runtime using the LD_PRELOAD environment variable, rather than directly linking it to your application? There is some more detail about this approach on this page: http://wiki.mcs.anl.gov/Darshan/index.php/Dynamic_linking thanks, -Phil On 02/20/2012 01:28 PM, Harr, Cameron Contractor, SAIC wrote: I'm trying to get Darshan running on a RedHat/X86_64 cluster with a few different flavors of MPI available. I'm most recently trying Intel MPI and gfortran (Intel Compilers didn't seem to work) and after a fair bit of tweaking, got the executable recompiled with libdarshan.so successfully linked in. In my job run script, I verify that libdarshan is linked in and the job runs to completion; however no output is generated. I also tried setting DARSHAN_INTERNAL_TIMING=1, but still no output. Should I be able to use Intel MPI and/or is there a place that documents which MPI stacks are supported? _______________________________________________ Darshan-users mailing list [email protected]<mailto:[email protected]> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users _______________________________________________ Darshan-users mailing list [email protected]<mailto:[email protected]> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
what interfaces are you using for I/O? Fortran calls, MPI_File_* routines, something else? kevin On Feb 20, 2012, at 1:42 PM, Harr, Cameron Contractor, SAIC wrote:
Thanks Phil. Using the PRELOAD, I can get a sample output with one of the test C programs that comes with the Darshan source; however, I still don't get any output from my executable. Let me ask a basic question: My app is all Fortran; do I need to compile in any support into Darshan for Fortran compatibility? I see that it should support Fortran, but perhaps I'm doing something wrong.
On 02/20/2012 10:44 AM, Phil Carns wrote:
I don't have any first-hand experience with Intel MPI, but in principle it should work.
Does your MPI compiler produces dynamic executables by default? If so, can you try leaving your executable/binary unmodified and load the libdarshan.so library at runtime using the LD_PRELOAD environment variable, rather than directly linking it to your application? There is some more detail about this approach on this page:
http://wiki.mcs.anl.gov/Darshan/index.php/Dynamic_linking
thanks, -Phil
On 02/20/2012 01:28 PM, Harr, Cameron Contractor, SAIC wrote:
I'm trying to get Darshan running on a RedHat/X86_64 cluster with a few different flavors of MPI available. I'm most recently trying Intel MPI and gfortran (Intel Compilers didn't seem to work) and after a fair bit of tweaking, got the executable recompiled with libdarshan.so successfully linked in. In my job run script, I verify that libdarshan is linked in and the job runs to completion; however no output is generated. I also tried setting DARSHAN_INTERNAL_TIMING=1, but still no output. Should I be able to use Intel MPI and/or is there a place that documents which MPI stacks are supported?
_______________________________________________ Darshan-users mailing list
[email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
_______________________________________________ Darshan-users mailing list
[email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
Interface is all Fortran calls, not using MPI_IO. On 02/20/2012 11:48 AM, Kevin Harms wrote:
what interfaces are you using for I/O? Fortran calls, MPI_File_* routines, something else?
kevin
On Feb 20, 2012, at 1:42 PM, Harr, Cameron Contractor, SAIC wrote:
Thanks Phil. Using the PRELOAD, I can get a sample output with one of the test C programs that comes with the Darshan source; however, I still don't get any output from my executable. Let me ask a basic question: My app is all Fortran; do I need to compile in any support into Darshan for Fortran compatibility? I see that it should support Fortran, but perhaps I'm doing something wrong.
On 02/20/2012 10:44 AM, Phil Carns wrote:
I don't have any first-hand experience with Intel MPI, but in principle it should work.
Does your MPI compiler produces dynamic executables by default? If so, can you try leaving your executable/binary unmodified and load the libdarshan.so library at runtime using the LD_PRELOAD environment variable, rather than directly linking it to your application? There is some more detail about this approach on this page:
http://wiki.mcs.anl.gov/Darshan/index.php/Dynamic_linking
thanks, -Phil
On 02/20/2012 01:28 PM, Harr, Cameron Contractor, SAIC wrote:
I'm trying to get Darshan running on a RedHat/X86_64 cluster with a few different flavors of MPI available. I'm most recently trying Intel MPI and gfortran (Intel Compilers didn't seem to work) and after a fair bit of tweaking, got the executable recompiled with libdarshan.so successfully linked in. In my job run script, I verify that libdarshan is linked in and the job runs to completion; however no output is generated. I also tried setting DARSHAN_INTERNAL_TIMING=1, but still no output. Should I be able to use Intel MPI and/or is there a place that documents which MPI stacks are supported?
_______________________________________________ Darshan-users mailing list
[email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
_______________________________________________ Darshan-users mailing list
[email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
The program is calling MPI_INIT and MPI_FINALIZE, though right? Fortran I/O should be captured just fine, but libdarshan.so still relies on the MPI initialization and finalization routines to start the tool up and write the log file. Assuming that the code is calling both of the above routines, it may be that there is some additional step needed with Intel MPI for it to use a library that relies on the MPI profiling (PMPI) interface. When Darshan is preloaded with LD_PRELOAD, it intercepts POSIX (Fortran I/O in this case) calls directly, but for the MPI calls it still uses the profiling interface to MPI. Have you tried using any more generic profiling libraries (like mpiP or mpitrace) by any chance? -Phil On 02/20/2012 02:55 PM, Harr, Cameron Contractor, SAIC wrote:
Interface is all Fortran calls, not using MPI_IO.
On 02/20/2012 11:48 AM, Kevin Harms wrote:
what interfaces are you using for I/O? Fortran calls, MPI_File_* routines, something else?
kevin
On Feb 20, 2012, at 1:42 PM, Harr, Cameron Contractor, SAIC wrote:
Thanks Phil. Using the PRELOAD, I can get a sample output with one of the test C programs that comes with the Darshan source; however, I still don't get any output from my executable. Let me ask a basic question: My app is all Fortran; do I need to compile in any support into Darshan for Fortran compatibility? I see that it should support Fortran, but perhaps I'm doing something wrong.
On 02/20/2012 10:44 AM, Phil Carns wrote:
I don't have any first-hand experience with Intel MPI, but in principle it should work.
Does your MPI compiler produces dynamic executables by default? If so, can you try leaving your executable/binary unmodified and load the libdarshan.so library at runtime using the LD_PRELOAD environment variable, rather than directly linking it to your application? There is some more detail about this approach on this page:
http://wiki.mcs.anl.gov/Darshan/index.php/Dynamic_linking
thanks, -Phil
On 02/20/2012 01:28 PM, Harr, Cameron Contractor, SAIC wrote:
I'm trying to get Darshan running on a RedHat/X86_64 cluster with a few different flavors of MPI available. I'm most recently trying Intel MPI and gfortran (Intel Compilers didn't seem to work) and after a fair bit of tweaking, got the executable recompiled with libdarshan.so successfully linked in. In my job run script, I verify that libdarshan is linked in and the job runs to completion; however no output is generated. I also tried setting DARSHAN_INTERNAL_TIMING=1, but still no output. Should I be able to use Intel MPI and/or is there a place that documents which MPI stacks are supported?
_______________________________________________ Darshan-users mailing list
[email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
_______________________________________________ Darshan-users mailing list
[email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
_______________________________________________ Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
Yes, it does call MPI_INIT and _FINALIZE. Following your comments, I looked at several different versions of the executable compiled with different MPI stacks. Only the one compiled with MVAPICH had PMPI symbols (per the nm -A command). Are those symbols necessary for Darshan to work? On 02/20/2012 01:08 PM, Phil Carns wrote:
The program is calling MPI_INIT and MPI_FINALIZE, though right? Fortran I/O should be captured just fine, but libdarshan.so still relies on the MPI initialization and finalization routines to start the tool up and write the log file.
Assuming that the code is calling both of the above routines, it may be that there is some additional step needed with Intel MPI for it to use a library that relies on the MPI profiling (PMPI) interface. When Darshan is preloaded with LD_PRELOAD, it intercepts POSIX (Fortran I/O in this case) calls directly, but for the MPI calls it still uses the profiling interface to MPI.
Have you tried using any more generic profiling libraries (like mpiP or mpitrace) by any chance?
-Phil
On 02/20/2012 02:55 PM, Harr, Cameron Contractor, SAIC wrote:
Interface is all Fortran calls, not using MPI_IO.
On 02/20/2012 11:48 AM, Kevin Harms wrote:
what interfaces are you using for I/O? Fortran calls, MPI_File_* routines, something else?
kevin
On Feb 20, 2012, at 1:42 PM, Harr, Cameron Contractor, SAIC wrote:
Thanks Phil. Using the PRELOAD, I can get a sample output with one of the test C programs that comes with the Darshan source; however, I still don't get any output from my executable. Let me ask a basic question: My app is all Fortran; do I need to compile in any support into Darshan for Fortran compatibility? I see that it should support Fortran, but perhaps I'm doing something wrong.
On 02/20/2012 10:44 AM, Phil Carns wrote:
I don't have any first-hand experience with Intel MPI, but in principle it should work.
Does your MPI compiler produces dynamic executables by default? If so, can you try leaving your executable/binary unmodified and load the libdarshan.so library at runtime using the LD_PRELOAD environment variable, rather than directly linking it to your application? There is some more detail about this approach on this page:
http://wiki.mcs.anl.gov/Darshan/index.php/Dynamic_linking
thanks, -Phil
On 02/20/2012 01:28 PM, Harr, Cameron Contractor, SAIC wrote:
I'm trying to get Darshan running on a RedHat/X86_64 cluster with a few different flavors of MPI available. I'm most recently trying Intel MPI and gfortran (Intel Compilers didn't seem to work) and after a fair bit of tweaking, got the executable recompiled with libdarshan.so successfully linked in. In my job run script, I verify that libdarshan is linked in and the job runs to completion; however no output is generated. I also tried setting DARSHAN_INTERNAL_TIMING=1, but still no output. Should I be able to use Intel MPI and/or is there a place that documents which MPI stacks are supported?
_______________________________________________ Darshan-users mailing list
[email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
_______________________________________________ Darshan-users mailing list
[email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
_______________________________________________ Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
_______________________________________________ Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
No, that part might be normal. I think that stock builds with MPICH also do not produce any PMPI symbols unless you link in a library that is using the profiling interface. -Phil On 02/20/2012 04:50 PM, Harr, Cameron Contractor, SAIC wrote:
Yes, it does call MPI_INIT and _FINALIZE. Following your comments, I looked at several different versions of the executable compiled with different MPI stacks. Only the one compiled with MVAPICH had PMPI symbols (per the nm -A command). Are those symbols necessary for Darshan to work?
On 02/20/2012 01:08 PM, Phil Carns wrote:
The program is calling MPI_INIT and MPI_FINALIZE, though right? Fortran I/O should be captured just fine, but libdarshan.so still relies on the MPI initialization and finalization routines to start the tool up and write the log file.
Assuming that the code is calling both of the above routines, it may be that there is some additional step needed with Intel MPI for it to use a library that relies on the MPI profiling (PMPI) interface. When Darshan is preloaded with LD_PRELOAD, it intercepts POSIX (Fortran I/O in this case) calls directly, but for the MPI calls it still uses the profiling interface to MPI.
Have you tried using any more generic profiling libraries (like mpiP or mpitrace) by any chance?
-Phil
On 02/20/2012 02:55 PM, Harr, Cameron Contractor, SAIC wrote:
Interface is all Fortran calls, not using MPI_IO.
On 02/20/2012 11:48 AM, Kevin Harms wrote:
what interfaces are you using for I/O? Fortran calls, MPI_File_* routines, something else?
kevin
On Feb 20, 2012, at 1:42 PM, Harr, Cameron Contractor, SAIC wrote:
Thanks Phil. Using the PRELOAD, I can get a sample output with one of the test C programs that comes with the Darshan source; however, I still don't get any output from my executable. Let me ask a basic question: My app is all Fortran; do I need to compile in any support into Darshan for Fortran compatibility? I see that it should support Fortran, but perhaps I'm doing something wrong.
On 02/20/2012 10:44 AM, Phil Carns wrote:
I don't have any first-hand experience with Intel MPI, but in principle it should work.
Does your MPI compiler produces dynamic executables by default? If so, can you try leaving your executable/binary unmodified and load the libdarshan.so library at runtime using the LD_PRELOAD environment variable, rather than directly linking it to your application? There is some more detail about this approach on this page:
http://wiki.mcs.anl.gov/Darshan/index.php/Dynamic_linking
thanks, -Phil
On 02/20/2012 01:28 PM, Harr, Cameron Contractor, SAIC wrote:
I'm trying to get Darshan running on a RedHat/X86_64 cluster with a few different flavors of MPI available. I'm most recently trying Intel MPI and gfortran (Intel Compilers didn't seem to work) and after a fair bit of tweaking, got the executable recompiled with libdarshan.so successfully linked in. In my job run script, I verify that libdarshan is linked in and the job runs to completion; however no output is generated. I also tried setting DARSHAN_INTERNAL_TIMING=1, but still no output. Should I be able to use Intel MPI and/or is there a place that documents which MPI stacks are supported?
_______________________________________________ Darshan-users mailing list
[email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
Darshan-users mailing list
[email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
_______________________________________________ Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
Hi all, Cameron, Kevin, and I have continued this discussion off list, but I wanted to post a quick update here for the benefit of the mailing list archives. With the Darshan 2.2.0-pre1 pre-release we are able to instrument C and C++ programs built using Intel MPI just fine, but we are having trouble intercepting symbols from Fortran programs. We are checking with Intel support now to see if they can help. thanks, -Phil On 02/22/2012 09:25 AM, Phil Carns wrote:
No, that part might be normal. I think that stock builds with MPICH also do not produce any PMPI symbols unless you link in a library that is using the profiling interface.
-Phil
On 02/20/2012 04:50 PM, Harr, Cameron Contractor, SAIC wrote:
Yes, it does call MPI_INIT and _FINALIZE. Following your comments, I looked at several different versions of the executable compiled with different MPI stacks. Only the one compiled with MVAPICH had PMPI symbols (per the nm -A command). Are those symbols necessary for Darshan to work?
On 02/20/2012 01:08 PM, Phil Carns wrote:
The program is calling MPI_INIT and MPI_FINALIZE, though right? Fortran I/O should be captured just fine, but libdarshan.so still relies on the MPI initialization and finalization routines to start the tool up and write the log file.
Assuming that the code is calling both of the above routines, it may be that there is some additional step needed with Intel MPI for it to use a library that relies on the MPI profiling (PMPI) interface. When Darshan is preloaded with LD_PRELOAD, it intercepts POSIX (Fortran I/O in this case) calls directly, but for the MPI calls it still uses the profiling interface to MPI.
Have you tried using any more generic profiling libraries (like mpiP or mpitrace) by any chance?
-Phil
On 02/20/2012 02:55 PM, Harr, Cameron Contractor, SAIC wrote:
Interface is all Fortran calls, not using MPI_IO.
On 02/20/2012 11:48 AM, Kevin Harms wrote:
what interfaces are you using for I/O? Fortran calls, MPI_File_* routines, something else?
kevin
On Feb 20, 2012, at 1:42 PM, Harr, Cameron Contractor, SAIC wrote:
Thanks Phil. Using the PRELOAD, I can get a sample output with one of the test C programs that comes with the Darshan source; however, I still don't get any output from my executable. Let me ask a basic question: My app is all Fortran; do I need to compile in any support into Darshan for Fortran compatibility? I see that it should support Fortran, but perhaps I'm doing something wrong.
On 02/20/2012 10:44 AM, Phil Carns wrote:
I don't have any first-hand experience with Intel MPI, but in principle it should work.
Does your MPI compiler produces dynamic executables by default? If so, can you try leaving your executable/binary unmodified and load the libdarshan.so library at runtime using the LD_PRELOAD environment variable, rather than directly linking it to your application? There is some more detail about this approach on this page:
http://wiki.mcs.anl.gov/Darshan/index.php/Dynamic_linking
thanks, -Phil
On 02/20/2012 01:28 PM, Harr, Cameron Contractor, SAIC wrote:
I'm trying to get Darshan running on a RedHat/X86_64 cluster with a few different flavors of MPI available. I'm most recently trying Intel MPI and gfortran (Intel Compilers didn't seem to work) and after a fair bit of tweaking, got the executable recompiled with libdarshan.so successfully linked in. In my job run script, I verify that libdarshan is linked in and the job runs to completion; however no output is generated. I also tried setting DARSHAN_INTERNAL_TIMING=1, but still no output. Should I be able to use Intel MPI and/or is there a place that documents which MPI stacks are supported?
_______________________________________________ Darshan-users mailing list
[email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
Darshan-users mailing list
[email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
_______________________________________________ Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
_______________________________________________ Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users
participants (3)
-
Harr, Cameron Contractor, SAIC -
Kevin Harms -
Phil Carns