March 2015 - Darshan-users - mailman3.cels.anl.gov

Re: [Darshan-users] unable to LD_PRELOAD PMPI-based profiler with Fortran for Intel compilers?
by Phil Carns 20 Mar '15

20 Mar '15

That's right. The issue has to do with the MPI library bindings rather than the underlying compiler. Unfortunately, Fortran profiling has been problematic across multiple MPI implementations recently, though. To summarize all in one place, Darshan won't instrument Fortran in the following scenarios: - Fortran (77 or 90) on the MPICH 3.1.1 through 3.1.3 releases (see http://trac.mpich.org/projects/mpich/ticket/2209) - Fortran90 on recent IBM BG drivers (see http://lists.mcs.anl.gov/pipermail/darshan-users/2015-March/000265.html) Fortran77 is fine, though. - Fortran (77 or 90) on Intel MPI 4.x (see https://software.intel.com/en-us/forums/topic/279445) The MPICH issue has been resolved. The IBM BG issue is still present. I'm not sure of the status of Intel MPI. All of these are independent of which underlying compiler you use. thanks, -Phil On 03/20/2015 01:54 PM, Hedges, Richard M. wrote: > Should things work with Intel compilers but non-Intel MPI? > > Thanks, > - Richard > From: Phil Carns <carns(a)mcs.anl.gov <mailto:[email protected]>> > Date: Friday, March 20, 2015 at 10:35 AM > To: "darshan-users(a)lists.mcs.anl.gov > <mailto:[email protected]>" > <darshan-users(a)lists.mcs.anl.gov <mailto:[email protected]>> > Subject: Re: [Darshan-users] unable to LD_PRELOAD PMPI-based profiler > with Fortran for Intel compilers? > > Hi Richard, > > I'm honestly not sure at this point. There has been a major release > of Intel MPI (5.x) since then, so things might be different now. That > issue was observed in 4.0. > > thanks, > -Phil > > On 03/20/2015 01:26 PM, Hedges, Richard M. wrote: >> Is this still true? >> >> https://software.intel.com/en-us/forums/topic/279445 >> >> >> ==================================================== >> >> Richard Hedges >> Customer Support and Test - File Systems Project >> Development Environment Group - Livermore Computing >> Lawrence Livermore National Laboratory >> 7000 East Avenue, MS L-557 >> Livermore, CA 94551 >> >> v: (925) 423-2699 >> f: (925) 423-6961 >> E: richard-hedges(a)llnl.gov >

1 0

Re: [Darshan-users] unable to LD_PRELOAD PMPI-based profiler with Fortran for Intel compilers?
by Phil Carns 20 Mar '15

20 Mar '15

Hi Richard, I'm honestly not sure at this point. There has been a major release of Intel MPI (5.x) since then, so things might be different now. That issue was observed in 4.0. thanks, -Phil On 03/20/2015 01:26 PM, Hedges, Richard M. wrote: > Is this still true? > > https://software.intel.com/en-us/forums/topic/279445 > > > ==================================================== > > Richard Hedges > Customer Support and Test - File Systems Project > Development Environment Group - Livermore Computing > Lawrence Livermore National Laboratory > 7000 East Avenue, MS L-557 > Livermore, CA 94551 > > v: (925) 423-2699 > f: (925) 423-6961 > E: richard-hedges(a)llnl.gov

2 1

unable to LD_PRELOAD PMPI-based profiler with Fortran for Intel compilers?
by Hedges, Richard M. 20 Mar '15

20 Mar '15

Is this still true? https://software.intel.com/en-us/forums/topic/279445 ==================================================== Richard Hedges Customer Support and Test - File Systems Project Development Environment Group - Livermore Computing Lawrence Livermore National Laboratory 7000 East Avenue, MS L-557 Livermore, CA 94551 v: (925) 423-2699 f: (925) 423-6961 E: richard-hedges(a)llnl.gov

1 0

Darshan Fortran wrapper on Blue Gene Q
by Kay Thust 18 Mar '15

18 Mar '15

Hello, We installed Darshan in Juelich for our users to help them analyzing their I/O patterns and identifying potential bottlenecks. Our Blue Gene Q system (JUQUEEN) runs driver version V1R2M2. Everything works fine with the C compiler wrapper but the ones for Fortran and C++ don't seem to work. I tested both version 2.3.0 and 2.3.1-pre2. For the Fortran wrapper we have no libmpifort, libfmpich.cnk or libfmpich on our system but only libmpichf90-xl (and versions for f77, or gcc but non of the ). So I needed to change FMPICH in the wrapper. I tried different variations (empty, -lmpichf90-xl, "-lmpichf90-xl -lmpich-xl") but non of those made a simple Fortran test program use the instrumented calls of Darshan. An "nm" on the binary shows the same symbols with "darshan" in their names in both the C and the Fortran version, so it seems to be used during linking. Darshan is a great tool we would also like to offer to users using Fortran or C++ codes, too. Did anybody already successfully install Darshan on a platform with a similar setup? Since linking succeeds and the applications run I have no idea how to find the cause of this problem although the FMPICH variable seems most promising. Any ideas on how to fix this issue would be appreciated. Thanks, Kay -- Dr. Kay Thust Juelich Supercomputing Centre Institute for Advanced Simulation Forschungszentrum Juelich GmbH ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

2 2

Re: [Darshan-users] Darshan-users Digest, Vol 43, Issue 2
by Phil Carns 15 Mar '15

15 Mar '15

Hi Rajesh, I'm not familiar with those specific errors, but does this mean that the jobs did not exit cleanly? If so, that may be part of the problem. Darshan will not write a log file unless all processes call MPI_Finalize(). thanks, -Phil On 03/13/2015 02:32 PM, Rajesh Kumar S R wrote: > Hello Phil, > Yes, we found following errors in the log file. > > * [mpiexec(a)bv010.local] HYDU_sock_write (utils/sock/sock.c:286): > write error (Bad file descriptor) > [mpiexec(a)bv010.local] HYD_pmcd_pmiserv_send_signal > (pm/pmiserv/pmiserv_cb.c:169): unable to write data to proxy > [mpiexec(a)bv010.local] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:79): > unable to send signal downstream > [mpiexec(a)bv010.local] HYDT_dmxu_poll_wait_for_event > (tools/demux/demux_poll.c:76): callback returned error status > [mpiexec(a)bv010.local] HYDT_bscu_wait_for_completion > (tools/bootstrap/utils/bscu_wait.c:61): error waiting for event > [mpiexec(a)bv010.local] HYDT_bsci_wait_for_completion > (tools/bootstrap/src/bsci_wait.c:23): launcher returned error > waiting for completion > [mpiexec(a)bv010.local] HYD_pmci_wait_for_completion > (pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting > for completion > [mpiexec(a)bv010.local] main (ui/mpich/mpiexec.c:344): process > manager error waiting for completion > > Thanks, > Rajesh. > > On March 12, 2015 at 10:30 PM darshan-users-request(a)lists.mcs.anl.gov > wrote: > > Send Darshan-users mailing list submissions to > > darshan-users(a)lists.mcs.anl.gov > > > > To subscribe or unsubscribe via the World Wide Web, visit > > https://lists.mcs.anl.gov/mailman/listinfo/darshan-users > > or, via email, send a message with subject or body 'help' to > > darshan-users-request(a)lists.mcs.anl.gov > > > > You can reach the person managing the list at > > darshan-users-owner(a)lists.mcs.anl.gov > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of Darshan-users digest..." > > > > > > Today's Topics: > > > > 1. Re: No Log Files Generated - Reg. (Phil Carns) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Thu, 12 Mar 2015 10:57:00 -0400 > > From: Phil Carns <carns(a)mcs.anl.gov> > > To: <darshan-users(a)lists.mcs.anl.gov> > > Subject: Re: [Darshan-users] No Log Files Generated - Reg. > > Message-ID: <5501A93C.2010607(a)mcs.anl.gov> > > Content-Type: text/plain; charset="windows-1252"; format=flowed > > > > Hi Rajesh, > > > > Are you aware of any environment changes for those applications, like a > > new version of MPI? > > > > Also, are any error messages produced? > > > > thanks, > > -Phil > > > > On 03/12/2015 07:46 AM, SR Rajesh Kumar wrote: > > > Hello ! > > > > > > We are analyzing workloads of molecular dynamics codes like > > > gromacs, amber etc. We were successful in getting logs of them. But > > > same jobs which generated logs are not generating logs now. We are > > > using DARSHAN_LOG environment variable exported in the pbs script > along > > > with LD_PRELOAD. For some applications there are no logs generated, > > > though they are dynamically linked mpi applications. Have a look, > pls. > > > > > > Cheers, > > > Rajesh. > > > > > > > ------------------------------------------------------------------------------------------------------------------------------- > > > [ C-DAC is on Social-Media too. Kindly follow us at: > > > Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] > > > > > > This e-mail is for the sole use of the intended recipient(s) and may > > > contain confidential and privileged information. If you are not the > > > intended recipient, please contact the sender by reply e-mail and > destroy > > > all copies and the original message. Any unauthorized review, use, > > > disclosure, dissemination, forwarding, printing or copying of this > email > > > is strictly prohibited and appropriate legal action will be taken. > > > > ------------------------------------------------------------------------------------------------------------------------------- > > > > > > _______________________________________________ > > > Darshan-users mailing list > > > Darshan-users(a)lists.mcs.anl.gov > > > https://lists.mcs.anl.gov/mailman/listinfo/darshan-users > > > > > > > > ------------------------------ > > > > _______________________________________________ > > Darshan-users mailing list > > Darshan-users(a)lists.mcs.anl.gov > > https://lists.mcs.anl.gov/mailman/listinfo/darshan-users > > > > > > End of Darshan-users Digest, Vol 43, Issue 2 > > ******************************************** > > ------------------------------------------------------------------------------------------------------------------------------- > > [ C-DAC is on Social-Media too. Kindly follow us at: > Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] > > This e-mail is for the sole use of the intended recipient(s) and may > contain confidential and privileged information. If you are not the > intended recipient, please contact the sender by reply e-mail and destroy > all copies and the original message. Any unauthorized review, use, > disclosure, dissemination, forwarding, printing or copying of this email > is strictly prohibited and appropriate legal action will be taken. > -------------------------------------------------------------------------------------------------------------------------------

1 0

Re: [Darshan-users] Darshan-users Digest, Vol 43, Issue 2
by Rajesh Kumar S R 13 Mar '15

13 Mar '15

Hello Phil, Yes, we found following errors in the log file. * [mpiexec(a)bv010.local] HYDU_sock_write (utils/sock/sock.c:286): write error (Bad file descriptor) [mpiexec(a)bv010.local] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:169): unable to write data to proxy [mpiexec(a)bv010.local] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:79): unable to send signal downstream [mpiexec(a)bv010.local] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [mpiexec(a)bv010.local] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:61): error waiting for event [mpiexec(a)bv010.local] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec(a)bv010.local] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec(a)bv010.local] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion Thanks, Rajesh. On March 12, 2015 at 10:30 PM darshan-users-request(a)lists.mcs.anl.gov wrote: > Send Darshan-users mailing list submissions to > darshan-users(a)lists.mcs.anl.gov > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.mcs.anl.gov/mailman/listinfo/darshan-users > or, via email, send a message with subject or body 'help' to > darshan-users-request(a)lists.mcs.anl.gov > > You can reach the person managing the list at > darshan-users-owner(a)lists.mcs.anl.gov > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Darshan-users digest..." > > > Today's Topics: > > 1. Re: No Log Files Generated - Reg. (Phil Carns) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 12 Mar 2015 10:57:00 -0400 > From: Phil Carns <carns(a)mcs.anl.gov> > To: <darshan-users(a)lists.mcs.anl.gov> > Subject: Re: [Darshan-users] No Log Files Generated - Reg. > Message-ID: <5501A93C.2010607(a)mcs.anl.gov> > Content-Type: text/plain; charset="windows-1252"; format=flowed > > Hi Rajesh, > > Are you aware of any environment changes for those applications, like a > new version of MPI? > > Also, are any error messages produced? > > thanks, > -Phil > > On 03/12/2015 07:46 AM, SR Rajesh Kumar wrote: > > Hello ! > > > > We are analyzing workloads of molecular dynamics codes like > > gromacs, amber etc. We were successful in getting logs of them. But > > same jobs which generated logs are not generating logs now. We are > > using DARSHAN_LOG environment variable exported in the pbs script along > > with LD_PRELOAD. For some applications there are no logs generated, > > though they are dynamically linked mpi applications. Have a look, pls. > > > > Cheers, > > Rajesh. > > > > ------------------------------------------------------------------------------------------------------------------------------- > > [ C-DAC is on Social-Media too. Kindly follow us at: > > Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] > > > > This e-mail is for the sole use of the intended recipient(s) and may > > contain confidential and privileged information. If you are not the > > intended recipient, please contact the sender by reply e-mail and destroy > > all copies and the original message. Any unauthorized review, use, > > disclosure, dissemination, forwarding, printing or copying of this email > > is strictly prohibited and appropriate legal action will be taken. > > ------------------------------------------------------------------------------------------------------------------------------- > > > > _______________________________________________ > > Darshan-users mailing list > > Darshan-users(a)lists.mcs.anl.gov > > https://lists.mcs.anl.gov/mailman/listinfo/darshan-users > > > > ------------------------------ > > _______________________________________________ > Darshan-users mailing list > Darshan-users(a)lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/darshan-users > > > End of Darshan-users Digest, Vol 43, Issue 2 > ******************************************** ------------------------------------------------------------------------------------------------------------------------------- [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. -------------------------------------------------------------------------------------------------------------------------------

1 0

Darshan 2.3.1 release
by Phil Carns 13 Mar '15

13 Mar '15

Darshan version 2.3.1 is now available for download <http://press3.mcs.anl.gov/darshan/download/>. This release includes several bug fixes and enhancements, including improvements for Cray, IBM Blue Gene, and Linux cluster environments. One of the most notable new features is that Darshan now supports the MPICH profiling configuration file method for instrumentation, which simplifies installation on some platforms. This release also includes a new regression testing framework. The full changelog since 2.3.0: darshan-2.3.1 ============= * added documentation and example configuration files for using the -profile or $MPICC_PROFILE hooks to add instrumentation to MPICH-based MPI implementations without generating custom wrapper scripts * Add wrappers for mkstemp(), mkostemp(), mkstemps(), and mkostemps() (reported by Tom Peterka) * Change OPEN_TIMESTAMP field to report timestamp right before open() is invoked rather than after timestamp after open is completed. NOTE: updated log format version to 2.06 to reflect this change. * Change start_time and end_time fields in job record to use min and max (respectively) across all ranks * Fix bug in write volume data reported in file system table in darshan-job-summary.pl (reported by Matthieu Dorier) * Clean up autoconf test for zlib and make zlib mandatory (reported by Kalyana Chadalavada) * add –start-group and –end-group notation to Darshan libraries for Cray PE 2.x environment to fix link-time corner cases (Yushu Yao) * improve y axis labels on time interval graphs in darshan-job-summary.pl (reported by Tom Peterka) * misc. improvements to darshan-parser –perf output (reported by Shane Snyder) – indicate which rank was slowest in unique file results – label I/O vs. meta time more clearly – include unique file meta time in agg_perf_by_slowest calculation * added regression test script framework in darshan-test/regression/ – currently support platforms include: – Linux with static linking and generated compiler wrappers – Linux with static linking and profiler configuration files – Linux with dynamic linking and LD_PRELOAD – Blue Gene/Q with static linking and profiler configuration files * update darshan-gen-fortran.pl and darshan-gen-cxx.pl to support new library naming conventions in MPICH 3.1.1 and higher * update documentation to reflect known issues with some versions of MPICH * Cray platforms: modify darshan-runtime so that link-time instrumentation options are only used when statically linking via Libs.private. (reported by Kalyana Chadalavada) thanks, -Darshan development team

1 0

Re: [Darshan-users] No Log Files Generated - Reg.
by Phil Carns 12 Mar '15

12 Mar '15

Hi Rajesh, Are you aware of any environment changes for those applications, like a new version of MPI? Also, are any error messages produced? thanks, -Phil On 03/12/2015 07:46 AM, SR Rajesh Kumar wrote: > Hello ! > > We are analyzing workloads of molecular dynamics codes like > gromacs, amber etc. We were successful in getting logs of them. But > same jobs which generated logs are not generating logs now. We are > using DARSHAN_LOG environment variable exported in the pbs script along > with LD_PRELOAD. For some applications there are no logs generated, > though they are dynamically linked mpi applications. Have a look, pls. > > Cheers, > Rajesh. > > ------------------------------------------------------------------------------------------------------------------------------- > [ C-DAC is on Social-Media too. Kindly follow us at: > Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] > > This e-mail is for the sole use of the intended recipient(s) and may > contain confidential and privileged information. If you are not the > intended recipient, please contact the sender by reply e-mail and destroy > all copies and the original message. Any unauthorized review, use, > disclosure, dissemination, forwarding, printing or copying of this email > is strictly prohibited and appropriate legal action will be taken. > ------------------------------------------------------------------------------------------------------------------------------- > > _______________________________________________ > Darshan-users mailing list > Darshan-users(a)lists.mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/darshan-users

1 0

No Log Files Generated - Reg.
by SR Rajesh Kumar 12 Mar '15

12 Mar '15

Hello ! We are analyzing workloads of molecular dynamics codes like gromacs, amber etc. We were successful in getting logs of them. But same jobs which generated logs are not generating logs now. We are using DARSHAN_LOG environment variable exported in the pbs script along with LD_PRELOAD. For some applications there are no logs generated, though they are dynamically linked mpi applications. Have a look, pls. Cheers, Rajesh. ------------------------------------------------------------------------------------------------------------------------------- [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. -------------------------------------------------------------------------------------------------------------------------------

1 0