February 2015 - Darshan-users - mailman3.cels.anl.gov

Darshan unable to write log files
by Gunter, David O 26 Feb '15

26 Feb '15

I’m a new user to Darshan and I’m hitting a serious problem. Darshan always fails with an “unable to open log file” error message. I have tried to use a system-installed build of 2.3.0 and I’ve recently downloaded both 2.3.0 and 2.3.1-pre2 and built the runtime myself. Nothing appears to work. Here are the details. My application is built with gcc 4.4.7 and openmpi-1.6.5. The application does strict MPI I/O. It was built dynamically so I’m using the LD_PRELOAD method to get info. I used the same compiler and Open-MPI version to build Darshan. Here is the Darshan runtime config line: ./configure --with-mem-align=8 --with-log-path-by-env=DARSHAN_LOGPATH --with-jobid-env=SLURM_JOBID CC=mpicc It configures, builds and installs without error. My app writes to a Panasas file system, /scratch/dog/test_prob and I have set DARSHAN_LOGPATH to /scratch/dog/darshan_logs/ The permissions on the directory are good. My mpi job runs to completion and then I get the error message. $ mpirun -n 16 ./higrad_driver_noCBE ./params.16pe.in Total time for simulation = 69.468311 HiGrad simulation is complete!! Shutting down MPI environment! darshan library warning: unable to open log file /scratch/dog/darshan_logs/dog_higrad_driver_noCBE_id501442_2-26-49816-14771330631875741401.darshan_partial: MPI_ERR_IO: input/output error darshan library warning: unable to write log file /scratch/dog/darshan_logs/dog_higrad_driver_noCBE_id501442_2-26-49816-14771330631875741401.darshan_partial -- David Gunter HPC-5: Applications Readiness Team

3 5

Re: [Darshan-users] Darshan unable to write log files
by Phil Carns 26 Feb '15

26 Feb '15

On 02/26/2015 04:54 PM, Latham, Robert J. wrote: > > On 02/26/2015 03:16 PM, Gunter, David O wrote: >> Good call, Rob. Thanks! >> >> Saving to a non-Panasas directory worked, as did writing to Panasas with the ufs: prefix. > Ok, super strange. When darshan writes the log files, it does it > through MPI-IO, so if one can write via MPI-IO to your panasass file > system, why can't darshan? The only unusual thing that I can think of in Darshan's log-writing path is that it sets a few MPI-IO hints. Specifically, it uses this by default: "romio_no_indep_rw=true;cb_nodes=4". That triggers the use of deferred opens and limits the number of aggregators to 4. You could try turning those hints off by setting the CP_LOG_HINTS_OVERRIDE="" environment variable before running your application (and of course aiming the logs at your Panasas volume without using a ufs: prefix). Those hints should be harmless in theory, but it might be worth a shot. thanks, -Phil > the error "MPI_ERR_IO: input/output error" is not very helpful. It's > the error you get if it was not ENAMETOOLONG, ENOENT, ENOTDIR, ELOOP, > EACCES, EROFS. > > Last summer I commited some code to ROMIO to also catch EDQUOT, ENOSPC, > and EEXIST. > > But even in the "all other cases" case, ROMIO's supposed to call > strerror() and give you something -- anything! -- more helpful than "Uh, > some IO error happened" > > What MPI implementation are you using? > > ==rob > > > >> -david >> -- >> David Gunter >> HPC-5: Applications Readiness Team >> >> >> >> >>> On Feb 26, 2015, at 2:04 PM, Rob Latham <robl(a)mcs.anl.gov> wrote: >>> >>> >>> >>> On 02/26/2015 02:54 PM, Gunter, David O wrote: >>> >>>> My app writes to a Panasas file system, /scratch/dog/test_prob and I have set >>>> DARSHAN_LOGPATH to /scratch/dog/darshan_logs/ >>> this bit, about Panasas, is the only thing that looks out of the ordinary to me. >>> >>> Can you try a non-panasas file system? If not, can you try prefixing the file with ufs: (DARSHAN_LOGPATH=ufs:/scratch/dog/darshan_logs/ >>> >>> >>> Did you set up the year/month/day directories? >>> (darshan-runtime/darshan-mk-log-dirs.pl ) >>> >>> ==rob >>> >>>> The permissions on the directory are good. My mpi job runs to completion and then I get the error message. >>>> >>>> $ mpirun -n 16 ./higrad_driver_noCBE ./params.16pe.in >>>> >>>> Total time for simulation = 69.468311 >>>> HiGrad simulation is complete!! >>>> Shutting down MPI environment! >>>> darshan library warning: unable to open log file /scratch/dog/darshan_logs/dog_higrad_driver_noCBE_id501442_2-26-49816-14771330631875741401.darshan_partial: MPI_ERR_IO: input/output error >>>> darshan library warning: unable to write log file /scratch/dog/darshan_logs/dog_higrad_driver_noCBE_id501442_2-26-49816-14771330631875741401.darshan_partial >>>> >>>> -- >>>> David Gunter >>>> HPC-5: Applications Readiness Team >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Darshan-users mailing list >>>> Darshan-users(a)lists.mcs.anl.gov >>>> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users >>>> >>> -- >>> Rob Latham >>> Mathematics and Computer Science Division >>> Argonne National Lab, IL USA >> _______________________________________________ >> Darshan-users mailing list >> Darshan-users(a)lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/darshan-users >>

1 0

Darshan 2.3.1-pre2 release
by Phil Carns 08 Feb '15

08 Feb '15

Darshan 2.3.1-pre2 is now available on the download <http://press3.mcs.anl.gov/darshan/download/> page for testing and feedback. The most significant changes since 2.3.1-pre1 are better support for recent MPICH releases, examples and documentation for the -profile hooks in MPICH, and the addition of a regression testing framework. Our plan is to feature freeze at this point and focus on platform testing for the final 2.3.1 release. The full changelog since 2.3.0 is as follows: darshan-2.3.1-pre2 ============= * added documentation and example configuration files for using the -profile or $MPICC_PROFILE hooks to add instrumentation to MPICH-based MPI implementations without generating custom wrapper scripts * Add wrappers for mkstemp(), mkostemp(), mkstemps(), and mkostemps() (reported by Tom Peterka) * Change OPEN_TIMESTAMP field to report timestamp right before open() is invoked rather than after timestamp after open is completed. NOTE: updated log format version to 2.06 to reflect this change. * Change start_time and end_time fields in job record to use min and max (respectively) across all ranks * Fix bug in write volume data reported in file system table in darshan-job-summary.pl (reported by Matthieu Dorier) * Clean up autoconf test for zlib and make zlib mandatory (reported by Kalyana Chadalavada) * add --start-group and --end-group notation to Darshan libraries for Cray PE 2.x environment to fix link-time corner cases (Yushu Yao) * improve y axis labels on time interval graphs in darshan-job-summary.pl (reported by Tom Peterka) * misc. improvements to darshan-parser --perf output (reported by Shane Snyder) - indicate which rank was slowest in unique file results - label I/O vs. meta time more clearly - include unique file meta time in agg_perf_by_slowest calculation * added regression test script framework in darshan-test/regression/ - workstation-static and workstation-dynamic test environments supported * update darshan-gen-fortran.pl and darshan-gen-cxx.pl to support new library naming conventions in MPICH 3.1.1 and higher * update documentation to reflect known issues with some versions of MPICH thanks, -Darshan development team

1 0