On 02/26/2015 04:54 PM, Latham, Robert J. wrote:

On 02/26/2015 03:16 PM, Gunter, David O wrote:

...

Good call, Rob. Thanks!

Saving to a non-Panasas directory worked, as did writing to Panasas with the ufs: prefix. Ok, super strange. When darshan writes the log files, it does it through MPI-IO, so if one can write via MPI-IO to your panasass file system, why can't darshan?

The only unusual thing that I can think of in Darshan's log-writing path is that it sets a few MPI-IO hints. Specifically, it uses this by default: "romio_no_indep_rw=true;cb_nodes=4". That triggers the use of deferred opens and limits the number of aggregators to 4. You could try turning those hints off by setting the CP_LOG_HINTS_OVERRIDE="" environment variable before running your application (and of course aiming the logs at your Panasas volume without using a ufs: prefix). Those hints should be harmless in theory, but it might be worth a shot. thanks, -Phil

the error "MPI_ERR_IO: input/output error" is not very helpful. It's the error you get if it was not ENAMETOOLONG, ENOENT, ENOTDIR, ELOOP, EACCES, EROFS.

Last summer I commited some code to ROMIO to also catch EDQUOT, ENOSPC, and EEXIST.

But even in the "all other cases" case, ROMIO's supposed to call strerror() and give you something -- anything! -- more helpful than "Uh, some IO error happened"

What MPI implementation are you using?

==rob

...

-david -- David Gunter HPC-5: Applications Readiness Team

...

On Feb 26, 2015, at 2:04 PM, Rob Latham <[email protected]> wrote:

On 02/26/2015 02:54 PM, Gunter, David O wrote:

...

My app writes to a Panasas file system, /scratch/dog/test_prob and I have set DARSHAN_LOGPATH to /scratch/dog/darshan_logs/ this bit, about Panasas, is the only thing that looks out of the ordinary to me.

Can you try a non-panasas file system? If not, can you try prefixing the file with ufs: (DARSHAN_LOGPATH=ufs:/scratch/dog/darshan_logs/

Did you set up the year/month/day directories? (darshan-runtime/darshan-mk-log-dirs.pl )

==rob

...

The permissions on the directory are good. My mpi job runs to completion and then I get the error message.

$ mpirun -n 16 ./higrad_driver_noCBE ./params.16pe.in

Total time for simulation = 69.468311 HiGrad simulation is complete!! Shutting down MPI environment! darshan library warning: unable to open log file /scratch/dog/darshan_logs/dog_higrad_driver_noCBE_id501442_2-26-49816-14771330631875741401.darshan_partial: MPI_ERR_IO: input/output error darshan library warning: unable to write log file /scratch/dog/darshan_logs/dog_higrad_driver_noCBE_id501442_2-26-49816-14771330631875741401.darshan_partial

-- David Gunter HPC-5: Applications Readiness Team

_______________________________________________ Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users

-- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

---

Darshan-users mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/darshan-users