On Mar 16, 2020, at 12:12 PM, Lisandro Dalcin <dalcinl@gmail.com> wrote:

On Mon, 16 Mar 2020 at 16:35, Jed Brown <jed@jedbrown.org> wrote:

Lisandro Dalcin <dalcinl@gmail.com> writes:

> Currently, binary viewers using POSIX file descriptors with READ mode open
> the file in ALL processes in the communicator. For WRITE mode, only process
> zero opens the file.
>
> The current PetscViewerBynaryXXX APIs make it really unnecessary to open
> the file in all processes for READ. I would like to get rid of that and
> always open on rank 0 for both READ or WROTE. 

I think we should use MPI-IO by default, and advise that people use it
whenever they can.

OK, let me look again to the details, I think a few minor things should be done before using MPI-IO as a default (like a proper subviewer implementation)

 

  I'm not sure of this suggested change, in that a
"bad for MPI-IO" workload (like each rank randomly seeking around a big
file) might not be better with rank 0 acting as a service rank.

Please note my main question is unrelated to MPI-IO. It is about the original POSIX-based implementation of binary viewers. For mode READ, all processes open the file (with the open() system call),

but in the current implementation, only process zero ever reads the file (unless the user gets the file descriptor and start issuing low-level PetscBinaryRead() calls). So I do not see the point of opening the file on all processes (and then stress metadata servers on parallel filesystem), if we are not going to ever read from rank != 0. Let's just fix things to open the file at rank==0 only. If users ever need to read in rank != 0, then can very well create the viewer on COMM_SELF, or whatever.

-- 

Lisandro Dalcin
============
Research Scientist
Extreme Computing Research Center (ECRC)
King Abdullah University of Science and Technology (KAUST)
http://ecrc.kaust.edu.sa/