March 2020 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] print statements in fortran interface functions
by Jacob Faibussowitsch 16 Mar '20

16 Mar '20

May be an issue printing to STDOUT, HPC sometimes redirect STDOUT to some other log files. Maybe try redirecting your PetscInfo() or PetscPrintf() output to a file. For PetscInfo its just a filename after the info flag: ./yourCode -info somefile . As for the PetscPrintf you will have to change those calls in your src to PetscFPrintf(). Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) Cell: (312) 694-3391 > On Mar 16, 2020, at 9:26 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > On Mon, Mar 16, 2020 at 9:50 PM Jacob Faibussowitsch <jacob.fai(a)gmail.com <mailto:[email protected]>> wrote: > Hello Mark, > > If you are using PetscInfo to print, PetscInfo requires that you pass “-info” as a command line flag in order to print anything. Sounds simple enough to miss, did you run your code with it? i.e. ./yourCode -info? > > Yea, I added -info. > > > For what its worth, PetscPrintf on PETSC_COMM_WORLD will always print but only from proc 0. PetscSynchronizedPrintf prints from all in order. Might be more useful for debugging than PetscInfo! > > Yea, I started with PetscPrintf. Started with WORLD then tried SELF. Still no output but I can step through the code and see the print statements, but nothing. > > This is on Cori/Haswell. I have now used printf on my Mac and it works. I did not do an install on my Mac, but I would not think that would matter... I should test that I suppose. > > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > Cell: (312) 694-3391 > >> On Mar 16, 2020, at 7:54 PM, Mark Adams <mfadams(a)lbl.gov <mailto:[email protected]>> wrote: >> >> I am trying to debug an application code that works with v3.7 but fails with master. The code works for "normal" solvers but for a solver that uses FieldSplit it fails. It looks like vectors are not getting created from MatCreateVecs with a matrix that is a MatNest (I can't run the code). >> >> I have put print statements in the interface files to try to see what is going on but the print statements seem to disappear somehow. (I added the same print statements in my installation of v3.7 to see what is different) I was using PetscPrint and then found and example of a Fortran PetscInfo and switched to that. I stepped through the code with DDT and see the print statements but nothing goes to the output. I did this DDT stuff with ex73f90t, which has a MatCreateNest. >> >> I am completely baffled. Does anyone have any ideas? >> >> Thanks, >> Mark >

1 0

Re: [petsc-dev] print statements in fortran interface functions
by Mark Adams 16 Mar '20

16 Mar '20

On Mon, Mar 16, 2020 at 10:04 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > Wrt fortran I/O - you can try adding calls to flush() after that. > Similarly C has fflush() > This is in zmatnestf.c. So C code that is the fortran stubs. Custom in this case. > > Satish > > > On Mon, 16 Mar 2020, Mark Adams wrote: > > > I am trying to debug an application code that works with v3.7 but fails > > with master. The code works for "normal" solvers but for a solver that > uses > > FieldSplit it fails. It looks like vectors are not getting created from > > MatCreateVecs with a matrix that is a MatNest (I can't run the code). > > > > I have put print statements in the interface files to try to see what is > > going on but the print statements seem to disappear somehow. (I added the > > same print statements in my installation of v3.7 to see what is > different) > > I was using PetscPrint and then found and example of a Fortran PetscInfo > > and switched to that. I stepped through the code with DDT and see the > print > > statements but nothing goes to the output. I did this DDT stuff with > > ex73f90t, which has a MatCreateNest. > > > > I am completely baffled. Does anyone have any ideas? > > > > Thanks, > > Mark > > > >

1 0

Re: [petsc-dev] print statements in fortran interface functions
by Mark Adams 16 Mar '20

16 Mar '20

On Mon, Mar 16, 2020 at 9:50 PM Jacob Faibussowitsch <jacob.fai(a)gmail.com> wrote: > Hello Mark, > > If you are using PetscInfo to print, PetscInfo requires that you pass > “-info” as a command line flag in order to print anything. Sounds simple > enough to miss, did you run your code with it? i.e. ./yourCode -info? > Yea, I added -info. > > For what its worth, PetscPrintf on PETSC_COMM_WORLD will always print but > only from proc 0. PetscSynchronizedPrintf prints from all in order. Might > be more useful for debugging than PetscInfo! > Yea, I started with PetscPrintf. Started with WORLD then tried SELF. Still no output but I can step through the code and see the print statements, but nothing. This is on Cori/Haswell. I have now used printf on my Mac and it works. I did not do an install on my Mac, but I would not think that would matter... I should test that I suppose. > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > Cell: (312) 694-3391 > > On Mar 16, 2020, at 7:54 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > I am trying to debug an application code that works with v3.7 but fails > with master. The code works for "normal" solvers but for a solver that uses > FieldSplit it fails. It looks like vectors are not getting created from > MatCreateVecs with a matrix that is a MatNest (I can't run the code). > > I have put print statements in the interface files to try to see what is > going on but the print statements seem to disappear somehow. (I added the > same print statements in my installation of v3.7 to see what is different) > I was using PetscPrint and then found and example of a Fortran PetscInfo > and switched to that. I stepped through the code with DDT and see the print > statements but nothing goes to the output. I did this DDT stuff with > ex73f90t, which has a MatCreateNest. > > I am completely baffled. Does anyone have any ideas? > > Thanks, > Mark > > >

1 0

Re: [petsc-dev] print statements in fortran interface functions
by Satish Balay 16 Mar '20

16 Mar '20

Wrt fortran I/O - you can try adding calls to flush() after that. Similarly C has fflush() Satish On Mon, 16 Mar 2020, Mark Adams wrote: > I am trying to debug an application code that works with v3.7 but fails > with master. The code works for "normal" solvers but for a solver that uses > FieldSplit it fails. It looks like vectors are not getting created from > MatCreateVecs with a matrix that is a MatNest (I can't run the code). > > I have put print statements in the interface files to try to see what is > going on but the print statements seem to disappear somehow. (I added the > same print statements in my installation of v3.7 to see what is different) > I was using PetscPrint and then found and example of a Fortran PetscInfo > and switched to that. I stepped through the code with DDT and see the print > statements but nothing goes to the output. I did this DDT stuff with > ex73f90t, which has a MatCreateNest. > > I am completely baffled. Does anyone have any ideas? > > Thanks, > Mark >

1 0

Re: [petsc-dev] print statements in fortran interface functions
by Jacob Faibussowitsch 16 Mar '20

16 Mar '20

Hello Mark, If you are using PetscInfo to print, PetscInfo requires that you pass “-info” as a command line flag in order to print anything. Sounds simple enough to miss, did you run your code with it? i.e. ./yourCode -info? For what its worth, PetscPrintf on PETSC_COMM_WORLD will always print but only from proc 0. PetscSynchronizedPrintf prints from all in order. Might be more useful for debugging than PetscInfo! Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) Cell: (312) 694-3391 > On Mar 16, 2020, at 7:54 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > I am trying to debug an application code that works with v3.7 but fails with master. The code works for "normal" solvers but for a solver that uses FieldSplit it fails. It looks like vectors are not getting created from MatCreateVecs with a matrix that is a MatNest (I can't run the code). > > I have put print statements in the interface files to try to see what is going on but the print statements seem to disappear somehow. (I added the same print statements in my installation of v3.7 to see what is different) I was using PetscPrint and then found and example of a Fortran PetscInfo and switched to that. I stepped through the code with DDT and see the print statements but nothing goes to the output. I did this DDT stuff with ex73f90t, which has a MatCreateNest. > > I am completely baffled. Does anyone have any ideas? > > Thanks, > Mark

1 0

print statements in fortran interface functions
by Mark Adams 16 Mar '20

16 Mar '20

I am trying to debug an application code that works with v3.7 but fails with master. The code works for "normal" solvers but for a solver that uses FieldSplit it fails. It looks like vectors are not getting created from MatCreateVecs with a matrix that is a MatNest (I can't run the code). I have put print statements in the interface files to try to see what is going on but the print statements seem to disappear somehow. (I added the same print statements in my installation of v3.7 to see what is different) I was using PetscPrint and then found and example of a Fortran PetscInfo and switched to that. I stepped through the code with DDT and see the print statements but nothing goes to the output. I did this DDT stuff with ex73f90t, which has a MatCreateNest. I am completely baffled. Does anyone have any ideas? Thanks, Mark

1 0

Re: [petsc-dev] Question about Binary-IO in READ mode with POSIX APIs
by Lisandro Dalcin 16 Mar '20

16 Mar '20

On Mon, 16 Mar 2020 at 20:26, Zhang, Hong <hongzhang(a)anl.gov> wrote: > > > I am a bit confused. Isn’t this required when one uses MPI-IO? > For MPI-IO, of course! Every process must call MPI_File_open(). I'm NOT talking/asking about the MPI-IO path. Our MPI-IO code is just fine, no objections/concerns about it. > But of course, when not using MPI-IO, only process zero should open the > file. > > That's what I was asking about. So it seems we all are agree about this. I'll go ahead and submit a MR. -- Lisandro Dalcin ============ Research Scientist Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/

1 0

Re: [petsc-dev] Question about Binary-IO in READ mode with POSIX APIs
by Jed Brown 16 Mar '20

16 Mar '20

Lisandro Dalcin <dalcinl(a)gmail.com> writes: >> I'm not sure of this suggested change, in that a >> "bad for MPI-IO" workload (like each rank randomly seeking around a big >> file) might not be better with rank 0 acting as a service rank. >> > > Please note my main question is unrelated to MPI-IO. It is about the > original POSIX-based implementation of binary viewers. For mode READ, all > processes open the file (with the open() system call), but in the current > implementation, only process zero ever reads the file (unless the user gets > the file descriptor and start issuing low-level PetscBinaryRead() calls). > So I do not see the point of opening the file on all processes (and then > stress metadata servers on parallel filesystem), if we are not going to > ever read from rank != 0. Let's just fix things to open the file at rank==0 > only. Okay, this much I agree on. If a rank doesn't actually read, it shouldn't open the file to read.

1 0

Re: [petsc-dev] Question about Binary-IO in READ mode with POSIX APIs
by Zhang, Hong 16 Mar '20

16 Mar '20

On Mar 16, 2020, at 12:12 PM, Lisandro Dalcin <dalcinl(a)gmail.com<mailto:[email protected]>> wrote: On Mon, 16 Mar 2020 at 16:35, Jed Brown <jed(a)jedbrown.org<mailto:[email protected]>> wrote: Lisandro Dalcin <dalcinl(a)gmail.com<mailto:[email protected]>> writes: > Currently, binary viewers using POSIX file descriptors with READ mode open > the file in ALL processes in the communicator. For WRITE mode, only process > zero opens the file. > > The current PetscViewerBynaryXXX APIs make it really unnecessary to open > the file in all processes for READ. I would like to get rid of that and > always open on rank 0 for both READ or WROTE. I think we should use MPI-IO by default, and advise that people use it whenever they can. OK, let me look again to the details, I think a few minor things should be done before using MPI-IO as a default (like a proper subviewer implementation) I'm not sure of this suggested change, in that a "bad for MPI-IO" workload (like each rank randomly seeking around a big file) might not be better with rank 0 acting as a service rank. Please note my main question is unrelated to MPI-IO. It is about the original POSIX-based implementation of binary viewers. For mode READ, all processes open the file (with the open() system call), I am a bit confused. Isn’t this required when one uses MPI-IO? But of course, when not using MPI-IO, only process zero should open the file. Hong (Mr.) but in the current implementation, only process zero ever reads the file (unless the user gets the file descriptor and start issuing low-level PetscBinaryRead() calls). So I do not see the point of opening the file on all processes (and then stress metadata servers on parallel filesystem), if we are not going to ever read from rank != 0. Let's just fix things to open the file at rank==0 only. If users ever need to read in rank != 0, then can very well create the viewer on COMM_SELF, or whatever. -- Lisandro Dalcin ============ Research Scientist Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/

1 0

Re: [petsc-dev] Question about Binary-IO in READ mode with POSIX APIs
by Lisandro Dalcin 16 Mar '20

16 Mar '20

On Mon, 16 Mar 2020 at 16:35, Jed Brown <jed(a)jedbrown.org> wrote: > Lisandro Dalcin <dalcinl(a)gmail.com> writes: > > > Currently, binary viewers using POSIX file descriptors with READ mode > open > > the file in ALL processes in the communicator. For WRITE mode, only > process > > zero opens the file. > > > > The current PetscViewerBynaryXXX APIs make it really unnecessary to open > > the file in all processes for READ. I would like to get rid of that and > > always open on rank 0 for both READ or WROTE. > > I think we should use MPI-IO by default, and advise that people use it > whenever they can. OK, let me look again to the details, I think a few minor things should be done before using MPI-IO as a default (like a proper subviewer implementation) > I'm not sure of this suggested change, in that a > "bad for MPI-IO" workload (like each rank randomly seeking around a big > file) might not be better with rank 0 acting as a service rank. > Please note my main question is unrelated to MPI-IO. It is about the original POSIX-based implementation of binary viewers. For mode READ, all processes open the file (with the open() system call), but in the current implementation, only process zero ever reads the file (unless the user gets the file descriptor and start issuing low-level PetscBinaryRead() calls). So I do not see the point of opening the file on all processes (and then stress metadata servers on parallel filesystem), if we are not going to ever read from rank != 0. Let's just fix things to open the file at rank==0 only. If users ever need to read in rank != 0, then can very well create the viewer on COMM_SELF, or whatever. -- Lisandro Dalcin ============ Research Scientist Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/

1 0