July 2016 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] PCFIELDSPLIT MATNEST and NULL blocks
by Barry Smith 31 Jul '16

31 Jul '16

> On Jul 31, 2016, at 4:58 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > > On Sun, Jul 31, 2016 at 4:56 PM, Stefano Zampini <stefano.zampini(a)gmail.com> wrote: > It’s the first time I’m trying to setup field split so maybe I’m missing something > > I have a MatNest with NULL (1,1) block. I setup the field split using the row ISs from the MatNest (a note aside: why they cannot be inferred by FieldSplit if pc->pmat is of type MatNEST?) > Options used are -pc_fieldsplit_type additive and -pc_fieldsplit_detect_saddle_point > > You don't need ISes is you use detect_saddle_point, so use one or the other. > > I then get this error, because MatGetSubMatrix_Nest returns a NULL pointer. > What is the solution? having MatGetSubMatrix_Nest return a zero-valued matrix of correct size? any command line option I’m not using? Or instead having Fieldsplit handle the case? > > I think have Nest return a real matrix is the right solution, since this is what will happen with other types. We could perhaps provide a MATNULL type which is an implementation of Mat that supports MatMult() etc for an empty matrix and get rid of the concept of a NULL being in the Nested structure. I don't like MatCreateNest(MPI_Comm comm,PetscInt nr,const IS is_row[],PetscInt nc,const IS is_col[],const Mat a[],Mat *B) taking in Mat arguments that you built; rather based on the ISs it should create properly sized empty matrices with a MATNULL type set and then I think you should call MatNestGetSubMat(s) to set the types you want and provide preallocation information etc. Barry > > Matt > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Null argument, when expecting valid pointer > [0]PETSC ERROR: Null Object: Parameter # 1 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.7.2-744-gb54d608 GIT Date: 2016-06-28 18:20:20 +0300 > [0]PETSC ERROR: Unknown Name on a debug named kl-13916.local by szampini Sun Jul 31 14:47:53 2016 > [0]PETSC ERROR: Configure options --download-chaco --download-ctetgen --download-exodusii --download-hdf5 --download-hypre --download-metis --download-mpich --download-mumps --download-netcdf --download-p4est --download-parmetis --download-ptscotch --download-scalapack --download-suitesparse --download-triangle --with-valgrind-dir=/usr/local/ CC=/usr/local/bin/gcc-5 CXX=/usr/local/bin/g++-5 FC=/usr/local/bin/gfortran-5 PETSC_ARCH=debug > [0]PETSC ERROR: #1 MatSetOptionsPrefix() line 654 in /Users/szampini/software/petsc/src/mat/interface/matrix.c > [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 558 in /Users/szampini/software/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: #3 PCSetUp() line 968 in /Users/szampini/software/petsc/src/ksp/pc/interface/precon.c > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener

1 0

Re: [petsc-dev] PCFIELDSPLIT MATNEST and NULL blocks
by Stefano Zampini 31 Jul '16

31 Jul '16

Thanks for the quick answers. For the moment, I’ll setup a MatNest with zero valued block and wait for Jed's answer for the correct solution. On Jul 31, 2016, at 3:03 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > >> On Jul 31, 2016, at 4:56 PM, Stefano Zampini <stefano.zampini(a)gmail.com> wrote: >> >> It’s the first time I’m trying to setup field split so maybe I’m missing something >> >> I have a MatNest with NULL (1,1) block. I setup the field split using the row ISs from the MatNest (a note aside: why they cannot be inferred by FieldSplit if pc->pmat is of type MatNEST?) > > Good question, something Jed can fix :-) > >> Options used are -pc_fieldsplit_type additive and -pc_fieldsplit_detect_saddle_point > > Hmm, I don't think -pc_fieldsplit_detect_saddle_point makes sense here because that option determines the two index sets based on zeros on the diagonal in the original matrix while you are providing the two index sets yourself directly. Anyways it is probably being ignored, but don't use it. > >> >> I then get this error, because MatGetSubMatrix_Nest returns a NULL pointer. > > Yes I suspect this is a bug; I don't know what Jed's intentions is. The fieldsplit is already way to complicated I hate to see added a bunch of if (!mat) stuff to it. For now you could, as you say, pass in the correctly sized empty matrix. > > Barry > >> What is the solution? having MatGetSubMatrix_Nest return a zero-valued matrix of correct size? any command line option I’m not using? Or instead having Fieldsplit handle the case? >> >> [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> [0]PETSC ERROR: Null argument, when expecting valid pointer >> [0]PETSC ERROR: Null Object: Parameter # 1 >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.7.2-744-gb54d608 GIT Date: 2016-06-28 18:20:20 +0300 >> [0]PETSC ERROR: Unknown Name on a debug named kl-13916.local by szampini Sun Jul 31 14:47:53 2016 >> [0]PETSC ERROR: Configure options --download-chaco --download-ctetgen --download-exodusii --download-hdf5 --download-hypre --download-metis --download-mpich --download-mumps --download-netcdf --download-p4est --download-parmetis --download-ptscotch --download-scalapack --download-suitesparse --download-triangle --with-valgrind-dir=/usr/local/ CC=/usr/local/bin/gcc-5 CXX=/usr/local/bin/g++-5 FC=/usr/local/bin/gfortran-5 PETSC_ARCH=debug >> [0]PETSC ERROR: #1 MatSetOptionsPrefix() line 654 in /Users/szampini/software/petsc/src/mat/interface/matrix.c >> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 558 in /Users/szampini/software/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c >> [0]PETSC ERROR: #3 PCSetUp() line 968 in /Users/szampini/software/petsc/src/ksp/pc/interface/precon.c >> >

1 0

Re: [petsc-dev] PCFIELDSPLIT MATNEST and NULL blocks
by Barry Smith 31 Jul '16

31 Jul '16

> On Jul 31, 2016, at 4:56 PM, Stefano Zampini <stefano.zampini(a)gmail.com> wrote: > > It’s the first time I’m trying to setup field split so maybe I’m missing something > > I have a MatNest with NULL (1,1) block. I setup the field split using the row ISs from the MatNest (a note aside: why they cannot be inferred by FieldSplit if pc->pmat is of type MatNEST?) Good question, something Jed can fix :-) > Options used are -pc_fieldsplit_type additive and -pc_fieldsplit_detect_saddle_point Hmm, I don't think -pc_fieldsplit_detect_saddle_point makes sense here because that option determines the two index sets based on zeros on the diagonal in the original matrix while you are providing the two index sets yourself directly. Anyways it is probably being ignored, but don't use it. > > I then get this error, because MatGetSubMatrix_Nest returns a NULL pointer. Yes I suspect this is a bug; I don't know what Jed's intentions is. The fieldsplit is already way to complicated I hate to see added a bunch of if (!mat) stuff to it. For now you could, as you say, pass in the correctly sized empty matrix. Barry > What is the solution? having MatGetSubMatrix_Nest return a zero-valued matrix of correct size? any command line option I’m not using? Or instead having Fieldsplit handle the case? > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Null argument, when expecting valid pointer > [0]PETSC ERROR: Null Object: Parameter # 1 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.7.2-744-gb54d608 GIT Date: 2016-06-28 18:20:20 +0300 > [0]PETSC ERROR: Unknown Name on a debug named kl-13916.local by szampini Sun Jul 31 14:47:53 2016 > [0]PETSC ERROR: Configure options --download-chaco --download-ctetgen --download-exodusii --download-hdf5 --download-hypre --download-metis --download-mpich --download-mumps --download-netcdf --download-p4est --download-parmetis --download-ptscotch --download-scalapack --download-suitesparse --download-triangle --with-valgrind-dir=/usr/local/ CC=/usr/local/bin/gcc-5 CXX=/usr/local/bin/g++-5 FC=/usr/local/bin/gfortran-5 PETSC_ARCH=debug > [0]PETSC ERROR: #1 MatSetOptionsPrefix() line 654 in /Users/szampini/software/petsc/src/mat/interface/matrix.c > [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 558 in /Users/szampini/software/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: #3 PCSetUp() line 968 in /Users/szampini/software/petsc/src/ksp/pc/interface/precon.c >

1 0

Re: [petsc-dev] PCFIELDSPLIT MATNEST and NULL blocks
by Matthew Knepley 31 Jul '16

31 Jul '16

On Sun, Jul 31, 2016 at 4:56 PM, Stefano Zampini <stefano.zampini(a)gmail.com> wrote: > It’s the first time I’m trying to setup field split so maybe I’m missing > something > > I have a MatNest with NULL (1,1) block. I setup the field split using the > row ISs from the MatNest (a note aside: why they cannot be inferred by > FieldSplit if pc->pmat is of type MatNEST?) > Options used are -pc_fieldsplit_type additive and > -pc_fieldsplit_detect_saddle_point > You don't need ISes is you use detect_saddle_point, so use one or the other. > I then get this error, because MatGetSubMatrix_Nest returns a NULL pointer. > What is the solution? having MatGetSubMatrix_Nest return a zero-valued > matrix of correct size? any command line option I’m not using? Or instead > having Fieldsplit handle the case? > I think have Nest return a real matrix is the right solution, since this is what will happen with other types. Matt > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Null argument, when expecting valid pointer > [0]PETSC ERROR: Null Object: Parameter # 1 > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.7.2-744-gb54d608 GIT > Date: 2016-06-28 18:20:20 +0300 > [0]PETSC ERROR: Unknown Name on a debug named kl-13916.local by szampini > Sun Jul 31 14:47:53 2016 > [0]PETSC ERROR: Configure options --download-chaco --download-ctetgen > --download-exodusii --download-hdf5 --download-hypre --download-metis > --download-mpich --download-mumps --download-netcdf --download-p4est > --download-parmetis --download-ptscotch --download-scalapack > --download-suitesparse --download-triangle --with-valgrind-dir=/usr/local/ > CC=/usr/local/bin/gcc-5 CXX=/usr/local/bin/g++-5 > FC=/usr/local/bin/gfortran-5 PETSC_ARCH=debug > [0]PETSC ERROR: #1 MatSetOptionsPrefix() line 654 in > /Users/szampini/software/petsc/src/mat/interface/matrix.c > [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 558 in > /Users/szampini/software/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: #3 PCSetUp() line 968 in > /Users/szampini/software/petsc/src/ksp/pc/interface/precon.c > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

PCFIELDSPLIT MATNEST and NULL blocks
by Stefano Zampini 31 Jul '16

31 Jul '16

It’s the first time I’m trying to setup field split so maybe I’m missing something I have a MatNest with NULL (1,1) block. I setup the field split using the row ISs from the MatNest (a note aside: why they cannot be inferred by FieldSplit if pc->pmat is of type MatNEST?) Options used are -pc_fieldsplit_type additive and -pc_fieldsplit_detect_saddle_point I then get this error, because MatGetSubMatrix_Nest returns a NULL pointer. What is the solution? having MatGetSubMatrix_Nest return a zero-valued matrix of correct size? any command line option I’m not using? Or instead having Fieldsplit handle the case? [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Null argument, when expecting valid pointer [0]PETSC ERROR: Null Object: Parameter # 1 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.7.2-744-gb54d608 GIT Date: 2016-06-28 18:20:20 +0300 [0]PETSC ERROR: Unknown Name on a debug named kl-13916.local by szampini Sun Jul 31 14:47:53 2016 [0]PETSC ERROR: Configure options --download-chaco --download-ctetgen --download-exodusii --download-hdf5 --download-hypre --download-metis --download-mpich --download-mumps --download-netcdf --download-p4est --download-parmetis --download-ptscotch --download-scalapack --download-suitesparse --download-triangle --with-valgrind-dir=/usr/local/ CC=/usr/local/bin/gcc-5 CXX=/usr/local/bin/g++-5 FC=/usr/local/bin/gfortran-5 PETSC_ARCH=debug [0]PETSC ERROR: #1 MatSetOptionsPrefix() line 654 in /Users/szampini/software/petsc/src/mat/interface/matrix.c [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 558 in /Users/szampini/software/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #3 PCSetUp() line 968 in /Users/szampini/software/petsc/src/ksp/pc/interface/precon.c

1 0

Re: [petsc-dev] memory usage with GPUs
by Karl Rupp 29 Jul '16

29 Jul '16

>> >> The information is readily available through the nvidia-smi command line tool. > > But what about from inside the program? >> >> Is there a good reason for providing the total allocated memory on GPUs (including non-PETSc memory) through PETSc? If so, why don't we already print the total system memory usage? > > Not sure what you mean. PetscMemoryGetCurrentUsage() returns the current usage as reported by /proc or getrusage and PetscMemoryGetMaxiumUsage() reports the high-water mark. The command line option -memory_view reports this information at the end of the run. So couldn't you easily rig up the same thing for the GPU memory? apparently PETSc already *is* a computational science operating system :-) Shame on me, I wasn't aware that these functions exists. In such case it only makes sense to add the same thing for GPUs (should be easy for CUDA, but may be hard for OpenCL). Best regards, Karli

1 0

Re: [petsc-dev] memory usage with GPUs
by Barry Smith 29 Jul '16

29 Jul '16

> On Jul 29, 2016, at 11:58 AM, Karl Rupp <rupp(a)iue.tuwien.ac.at> wrote: > > >> Do you know if Nvidia provides an API to determine how much GPU >> memory has been currently allocated? >> >> >> Yes it does > > The information is readily available through the nvidia-smi command line tool. But what about from inside the program? > > Is there a good reason for providing the total allocated memory on GPUs (including non-PETSc memory) through PETSc? If so, why don't we already print the total system memory usage? Not sure what you mean. PetscMemoryGetCurrentUsage() returns the current usage as reported by /proc or getrusage and PetscMemoryGetMaxiumUsage() reports the high-water mark. The command line option -memory_view reports this information at the end of the run. So couldn't you easily rig up the same thing for the GPU memory? Barry > > Best regards, > Karli > >

1 0

Re: [petsc-dev] hypre and threads
by Barry Smith 29 Jul '16

29 Jul '16

Well at runtime you are going to need to tell OpenMP how many threads to use per process which presumably is done via an OpenMP environmental variable. Barry > On Jul 28, 2016, at 7:54 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > Cool, so just --with-openmp and we are off. I was hoping (and sort of claimed) it would easy, > Thanks, > > On Thu, Jul 28, 2016 at 3:08 PM, Satish Balay <balay(a)mcs.anl.gov> wrote: > https://bitbucket.org/petsc/petsc/commits/e872ca4eda51751516e9741c875849477… > > satish > > On Thu, 28 Jul 2016, Mark Adams wrote: > > > I have a project (XGC) that has heard that hypre supports threads and they > > are interested in using it. They would call PETSc outside of a threaded > > section, of course, but can hypre pick up the threads in some way? > > > > Mark > > > >

1 0

Re: [petsc-dev] memory usage with GPUs
by Karl Rupp 29 Jul '16

29 Jul '16

> Do you know if Nvidia provides an API to determine how much GPU > memory has been currently allocated? > > > Yes it does The information is readily available through the nvidia-smi command line tool. Is there a good reason for providing the total allocated memory on GPUs (including non-PETSc memory) through PETSc? If so, why don't we already print the total system memory usage? Best regards, Karli

1 0

Re: [petsc-dev] memory usage with GPUs
by Dave May 29 Jul '16

29 Jul '16

On Friday, 29 July 2016, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Do you know if Nvidia provides an API to determine how much GPU memory > has been currently allocated? Yes it does > > > If it does then that could easily be hooked up in same style as the > PetscMemory...() routines for example PetscGPUMemory...(). If they don't > provide an API then obviously there is no way to get that data. > > Barry > > > On Jul 27, 2016, at 8:18 PM, Mark Adams <mfadams(a)lbl.gov <javascript:;>> > wrote: > > > > Note, I wanted to use this for a Petsc code that uses GPUs but not > through at Petsc. > > > > So my case is a little odd and I did not expect support but I just > wanted to check. > > > > On Wednesday, July 27, 2016, Karl Rupp <rupp(a)iue.tuwien.ac.at > <javascript:;>> wrote: > > > > We (meanings someone) should add logging of GPU allocations > separately. > > > > Agreed. I'll do it (though not immediately, as there are other things > with higher priority to make GPU support in PETSc more useful) > > > > Best regards, > > Karli > > > > > > > > On Jul 27, 2016, at 1:41 PM, Matthew Knepley <knepley(a)gmail.com > <javascript:;>> wrote: > > > > No, although since we mirror storage you could get an idea if you broke > out the GPU vecs into a separate stage. > > > > Matt > > > > On Wed, Jul 27, 2016 at 9:05 AM, Mark Adams <mfadams(a)lbl.gov > <javascript:;>> wrote: > > Would/could PETSc memory usage methods (PetscMemoryGetMaximumUsage) pick > up GPU memory usage? > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > > >

1 0