February 2013 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] questions on new include organization
by Jed Brown 13 Feb '13

13 Feb '13

On Wed, Feb 13, 2013 at 7:31 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > 1) ISGetIndices(is,int group_number, &indices) gives the indices for a > group or > > > > So by convention, group 0 is "offset" and group 1 is "size"? > > No, group_number is the number of the clump. and indices are all the > values in that clump Then you need to also access the size of the clump, but since the primary purpose is for indexing into another array, you have to copy out a buffer (thus requiring a matching Restore). When using (int) -> (offset, size), we get direct access. It makes this inner loop code a lot simpler and faster. Consider DMPlexPointLocalRef(), which is effectively ierr = PetscSectionGetOffset(dm->defaultGlobalSection,point,&offset);CHKERRQ(ierr); *(PetscScalar**)ptr = (start >= 0) ? array + start - dm->map->rstart : NULL; This gives read+write access to that piece of the array. If PetscSectionGetOffset instead returned discontiguous indices, we'd need a work buffer and copy-back semantics, or we'd need to push that unstructured indexing burden onto the user.

1 0

Re: [petsc-dev] questions on new include organization
by Barry Smith 13 Feb '13

13 Feb '13

Regardless of exactly how this shakes out I think you both have to agree that PetscSection is a bit of an oddball and it should be more "integrated" with the "IS stuff" in that we have a single source code location (directory) and set of concepts related to indexing things. And don't have some in the Vec directory. So, for now, I won't change names or functionality but would like permission to move source around. Who knows, maybe in the end the is directory will get a more suitable name. Barry As you know I really really like having names that convey connections left to right, KSPGMRES, PC_ILU etc. I think this helps make the learning and understanding curve lower. Now people see IS and PetscSection and they are two completely unrelated things to their eyes but in fact they are not unrelated and I would like to convey that somehow in the future. BTW: I consider it a terrible tragedy that in (for example C++ and Java) one can define a subclass of a class and just use a completely arbitrary ASCII name for the subclass completely unrelated to the class it is derived from, talk about losing information. On Feb 13, 2013, at 6:40 AM, Matthew Knepley <knepley(a)gmail.com> wrote: > On Wed, Feb 13, 2013 at 12:45 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > > On Tue, Feb 12, 2013 at 11:34 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > Is it? Maybe a more general definition of PetscSection would be better as map from (int) -> IS Matt's current usage is very restricted to only contiguous blocks of elements in a group. Need that be it? Should that be it? > > There is some hackery to ascribe "field" structure to the output of the map, but I'm skeptical of whether it belongs at this level. > > The granularity is normally very small so we certainly don't want to pay the memory or performance overhead of an IS for each output. Although I'm sure we can think up scenarios where that flexibility would be useful, I think generality in the outputs here would add significant complexity with limited practical return. > > Note that PetscSection is currently accessed using function calls in inner loops. This already accounts for a noticeable amount of time for a CFD application. So we either need a coarser grained access API or we need a non-polymorphic accessor for use in hot access sites. (This is a rare case where indirect function call overhead matters.) > > I think its helpful to look at the one universal use case right now: > > You would like to describe a function over a mesh, so > associate functions with each part of the mesh and > store coefficients of these functions. > This happens for: > mesh geometry > PDE solutions and auxiliary fields (like viscosity) > the mesh itself (the adjacency is a field over the points) > > The inner loops Jed is referring to are structured like this > > VecGetArray(sol, &solArray); > Loop over pieces (like cell, face, etc.) > Get solution on this piece: > PetscSectionGetDof(solSection, piece, &dof), PetscSectionGetOffset(solSection, piece, &off) > pieceArray = &solArray[off]; > Jed condenses this to DMPlexPointLocalRead(solDM, piece, solArray, &pieceArray) > > Thus this adds structure to the Vec sol by breaking it into irregular pieces. The lower level > function Dof/Offset() are definitely necessary so you can reason about the operations. I think > you could get away with only the array offsetting being fast, but I don't see how you make that > fast and those slow. > > Barry: > Is it? Maybe a more general definition of PetscSection would be better as map from (int) -> IS Matt's current usage is very restricted to only contiguous blocks of elements in a group. Need that be it? Should that be it? > > I think you are missing the point here. I use this all over the place. This is PetscSection+IS. You put your > non-contiguous indices in the IS and index into it using PetscSection. PetscSection is the composable part > of this indexing-by-groups. It can be combined with IS or Vec or anything. I think it is the right building block. > > Matt > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener

1 0

Re: [petsc-dev] questions on new include organization
by Barry Smith 13 Feb '13

13 Feb '13

On Feb 12, 2013, at 11:35 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > > On Tue, Feb 12, 2013 at 11:31 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > Then would depend on how we would define how the grouping is handled and how the GetIndices would be used. Some possibilities > > 1) ISGetIndices(is,int group_number, &indices) gives the indices for a group or > > So by convention, group 0 is "offset" and group 1 is "size"? No, group_number is the number of the clump. and indices are all the values in that clump > Do we need an arbitrary number of groups? (Making the object more general makes it harder to reason about what it does and harder to write non-implementation code that manipulates these objects.) > > > 2) ISGetIndices(is,&indices) simply returns all the indices for all the groups > > That would need to be packed somehow.

1 0

Re: [petsc-dev] removing PETSC_NULL from petsc-dev
by Barry Smith 13 Feb '13

13 Feb '13

On Feb 13, 2013, at 6:43 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > > On Thu, Feb 7, 2013 at 9:56 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > I have removed the use of PETSC_NULL from all PETSc C code. Please remember to just use NULL in PETSc source code in the future (or Karl's script will find you). > > I see you changed to > > #define PETSC_IGNORE NULL > > but kept > > #define PETSC_NULL 0 > > Should the second be updated to > > #define PETSC_NULL NULL > > because it is stricter about passing PETSC_NULL for an integer argument? Yes, I was hesitant to change too much at once. It is fine to change and fine to add the damn deprecated message. Barry > > Also, we could do something like this (properly guarded, of course): > > __attribute__((deprecated)) PETSC_UNUSED > PETSC_STATIC_INLINE void PETSC_NULL_IS_DEPRECATED_JUST_USE_NULL() {}; > #define PETSC_NULL (PETSC_NULL_IS_DEPRECATED_JUST_USE_NULL(),NULL) > > which can be used for any compiler that supports non-static statements in declarations (everything but Sun and MSVC?) and has __attribute__((deprecated)). This produces warnings like this under Clang > > ex5.c:102:36: warning: 'PETSC_NULL_IS_DEPRECATED_JUST_USE_NULL' is deprecated [-Wdeprecated-declarations] > ierr = PetscOptionsGetReal(PETSC_NULL,"-par",&user.param,NULL);CHKERRQ(ierr); > ^ > /home/jed/petsc/include/petscsys.h:387:31: note: expanded from macro 'PETSC_NULL' > #define PETSC_NULL (PETSC_NULL_IS_DEPRECATED_JUST_USE_NULL(),NULL) > ^ > /home/jed/petsc/include/petscsys.h:385:33: note: 'PETSC_NULL_IS_DEPRECATED_JUST_USE_NULL' declared here > PETSC_UNUSED static inline void PETSC_NULL_IS_DEPRECATED_JUST_USE_NULL() {}; > ^ > 1 warning generated. > > and like this with GCC: > > ex5.c: In function ‘main’: > ex5.c:102:3: warning: ‘PETSC_NULL_IS_DEPRECATED_JUST_USE_NULL’ is deprecated (declared at /home/jed/petsc/include/petscsys.h:385) [-Wdeprecated-declarations] > > > We can make the error message cleaner (but more confusing) with > > __attribute__((deprecated)) > PETSC_UNUSED static inline void PETSC_NULL() {}; > #define PETSC_NULL (PETSC_NULL(),NULL) > > which produces: > > ex5.c: In function ‘main’: > ex5.c:102:3: warning: ‘PETSC_NULL’ is deprecated (declared at /home/jed/petsc/include/petscsys.h:385) [-Wdeprecated-declarations] > > > Any sense in this stuff?

1 0

Re: [petsc-dev] removing PETSC_NULL from petsc-dev
by Jed Brown 13 Feb '13

13 Feb '13

On Thu, Feb 7, 2013 at 9:56 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > I have removed the use of PETSC_NULL from all PETSc C code. Please > remember to just use NULL in PETSc source code in the future (or Karl's > script will find you). > I see you changed to #define PETSC_IGNORE NULL but kept #define PETSC_NULL 0 Should the second be updated to #define PETSC_NULL NULL because it is stricter about passing PETSC_NULL for an integer argument? Also, we could do something like this (properly guarded, of course): __attribute__((deprecated)) PETSC_UNUSED PETSC_STATIC_INLINE void PETSC_NULL_IS_DEPRECATED_JUST_USE_NULL() {}; #define PETSC_NULL (PETSC_NULL_IS_DEPRECATED_JUST_USE_NULL(),NULL) which can be used for any compiler that supports non-static statements in declarations (everything but Sun and MSVC?) and has __attribute__((deprecated)). This produces warnings like this under Clang ex5.c:102:36: warning: 'PETSC_NULL_IS_DEPRECATED_JUST_USE_NULL' is deprecated [-Wdeprecated-declarations] ierr = PetscOptionsGetReal(PETSC_NULL,"-par",&user.param,NULL);CHKERRQ(ierr); ^ /home/jed/petsc/include/petscsys.h:387:31: note: expanded from macro 'PETSC_NULL' #define PETSC_NULL (PETSC_NULL_IS_DEPRECATED_JUST_USE_NULL(),NULL) ^ /home/jed/petsc/include/petscsys.h:385:33: note: 'PETSC_NULL_IS_DEPRECATED_JUST_USE_NULL' declared here PETSC_UNUSED static inline void PETSC_NULL_IS_DEPRECATED_JUST_USE_NULL() {}; ^ 1 warning generated. and like this with GCC: ex5.c: In function ‘main’: ex5.c:102:3: warning: ‘PETSC_NULL_IS_DEPRECATED_JUST_USE_NULL’ is deprecated (declared at /home/jed/petsc/include/petscsys.h:385) [-Wdeprecated-declarations] We can make the error message cleaner (but more confusing) with __attribute__((deprecated)) PETSC_UNUSED static inline void PETSC_NULL() {}; #define PETSC_NULL (PETSC_NULL(),NULL) which produces: ex5.c: In function ‘main’: ex5.c:102:3: warning: ‘PETSC_NULL’ is deprecated (declared at /home/jed/petsc/include/petscsys.h:385) [-Wdeprecated-declarations] Any sense in this stuff?

1 0

Re: [petsc-dev] questions on new include organization
by Matthew Knepley 13 Feb '13

13 Feb '13

On Wed, Feb 13, 2013 at 12:45 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > > On Tue, Feb 12, 2013 at 11:34 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > >> Is it? Maybe a more general definition of PetscSection would be better >> as map from (int) -> IS Matt's current usage is very restricted to only >> contiguous blocks of elements in a group. Need that be it? Should that be >> it? > > > There is some hackery to ascribe "field" structure to the output of the > map, but I'm skeptical of whether it belongs at this level. > > The granularity is normally very small so we certainly don't want to pay > the memory or performance overhead of an IS for each output. Although I'm > sure we can think up scenarios where that flexibility would be useful, I > think generality in the outputs here would add significant complexity with > limited practical return. > > Note that PetscSection is currently accessed using function calls in inner > loops. This already accounts for a noticeable amount of time for a CFD > application. So we either need a coarser grained access API or we need a > non-polymorphic accessor for use in hot access sites. (This is a rare case > where indirect function call overhead matters.) > I think its helpful to look at the one universal use case right now: You would like to describe a function over a mesh, so associate functions with each part of the mesh and store coefficients of these functions. This happens for: mesh geometry PDE solutions and auxiliary fields (like viscosity) the mesh itself (the adjacency is a field over the points) The inner loops Jed is referring to are structured like this VecGetArray(sol, &solArray); Loop over pieces (like cell, face, etc.) Get solution on this piece: PetscSectionGetDof(solSection, piece, &dof), PetscSectionGetOffset(solSection, piece, &off) pieceArray = &solArray[off]; Jed condenses this to DMPlexPointLocalRead(solDM, piece, solArray, &pieceArray) Thus this adds structure to the Vec sol by breaking it into irregular pieces. The lower level function Dof/Offset() are definitely necessary so you can reason about the operations. I think you could get away with only the array offsetting being fast, but I don't see how you make that fast and those slow. Barry: > Is it? Maybe a more general definition of PetscSection would be better as map from (int) -> IS Matt's current usage is very restricted to only contiguous blocks of elements in a group. Need that be it? Should that be it? I think you are missing the point here. I use this all over the place. This is PetscSection+IS. You put your non-contiguous indices in the IS and index into it using PetscSection. PetscSection is the composable part of this indexing-by-groups. It can be combined with IS or Vec or anything. I think it is the right building block. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Fix for mataijcusp.cu
by Jose E. Roman 13 Feb '13

13 Feb '13

mataijcusp.cu does not currently compile (MatGetVecs_MPIAIJCUSP accesses private Vec members). Here is a fix. I did not push because maybe you prefer to use VecGetLayout here. Jose --- a/src/mat/impls/aij/mpi/mpicusp/mpiaijcusp.cu Tue Feb 12 23:21:30 2013 -0600 +++ b/src/mat/impls/aij/mpi/mpicusp/mpiaijcusp.cu Wed Feb 13 10:42:59 2013 +0100 @@ -3,6 +3,7 @@ #include <../src/mat/impls/aij/mpi/mpiaij.h> /*I "petscmat.h" I*/ PETSC_CUDA_EXTERN_C_END #include "mpicuspmatimpl.h" +#include <petsc-private/vecimpl.h> #undef __FUNCT__ #define __FUNCT__ "MatMPIAIJSetPreallocation_MPIAIJCUSP"

1 0

Re: [petsc-dev] questions on new include organization
by Jed Brown 12 Feb '13

12 Feb '13

On Tue, Feb 12, 2013 at 11:34 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > Is it? Maybe a more general definition of PetscSection would be better as > map from (int) -> IS Matt's current usage is very restricted to only > contiguous blocks of elements in a group. Need that be it? Should that be > it? There is some hackery to ascribe "field" structure to the output of the map, but I'm skeptical of whether it belongs at this level. The granularity is normally very small so we certainly don't want to pay the memory or performance overhead of an IS for each output. Although I'm sure we can think up scenarios where that flexibility would be useful, I think generality in the outputs here would add significant complexity with limited practical return. Note that PetscSection is currently accessed using function calls in inner loops. This already accounts for a noticeable amount of time for a CFD application. So we either need a coarser grained access API or we need a non-polymorphic accessor for use in hot access sites. (This is a rare case where indirect function call overhead matters.)

1 0

Re: [petsc-dev] questions on new include organization
by Jed Brown 12 Feb '13

12 Feb '13

On Tue, Feb 12, 2013 at 11:31 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > Then would depend on how we would define how the grouping is handled and > how the GetIndices would be used. Some possibilities > > 1) ISGetIndices(is,int group_number, &indices) gives the indices for a > group or > So by convention, group 0 is "offset" and group 1 is "size"? Do we need an arbitrary number of groups? (Making the object more general makes it harder to reason about what it does and harder to write non-implementation code that manipulates these objects.) > > 2) ISGetIndices(is,&indices) simply returns all the indices for all the > groups > That would need to be packed somehow.

1 0

Re: [petsc-dev] questions on new include organization
by Barry Smith 12 Feb '13

12 Feb '13

On Feb 12, 2013, at 11:24 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > > On Tue, Feb 12, 2013 at 11:17 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > Yes. PetscSection represents groups of numbers, and you can see the grouping. > > IS is a map from (int) -> (int), but PetscSection is a map from (int) -> (int,int) where the pair is interpreted to mean an interval of non-negative extent. If it's any use, an IS could be thought of as the special case (int) -> (int,1). Is it? Maybe a more general definition of PetscSection would be better as map from (int) -> IS Matt's current usage is very restricted to only contiguous blocks of elements in a group. Need that be it? Should that be it? Barry > > > What queries "do not just return members of this list" in PETSc section? (That is queries we wish to retain anyways). > > For example, you can ask "how many items are in group p?". This is crucial, and the whole point of the structure. > I don't see how you can recover that information from just the list of dof numbers. > >

1 0