February 2012 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] SNES, Coloring, DM, and TS
by Matthew Knepley 28 Feb '12

28 Feb '12

On Tue, Feb 28, 2012 at 12:11 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > The DM public interface is the _single_ interface that provides all the > information the algebraic solvers need to do their thing. > > Barry > > Yes it involves bits and pieces of information about the topology, the > geometry, the function spaces, and the particular equations being solved. > This does not mean that all those things need be wrapped up inside one > monolithic class on the back end (though they are for DA). But this is what is causing the trouble. Vecs need to hold a DM to tell them how to write themselves, but DMs need those Vecs to pass to sovlers. These are very different jobs, and thus cause loops. Matt > On Feb 28, 2012, at 11:50 AM, Matthew Knepley wrote: > > > On Tue, Feb 28, 2012 at 11:47 AM, Jed Brown <jedbrown(a)mcs.anl.gov> > wrote: > > On Tue, Feb 28, 2012 at 11:03, Matthew Knepley <knepley(a)gmail.com> > wrote: > > So you think this has nothing to do with > > - Refinement/coarsening > > - Local evaluation > > - Partitioning > > It is all implied, and in fact only really works completely for > Cartesian stuff. > > > > It's not in the DM *interface*. Implementing the interface for an > unstructured grid, etc, will involve *doing* these things, but it doesn't > mean that the logic needs to be in your DM_XXX() or that we need another > public object to share stuff between DMs. > > > > > > > > - Modeling approximation (like function spaces, projectors) > > > > This is only in the DM interface through coarsening and interpolation. > > > > You also need to know this to piece together local evaluations. > > > > *If* that is the interface you want DMXX users to implement to expose > their physics. The concept is not in the DM interface. > > > > > > > > - Modeling equations (including decompositions, variable substitutions) > > > > This is only in the DM interface with collective semantics. (For > nonlinear ASM, I would be in favor of getting a subdomain DM which would > have collective semantics on the subcommunicator instead of putting "local" > evaluation into the public interface.) > > > > The equations are there explicitly in all these function pointers it is > holding. What are you talking about? > > > > The local function stuff is in DM_DA, not in DM. The DM interface _only_ > exposes collective evaluation. > > > > Then I think you are mixing levels, but it appears that the DMs are > being passed around to provide this > > "hidden" information which is actually necessary to make everything work > underneath, and it is this that > > creates dependency loops. > > > > Matt > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] SNES, Coloring, DM, and TS
by Barry Smith 28 Feb '12

28 Feb '12

The DM public interface is the _single_ interface that provides all the information the algebraic solvers need to do their thing. Barry Yes it involves bits and pieces of information about the topology, the geometry, the function spaces, and the particular equations being solved. This does not mean that all those things need be wrapped up inside one monolithic class on the back end (though they are for DA). On Feb 28, 2012, at 11:50 AM, Matthew Knepley wrote: > On Tue, Feb 28, 2012 at 11:47 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Tue, Feb 28, 2012 at 11:03, Matthew Knepley <knepley(a)gmail.com> wrote: > So you think this has nothing to do with > - Refinement/coarsening > - Local evaluation > - Partitioning > It is all implied, and in fact only really works completely for Cartesian stuff. > > It's not in the DM *interface*. Implementing the interface for an unstructured grid, etc, will involve *doing* these things, but it doesn't mean that the logic needs to be in your DM_XXX() or that we need another public object to share stuff between DMs. > > > > - Modeling approximation (like function spaces, projectors) > > This is only in the DM interface through coarsening and interpolation. > > You also need to know this to piece together local evaluations. > > *If* that is the interface you want DMXX users to implement to expose their physics. The concept is not in the DM interface. > > > > - Modeling equations (including decompositions, variable substitutions) > > This is only in the DM interface with collective semantics. (For nonlinear ASM, I would be in favor of getting a subdomain DM which would have collective semantics on the subcommunicator instead of putting "local" evaluation into the public interface.) > > The equations are there explicitly in all these function pointers it is holding. What are you talking about? > > The local function stuff is in DM_DA, not in DM. The DM interface _only_ exposes collective evaluation. > > Then I think you are mixing levels, but it appears that the DMs are being passed around to provide this > "hidden" information which is actually necessary to make everything work underneath, and it is this that > creates dependency loops. > > Matt > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener

1 0

Re: [petsc-dev] SNES, Coloring, DM, and TS
by Matthew Knepley 28 Feb '12

28 Feb '12

On Tue, Feb 28, 2012 at 11:47 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Tue, Feb 28, 2012 at 11:03, Matthew Knepley <knepley(a)gmail.com> wrote: > >> So you think this has nothing to do with >> - Refinement/coarsening >> - Local evaluation >> - Partitioning >> It is all implied, and in fact only really works completely for Cartesian >> stuff. >> > > It's not in the DM *interface*. Implementing the interface for an > unstructured grid, etc, will involve *doing* these things, but it doesn't > mean that the logic needs to be in your DM_XXX() or that we need another > public object to share stuff between DMs. > > >> >> >>> >>> >>>> - Modeling approximation (like function spaces, projectors) >>>> >>> >>> This is only in the DM interface through coarsening and interpolation. >>> >> >> You also need to know this to piece together local evaluations. >> > > *If* that is the interface you want DMXX users to implement to expose > their physics. The concept is not in the DM interface. > > >> >> >>> >>> >> - Modeling equations (including decompositions, variable substitutions) >>>> >>> >>> This is only in the DM interface with collective semantics. (For >>> nonlinear ASM, I would be in favor of getting a subdomain DM which would >>> have collective semantics on the subcommunicator instead of putting "local" >>> evaluation into the public interface.) >>> >> >> The equations are there explicitly in all these function pointers it is >> holding. What are you talking about? >> > > The local function stuff is in DM_DA, not in DM. The DM interface _only_ > exposes collective evaluation. > Then I think you are mixing levels, but it appears that the DMs are being passed around to provide this "hidden" information which is actually necessary to make everything work underneath, and it is this that creates dependency loops. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] SNES, Coloring, DM, and TS
by Jed Brown 28 Feb '12

28 Feb '12

On Tue, Feb 28, 2012 at 11:03, Matthew Knepley <knepley(a)gmail.com> wrote: > So you think this has nothing to do with > - Refinement/coarsening > - Local evaluation > - Partitioning > It is all implied, and in fact only really works completely for Cartesian > stuff. > It's not in the DM *interface*. Implementing the interface for an unstructured grid, etc, will involve *doing* these things, but it doesn't mean that the logic needs to be in your DM_XXX() or that we need another public object to share stuff between DMs. > > >> >> >>> - Modeling approximation (like function spaces, projectors) >>> >> >> This is only in the DM interface through coarsening and interpolation. >> > > You also need to know this to piece together local evaluations. > *If* that is the interface you want DMXX users to implement to expose their physics. The concept is not in the DM interface. > > >> >> > - Modeling equations (including decompositions, variable substitutions) >>> >> >> This is only in the DM interface with collective semantics. (For >> nonlinear ASM, I would be in favor of getting a subdomain DM which would >> have collective semantics on the subcommunicator instead of putting "local" >> evaluation into the public interface.) >> > > The equations are there explicitly in all these function pointers it is > holding. What are you talking about? > The local function stuff is in DM_DA, not in DM. The DM interface _only_ exposes collective evaluation.

1 0

Re: [petsc-dev] SNES, Coloring, DM, and TS
by Matthew Knepley 28 Feb '12

28 Feb '12

On Tue, Feb 28, 2012 at 10:45 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Tue, Feb 28, 2012 at 10:40, Matthew Knepley <knepley(a)gmail.com> wrote: > >> Right now, DMs wrap up >> - Domain Topology >> > - Domain Geometry >> > > These are nowhere in the DM interface. > So you think this has nothing to do with - Refinement/coarsening - Local evaluation - Partitioning It is all implied, and in fact only really works completely for Cartesian stuff. > > >> - Modeling approximation (like function spaces, projectors) >> > > This is only in the DM interface through coarsening and interpolation. > You also need to know this to piece together local evaluations. > > - Modeling equations (including decompositions, variable substitutions) >> > > This is only in the DM interface with collective semantics. (For nonlinear > ASM, I would be in favor of getting a subdomain DM which would have > collective semantics on the subcommunicator instead of putting "local" > evaluation into the public interface.) > The equations are there explicitly in all these function pointers it is holding. What are you talking about? Matt > >> and I may have forgotten some stuff. That is a huge amount of crap for >> one object. >> > > You seem to be confusing the DM interface from a particular way of packing > data into an implementation of DM. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] SNES, Coloring, DM, and TS
by Jed Brown 28 Feb '12

28 Feb '12

On Tue, Feb 28, 2012 at 10:40, Matthew Knepley <knepley(a)gmail.com> wrote: > Right now, DMs wrap up > - Domain Topology > - Domain Geometry > These are nowhere in the DM interface. > - Modeling approximation (like function spaces, projectors) > This is only in the DM interface through coarsening and interpolation. > - Modeling equations (including decompositions, variable substitutions) > This is only in the DM interface with collective semantics. (For nonlinear ASM, I would be in favor of getting a subdomain DM which would have collective semantics on the subcommunicator instead of putting "local" evaluation into the public interface.) > > and I may have forgotten some stuff. That is a huge amount of crap for one > object. > You seem to be confusing the DM interface from a particular way of packing data into an implementation of DM.

1 0

Re: [petsc-dev] OpenMP/Vec
by Lawrence Mitchell 28 Feb '12

28 Feb '12

Dear all, I've been doing some of the programming associated with the work Gerard's describing, so here's an attempt of a bit of a response with some summary of where we'd go next. On 28/02/12 04:39, Jed Brown wrote: > On Mon, Feb 27, 2012 at 16:31, Gerard Gorman <g.gorman(a)imperial.ac.uk > <mailto:[email protected]>> wrote: > > I had a quick go at trying to get some sensible benchmarks for this but > I was getting too much system noise. I am particularly interested in > seeing if the overhead goes to zero if num_threads(1) is used. > > What timing method did you use? I did not see overhead going to zero > when num_threads goes to 1 when using GCC compilers, but Intel seems to > do fairly well. It's possible that if one asks the gcc folk nicely enough that the overhead might go to zero. Alternately, if it turns out that overheads are unavoidable, and we've gone down the macro-ification route anyway (see below) we could explicitly make a runtime decision based on numthreads in PETSc. So that: PetscOMPParallelFor(..., numthreads) for ( ... ) ; would turn into if ( numthreads == 1 ) for ( ... ) ; else #pragma omp parallel for for ( ... ) ; The branch should hopefully be cheaper than the thread sync overhead. > I'm surprised by this. I not aware of any compiler that doesn't have > OpenMP support - and then you do not actually enable OpenMP compilers > generally just ignore the pragma. Do you know of any compiler that does > not have OpenMP support which will complain? > > > Sean points out that omp.h might not be available, but that misses the > point. As far as I know, recent mainstream compilers have enough sense > to at least ignore these directives, but I'm sure there are still cases > where it would be an issue. More importantly, #pragma was a misfeature > that should never be used now that _Pragma() exists. The latter is > better not just because it can be turned off, but because it can be > manipulated using macros and can be explicitly compiled out. > > This may not be flexible enough. You frequently want to have a parallel > region, and then have multiple omp for's within that one region. > > > PetscPragmaOMPObject(obj, parallel) > { > PetscPragmaOMP(whetever you normally write for this loop) > for (....) { } > ... > and so on > } So this is easy to do as you say, the exact best way to do it will probably become clear when writing the code. As far as attaching the threading information to the object goes, it would be nice to do so staying within the sequential implementation (rather than copying all the code as is done for pthreads right now). I'd envisage something like this: PetscPragmaOMPObject(vec, parallel) { PetscPragmaOMP(for...) for ( i = PetscVecLowerBound(vec); i < PetscVecUpperBound(vec); i++ ) ...; } Where one has: #if defined(PETSC_HAVE_OPENMP) #define PetscVecLowerBound(vec) compute_lower_bound #define PetscVecUpperBound(vec) compute_upper_bound #else #define PetscVecLowerBound(vec) 0 #define PetscVecUpperBound(vec) vec->map->n #endif Or is this too ugly for words? Computing the lower and upper bounds could either be done for every loop, or, if it's only a property of the vector length, we could stash it in some extra slots at creation time. For a first cut, computation of the upper and lower bounds will do exactly what a static schedule does (i.e. chunking by size/nthread). > I think what you describe is close to Fig 3 of this paper written by > your neighbours: > http://greg.bronevetsky.com/papers/2008IWOMP.pdf > However, before making the implementation more complex, it would be good > to benchmark the current approach and use a tool like likwid to measure > the NUMA traffic so we can get a good handle on the costs. > > > Sure. > > Well this is where the implementation details get richer and there are > many options - they also become less portable. For example, what does > all this mean for the sparc64 processors which are UMA. > > > Delay to runtime, use an ignorant partition for UMA. (Blue Gene/Q is > also essentially uniform.) But note that even with uniform memory, cache > still makes it somewhat hierarchical. > > Not to mention > Intel MIC which also supports OpenMP. I guess I am cautious about > getting too bogged down with very invasive optimisations until we have > benchmarked the basic approach which in a wide range of use cases will > achieve good thread/page locality as illustrated previously. > > > I guess I'm just interested in exposing enough semantic information to > be able to schedule a few different ways using run-time (or, if > absolutely necessary, configure-time) options. I don't want to have to > revisit individual loops. So to take a single, representative, kernel we move from something like: PetscErrorCode VecConjugate_Seq(Vec xin) { PetscScalar *x; PetscInt n = xin->map->n; PetscInt i; PetscErrorCode ierr; PetscFunctionBegin; ierr = VecGetArray(xin,&x);CHKERRQ(ierr); #pragma omp parallel for schedule(static) default(none) private(i) shared(n, x) for (i=0; i<n; i++) x[i] = PetscConj(x[i]); ierr = VecRestoreArray(xin,&x);CHKERRQ(ierr); PetscFunctionReturn(0); } To something like this: PetscErrorCode VecConjugate_Seq(Vec xin) { PetscScalar *x; PetscInt i; PetscErrorCode ierr; PetscFunctionBegin; ierr = VecGetArray(xin,&x);CHKERRQ(ierr); PetscPragmaOMPObject(xin, parallel private(i) shared(xin, x)) { for (i=PetscVecLowerBound(xin); i<PetscVecUpperBound(xin); i++) x[i] = PetscConj(x[i]); } ierr = VecRestoreArray(xin,&x);CHKERRQ(ierr); PetscFunctionReturn(0); } I prefer inlining the loop bounds into macro/function calls at point of use, rather than relying on magic names from a macro definition, but I'm not completely wedded to the idea. Lawrence -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

Re: [petsc-dev] SNES, Coloring, DM, and TS
by Matthew Knepley 28 Feb '12

28 Feb '12

On Tue, Feb 28, 2012 at 6:00 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Mon, Feb 27, 2012 at 23:31, Matthew Knepley <knepley(a)gmail.com> wrote: > >> I like this because we are using DM for our equation abstraction. I think >> the dependency loops are due to refusal to >> decompose the DM. >> > > What do you mean by "decompose the DM"? > Right now, DMs wrap up - Domain Topology - Domain Geometry - Modeling approximation (like function spaces, projectors) - Modeling equations (including decompositions, variable substitutions) and I may have forgotten some stuff. That is a huge amount of crap for one object. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] Regarding an error
by Jed Brown 28 Feb '12

28 Feb '12

On Tue, Feb 28, 2012 at 07:18, Chen S <chensp2602(a)gmail.com> wrote: > I've built it with the fixed files and got the following errors (attached) These line numbers don't match up at all and the errors look quite nonsensical. This smells of a Windows file encoding issue. C:\cygwin\home\PETSC-~4\src\mat\coarsen\impls\mis\mis.c(1) : error C2059: syntax error : '<' C:\cygwin\home\PETSC-~4\src\mat\coarsen\impls\mis\mis.c(10) : error C2018: unknown character '0x40' C:\cygwin\home\PETSC-~4\src\mat\coarsen\impls\mis\mis.c(48) : error C2014: preprocessor command must start as first nonwhite space mis.c line 1 is an empty line, line 2 is definitely valid C: #include <private/matimpl.h> /*I "petscmat.h" I*/ ASCII 0x40 is "@" which does not even appear in that file. There is not a preprocessor directive anywhere near line 48.

1 0

Re: [petsc-dev] Regarding an error
by Jed Brown 28 Feb '12

28 Feb '12

I have pushed fixes for these issues, though I can't test them locally. Can you check that it works now? http://petsc.cs.iit.edu/petsc/petsc-dev/rev/fd97cbc27dcc On Tue, Feb 28, 2012 at 04:27, Chen S <chensp2602(a)gmail.com> wrote: > Hi , > I'm building petsc using cygwin (on a windows machine), and I got the > following errors while running "make all", > (this is the latest build, i've updated on Feb 28th. ): > > C:\cygwin\home\PETSC-~4\src\mat\coarsen\impls\mis\mis.c(352) : error > C2016: C requires that a struct or union has at least one member > C:\cygwin\home\PETSC-~4\src\mat\coarsen\impls\hem\hem.c(322) : error > C2016: C requires that a struct or union has at least one member > > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(421) : error C2143: syntax > error : missing ';' before 'type' > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(424) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(424) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(424) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(432) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(432) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(432) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(434) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(434) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(434) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(436) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(436) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(436) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(437) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(437) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(437) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(438) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(438) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(438) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(439) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(439) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(439) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(440) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(440) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(440) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(442) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(442) : error C2065: 'ierr' > : undeclared identifier > C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(442) : error C2065: 'ierr' > : undeclared identifier > > C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(516) : error C2143: > syntax error : missing ';' before 'type' > C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(519) : error C2065: > 'selected' : undeclared identifier > C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(519) : warning C4047: > 'function' : 'IS *' differs in levels of indirection from 'int *' > C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(519) : warning C4024: > 'function through pointer' : different types for formal and actual > parameter 3 > C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(519) : error C2065: > 'llist' : undeclared identifier > C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(519) : warning C4047: > 'function' : 'IS *' differs in levels of indirection from 'int *' > C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(519) : warning C4024: > 'function through pointer' : different types for formal and actual > parameter 4 > C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(520) : error C2065: > 'selected' : undeclared identifier > C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(520) : warning C4047: > 'function' : 'IS' differs in levels of indirection from 'int' > C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(520) : warning C4024: > 'function through pointer' : different types for formal and actual > parameter 4 > C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(520) : error C2065: > 'llist' : undeclared identifier > C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(520) : warning C4047: > 'function' : 'IS' differs in levels of indirection from 'int' > C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(520) : warning C4024: > 'function through pointer' : different types for formal and actual > parameter 5 > C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(538) : error C2065: > 'selected' : undeclared identifier > C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(538) : warning C4047: > 'function' : 'IS *' differs in levels of indirection from 'int *' > C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(538) : warning C4024: > 'ISDestroy' : different types for formal and actual parameter 1 > C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(539) : error C2065: > 'llist' : undeclared identifier > C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(539) : warning C4047: > 'function' : 'IS *' differs in levels of indirection from 'int *' > C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(539) : warning C4024: > 'ISDestroy' : different types for formal and actual parameter 1 > >

1 0