April 2011 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Preconditioner for Schur complement (pcfieldsplit)
by Thomas Witkowski 07 Apr '11

07 Apr '11

Jed Brown wrote: > On Wed, Apr 6, 2011 at 11:36, Thomas Witkowski > <thomas.witkowski(a)tu-dresden.de > <mailto:[email protected]>> wrote: > > This brings me back to the starting point of my current problem: > how to set splits for MatNest correctly? PCFieldSplitSetFields > seems not to work in my case. > > > PCFieldSplitSetFields() is a special case that only works when fields > are interlaced. I would like to get rid of it, but there are still > some cases where it can provide a faster implementation in the special > case so we'll wait until the faster implementation can be accessed > through the generic interface. > > > When I want to make use of PCFieldSplitSetIS for splitting > MatNest, which indices should I provide? The MatNest I've created > consists of the to main diagonal blocks with indices 0 to n and 0 > to m. > > > This does not make sense because these index sets overlap. Maybe you > want to use (0:m) and (m:m+n), then these are the index sets you pass > to PCFieldSplitSetIS(). With two processes, you would usually have > > is0 = [(0:m0), (m0+n0:m0+n0+m1)] > is1 = [(m0:m0+n0), (m0+n0+m1:m0+n0+m1+n1)] > > This is the decomposition that is set up automatically by > MatCreateNest when you pass PETSC_NULL for the index sets. > > You should be able to retrieve the index sets that MatNest created > using MatNestGetSubMats(), but this is not implemented (Dave and I > discussed it and agree, but a patch has not been pushed yet). Doing it this way seems to work correctly, at least for the initialization. When solving the system, I get a segdev. Here is the stack (I make use of the current petsc-dev): #0 0x00000000008ce6b3 in PetscMemcpy (a=0x188bba0, b=0x2, n=128) at /fastfs/witkowsk/software/petsc-dev/include/petscsys.h:1777 #1 0x00000000008d0935 in VecScatterBegin_1 (ctx=0x18a4460, xin=0x17b1d70, yin=0x1889840, addv=INSERT_VALUES, mode=SCATTER_FORWARD) at /fastfs/witkowsk/software/petsc-dev/src/vec/vec/utils/vpscat.h:11 4 #2 0x00000000008c6953 in VecScatterBegin (inctx=0x18a4460, x=0x17b1d70, y=0x1889840, addv=INSERT_VALUES, mode=SCATTER_FORWARD) at /fastfs/witkowsk/software/petsc-dev/src/vec/vec/utils/vscat.c:1575 #3 0x00000000007160b6 in PCApply_FieldSplit_Schur (pc=0x182f4e0, x=0x17b1d70, y=0x17c2640) at /fastfs/witkowsk/software/petsc-dev/src/ksp/pc/impls/fieldsplit/fieldsplit.c:565 #4 0x0000000000c7015c in PCApply (pc=0x182f4e0, x=0x17b1d70, y=0x17c2640) at /fastfs/witkowsk/software/petsc-dev/src/ksp/pc/interface/precon.c:384 #5 0x00000000007ff6e3 in KSPSolve_PREONLY (ksp=0x17fe730) at /fastfs/witkowsk/software/petsc-dev/src/ksp/ksp/impls/preonly/preonly.c:26 #6 0x000000000077b246 in KSPSolve (ksp=0x17fe730, b=0x17b1d70, x=0x17c2640) at /fastfs/witkowsk/software/petsc-dev/src/ksp/ksp/interface/itfunc.c:426 .... The problem is that the data field (vec->data) of the nested vectors is empty. I created the nested vectors exactly in the same way as in /src/ksp/ksp/examples/tests/ex22.c. When I would make a VecGetArray(myvec, &vecptr) on a nested vector myvec, than vecptr is not a vailde pointer. This is the reason for the segdev. Is it a bug in PETSc or some problem of my usage of PETSc's interface? Thomas

1 0

Re: [petsc-dev] Pflotran perf on GPU
by Barry Smith 06 Apr '11

06 Apr '11

Jeswin, Thanks for the numbers. The linear solver is doing very well and you can really see Amdahl's Law kicking in, even when the KSPSolve is several times faster because of the GPU the whole application code is NOT several times faster because the function evaluation and Jacobian evaluations which are not on the GPU are hold the speedup down. Look for example at the row labeled KSPSolve() and the row labeled SNESSolve(). The reason the tracer code is not showing great improvement is because the problem is WAY WAY to small to run on the GPU. GPU's can only work well with lots and lots of data, this is a fact of life and cannot be changed by better software. This is very exciting, the GPU is really helping the linear solver as much as one could hope for. I'm glad you got it running, Barry On Apr 6, 2011, at 3:46 PM, Jeswin Godwin wrote: > > > Petsc Team, > > I am student from Ohio-State University working in Hpcrl lab. > > > > --This shows the performance of Pflotran on the GPU. > > I am also attaching the –log_summary output for 1st case of both the example cases. > > --I built Petsc with-debugging=0. > > -- I did not use the preconditioner for any of the cases.... > > -- Would you people be able to explain the performance in the these cases. (i.e) The performance of 100_100_100 is better in the GPU case compared to the ID tracer. > > > > > > > > Performance of Pflotran on the GPU. > > example 100_100_100 > > No Gpu > > Gpu > > Nx > > Ny > > Ny > > time sec > > Gflops > > Time sec > > Gflops > > 32 > > 32 > > 32 > > 4.58E+01 > > 1.78E+08 > > 3.82E+01 > > 2.19E+08 > > 64 > > 64 > > 64 > > 6.83E+02 > > 2.66E+08 > > 3.64E+02 > > 5.14E+08 > > 96 > > 96 > > 96 > > 3.56E+03 > > 3.32E+08 > > 1.27E+03 > > 9.53E+08 > > example 1-D Tracer > > No Gpu > > Gpu > > Nx > > Ny > > Nz > > time sec > > Gflops > > Time sec > > Gflops > > 1000 > > 1 > > 1 > > 3.43E+00 > > 5.04E+08 > > 2.62E+01 > > 7.21E+07 > > 5000 > > 1 > > 1 > > 1.12E+02 > > 6.51E+08 > > 2.02E+02 > > 3.74E+08 > > 8000 > > 1 > > 1 > > 3.97E+02 > > 6.41E+08 > > 4.73E+02 > > 5.67E+08 > > > > > > > > > > Thank you, > > Jeswin > > > > <output_100_100_100.txt><output_seqcusp100_100_100.txt><output_seqcuspcuda.txt><output_tracer.txt>

1 0

Re: [petsc-dev] Preconditioner for Schur complement (pcfieldsplit)
by Jed Brown 06 Apr '11

06 Apr '11

On Wed, Apr 6, 2011 at 11:36, Thomas Witkowski < thomas.witkowski(a)tu-dresden.de> wrote: > This brings me back to the starting point of my current problem: how to set > splits for MatNest correctly? PCFieldSplitSetFields seems not to work in my > case. PCFieldSplitSetFields() is a special case that only works when fields are interlaced. I would like to get rid of it, but there are still some cases where it can provide a faster implementation in the special case so we'll wait until the faster implementation can be accessed through the generic interface. > When I want to make use of PCFieldSplitSetIS for splitting MatNest, which > indices should I provide? The MatNest I've created consists of the to main > diagonal blocks with indices 0 to n and 0 to m. This does not make sense because these index sets overlap. Maybe you want to use (0:m) and (m:m+n), then these are the index sets you pass to PCFieldSplitSetIS(). With two processes, you would usually have is0 = [(0:m0), (m0+n0:m0+n0+m1)] is1 = [(m0:m0+n0), (m0+n0+m1:m0+n0+m1+n1)] This is the decomposition that is set up automatically by MatCreateNest when you pass PETSC_NULL for the index sets. You should be able to retrieve the index sets that MatNest created using MatNestGetSubMats(), but this is not implemented (Dave and I discussed it and agree, but a patch has not been pushed yet).

1 0

Re: [petsc-dev] Preconditioner for Schur complement (pcfieldsplit)
by Thomas Witkowski 06 Apr '11

06 Apr '11

Jed Brown wrote: > On Wed, Apr 6, 2011 at 09:34, Thomas Witkowski > <thomas.witkowski(a)tu-dresden.de > <mailto:[email protected]>> wrote: > > I have not a very good overview of PETSc's internal matters, but > wouldn't be it correct that MatGetBlockSize returns the number of > nested matrices of a MatNest? > > > No, and it would not make sense to do so because that is semantically > different information. > > We can add support for automatically detecting splits when MatNest is > used, but it really is better to set the splits using > PCFieldSplitSetIS() or by using DMComposite because then it will do > the correct thing regardless of whether MatNest is used underneath. This brings me back to the starting point of my current problem: how to set splits for MatNest correctly? PCFieldSplitSetFields seems not to work in my case. When I want to make use of PCFieldSplitSetIS for splitting MatNest, which indices should I provide? The MatNest I've created consists of the to main diagonal blocks with indices 0 to n and 0 to m. I'm not sure which index sets must be provided for PCFieldSplitSetIS. Thomas

1 0

Re: [petsc-dev] Preconditioner for Schur complement (pcfieldsplit)
by Jed Brown 06 Apr '11

06 Apr '11

On Wed, Apr 6, 2011 at 09:34, Thomas Witkowski < thomas.witkowski(a)tu-dresden.de> wrote: > I have not a very good overview of PETSc's internal matters, but wouldn't > be it correct that MatGetBlockSize returns the number of nested matrices of > a MatNest? No, and it would not make sense to do so because that is semantically different information. We can add support for automatically detecting splits when MatNest is used, but it really is better to set the splits using PCFieldSplitSetIS() or by using DMComposite because then it will do the correct thing regardless of whether MatNest is used underneath.

1 0

Re: [petsc-dev] Preconditioner for Schur complement (pcfieldsplit)
by Thomas Witkowski 06 Apr '11

06 Apr '11

Matthew Knepley wrote: > On Tue, Apr 5, 2011 at 8:38 AM, Thomas Witkowski > <thomas.witkowski(a)tu-dresden.de > <mailto:[email protected]>> wrote: > > Matthew Knepley wrote: > > On Tue, Apr 5, 2011 at 4:28 AM, Thomas Witkowski > <thomas.witkowski(a)tu-dresden.de > <mailto:[email protected]> > <mailto:[email protected] > <mailto:[email protected]>>> wrote: > > Jed Brown wrote: > > On Tue, Mar 29, 2011 at 15:29, Thomas Witkowski > <thomas.witkowski(a)tu-dresden.de > <mailto:[email protected]> > <mailto:[email protected] > <mailto:[email protected]>> > <mailto:[email protected] > <mailto:[email protected]> > <mailto:[email protected] > <mailto:[email protected]>>>> wrote: > > Is there any example which makes use of it? > > > src/snes/examples/tutorials/ex28.c is a (contrived) > multi-physics prototype that uses MatNest tranparently > if you > run with -pack_dm_mat_type nest (currently, you should add > -pc_type fieldsplit -pc_fieldsplit_type additive because > off-diagonal blocks are not automatically preallocated). > > There are also two tests that use the low-level API: > > $ grep -l MatNest src/**/tests/*.c > src/ksp/ksp/examples/tests/ex22.c > src/snes/examples/tests/ex17.c > > So, I reimplemented my code to assemble the matrices to a > MatNest > matrix. This seems to work fine. But how to define the > FieldSplits? I did it in the following way: > > KSP solver; > PC pc; > KSPCreate(PETSC_COMM_WORLD, &solver); > KSPGetPC(solver, &pc); > const PetscInt interiorField[1] = {0}; > const PetscInt boundaryField[1] = {1}; > PCFieldSplitSetFields(pc, "interior", 1, interiorField); > PCFieldSplitSetFields(pc, "boundary", 1, boundaryField); > > When I run my code with the options "-ksp_type preonly -pc_type > fieldsplit -pc_fieldsplit_type schur", I get the following > errors > in KSPSolve: > > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: Petsc has generated inconsistent data! > [0]PETSC ERROR: Unhandled case, must have at least two fields! > > Is the creation of the split wrong? If not, what is a good > way to > figure out the problem? > > > This is almost certainly a problem with the setup order. In > this code, the PC type is not yet set > so the FieldSplit calls are ignored. When it finally is set, > there are no fields defined. I would call > KSPSetFromOptions() before retrieving the PC. > > This generates exactly the same error message. > > > Then there is no choice but to use the debugger to see what is going on. I did so and the problem is as you have described before. But, when I understood the implementation of fieldsplit right, than it should not be necessary to set the splits using PCFieldSplitSetFields. This could be done in PCFieldSplitSetDefaults, when I make use of the parameter "-pc_fieldsplit_defaults". The problem is that MatGetBlockSize on my MatNest (2x2 blocks) returns 1. Therefore PCFieldSplitSetDefaults creates only one split and exits with the error due to if (jac->nsplits < 2) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_PLIB,"Unhandled case, must have at least two fields"); I have not a very good overview of PETSc's internal matters, but wouldn't be it correct that MatGetBlockSize returns the number of nested matrices of a MatNest? Thomas > > Matt > > > > Thomas > > > Matt > > > Thomas > > > > > -- > What most experimenters take for granted before they begin > their experiments is infinitely more interesting than any > results to which their experiments lead. > -- Norbert Wiener > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener

1 0

Re: [petsc-dev] buildsystem package dependencies
by Satish Balay 05 Apr '11

05 Apr '11

On Tue, 5 Apr 2011, Lisandro Dalcin wrote: > On 5 April 2011 13:20, Satish Balay <balay(a)mcs.anl.gov> wrote: > > Not sure I understand. If I say 'mpi depends on mpe' - then the link line is: > > > > Sorry, I was thinking in terms of shared libraries. > > > -lhypre -lmpi -lmpe > > > > But thats incorrect usage - and not what I want.. The correct usage is: > > -lhypre -lmpe -lmpi > > > > The issue comes up with -llmpe - which traps MPI_xx functions - and does some > > logging - and then calls MPI_xx functions from -lmpi. i.e usage is: > > > > If you are going to use MPE from MPICH2, "mpicc -mpe=mpilog" does the > right thing: > > $ mpicc -mpe=mpilog -LLIBDIR -lLIBNAME -link_info > gcc -I/usr/include/mpich2-i386 -m32 -O2 -Wl,-z,noexecstack > -Wl,--export-dynamic -LLIBDIR -lLIBNAME -I/usr/include/mpich2-i386 > -L/usr/lib/mpich2/lib -L/usr/lib/mpich2/lib -L/usr/lib/mpich2/lib > -llmpe -lmpe -lmpich -lopa -lpthread -lrt I guess I'll just leave it at that.. [i.e Its users responsibility to use "mpicc -mpe=mpilog" if they want to log MPI messages.] Currently configure supports --with-mpe=1 or --download-mpe=1. This behaves as any other package. wrt build/usage. If you run 'ex2 -log_mpe' - you get a ex2.clog with all petsc events logged. ./ex2 behaves as before. But once you want to log petsc events + MPI messages - then it gets complicated. [thats when MPE becomes MPI replacement - as Matt says]. I was trying to see if there is a easy way to fit petsc+mpi logging in the current 'package' model. But looks like its easier to say: Logging PETSc events? use --with-mpe=1 or --download-mpe=1 [and run with 'ex2 -log_mpe'] Logging MPI calls? build PETSc with "mpicc -mpe=mpilog" and "mpif90 -mpe=mpilog" or equivalent wrappers - and run as 'ex2' [it always creates a logfile] Logging Both PETSc events and MPI calls? build PETSc with "mpicc -mpe=mpilog" and "mpif90 -mpe=mpilog" --with-mpe=1 ['ex2' creates MPI only log, while 'ex2 -log_mpe' will crate petsc+mpi log]. [I was hoping to simplify the 3rd part as: build PETSc with --with-mpe=1 or --download-mpe=1 and --with-mpe-enalbe-mpilog=1, so that ./'ex2' always creates mpi+petsc log - but that appears to need merging mpi.py with mpe.py. with mpi.py already being compliated enought - I' avoid this] Satish

1 0

Re: [petsc-dev] Compiling C code with C compiler
by Lisandro Dalcin 05 Apr '11

05 Apr '11

On 5 April 2011 10:22, Matthew Knepley <knepley(a)gmail.com> wrote: > On Tue, Apr 5, 2011 at 7:36 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >> >> Jed, >> >> If it is as simple as you say then I am fine with adding support BUT >> it will require "yet another flag" and hence increase complexity. >> --with-c-for-c-code=1 or something and either require a compiler that is >> smart enough to switch based on extension or two lists of files to compile >> with the two compilers. These choices are already complicated for some users >> and the logic in PETSc will get messy. > > I actually want the code compiled with the C++ compiler. That is what the > option said and that is what I want. You can > want something else, so make a different option for it. This is exactly what > users requested and we implemented. --with-c++-support=1 ? This should compile .c with C and .cpp/.cxx with C++, and it should be enabled by default. > Can someone give a 1 line description of the dlopen() test they want and I > will put it in today. > Matt I think that the following code should do #include <dlfcn.h> int main() { void* handle; dlerror(); handle = dlopen(0, RTLD_LAZY|RTLD_LOCAL); if (!handle || dlerror()) return 1; /*fail, cannot dlopen executable*/ return 0; } > >> >> Barry >> >> On Apr 5, 2011, at 2:03 AM, Jed Brown wrote: >> >> > On Tue, Apr 5, 2011 at 04:33, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >> > Since the sieve c++ compiled code uses the C code (like VecXXX) the C++ >> > compiler would have to know that the other parts of C are not compiled with >> > C++. At a minimum you would have to extern c it in the header files other >> > wise the C++ compiled code will call the C routines with name-mangling >> > turned on while the C compiler doesn't know about name mangling and hence >> > would generate routines without name mangling and the c++ code wouldn't find >> > the c routines. >> > >> > But this is already what happens --with-c-support. Why is it ever >> > necessary to compile the C source with a C++ compiler (that is, why support >> > with-c-support=0)? The C++ convenience functions (PetscPolymorphicFunction) >> > can easily be defined whenever a C++ compiler is used, they will be name >> > mangled (and usually inlined). I think all you would need to do is remove >> > the PETSC_USE_EXTERN_CXX clause from >> > >> > #if !defined(PETSC_USE_EXTERN_CXX) && defined(__cplusplus) >> > >> > In other words, even without issues of struct layouts currently >> > everything in PETSc needs to be built with the same compiler. If sieve was >> > made a completely separate package form PETSc then one could easily arrange >> > things so that sieve compiled with c++ and PETSc with c. >> > >> > I don't see any complexity in always building C source with a C compiler >> > and C++ source with a C++ compiler. It seems simpler to me. And then people >> > could build a PETSc that includes Sieve and still call it from C (e.g. when >> > they happen to not be using Sieve). >> > >> > >> > The only problem I can think of, and which might be justification for >> > building C code with a C++ compiler, is that C++ compilers seem to be >> > stricter about complex. C99 complex is binary compatible with C++ complex, >> > but type checking is weaker. I don't know if this is implicit in the >> > standard or if the GNU compilers are unnecessarily lenient about C99 >> > complex. This is purely a matter of quality error/warning messages, but I'm >> > loathe to give up any static type checking. >> > > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

1 0

Re: [petsc-dev] buildsystem package dependencies
by Satish Balay 05 Apr '11

05 Apr '11

On Tue, 5 Apr 2011, Lisandro Dalcin wrote: > On 5 April 2011 13:29, Matthew Knepley <knepley(a)gmail.com> wrote: > > On Tue, Apr 5, 2011 at 11:20 AM, Satish Balay <balay(a)mcs.anl.gov> wrote: > >> > >> Not sure I understand. If I say 'mpi depends on mpe' - then the link line > >> is: > >> > >> -lhypre -lmpi -lmpe > >> > >> But thats incorrect usage - and not what I want.. The correct usage is: > >> -lhypre -lmpe -lmpi > >> > >> The issue comes up with -llmpe - which traps MPI_xx functions - and does > >> some > >> logging - and then calls MPI_xx functions from -lmpi. i.e usage is: > >> > >> -lhypre -llmpe -lmpe -lmpi > > > > Right, but this MPE package is changing the MPI interface. All those other > > packages, > > like Hypre, depend on MPI. MPI depends on MPE, but the MPE module may modify > > the MPI lib to be '-lmpe -lmpi'. I do not see another way to do this. > > Matt > > > > We could also build a custom libmpe.so, load it at runtime from > petscmpiexec by passing a -mpe flag. This way, we can get MPE logging > for free, using a regular PETSc build. This is more or less described > at the end of this ticket: > https://trac.mcs.anl.gov/projects/mpich2/ticket/573. > > What do you think? Hardly anyone uses petscmpiexec. Satish

1 0

Re: [petsc-dev] buildsystem package dependencies
by Lisandro Dalcin 05 Apr '11

05 Apr '11

On 5 April 2011 13:29, Matthew Knepley <knepley(a)gmail.com> wrote: > On Tue, Apr 5, 2011 at 11:20 AM, Satish Balay <balay(a)mcs.anl.gov> wrote: >> >> Not sure I understand. If I say 'mpi depends on mpe' - then the link line >> is: >> >> -lhypre -lmpi -lmpe >> >> But thats incorrect usage - and not what I want.. The correct usage is: >> -lhypre -lmpe -lmpi >> >> The issue comes up with -llmpe - which traps MPI_xx functions - and does >> some >> logging - and then calls MPI_xx functions from -lmpi. i.e usage is: >> >> -lhypre -llmpe -lmpe -lmpi > > Right, but this MPE package is changing the MPI interface. All those other > packages, > like Hypre, depend on MPI. MPI depends on MPE, but the MPE module may modify > the MPI lib to be '-lmpe -lmpi'. I do not see another way to do this. > Matt > We could also build a custom libmpe.so, load it at runtime from petscmpiexec by passing a -mpe flag. This way, we can get MPE logging for free, using a regular PETSc build. This is more or less described at the end of this ticket: https://trac.mcs.anl.gov/projects/mpich2/ticket/573. What do you think? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

1 0