February 2012 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] [petsc-users] VECMPICUSP with ghosted vector
by Matthew Knepley 06 Feb '12

06 Feb '12

On Mon, Feb 6, 2012 at 12:40 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Mon, Feb 6, 2012 at 20:35, Matthew Knepley <knepley(a)gmail.com> wrote: > >> VecCreate() >>> VecSetType() >>> VecSetGhosted() >> >> > Is there a compelling reason for memory to be allocated before VecSetUp()? > Deferring it would make it easier to specify what memory to use or specify > ghosting in any order. > > My preference is that ghosting is orthogonal to type, to the extent that > it would make sense for the storage format (non-contiguous AMR grids might > not be). > I don't like this because it would mean calling VecSetUp() all over the place. Couldn't the ghosting flag be on the same level as the sizes? Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] [petsc-users] VECMPICUSP with ghosted vector
by Jed Brown 06 Feb '12

06 Feb '12

On Mon, Feb 6, 2012 at 20:35, Matthew Knepley <knepley(a)gmail.com> wrote: > VecCreate() >> VecSetType() >> VecSetGhosted() > > Is there a compelling reason for memory to be allocated before VecSetUp()? Deferring it would make it easier to specify what memory to use or specify ghosting in any order. My preference is that ghosting is orthogonal to type, to the extent that it would make sense for the storage format (non-contiguous AMR grids might not be).

1 0

Re: [petsc-dev] [petsc-users] VECMPICUSP with ghosted vector
by Matthew Knepley 06 Feb '12

06 Feb '12

On Mon, Feb 6, 2012 at 11:09 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > Fredrik, > > This question belongs on petsc-dev(a)mcs.anl.gov since it involves > additions/extensions to PETSc so I am moving the discussion over to there. > > We have not done the required work to have ghosted vectors work with > CUSP yet, so this will require some additions to PETSc. We can help you > with that process but since the PETSc team does not have a CUSP person > developing PETSc full time you will need to actual contribute some code but > I'll try to guide you in the right direction. > > The first observation is that ghosted vectors in PETSc are actually > handled with largely the same code as VECMPI vectors (with just no ghost > points by default) so in theory little work needs to be done to get the > functionality you need. What makes the needed changes non-trivial is the > current interface where one calls VecCreateGhost() to create the vectors. > This is one of our "easy" interfaces and it is somewhat legacy in that > there is no way to control the types of the vectors since it creates > everything about the vector in one step. Note that we have the same > issues with regard to the pthread versions of the PETSc vectors and > ghosting. > > So before we even talk about what code to change/add we need to decide > on the interface. Presumably you want to be able to decide at runtime > whether to use regular VECMPI, VECMPICUSP or VECMPIPTHREAD in your ghosted > vectors. How do we get that information in there? An additional argument to > VecCreateGhost() (ugly?)? Options database (by calling VecSetFromOptions() > ?), other ways? So for example one could have: > > VecCreateGhost(......) > VecSetFromOptions(......) > > to set the specific type cusp or pthread? What about > > VecCreateGhost(......) > VecSetType(......,VECMPICUSP); > > which as you note doesn't currently work. Note that the PTHREAD version > needs to do its own memory allocation so essentially has to undo much of > what VecCreateGhost() already did, is that a bad thing? > > Or do we get rid of VecCreateGhost() completely and change the model to > something like > > VecCreate() > VecSetType() > VecSetGhosted() > > or > > VecCreate() > VecSetTypeFromOptions() > VecSetGhosted() > > or even > > VecCreate() > VecSetGhosted() which will just default to regular MPI ghosted. > > this model allows a clean implementation that doesn't require undoing > previously built internals. > I am for the second model, just absorbing ghosting into the current implementations. Matt > > Everyone chime in with observations so we can figure out any > refactorizations needed. > > Barry > > > > > > On Feb 6, 2012, at 8:33 AM, Fredrik Heffer Valdmanis wrote: > > > Hi, > > > > In FEniCS, we use ghosted vectors for parallel computations, with the > functions > > > > VecCreateGhost > > VecGhostGetLocalForm > > > > As I am integrating the new GPU-based vectors and matrices in FEniCS, I > want the ghosted vectors to be of type VECMPICUSP. I have tried to do this > by calling VecSetType after creating the vector, but that makes > VecGhostGetLocalForm give an error. > > > > Is it possible to set the type to be mpicusp when using ghost vectors, > without changing much of the code? If so, how? > > > > If not, how would you recommend I proceed to work with mpicusp vectors > in this context? > > > > Thanks! > > > > -- > > Fredrik > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] [petsc-users] VECMPICUSP with ghosted vector
by Barry Smith 06 Feb '12

06 Feb '12

Fredrik, This question belongs on petsc-dev(a)mcs.anl.gov since it involves additions/extensions to PETSc so I am moving the discussion over to there. We have not done the required work to have ghosted vectors work with CUSP yet, so this will require some additions to PETSc. We can help you with that process but since the PETSc team does not have a CUSP person developing PETSc full time you will need to actual contribute some code but I'll try to guide you in the right direction. The first observation is that ghosted vectors in PETSc are actually handled with largely the same code as VECMPI vectors (with just no ghost points by default) so in theory little work needs to be done to get the functionality you need. What makes the needed changes non-trivial is the current interface where one calls VecCreateGhost() to create the vectors. This is one of our "easy" interfaces and it is somewhat legacy in that there is no way to control the types of the vectors since it creates everything about the vector in one step. Note that we have the same issues with regard to the pthread versions of the PETSc vectors and ghosting. So before we even talk about what code to change/add we need to decide on the interface. Presumably you want to be able to decide at runtime whether to use regular VECMPI, VECMPICUSP or VECMPIPTHREAD in your ghosted vectors. How do we get that information in there? An additional argument to VecCreateGhost() (ugly?)? Options database (by calling VecSetFromOptions() ?), other ways? So for example one could have: VecCreateGhost(......) VecSetFromOptions(......) to set the specific type cusp or pthread? What about VecCreateGhost(......) VecSetType(......,VECMPICUSP); which as you note doesn't currently work. Note that the PTHREAD version needs to do its own memory allocation so essentially has to undo much of what VecCreateGhost() already did, is that a bad thing? Or do we get rid of VecCreateGhost() completely and change the model to something like VecCreate() VecSetType() VecSetGhosted() or VecCreate() VecSetTypeFromOptions() VecSetGhosted() or even VecCreate() VecSetGhosted() which will just default to regular MPI ghosted. this model allows a clean implementation that doesn't require undoing previously built internals. Everyone chime in with observations so we can figure out any refactorizations needed. Barry On Feb 6, 2012, at 8:33 AM, Fredrik Heffer Valdmanis wrote: > Hi, > > In FEniCS, we use ghosted vectors for parallel computations, with the functions > > VecCreateGhost > VecGhostGetLocalForm > > As I am integrating the new GPU-based vectors and matrices in FEniCS, I want the ghosted vectors to be of type VECMPICUSP. I have tried to do this by calling VecSetType after creating the vector, but that makes VecGhostGetLocalForm give an error. > > Is it possible to set the type to be mpicusp when using ghost vectors, without changing much of the code? If so, how? > > If not, how would you recommend I proceed to work with mpicusp vectors in this context? > > Thanks! > > -- > Fredrik

1 0

implicit none error
by Mark F. Adams 06 Feb '12

06 Feb '12

I get this error with implicit none: [ 0%] Building Fortran object CMakeFiles/petsc.dir/src/dm/f90-mod/petscdmmod.F.o /Users/markadams/Codes/petsc-dev/include/finclude/ftn-custom/petscdmcomposite.h90:8.49: Included at /Users/markadams/Codes/petsc-dev/include/finclude/petscdmcomposite.h90:6: Included at /Users/markadams/Codes/petsc-dev/src/dm/f90-mod/petscdmmod.F:42: Subroutine DMCompositeGetEntries1(dm1, d1,ierr) 1 Error: Symbol 'd1' at (1) has no IMPLICIT type Mark

1 0

Re: [petsc-dev] MatMatMult gives different results
by Hong Zhang 06 Feb '12

06 Feb '12

MatMatMult() in petsc is not well-tested for complex - could be buggy. Can you send us the matrices A and B in petsc binary format for investigation? Hong On Mon, Feb 6, 2012 at 5:55 AM, Alexander Grayver <agrayver(a)gfz-potsdam.de>wrote: > Dear PETSc team, > > I try to use: > call MatMatMult(A,B,MAT_INITIAL_**MATRIX,PETSC_DEFAULT_DOUBLE_** > PRECISION,C,ierr);CHKERRQ(**ierr) > > Where both A and B are rectangular, but A is sparse and B is dense. Both > are double complex and distributed. > The product PETSc gives me contains some errors in some part of the matrix. > I output A, B and C then computed product in matlab. > > Attached you see figure plotted as: > imagesc(log10(abs(C-Cm))) > > Where Cm -- product computed in matlab. > > The pattern and amplitude vary depending on the number of cores I use. > This picture is obtained for 48 cores (I've tried 12, 64 cores as well). > > Where should I look for possible explanation? > > -- > Regards, > Alexander >

1 0

MatMatMult gives different results
by Alexander Grayver 06 Feb '12

06 Feb '12

Dear PETSc team, I try to use: call MatMatMult(A,B,MAT_INITIAL_MATRIX,PETSC_DEFAULT_DOUBLE_PRECISION,C,ierr);CHKERRQ(ierr) Where both A and B are rectangular, but A is sparse and B is dense. Both are double complex and distributed. The product PETSc gives me contains some errors in some part of the matrix. I output A, B and C then computed product in matlab. Attached you see figure plotted as: imagesc(log10(abs(C-Cm))) Where Cm -- product computed in matlab. The pattern and amplitude vary depending on the number of cores I use. This picture is obtained for 48 cores (I've tried 12, 64 cores as well). Where should I look for possible explanation? -- Regards, Alexander

1 0

Re: [petsc-dev] Error with DMCompositeGetISLocalToGlobalMappings
by domenico.borzacchiello＠univ-st-etienne.fr 05 Feb '12

05 Feb '12

> 1. Call DMSetUp(packer) once you are done adding DMs. I will make it > automatic in petsc-dev. > Ok calling DMSetUp did the job! thank you for your suggestions Jed

1 0

Re: [petsc-dev] error when compiling gamg
by Satish Balay 04 Feb '12

04 Feb '12

pushed a fix now. Satish On Sat, 4 Feb 2012, Stefano Zampini wrote: > Mark, > > I got a compiling error after my latest pull. Can you please fix this? > > gamg.c:1827: error: conflicting types for 'PCGAMGSetType' > /work/adz/zampini/MyWorkingCopyOfPetsc/petsc-dev/include/petscpc.h:451: > note: previous declaration of 'PCGAMGSetType' was here > make[1]: > [/work/adz/zampini/MyWorkingCopyOfPetsc/petsc-dev/gnu-4.4.3-debug-double-louis/lib/libpetsc.a(gamg.o)] > Error 1 (ignored) > > >

1 0

error when compiling gamg
by Stefano Zampini 04 Feb '12

04 Feb '12

Mark, I got a compiling error after my latest pull. Can you please fix this? gamg.c:1827: error: conflicting types for 'PCGAMGSetType' /work/adz/zampini/MyWorkingCopyOfPetsc/petsc-dev/include/petscpc.h:451: note: previous declaration of 'PCGAMGSetType' was here make[1]: [/work/adz/zampini/MyWorkingCopyOfPetsc/petsc-dev/gnu-4.4.3-debug-double-louis/lib/libpetsc.a(gamg.o)] Error 1 (ignored) -- Stefano

1 0