April 2012 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] MetSetBlockSize issue
by Barry Smith 24 Apr '12

24 Apr '12

On Apr 24, 2012, at 2:38 PM, Mark F. Adams wrote: > > On Apr 24, 2012, at 3:28 PM, Hong Zhang wrote: > >> Mark : >> Shall C=PtAP inherit the block size of A? >> Currently, MatPtAP() is only implemented with aij bs=1. > > No, as I said, algebraically it should inherit the column block size of P, but that is not available. What is the column block size of P and how do you know what it is? Barry > > Perhaps PtAP should take a parameter for the block size ... > > Mark > >> Hong >> >> On Apr 24, 2012, at 2:55 PM, Barry Smith wrote: >> >> > >> > On Apr 24, 2012, at 1:39 PM, Mark F. Adams wrote: >> > >> >> Now I'm getting this error with code like this: >> >> >> >> ierr = MatGetSubMatrix( Cmat, new_eq_indices, new_eq_indices, MAT_INITIAL_MATRIX, &mat ); >> >> CHKERRQ(ierr); >> >> ierr = MatSetBlockSize( mat, cbs ); CHKERRQ(ierr); >> >> >> >> and like this: >> >> >> >> ierr = MatPtAP( Amat_fine, Pold, MAT_INITIAL_MATRIX, 2.0, &Cmat ); CHKERRQ(ierr); >> >> ierr = MatSetBlockSize( Cmat, cbs ); CHKERRQ(ierr); >> > >> > Right you cannot do this. The matrix already exists so you cannot now set its size. >> > >> > Does Cmat have a block size of cbs >> > >> > What about Amat? By default these routines should create the new matrix with the correct blocksize. >> > >> > The harder part is if the blocksize of mat would be different than Cmat? >> > >> >> It seems like MatGetSubMatrix should just inherit the block size but PtAP is harder. The block size of the Cmat is the column block size of P. But I can not set a column block size (!= row block size) so I don't set block size on P at all. Here cbs is this column block size of P, or what it should be. >> >> Mark >> >> > Barry >> > >> >> >> >> Mark >> >> >> >> On Apr 23, 2012, at 9:09 PM, Barry Smith wrote: >> >> >> >>> >> >>> Yes, look, for example how ex32 is handled at the bottom of the makefile. >> >>> >> >>> Thanks >> >>> >> >>> Barry >> >>> >> >>> On Apr 23, 2012, at 5:50 PM, Mark F. Adams wrote: >> >>> >> >>>> >> >>>> On Apr 23, 2012, at 5:59 PM, Barry Smith wrote: >> >>>> >> >>>>> >> >>>>> It was changed a while ago that MatSetBlockSize() couldn't be set after the matrix was full instantiated. I guess that example did not get fixed because it is not listed in the makefile to run in the makeall. >> >>>> >> >>>> May I add it? >> >>>> >> >>>>> >> >>>>> I have fixed the example to call MatSetBlockSize() at a safe point. >> >>>>> >> >>>>> Barry >> >>>>> >> >>>>> On Apr 23, 2012, at 4:29 PM, Mark F. Adams wrote: >> >>>>> >> >>>>>> ex55.c in ksp is failing with: >> >>>>>> >> >>>>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >> >>>>>> [0]PETSC ERROR: Arguments are incompatible! >> >>>>>> [0]PETSC ERROR: Cannot change block size 1 to 2! >> >>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >> >>>>>> >> >>>>>> on this line 57 of ex55.c: >> >>>>>> >> >>>>>> ierr = MatSetBlockSize(Amat,2); CHKERRQ(ierr); >> >>>>>> >> >>>>>> Any idea what happened here? >> >>>>>> >> >>>>>> Mark >> >>>>> >> >>>>> >> >>>> >> >>> >> >>> >> >> >> > >> > >> >> >

1 0

Re: [petsc-dev] PetscTypeCompare
by Barry Smith 24 Apr '12

24 Apr '12

Yes On Apr 24, 2012, at 2:37 PM, Dmitry Karpeev wrote: > Should this be renamed PetscObjectTypeCompare, since it operates on PetscObjects? > Dmitry.

1 0

Re: [petsc-dev] Performance of VecMDot_SeqCUSP
by Daniel Lowell 24 Apr '12

24 Apr '12

Doing tree-wise parallel reductions on a single dot does leave most of the threads stalled by the end of the kernel execution. I keep this inefficiency built-in and take advantage of it by launching overlapping kernels and pipelining the dot operation, breaking it up into smaller kernels. Obviously this only makes sense for large vectors. I haven't tried implementing the entire MDot as one large kernel though. Might be worthwhile though.... On Tue, Apr 24, 2012 at 2:42 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Tue, Apr 24, 2012 at 14:29, Daniel Lowell <redratio1(a)gmail.com> wrote: > >> Launching smaller overlapping asynchronous kernels can have speed up if >> your vectors are large and you are doing reductions. This way warps stalls >> can be compensated for, and latencies can be hidden. Not sure what you mean >> "the way it currently is" though... > > > The reduction is only needed at the end. Any sequential launch adds > artificial synchronization. I'd be interested to see the performance > comparison, but I'd be surprised if independent kernel launches were faster > than a decent implementation with one kernel launch. > -- Daniel Lowell

1 0

Re: [petsc-dev] Performance of VecMDot_SeqCUSP
by Aron Ahmadia 24 Apr '12

24 Apr '12

I'm interested in seeing this too, especially if somebody can explain the results after they've been demonstrated :) A On Tue, Apr 24, 2012 at 10:42 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Tue, Apr 24, 2012 at 14:29, Daniel Lowell <redratio1(a)gmail.com> wrote: > >> Launching smaller overlapping asynchronous kernels can have speed up if >> your vectors are large and you are doing reductions. This way warps stalls >> can be compensated for, and latencies can be hidden. Not sure what you mean >> "the way it currently is" though... > > > The reduction is only needed at the end. Any sequential launch adds > artificial synchronization. I'd be interested to see the performance > comparison, but I'd be surprised if independent kernel launches were faster > than a decent implementation with one kernel launch. >

1 0

Re: [petsc-dev] Performance of VecMDot_SeqCUSP
by Jed Brown 24 Apr '12

24 Apr '12

On Tue, Apr 24, 2012 at 14:29, Daniel Lowell <redratio1(a)gmail.com> wrote: > Launching smaller overlapping asynchronous kernels can have speed up if > your vectors are large and you are doing reductions. This way warps stalls > can be compensated for, and latencies can be hidden. Not sure what you mean > "the way it currently is" though... The reduction is only needed at the end. Any sequential launch adds artificial synchronization. I'd be interested to see the performance comparison, but I'd be surprised if independent kernel launches were faster than a decent implementation with one kernel launch.

1 0

Re: [petsc-dev] MetSetBlockSize issue
by Mark F. Adams 24 Apr '12

24 Apr '12

On Apr 24, 2012, at 3:28 PM, Hong Zhang wrote: > Mark : > Shall C=PtAP inherit the block size of A? > Currently, MatPtAP() is only implemented with aij bs=1. No, as I said, algebraically it should inherit the column block size of P, but that is not available. Perhaps PtAP should take a parameter for the block size ... Mark > Hong > > On Apr 24, 2012, at 2:55 PM, Barry Smith wrote: > > > > > On Apr 24, 2012, at 1:39 PM, Mark F. Adams wrote: > > > >> Now I'm getting this error with code like this: > >> > >> ierr = MatGetSubMatrix( Cmat, new_eq_indices, new_eq_indices, MAT_INITIAL_MATRIX, &mat ); > >> CHKERRQ(ierr); > >> ierr = MatSetBlockSize( mat, cbs ); CHKERRQ(ierr); > >> > >> and like this: > >> > >> ierr = MatPtAP( Amat_fine, Pold, MAT_INITIAL_MATRIX, 2.0, &Cmat ); CHKERRQ(ierr); > >> ierr = MatSetBlockSize( Cmat, cbs ); CHKERRQ(ierr); > > > > Right you cannot do this. The matrix already exists so you cannot now set its size. > > > > Does Cmat have a block size of cbs > > > > What about Amat? By default these routines should create the new matrix with the correct blocksize. > > > > The harder part is if the blocksize of mat would be different than Cmat? > > > > It seems like MatGetSubMatrix should just inherit the block size but PtAP is harder. The block size of the Cmat is the column block size of P. But I can not set a column block size (!= row block size) so I don't set block size on P at all. Here cbs is this column block size of P, or what it should be. > > Mark > > > Barry > > > >> > >> Mark > >> > >> On Apr 23, 2012, at 9:09 PM, Barry Smith wrote: > >> > >>> > >>> Yes, look, for example how ex32 is handled at the bottom of the makefile. > >>> > >>> Thanks > >>> > >>> Barry > >>> > >>> On Apr 23, 2012, at 5:50 PM, Mark F. Adams wrote: > >>> > >>>> > >>>> On Apr 23, 2012, at 5:59 PM, Barry Smith wrote: > >>>> > >>>>> > >>>>> It was changed a while ago that MatSetBlockSize() couldn't be set after the matrix was full instantiated. I guess that example did not get fixed because it is not listed in the makefile to run in the makeall. > >>>> > >>>> May I add it? > >>>> > >>>>> > >>>>> I have fixed the example to call MatSetBlockSize() at a safe point. > >>>>> > >>>>> Barry > >>>>> > >>>>> On Apr 23, 2012, at 4:29 PM, Mark F. Adams wrote: > >>>>> > >>>>>> ex55.c in ksp is failing with: > >>>>>> > >>>>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > >>>>>> [0]PETSC ERROR: Arguments are incompatible! > >>>>>> [0]PETSC ERROR: Cannot change block size 1 to 2! > >>>>>> [0]PETSC ERROR: ------------------------------------------------------------------------ > >>>>>> > >>>>>> on this line 57 of ex55.c: > >>>>>> > >>>>>> ierr = MatSetBlockSize(Amat,2); CHKERRQ(ierr); > >>>>>> > >>>>>> Any idea what happened here? > >>>>>> > >>>>>> Mark > >>>>> > >>>>> > >>>> > >>> > >>> > >> > > > > > >

1 0

PetscTypeCompare
by Dmitry Karpeev 24 Apr '12

24 Apr '12

Should this be renamed PetscObjectTypeCompare, since it operates on PetscObjects? Dmitry.

1 0

Re: [petsc-dev] Performance of VecMDot_SeqCUSP
by Daniel Lowell 24 Apr '12

24 Apr '12

Launching smaller overlapping asynchronous kernels can have speed up if your vectors are large and you are doing reductions. This way warps stalls can be compensated for, and latencies can be hidden. Not sure what you mean "the way it currently is" though... On Tue, Apr 24, 2012 at 2:21 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Tue, Apr 24, 2012 at 14:12, Daniel Lowell <redratio1(a)gmail.com> wrote: > >> I'm writing a vector type with uses flag synching like you have in PETSc >> with Vec CUSP, however it uses asynchronous kernel launches >> (pipeling,etc..) and autotuned kernels. Not quite ready for primetime, but >> we have seen the value of it in terms of speed up. > > > Okay, but why do dozens of small kernel launches when all the data is > available up-front? I'm just skeptical that VecMDot should be implemented > for CUDA the way it currently is. > -- Daniel Lowell

1 0

Re: [petsc-dev] MetSetBlockSize issue
by Hong Zhang 24 Apr '12

24 Apr '12

Mark : Shall C=PtAP inherit the block size of A? Currently, MatPtAP() is only implemented with aij bs=1. Hong > > On Apr 24, 2012, at 2:55 PM, Barry Smith wrote: > > > > > On Apr 24, 2012, at 1:39 PM, Mark F. Adams wrote: > > > >> Now I'm getting this error with code like this: > >> > >> ierr = MatGetSubMatrix( Cmat, new_eq_indices, new_eq_indices, > MAT_INITIAL_MATRIX, &mat ); > >> CHKERRQ(ierr); > >> ierr = MatSetBlockSize( mat, cbs ); CHKERRQ(ierr); > >> > >> and like this: > >> > >> ierr = MatPtAP( Amat_fine, Pold, MAT_INITIAL_MATRIX, 2.0, &Cmat ); > CHKERRQ(ierr); > >> ierr = MatSetBlockSize( Cmat, cbs ); CHKERRQ(ierr); > > > > Right you cannot do this. The matrix already exists so you cannot now > set its size. > > > > Does Cmat have a block size of cbs > > > > What about Amat? By default these routines should create the new > matrix with the correct blocksize. > > > > The harder part is if the blocksize of mat would be different than > Cmat? > > > > It seems like MatGetSubMatrix should just inherit the block size but PtAP > is harder. The block size of the Cmat is the column block size of P. But > I can not set a column block size (!= row block size) so I don't set block > size on P at all. Here cbs is this column block size of P, or what it > should be. > > Mark > > > Barry > > > >> > >> Mark > >> > >> On Apr 23, 2012, at 9:09 PM, Barry Smith wrote: > >> > >>> > >>> Yes, look, for example how ex32 is handled at the bottom of the > makefile. > >>> > >>> Thanks > >>> > >>> Barry > >>> > >>> On Apr 23, 2012, at 5:50 PM, Mark F. Adams wrote: > >>> > >>>> > >>>> On Apr 23, 2012, at 5:59 PM, Barry Smith wrote: > >>>> > >>>>> > >>>>> It was changed a while ago that MatSetBlockSize() couldn't be set > after the matrix was full instantiated. I guess that example did not get > fixed because it is not listed in the makefile to run in the makeall. > >>>> > >>>> May I add it? > >>>> > >>>>> > >>>>> I have fixed the example to call MatSetBlockSize() at a safe point. > >>>>> > >>>>> Barry > >>>>> > >>>>> On Apr 23, 2012, at 4:29 PM, Mark F. Adams wrote: > >>>>> > >>>>>> ex55.c in ksp is failing with: > >>>>>> > >>>>>> [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > >>>>>> [0]PETSC ERROR: Arguments are incompatible! > >>>>>> [0]PETSC ERROR: Cannot change block size 1 to 2! > >>>>>> [0]PETSC ERROR: > ------------------------------------------------------------------------ > >>>>>> > >>>>>> on this line 57 of ex55.c: > >>>>>> > >>>>>> ierr = MatSetBlockSize(Amat,2); CHKERRQ(ierr); > >>>>>> > >>>>>> Any idea what happened here? > >>>>>> > >>>>>> Mark > >>>>> > >>>>> > >>>> > >>> > >>> > >> > > > > > >

1 0

Re: [petsc-dev] Performance of VecMDot_SeqCUSP
by Jed Brown 24 Apr '12

24 Apr '12

On Tue, Apr 24, 2012 at 14:12, Daniel Lowell <redratio1(a)gmail.com> wrote: > I'm writing a vector type with uses flag synching like you have in PETSc > with Vec CUSP, however it uses asynchronous kernel launches > (pipeling,etc..) and autotuned kernels. Not quite ready for primetime, but > we have seen the value of it in terms of speed up. Okay, but why do dozens of small kernel launches when all the data is available up-front? I'm just skeptical that VecMDot should be implemented for CUDA the way it currently is.

1 0