February 2016 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] [petsc-users] PETSc release soon, request for input on needed fixes or enhancements
by Barry Smith 29 Feb '16

29 Feb '16

Thanks, nothing alarming. Barry > On Feb 29, 2016, at 1:17 PM, Bhalla, Amneet Pal S <amneetb(a)live.unc.edu> wrote: > > > >> On Feb 29, 2016, at 8:51 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >> >> You could send me a trace file (like you did before) and I'll see if there is anything else that might be improved. > > Attached. > <Fieldsplit_Barry_branch.trace.zip> >

1 0

Re: [petsc-dev] [petsc-users] PETSc release soon, request for input on needed fixes or enhancements
by Bhalla, Amneet Pal S 29 Feb '16

29 Feb '16

On Feb 29, 2016, at 8:51 AM, Barry Smith <bsmith(a)mcs.anl.gov<mailto:[email protected]>> wrote: You could send me a trace file (like you did before) and I'll see if there is anything else that might be improved. Attached.

1 0

Re: [petsc-dev] [petsc-users] PETSc release soon, request for input on needed fixes or enhancements
by Barry Smith 29 Feb '16

29 Feb '16

> On Feb 29, 2016, at 12:28 PM, Bhalla, Amneet Pal S <amneetb(a)live.unc.edu> wrote: > > > >> On Feb 29, 2016, at 8:51 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >> >> The three options are all independent. Turn them all on for faster time --with-debugging=0 --with-viewfromoptions=0 --with-errorchecking=0 > > > I am trying out this. Just to be sure --with-viewfromoptions=0 does not however disable command line options given to various PETSc solvers, right? Correct. It does not turn off setting solver options from the command line. It only turns off things like -ksp_view -snes_view -ksp_view_mat etc that one doesn't usually want in production runs anyways. The reason to turn those off is that they are checked directly in the solver routines such as KSPSolve(), the solver options are checked in KSPSetFromOptions() etc which are not called for each solve. Barry

1 0

Re: [petsc-dev] [petsc-users] PETSc release soon, request for input on needed fixes or enhancements
by Bhalla, Amneet Pal S 29 Feb '16

29 Feb '16

On Feb 29, 2016, at 8:51 AM, Barry Smith <bsmith(a)mcs.anl.gov<mailto:[email protected]>> wrote: The three options are all independent. Turn them all on for faster time --with-debugging=0 --with-viewfromoptions=0 --with-errorchecking=0 I am trying out this. Just to be sure --with-viewfromoptions=0 does not however disable command line options given to various PETSc solvers, right?

1 0

Re: [petsc-dev] [petsc-users] PETSc release soon, request for input on needed fixes or enhancements
by Barry Smith 29 Feb '16

29 Feb '16

> On Feb 29, 2016, at 10:39 AM, Griffith, Boyce Eugene <boyceg(a)email.unc.edu> wrote: > >> >> On Feb 29, 2016, at 12:15 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >> >> >>> On Feb 27, 2016, at 8:22 PM, Boyce Griffith <griffith(a)cims.nyu.edu> wrote: >>> >>> Barry — >>> >>> I haven’t been following master, and so maybe these have already been addressed, but is there a chance that fixes for the performance issues Amneet found (options in setting up lots of KSPs in small-block ASM, >> >> Thanks for the reminder. I have added a new ./configure option--with-debugging=0 which will remove all of that overhead and likely perform better than the "hack" I suggested to apply earlier. This is in the branch barry/fixes-for-release-1 and will be in master if it passes all the test. >> >> I'll also remind you of the option --with-errorchecking=0 which turns off additional overhead that in theory is not needed in production runs and could help a bit more when using many small solvers. > > Thanks Barry! Is --with-errorchecking=0 implied by --with-debugging=0 (or vice versa), or are these somewhat independent? We were running the timing tests with --with-debugging=0 but not with --with-errorchecking=0. The three options are all independent. Turn them all on for faster time --with-debugging=0 --with-viewfromoptions=0 --with-errorchecking=0 > > Amneet, can you please verify that the performance issues you uncovered are fixed with a clean checkout of the PETSc branch Barry mentions above along with the foregoing PETSc configure options? You could send me a trace file (like you did before) and I'll see if there is anything else that might be improved. Barry > > Thanks, > > -- Boyce > >> >> Please let us know if there are any difficulties. >> >> Barry >> >> >>> and logging of IS objects) will make it into 3.7? >>> >>> Thanks, >>> >>> — Boyce >>> >>>> On Feb 27, 2016, at 3:36 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >>>> >>>> >>>> PETSc Users, >>>> >>>> We are planning the PETSc release 3.7 shortly. If you know of any bugs that need to be fixed or enhancements added before the release please let us know. >>>> >>>> You can think of the master branch of the PETSc repository obtainable with >>>> >>>> git clone https://bitbucket.org/petsc/petsc petsc >>>> >>>> as a release candidate for 3.7. Changes for the release are listed at http://www.mcs.anl.gov/petsc/documentation/changes/dev.html >>>> >>>> Thanks >>>> >>>> Barry

1 0

Re: [petsc-dev] [petsc-users] PETSc release soon, request for input on needed fixes or enhancements
by Griffith, Boyce Eugene 29 Feb '16

29 Feb '16

> On Feb 29, 2016, at 12:15 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > >> On Feb 27, 2016, at 8:22 PM, Boyce Griffith <griffith(a)cims.nyu.edu> wrote: >> >> Barry — >> >> I haven’t been following master, and so maybe these have already been addressed, but is there a chance that fixes for the performance issues Amneet found (options in setting up lots of KSPs in small-block ASM, > > Thanks for the reminder. I have added a new ./configure option --with-viewfromoptions=0 which will remove all of that overhead and likely perform better than the "hack" I suggested to apply earlier. This is in the branch barry/fixes-for-release-1 and will be in master if it passes all the test. > > I'll also remind you of the option --with-errorchecking=0 which turns off additional overhead that in theory is not needed in production runs and could help a bit more when using many small solvers. Thanks Barry! Is --with-errorchecking=0 implied by --with-debugging=0 (or vice versa), or are these somewhat independent? We were running the timing tests with --with-debugging=0 but not with --with-errorchecking=0. Amneet, can you please verify that the performance issues you uncovered are fixed with a clean checkout of the PETSc branch Barry mentions above along with the foregoing PETSc configure options? Thanks, -- Boyce > > Please let us know if there are any difficulties. > > Barry > > >> and logging of IS objects) will make it into 3.7? >> >> Thanks, >> >> — Boyce >> >>> On Feb 27, 2016, at 3:36 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >>> >>> >>> PETSc Users, >>> >>> We are planning the PETSc release 3.7 shortly. If you know of any bugs that need to be fixed or enhancements added before the release please let us know. >>> >>> You can think of the master branch of the PETSc repository obtainable with >>> >>> git clone https://bitbucket.org/petsc/petsc petsc >>> >>> as a release candidate for 3.7. Changes for the release are listed at http://www.mcs.anl.gov/petsc/documentation/changes/dev.html >>> >>> Thanks >>> >>> Barry >>> >>

1 0

Re: [petsc-dev] Not possible to do a VecPlaceArray for veccusp
by Karl Rupp 29 Feb '16

29 Feb '16

> This is not about performance, but about providing the ability for > users to 'implant' their own memory buffers. CUSP doesn't allow it > (which was the initial point of this thread). > > > Thanks. Sorry I missed that. Given CPU memcpy is an order of magnitude > more bandwidth than PCI offload, I still don't get the point, but I > don't need to. User-provided GPU memory buffers. CUDA buffers. Avoiding PCI Express ;-) Best regards, Karli

1 0

Re: [petsc-dev] Not possible to do a VecPlaceArray for veccusp
by Jeff Hammond 29 Feb '16

29 Feb '16

On Monday, February 29, 2016, Karl Rupp <rupp(a)iue.tuwien.ac.at> wrote: > Hi Jeff, > > > Ok, this is on fairly short notice considering the changes required. > >> I recommend to start with copying the CUSP sources and migrate it >> over to VECCUDA by replacing any use of cusp::array1d to a raw CUDA >> handle. Operations from CUSP should be replaced by CUBLAS calls. >> >> >> It's hard to imagine any performance benefit from this unless CUSP >> sucks. What am I missing? >> > > This is not about performance, but about providing the ability for users > to 'implant' their own memory buffers. CUSP doesn't allow it (which was the > initial point of this thread). > > Thanks. Sorry I missed that. Given CPU memcpy is an order of magnitude more bandwidth than PCI offload, I still don't get the point, but I don't need to. Jeff > Best regards, > Karli > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [petsc-dev] Not possible to do a VecPlaceArray for veccusp
by Karl Rupp 29 Feb '16

29 Feb '16

Hi Jeff, > Ok, this is on fairly short notice considering the changes required. > I recommend to start with copying the CUSP sources and migrate it > over to VECCUDA by replacing any use of cusp::array1d to a raw CUDA > handle. Operations from CUSP should be replaced by CUBLAS calls. > > > It's hard to imagine any performance benefit from this unless CUSP > sucks. What am I missing? This is not about performance, but about providing the ability for users to 'implant' their own memory buffers. CUSP doesn't allow it (which was the initial point of this thread). Best regards, Karli

1 0

Re: [petsc-dev] [petsc-users] PETSc release soon, request for input on needed fixes or enhancements
by Barry Smith 28 Feb '16

28 Feb '16

> On Feb 27, 2016, at 8:22 PM, Boyce Griffith <griffith(a)cims.nyu.edu> wrote: > > Barry — > > I haven’t been following master, and so maybe these have already been addressed, but is there a chance that fixes for the performance issues Amneet found (options in setting up lots of KSPs in small-block ASM, Thanks for the reminder. I have added a new ./configure option --with-viewfromoptions=0 which will remove all of that overhead and likely perform better than the "hack" I suggested to apply earlier. This is in the branch barry/fixes-for-release-1 and will be in master if it passes all the test. I'll also remind you of the option --with-errorchecking=0 which turns off additional overhead that in theory is not needed in production runs and could help a bit more when using many small solvers. Please let us know if there are any difficulties. Barry > and logging of IS objects) will make it into 3.7? > > Thanks, > > — Boyce > >> On Feb 27, 2016, at 3:36 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >> >> >> PETSc Users, >> >> We are planning the PETSc release 3.7 shortly. If you know of any bugs that need to be fixed or enhancements added before the release please let us know. >> >> You can think of the master branch of the PETSc repository obtainable with >> >> git clone https://bitbucket.org/petsc/petsc petsc >> >> as a release candidate for 3.7. Changes for the release are listed at http://www.mcs.anl.gov/petsc/documentation/changes/dev.html >> >> Thanks >> >> Barry >> >

1 0