April 2011 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] [petsc-maint #69143] petsc-dev performance issues
by Barry Smith 18 Apr '11

18 Apr '11

If anyone has started work on fixing this, please let me know. Otherwise I'll start shortly. Barry On Apr 18, 2011, at 8:24 AM, Barry Smith wrote: > > Matt, > > Yes, this is the problem. Since we cannot know if anyone has called a PetscSynchronizedPrintf() it has to call this flush to be safe and that, to be safe, has to have the extra message passing. Even if we handle 1,000 processes at a time, with 100,000 processes it will take 100 stages which is too long. > > I think we have to do a major rearchitecturing. Maybe we want to have a flag (set to false by default) in each ASCII viewer indicating that it has synchronize prints. When the flag is false it skips the entire flush and if a synchronize print is done it generates an error. We'll that is easily done. > > Barry > > > > > > > On Apr 18, 2011, at 8:10 AM, Matthew Knepley wrote: > >> On Mon, Apr 18, 2011 at 7:52 AM, Sebastian Steiger <steiger(a)purdue.edu>wrote: >> >>> >>> On 4/18/2011 7:25 AM, Matthew Knepley wrote: >>>> On Sun, Apr 17, 2011 at 11:59 PM, Sebastian Steiger <steiger(a)purdue.edu >>>> >>>> 17631 is in >>>> PetscSynchronizedFlush(), a routine which is not called from >>>> >>>> KSPMonitorDefault() >>>> ---> PetscViewerASCIIMonitorPrintf() >>> >> >> I am looking over the code, and I think that change is going to be bad for >> everyone >> because we have >> >> PetscViewerFlush_ASCII() >> --> PetscSynchronizedFlush() >> --> ping-pong between 0 and everyone >> >> Anytime someone calls flush, the process cannot just send 0 and move on, it >> has >> to wait. >> >> Matt >> >> >>>> and thus would not change the performance. My guess is that you have >>> another >>>> print statement using PetscSynchronizedPrintf/Flush() somewhere in the >>> code. >>>> >>>> Matt >>> >>> >>> In addition to the KSPMonitor, I've hooked up the petsc output with my >>> own output system as it is written on the webpage: >>> >>> >>> PetscErrorCode my_double_petsc_vfprintf(FILE * fd, const char format[], >>> va_list Argp) >>> { >>> PetscErrorCode ierr; >>> >>> //PetscFunctionBegin; // <ss> gave problems in petsc-3.2 - not >>> essential according to Barry and Matt >>> if (fd != stdout && fd != stderr) { /* handle regular files */ >>> ierr = PetscVFPrintfDefault(fd,format,Argp); >>> PetscMatrixNemo<double>::check_petsc_error(ierr); >>> } else { >>> char buff[2000]; >>> #if PETSC_VERSION_RELEASE == 1 >>> int length; >>> #else >>> size_t length; >>> #endif >>> ierr = PetscVSNPrintf(buff, 2000, format, &length, Argp); >>> PetscMatrixNemo<double>::check_petsc_error(ierr); >>> /* now send buff to whatever stream or whatever you want */ >>> msg << buff; >>> >>> // flush buffer every 2nd time >>> num_double_petsc_lines++; >>> if (num_double_petsc_lines==2) { >>> msg.flush(); >>> num_double_petsc_lines = 0; >>> } >>> } >>> //PetscFunctionReturn(0); >>> return 0; >>> } >>> >>> Is that connected to the problem? Please note that the "flush buffer >>> every 2nd time" does not cause the problems, as it is on my side. >>> >>> Sebastian >>> >>> >>> >>>> >>>> >>>> Sebastian >>>> >>>> On 4/17/2011 9:16 PM, Barry Smith wrote: >>>>> >>>>> On Apr 17, 2011, at 8:11 PM, Matthew Knepley wrote: >>>>> >>>>>> On Sun, Apr 17, 2011 at 8:05 PM, Barry Smith <bsmith(a)mcs.anl.gov >>>> <mailto:[email protected]>> wrote: >>>>>> >>>>>> I don't understand, what is different and why does it slow >>>> everything down? >>>>>> >>>>>> You introduced a ping-pong between 0 and every process for every >>>> call to >>>>>> PetscSynchronizedFlush() in order to "stop a flood of messages to >>>> 0". As >>>>>> process count increases, this is going to be horrible. >>>>> >>>>> Oh yes, this will do it Is this being used by one of our >>>> monitor routines? If so we need to fix our monitors to not use >>>> synchronized printf. So what in the code is calling this. >>>>> >>>>> >>>>>> >>>>>> This is just the kind of thing that MPI should be solving for us, >>>> instead of us >>>>>> putting in bullshit code like this to try and regulate message >>>> counts. Is there >>>>>> an MPI construct to manage this? >>>>> >>>>> One could use some kind of all to one to get the messages down >>>> to process zero. >>>>> >>>>> Barry >>>>> >>>>>> >>>>>> Matt >>>>>> >>>>>> Barry >>>>>> >>>>>> On Apr 17, 2011, at 6:50 PM, Sebastian Steiger wrote: >>>>>> >>>>>>> Barry, Matt, Satish, >>>>>>> >>>>>>> Found it. r16730 is good, r16731 is bad. Matt was right: when I >>>> turn off >>>>>>> the petsc messages generated by monitoring the iterations of the >>>> linear >>>>>>> solver, then the revisions have comparable speed. When I turn it >>> on, >>>>>>> then 16731 is 2-3 times slower. >>>>>>> >>>>>>> Please fix! >>>>>>> >>>>>>> Best >>>>>>> Sebastian >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 04/15/2011 07:31 AM, Matthew Knepley wrote: >>>>>>>> On Thu, Apr 14, 2011 at 11:33 PM, Satish Balay >>>> <petsc-maint(a)mcs.anl.gov <mailto:[email protected]> >>>>>>>> <mailto:[email protected] >>>> <mailto:[email protected]>>> wrote: >>>>>>>> >>>>>>>> It would be good to complete the bisection to confirm - but >>>> could it >>>>>>>> be this? >>>>>>>> >>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/rev/10be2c80ec26 >>>>>>>> >>>>>>>> CHKMEMQ is disabled only with --with-errorchecking=0 - but >>> the >>>>>>>> assumpton for this change was that it would be disabled for >>>>>>>> --with-debugging=0 ? >>>>>>>> >>>>>>>> >>>>>>>> This is my bet for bad set: >>>>>>>> >>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/rev/ee1bdc170b4d >>>>>>>> >>>>>>>> It introduces new messages, but only of length 1. Is your code >>>> printing >>>>>>>> anything at all >>>>>>>> during the run? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>> Satish >>>>>>>> >>>>>>>> >>>>>>>> On Thu, 14 Apr 2011, Sebastian Steiger wrote: >>>>>>>> >>>>>>>>> I'm sorry, actually 16721 is good and 16733 is bad. So it's 12 >>>>>>>> revisions. >>>>>>>>> >>>>>>>>> On 4/14/2011 9:12 PM, Sebastian Steiger wrote: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I've pinned it down to 16 revisions: 16721 is good, 16737 is >>>>>>>> bad. The >>>>>>>>>> next bisection step would be 16727, but from the log I see >>> that >>>>>>>>>> 16727-16736 might have some problems. Should I continue or do >>>>>>>> you guys >>>>>>>>>> now have a clue what could cause the factor 2-3 slowdown? >>>>>>>>>> >>>>>>>>>> Sebastian >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>> steiger@jaguarpf-login6 >>> :~/src/app-nemo/NEMO/libs/petsc-dev/petsc-dev$ >>>> hg >>>>>>>>>> log -r16721:16737 >>>>>>>>>> changeset: 16721:b788c229de4d >>>>>>>>>> parent: 16720:9aafffdcf68b >>>>>>>>>> parent: 16715:e53f46924236 >>>>>>>>>> user: Jed Brown <jed(a)59A2.org> >>>>>>>>>> date: Mon Aug 09 17:30:28 2010 -0800 >>>>>>>>>> summary: merge >>>>>>>>>> >>>>>>>>>> changeset: 16722:4ca941c57754 >>>>>>>>>> user: hong(a)vis-v410v166.mcs.anl-external.org >>>> <mailto:[email protected]> >>>>>>>> <mailto:[email protected] >>>> <mailto:[email protected]>> >>>>>>>>>> date: Tue Aug 10 11:57:11 2010 -0500 >>>>>>>>>> summary: add case of np=1 for >>> MatGetMultiProcBlock_MPIAIJ() >>>>>>>>>> >>>>>>>>>> changeset: 16723:4d6b54981472 >>>>>>>>>> user: hong(a)vis-v410v166.mcs.anl-external.org >>>> <mailto:[email protected]> >>>>>>>> <mailto:[email protected] >>>> <mailto:[email protected]>> >>>>>>>>>> date: Tue Aug 10 12:00:14 2010 -0500 >>>>>>>>>> summary: fix memory leak in MatHeaderCopy() and >>>>>>>> MatHeaderReplace() >>>>>>>>>> >>>>>>>>>> changeset: 16724:2680bde1c9b8 >>>>>>>>>> user: Shri(a)mcswl165.mcs.anl.gov >>>> <mailto:[email protected]> <mailto:[email protected] >>>> <mailto:[email protected]>> >>>>>>>>>> date: Thu Aug 12 14:37:43 2010 -0500 >>>>>>>>>> summary: Started adding semismooth stuff. >>>>>>>>>> >>>>>>>>>> changeset: 16725:50dceda7151c >>>>>>>>>> user: Shri(a)mcswl165.mcs.anl.gov >>>> <mailto:[email protected]> <mailto:[email protected] >>>> <mailto:[email protected]>> >>>>>>>>>> date: Thu Aug 12 14:45:13 2010 -0500 >>>>>>>>>> summary: Defined Infinity and epsilon >>>>>>>>>> >>>>>>>>>> changeset: 16726:c0f1b46d675f >>>>>>>>>> user: hzhang >>>>>>>>>> date: Fri Aug 13 11:10:37 2010 -0500 >>>>>>>>>> summary: update outputfiles for nightly tests >>>>>>>>>> >>>>>>>>>> changeset: 16727:67df4fdba5bd >>>>>>>>>> user: hong(a)hzhangmac.mcs.anl.gov >>>> <mailto:[email protected]> >>>>>>>> <mailto:[email protected] >>>> <mailto:[email protected]>> >>>>>>>>>> date: Fri Aug 13 13:25:56 2010 -0500 >>>>>>>>>> summary: add PetscSubcommType in PetscSubcommCreate(); add >>>>>>>>>> PetscSubcommCreate_contiguous(). >>>>>>>>>> >>>>>>>>>> changeset: 16728:10be2c80ec26 >>>>>>>>>> parent: 16723:4d6b54981472 >>>>>>>>>> user: Barry Smith bsmith(a)mcs.anl.gov >>>> <mailto:[email protected]> >>>>>>>> <mailto:[email protected] <mailto:[email protected]>> >>>>>>>>>> date: Tue Aug 10 18:15:10 2010 -0500 >>>>>>>>>> summary: merged CHKMEMQ; into PetscStackPush() and >>>>>>>> PetscStackPop() >>>>>>>>>> >>>>>>>>>> changeset: 16729:ef8e7a4583e6 >>>>>>>>>> user: Barry Smith bsmith(a)mcs.anl.gov >>>> <mailto:[email protected]> >>>>>>>> <mailto:[email protected] <mailto:[email protected]>> >>>>>>>>>> date: Thu Aug 12 14:16:40 2010 -0500 >>>>>>>>>> summary: changed MatHeaderCopy() to MatHeaderMerge() and >>>>>>>> improved >>>>>>>>>> comment >>>>>>>>>> >>>>>>>>>> changeset: 16730:50055ed8b725 >>>>>>>>>> parent: 16729:ef8e7a4583e6 >>>>>>>>>> parent: 16725:50dceda7151c >>>>>>>>>> user: Barry Smith bsmith(a)mcs.anl.gov >>>> <mailto:[email protected]> >>>>>>>> <mailto:[email protected] <mailto:[email protected]>> >>>>>>>>>> date: Thu Aug 12 16:24:48 2010 -0500 >>>>>>>>>> summary: commit after merge >>>>>>>>>> >>>>>>>>>> changeset: 16731:ee1bdc170b4d >>>>>>>>>> user: Barry Smith bsmith(a)mcs.anl.gov >>>> <mailto:[email protected]> >>>>>>>> <mailto:[email protected] <mailto:[email protected]>> >>>>>>>>>> date: Fri Aug 13 13:50:30 2010 -0500 >>>>>>>>>> summary: minor formating of short lines >>>>>>>>>> >>>>>>>>>> changeset: 16732:a49d9a4b41bc >>>>>>>>>> parent: 16731:ee1bdc170b4d >>>>>>>>>> parent: 16727:67df4fdba5bd >>>>>>>>>> user: Barry Smith bsmith(a)mcs.anl.gov >>>> <mailto:[email protected]> >>>>>>>> <mailto:[email protected] <mailto:[email protected]>> >>>>>>>>>> date: Fri Aug 13 13:50:43 2010 -0500 >>>>>>>>>> summary: commit after merge >>>>>>>>>> >>>>>>>>>> changeset: 16733:804ce43d15c9 >>>>>>>>>> user: Jed Brown <jed(a)59A2.org> >>>>>>>>>> date: Fri Aug 13 11:01:24 2010 -0800 >>>>>>>>>> summary: Update converged reason for Rosenbrock demo to be >>>>>>>> more portable >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to >>>> which >>>>>>>> their experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> >>>>>>> <bisection-conclusion.txt> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which >>>> their experiments lead. >>>>>> -- Norbert Wiener >>>>> >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which >>>> their experiments lead. >>>> -- Norbert Wiener >>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their experiments >> is infinitely more interesting than any results to which their experiments >> lead. >> -- Norbert Wiener >> >> On Mon, Apr 18, 2011 at 7:52 AM, Sebastian Steiger <steiger(a)purdue.edu> wrote: >> >> On 4/18/2011 7:25 AM, Matthew Knepley wrote: >>> On Sun, Apr 17, 2011 at 11:59 PM, Sebastian Steiger <steiger(a)purdue.edu >>> <mailto:[email protected]>> wrote: >>> >>> On a side note, why did that PetscPrintf function not show up in the log >>> summary? That could have spared me the whole bisection process. >>> >>> >>> I cannot make sense of the problem from your report. The only change in >>> 17631 is in >>> PetscSynchronizedFlush(), a routine which is not called from >>> >>> KSPMonitorDefault() >>> ---> PetscViewerASCIIMonitorPrintf() >> >> I am looking over the code, and I think that change is going to be bad for everyone >> because we have >> >> PetscViewerFlush_ASCII() >> --> PetscSynchronizedFlush() >> --> ping-pong between 0 and everyone >> >> Anytime someone calls flush, the process cannot just send 0 and move on, it has >> to wait. >> >> Matt >> >>> and thus would not change the performance. My guess is that you have another >>> print statement using PetscSynchronizedPrintf/Flush() somewhere in the code. >>> >>> Matt >> >> >> In addition to the KSPMonitor, I've hooked up the petsc output with my >> own output system as it is written on the webpage: >> >> >> PetscErrorCode my_double_petsc_vfprintf(FILE * fd, const char format[], >> va_list Argp) >> { >> PetscErrorCode ierr; >> >> //PetscFunctionBegin; // <ss> gave problems in petsc-3.2 - not >> essential according to Barry and Matt >> if (fd != stdout && fd != stderr) { /* handle regular files */ >> ierr = PetscVFPrintfDefault(fd,format,Argp); >> PetscMatrixNemo<double>::check_petsc_error(ierr); >> } else { >> char buff[2000]; >> #if PETSC_VERSION_RELEASE == 1 >> int length; >> #else >> size_t length; >> #endif >> ierr = PetscVSNPrintf(buff, 2000, format, &length, Argp); >> PetscMatrixNemo<double>::check_petsc_error(ierr); >> /* now send buff to whatever stream or whatever you want */ >> msg << buff; >> >> // flush buffer every 2nd time >> num_double_petsc_lines++; >> if (num_double_petsc_lines==2) { >> msg.flush(); >> num_double_petsc_lines = 0; >> } >> } >> //PetscFunctionReturn(0); >> return 0; >> } >> >> Is that connected to the problem? Please note that the "flush buffer >> every 2nd time" does not cause the problems, as it is on my side. >> >> Sebastian >> >> >> >>> >>> >>> Sebastian >>> >>> On 4/17/2011 9:16 PM, Barry Smith wrote: >>>> >>>> On Apr 17, 2011, at 8:11 PM, Matthew Knepley wrote: >>>> >>>>> On Sun, Apr 17, 2011 at 8:05 PM, Barry Smith <bsmith(a)mcs.anl.gov >>> <mailto:[email protected]>> wrote: >>>>> >>>>> I don't understand, what is different and why does it slow >>> everything down? >>>>> >>>>> You introduced a ping-pong between 0 and every process for every >>> call to >>>>> PetscSynchronizedFlush() in order to "stop a flood of messages to >>> 0". As >>>>> process count increases, this is going to be horrible. >>>> >>>> Oh yes, this will do it Is this being used by one of our >>> monitor routines? If so we need to fix our monitors to not use >>> synchronized printf. So what in the code is calling this. >>>> >>>> >>>>> >>>>> This is just the kind of thing that MPI should be solving for us, >>> instead of us >>>>> putting in bullshit code like this to try and regulate message >>> counts. Is there >>>>> an MPI construct to manage this? >>>> >>>> One could use some kind of all to one to get the messages down >>> to process zero. >>>> >>>> Barry >>>> >>>>> >>>>> Matt >>>>> >>>>> Barry >>>>> >>>>> On Apr 17, 2011, at 6:50 PM, Sebastian Steiger wrote: >>>>> >>>>>> Barry, Matt, Satish, >>>>>> >>>>>> Found it. r16730 is good, r16731 is bad. Matt was right: when I >>> turn off >>>>>> the petsc messages generated by monitoring the iterations of the >>> linear >>>>>> solver, then the revisions have comparable speed. When I turn it on, >>>>>> then 16731 is 2-3 times slower. >>>>>> >>>>>> Please fix! >>>>>> >>>>>> Best >>>>>> Sebastian >>>>>> >>>>>> >>>>>> >>>>>> On 04/15/2011 07:31 AM, Matthew Knepley wrote: >>>>>>> On Thu, Apr 14, 2011 at 11:33 PM, Satish Balay >>> <petsc-maint(a)mcs.anl.gov <mailto:[email protected]> >>>>>>> <mailto:[email protected] >>> <mailto:[email protected]>>> wrote: >>>>>>> >>>>>>> It would be good to complete the bisection to confirm - but >>> could it >>>>>>> be this? >>>>>>> >>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/rev/10be2c80ec26 >>>>>>> >>>>>>> CHKMEMQ is disabled only with --with-errorchecking=0 - but the >>>>>>> assumpton for this change was that it would be disabled for >>>>>>> --with-debugging=0 ? >>>>>>> >>>>>>> >>>>>>> This is my bet for bad set: >>>>>>> >>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/rev/ee1bdc170b4d >>>>>>> >>>>>>> It introduces new messages, but only of length 1. Is your code >>> printing >>>>>>> anything at all >>>>>>> during the run? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>> Satish >>>>>>> >>>>>>> >>>>>>> On Thu, 14 Apr 2011, Sebastian Steiger wrote: >>>>>>> >>>>>>>> I'm sorry, actually 16721 is good and 16733 is bad. So it's 12 >>>>>>> revisions. >>>>>>>> >>>>>>>> On 4/14/2011 9:12 PM, Sebastian Steiger wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I've pinned it down to 16 revisions: 16721 is good, 16737 is >>>>>>> bad. The >>>>>>>>> next bisection step would be 16727, but from the log I see that >>>>>>>>> 16727-16736 might have some problems. Should I continue or do >>>>>>> you guys >>>>>>>>> now have a clue what could cause the factor 2-3 slowdown? >>>>>>>>> >>>>>>>>> Sebastian >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>> steiger@jaguarpf-login6:~/src/app-nemo/NEMO/libs/petsc-dev/petsc-dev$ >>> hg >>>>>>>>> log -r16721:16737 >>>>>>>>> changeset: 16721:b788c229de4d >>>>>>>>> parent: 16720:9aafffdcf68b >>>>>>>>> parent: 16715:e53f46924236 >>>>>>>>> user: Jed Brown <jed(a)59A2.org> >>>>>>>>> date: Mon Aug 09 17:30:28 2010 -0800 >>>>>>>>> summary: merge >>>>>>>>> >>>>>>>>> changeset: 16722:4ca941c57754 >>>>>>>>> user: hong(a)vis-v410v166.mcs.anl-external.org >>> <mailto:[email protected]> >>>>>>> <mailto:[email protected] >>> <mailto:[email protected]>> >>>>>>>>> date: Tue Aug 10 11:57:11 2010 -0500 >>>>>>>>> summary: add case of np=1 for MatGetMultiProcBlock_MPIAIJ() >>>>>>>>> >>>>>>>>> changeset: 16723:4d6b54981472 >>>>>>>>> user: hong(a)vis-v410v166.mcs.anl-external.org >>> <mailto:[email protected]> >>>>>>> <mailto:[email protected] >>> <mailto:[email protected]>> >>>>>>>>> date: Tue Aug 10 12:00:14 2010 -0500 >>>>>>>>> summary: fix memory leak in MatHeaderCopy() and >>>>>>> MatHeaderReplace() >>>>>>>>> >>>>>>>>> changeset: 16724:2680bde1c9b8 >>>>>>>>> user: Shri(a)mcswl165.mcs.anl.gov >>> <mailto:[email protected]> <mailto:[email protected] >>> <mailto:[email protected]>> >>>>>>>>> date: Thu Aug 12 14:37:43 2010 -0500 >>>>>>>>> summary: Started adding semismooth stuff. >>>>>>>>> >>>>>>>>> changeset: 16725:50dceda7151c >>>>>>>>> user: Shri(a)mcswl165.mcs.anl.gov >>> <mailto:[email protected]> <mailto:[email protected] >>> <mailto:[email protected]>> >>>>>>>>> date: Thu Aug 12 14:45:13 2010 -0500 >>>>>>>>> summary: Defined Infinity and epsilon >>>>>>>>> >>>>>>>>> changeset: 16726:c0f1b46d675f >>>>>>>>> user: hzhang >>>>>>>>> date: Fri Aug 13 11:10:37 2010 -0500 >>>>>>>>> summary: update outputfiles for nightly tests >>>>>>>>> >>>>>>>>> changeset: 16727:67df4fdba5bd >>>>>>>>> user: hong(a)hzhangmac.mcs.anl.gov >>> <mailto:[email protected]> >>>>>>> <mailto:[email protected] >>> <mailto:[email protected]>> >>>>>>>>> date: Fri Aug 13 13:25:56 2010 -0500 >>>>>>>>> summary: add PetscSubcommType in PetscSubcommCreate(); add >>>>>>>>> PetscSubcommCreate_contiguous(). >>>>>>>>> >>>>>>>>> changeset: 16728:10be2c80ec26 >>>>>>>>> parent: 16723:4d6b54981472 >>>>>>>>> user: Barry Smith bsmith(a)mcs.anl.gov >>> <mailto:[email protected]> >>>>>>> <mailto:[email protected] <mailto:[email protected]>> >>>>>>>>> date: Tue Aug 10 18:15:10 2010 -0500 >>>>>>>>> summary: merged CHKMEMQ; into PetscStackPush() and >>>>>>> PetscStackPop() >>>>>>>>> >>>>>>>>> changeset: 16729:ef8e7a4583e6 >>>>>>>>> user: Barry Smith bsmith(a)mcs.anl.gov >>> <mailto:[email protected]> >>>>>>> <mailto:[email protected] <mailto:[email protected]>> >>>>>>>>> date: Thu Aug 12 14:16:40 2010 -0500 >>>>>>>>> summary: changed MatHeaderCopy() to MatHeaderMerge() and >>>>>>> improved >>>>>>>>> comment >>>>>>>>> >>>>>>>>> changeset: 16730:50055ed8b725 >>>>>>>>> parent: 16729:ef8e7a4583e6 >>>>>>>>> parent: 16725:50dceda7151c >>>>>>>>> user: Barry Smith bsmith(a)mcs.anl.gov >>> <mailto:[email protected]> >>>>>>> <mailto:[email protected] <mailto:[email protected]>> >>>>>>>>> date: Thu Aug 12 16:24:48 2010 -0500 >>>>>>>>> summary: commit after merge >>>>>>>>> >>>>>>>>> changeset: 16731:ee1bdc170b4d >>>>>>>>> user: Barry Smith bsmith(a)mcs.anl.gov >>> <mailto:[email protected]> >>>>>>> <mailto:[email protected] <mailto:[email protected]>> >>>>>>>>> date: Fri Aug 13 13:50:30 2010 -0500 >>>>>>>>> summary: minor formating of short lines >>>>>>>>> >>>>>>>>> changeset: 16732:a49d9a4b41bc >>>>>>>>> parent: 16731:ee1bdc170b4d >>>>>>>>> parent: 16727:67df4fdba5bd >>>>>>>>> user: Barry Smith bsmith(a)mcs.anl.gov >>> <mailto:[email protected]> >>>>>>> <mailto:[email protected] <mailto:[email protected]>> >>>>>>>>> date: Fri Aug 13 13:50:43 2010 -0500 >>>>>>>>> summary: commit after merge >>>>>>>>> >>>>>>>>> changeset: 16733:804ce43d15c9 >>>>>>>>> user: Jed Brown <jed(a)59A2.org> >>>>>>>>> date: Fri Aug 13 11:01:24 2010 -0800 >>>>>>>>> summary: Update converged reason for Rosenbrock demo to be >>>>>>> more portable >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to >>> which >>>>>>> their experiments lead. >>>>>>> -- Norbert Wiener >>>>>> >>>>>> >>>>>> <bisection-conclusion.txt> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which >>> their experiments lead. >>>>> -- Norbert Wiener >>>> >>> >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which >>> their experiments lead. >>> -- Norbert Wiener >> >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >

1 0

Re: [petsc-dev] Matrix Options shown too many times in -help
by Barry Smith 18 Apr '11

18 Apr '11

Jed, If you are going to do this for Mat then you should do it universally for all objects. Each object prints its help messages only once. All of this can be hidden inside the PetscOptionsBegin .... infrastructure so you do not need to modify the Mat source code or any other object source code. Now finish your dang dissertation, if we keep this up you'll have totally rewritten PETSc during the time you should have written your dissertation. Good for us but not so good for you. I'll point out that this disease is contagious, I'm finishing up the massive destructor change to PETSc when I should be working on some dang SciDAC proposals. Barry On Apr 18, 2011, at 8:15 AM, Aron Ahmadia wrote: > Agreed, the overwhelming output from -h is tremendously annoying, and I cannot always grep because I don't know what I'm grepping for :( > > A > > On Mon, Apr 18, 2011 at 3:43 PM, Jed Brown <jed(a)59a2.org> wrote: > Currently Matrix Options are printed in MatView_Private which is called from many places, e.g. > > MatAssemblyEnd > MatZeroRowsColumns > MatZeroRows > MatLUFactorNumeric > MatCholeskyFactorNumeric > > The problem here is that in time-stepping or a multigrid solve, it's common to have these functions called hundreds or thousands of times which is really overwhelming to -help output. One solution would be to store on a per-matrix basis whether help has been displayed and only view it once (this implies caching all the -mat_view_* options in the Mat, or calling with PetscOptionsGetBool after the first time). The options could even be moved to MatSetFromOptions, but that doesn't get called nearly as much as we would probably like. > > There is also the somewhat deeper problem of multiple objects with the same prefix (often no prefix). This is most visible for matrices inside of multigrid and/or fieldsplit. It is easy to have more than 20 matrices with no prefix so even if the multiple-output problem above was solved, it would still be fairly overwhelming. I think this is somewhat harder to fix because it would imply a per-prefix log of whether the option has been printed. I think that the one special case of no-prefix Mat would already be a significant usability improvement. >

1 0

Re: [petsc-dev] [petsc-maint #69143] petsc-dev performance issues
by Barry Smith 18 Apr '11

18 Apr '11

Matt, Yes, this is the problem. Since we cannot know if anyone has called a PetscSynchronizedPrintf() it has to call this flush to be safe and that, to be safe, has to have the extra message passing. Even if we handle 1,000 processes at a time, with 100,000 processes it will take 100 stages which is too long. I think we have to do a major rearchitecturing. Maybe we want to have a flag (set to false by default) in each ASCII viewer indicating that it has synchronize prints. When the flag is false it skips the entire flush and if a synchronize print is done it generates an error. We'll that is easily done. Barry On Apr 18, 2011, at 8:10 AM, Matthew Knepley wrote: > On Mon, Apr 18, 2011 at 7:52 AM, Sebastian Steiger <steiger(a)purdue.edu>wrote: > >> >> On 4/18/2011 7:25 AM, Matthew Knepley wrote: >>> On Sun, Apr 17, 2011 at 11:59 PM, Sebastian Steiger <steiger(a)purdue.edu >>> >>> 17631 is in >>> PetscSynchronizedFlush(), a routine which is not called from >>> >>> KSPMonitorDefault() >>> ---> PetscViewerASCIIMonitorPrintf() >> > > I am looking over the code, and I think that change is going to be bad for > everyone > because we have > > PetscViewerFlush_ASCII() > --> PetscSynchronizedFlush() > --> ping-pong between 0 and everyone > > Anytime someone calls flush, the process cannot just send 0 and move on, it > has > to wait. > > Matt > > >>> and thus would not change the performance. My guess is that you have >> another >>> print statement using PetscSynchronizedPrintf/Flush() somewhere in the >> code. >>> >>> Matt >> >> >> In addition to the KSPMonitor, I've hooked up the petsc output with my >> own output system as it is written on the webpage: >> >> >> PetscErrorCode my_double_petsc_vfprintf(FILE * fd, const char format[], >> va_list Argp) >> { >> PetscErrorCode ierr; >> >> //PetscFunctionBegin; // <ss> gave problems in petsc-3.2 - not >> essential according to Barry and Matt >> if (fd != stdout && fd != stderr) { /* handle regular files */ >> ierr = PetscVFPrintfDefault(fd,format,Argp); >> PetscMatrixNemo<double>::check_petsc_error(ierr); >> } else { >> char buff[2000]; >> #if PETSC_VERSION_RELEASE == 1 >> int length; >> #else >> size_t length; >> #endif >> ierr = PetscVSNPrintf(buff, 2000, format, &length, Argp); >> PetscMatrixNemo<double>::check_petsc_error(ierr); >> /* now send buff to whatever stream or whatever you want */ >> msg << buff; >> >> // flush buffer every 2nd time >> num_double_petsc_lines++; >> if (num_double_petsc_lines==2) { >> msg.flush(); >> num_double_petsc_lines = 0; >> } >> } >> //PetscFunctionReturn(0); >> return 0; >> } >> >> Is that connected to the problem? Please note that the "flush buffer >> every 2nd time" does not cause the problems, as it is on my side. >> >> Sebastian >> >> >> >>> >>> >>> Sebastian >>> >>> On 4/17/2011 9:16 PM, Barry Smith wrote: >>>> >>>> On Apr 17, 2011, at 8:11 PM, Matthew Knepley wrote: >>>> >>>>> On Sun, Apr 17, 2011 at 8:05 PM, Barry Smith <bsmith(a)mcs.anl.gov >>> <mailto:[email protected]>> wrote: >>>>> >>>>> I don't understand, what is different and why does it slow >>> everything down? >>>>> >>>>> You introduced a ping-pong between 0 and every process for every >>> call to >>>>> PetscSynchronizedFlush() in order to "stop a flood of messages to >>> 0". As >>>>> process count increases, this is going to be horrible. >>>> >>>> Oh yes, this will do it Is this being used by one of our >>> monitor routines? If so we need to fix our monitors to not use >>> synchronized printf. So what in the code is calling this. >>>> >>>> >>>>> >>>>> This is just the kind of thing that MPI should be solving for us, >>> instead of us >>>>> putting in bullshit code like this to try and regulate message >>> counts. Is there >>>>> an MPI construct to manage this? >>>> >>>> One could use some kind of all to one to get the messages down >>> to process zero. >>>> >>>> Barry >>>> >>>>> >>>>> Matt >>>>> >>>>> Barry >>>>> >>>>> On Apr 17, 2011, at 6:50 PM, Sebastian Steiger wrote: >>>>> >>>>>> Barry, Matt, Satish, >>>>>> >>>>>> Found it. r16730 is good, r16731 is bad. Matt was right: when I >>> turn off >>>>>> the petsc messages generated by monitoring the iterations of the >>> linear >>>>>> solver, then the revisions have comparable speed. When I turn it >> on, >>>>>> then 16731 is 2-3 times slower. >>>>>> >>>>>> Please fix! >>>>>> >>>>>> Best >>>>>> Sebastian >>>>>> >>>>>> >>>>>> >>>>>> On 04/15/2011 07:31 AM, Matthew Knepley wrote: >>>>>>> On Thu, Apr 14, 2011 at 11:33 PM, Satish Balay >>> <petsc-maint(a)mcs.anl.gov <mailto:[email protected]> >>>>>>> <mailto:[email protected] >>> <mailto:[email protected]>>> wrote: >>>>>>> >>>>>>> It would be good to complete the bisection to confirm - but >>> could it >>>>>>> be this? >>>>>>> >>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/rev/10be2c80ec26 >>>>>>> >>>>>>> CHKMEMQ is disabled only with --with-errorchecking=0 - but >> the >>>>>>> assumpton for this change was that it would be disabled for >>>>>>> --with-debugging=0 ? >>>>>>> >>>>>>> >>>>>>> This is my bet for bad set: >>>>>>> >>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/rev/ee1bdc170b4d >>>>>>> >>>>>>> It introduces new messages, but only of length 1. Is your code >>> printing >>>>>>> anything at all >>>>>>> during the run? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>> Satish >>>>>>> >>>>>>> >>>>>>> On Thu, 14 Apr 2011, Sebastian Steiger wrote: >>>>>>> >>>>>>>> I'm sorry, actually 16721 is good and 16733 is bad. So it's 12 >>>>>>> revisions. >>>>>>>> >>>>>>>> On 4/14/2011 9:12 PM, Sebastian Steiger wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I've pinned it down to 16 revisions: 16721 is good, 16737 is >>>>>>> bad. The >>>>>>>>> next bisection step would be 16727, but from the log I see >> that >>>>>>>>> 16727-16736 might have some problems. Should I continue or do >>>>>>> you guys >>>>>>>>> now have a clue what could cause the factor 2-3 slowdown? >>>>>>>>> >>>>>>>>> Sebastian >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>> steiger@jaguarpf-login6 >> :~/src/app-nemo/NEMO/libs/petsc-dev/petsc-dev$ >>> hg >>>>>>>>> log -r16721:16737 >>>>>>>>> changeset: 16721:b788c229de4d >>>>>>>>> parent: 16720:9aafffdcf68b >>>>>>>>> parent: 16715:e53f46924236 >>>>>>>>> user: Jed Brown <jed(a)59A2.org> >>>>>>>>> date: Mon Aug 09 17:30:28 2010 -0800 >>>>>>>>> summary: merge >>>>>>>>> >>>>>>>>> changeset: 16722:4ca941c57754 >>>>>>>>> user: hong(a)vis-v410v166.mcs.anl-external.org >>> <mailto:[email protected]> >>>>>>> <mailto:[email protected] >>> <mailto:[email protected]>> >>>>>>>>> date: Tue Aug 10 11:57:11 2010 -0500 >>>>>>>>> summary: add case of np=1 for >> MatGetMultiProcBlock_MPIAIJ() >>>>>>>>> >>>>>>>>> changeset: 16723:4d6b54981472 >>>>>>>>> user: hong(a)vis-v410v166.mcs.anl-external.org >>> <mailto:[email protected]> >>>>>>> <mailto:[email protected] >>> <mailto:[email protected]>> >>>>>>>>> date: Tue Aug 10 12:00:14 2010 -0500 >>>>>>>>> summary: fix memory leak in MatHeaderCopy() and >>>>>>> MatHeaderReplace() >>>>>>>>> >>>>>>>>> changeset: 16724:2680bde1c9b8 >>>>>>>>> user: Shri(a)mcswl165.mcs.anl.gov >>> <mailto:[email protected]> <mailto:[email protected] >>> <mailto:[email protected]>> >>>>>>>>> date: Thu Aug 12 14:37:43 2010 -0500 >>>>>>>>> summary: Started adding semismooth stuff. >>>>>>>>> >>>>>>>>> changeset: 16725:50dceda7151c >>>>>>>>> user: Shri(a)mcswl165.mcs.anl.gov >>> <mailto:[email protected]> <mailto:[email protected] >>> <mailto:[email protected]>> >>>>>>>>> date: Thu Aug 12 14:45:13 2010 -0500 >>>>>>>>> summary: Defined Infinity and epsilon >>>>>>>>> >>>>>>>>> changeset: 16726:c0f1b46d675f >>>>>>>>> user: hzhang >>>>>>>>> date: Fri Aug 13 11:10:37 2010 -0500 >>>>>>>>> summary: update outputfiles for nightly tests >>>>>>>>> >>>>>>>>> changeset: 16727:67df4fdba5bd >>>>>>>>> user: hong(a)hzhangmac.mcs.anl.gov >>> <mailto:[email protected]> >>>>>>> <mailto:[email protected] >>> <mailto:[email protected]>> >>>>>>>>> date: Fri Aug 13 13:25:56 2010 -0500 >>>>>>>>> summary: add PetscSubcommType in PetscSubcommCreate(); add >>>>>>>>> PetscSubcommCreate_contiguous(). >>>>>>>>> >>>>>>>>> changeset: 16728:10be2c80ec26 >>>>>>>>> parent: 16723:4d6b54981472 >>>>>>>>> user: Barry Smith bsmith(a)mcs.anl.gov >>> <mailto:[email protected]> >>>>>>> <mailto:[email protected] <mailto:[email protected]>> >>>>>>>>> date: Tue Aug 10 18:15:10 2010 -0500 >>>>>>>>> summary: merged CHKMEMQ; into PetscStackPush() and >>>>>>> PetscStackPop() >>>>>>>>> >>>>>>>>> changeset: 16729:ef8e7a4583e6 >>>>>>>>> user: Barry Smith bsmith(a)mcs.anl.gov >>> <mailto:[email protected]> >>>>>>> <mailto:[email protected] <mailto:[email protected]>> >>>>>>>>> date: Thu Aug 12 14:16:40 2010 -0500 >>>>>>>>> summary: changed MatHeaderCopy() to MatHeaderMerge() and >>>>>>> improved >>>>>>>>> comment >>>>>>>>> >>>>>>>>> changeset: 16730:50055ed8b725 >>>>>>>>> parent: 16729:ef8e7a4583e6 >>>>>>>>> parent: 16725:50dceda7151c >>>>>>>>> user: Barry Smith bsmith(a)mcs.anl.gov >>> <mailto:[email protected]> >>>>>>> <mailto:[email protected] <mailto:[email protected]>> >>>>>>>>> date: Thu Aug 12 16:24:48 2010 -0500 >>>>>>>>> summary: commit after merge >>>>>>>>> >>>>>>>>> changeset: 16731:ee1bdc170b4d >>>>>>>>> user: Barry Smith bsmith(a)mcs.anl.gov >>> <mailto:[email protected]> >>>>>>> <mailto:[email protected] <mailto:[email protected]>> >>>>>>>>> date: Fri Aug 13 13:50:30 2010 -0500 >>>>>>>>> summary: minor formating of short lines >>>>>>>>> >>>>>>>>> changeset: 16732:a49d9a4b41bc >>>>>>>>> parent: 16731:ee1bdc170b4d >>>>>>>>> parent: 16727:67df4fdba5bd >>>>>>>>> user: Barry Smith bsmith(a)mcs.anl.gov >>> <mailto:[email protected]> >>>>>>> <mailto:[email protected] <mailto:[email protected]>> >>>>>>>>> date: Fri Aug 13 13:50:43 2010 -0500 >>>>>>>>> summary: commit after merge >>>>>>>>> >>>>>>>>> changeset: 16733:804ce43d15c9 >>>>>>>>> user: Jed Brown <jed(a)59A2.org> >>>>>>>>> date: Fri Aug 13 11:01:24 2010 -0800 >>>>>>>>> summary: Update converged reason for Rosenbrock demo to be >>>>>>> more portable >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to >>> which >>>>>>> their experiments lead. >>>>>>> -- Norbert Wiener >>>>>> >>>>>> >>>>>> <bisection-conclusion.txt> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which >>> their experiments lead. >>>>> -- Norbert Wiener >>>> >>> >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which >>> their experiments lead. >>> -- Norbert Wiener >> >> >> > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > > On Mon, Apr 18, 2011 at 7:52 AM, Sebastian Steiger <steiger(a)purdue.edu> wrote: > > On 4/18/2011 7:25 AM, Matthew Knepley wrote: > > On Sun, Apr 17, 2011 at 11:59 PM, Sebastian Steiger <steiger(a)purdue.edu > > <mailto:[email protected]>> wrote: > > > > On a side note, why did that PetscPrintf function not show up in the log > > summary? That could have spared me the whole bisection process. > > > > > > I cannot make sense of the problem from your report. The only change in > > 17631 is in > > PetscSynchronizedFlush(), a routine which is not called from > > > > KSPMonitorDefault() > > ---> PetscViewerASCIIMonitorPrintf() > > I am looking over the code, and I think that change is going to be bad for everyone > because we have > > PetscViewerFlush_ASCII() > --> PetscSynchronizedFlush() > --> ping-pong between 0 and everyone > > Anytime someone calls flush, the process cannot just send 0 and move on, it has > to wait. > > Matt > > > and thus would not change the performance. My guess is that you have another > > print statement using PetscSynchronizedPrintf/Flush() somewhere in the code. > > > > Matt > > > In addition to the KSPMonitor, I've hooked up the petsc output with my > own output system as it is written on the webpage: > > > PetscErrorCode my_double_petsc_vfprintf(FILE * fd, const char format[], > va_list Argp) > { > PetscErrorCode ierr; > > //PetscFunctionBegin; // <ss> gave problems in petsc-3.2 - not > essential according to Barry and Matt > if (fd != stdout && fd != stderr) { /* handle regular files */ > ierr = PetscVFPrintfDefault(fd,format,Argp); > PetscMatrixNemo<double>::check_petsc_error(ierr); > } else { > char buff[2000]; > #if PETSC_VERSION_RELEASE == 1 > int length; > #else > size_t length; > #endif > ierr = PetscVSNPrintf(buff, 2000, format, &length, Argp); > PetscMatrixNemo<double>::check_petsc_error(ierr); > /* now send buff to whatever stream or whatever you want */ > msg << buff; > > // flush buffer every 2nd time > num_double_petsc_lines++; > if (num_double_petsc_lines==2) { > msg.flush(); > num_double_petsc_lines = 0; > } > } > //PetscFunctionReturn(0); > return 0; > } > > Is that connected to the problem? Please note that the "flush buffer > every 2nd time" does not cause the problems, as it is on my side. > > Sebastian > > > > > > > > > Sebastian > > > > On 4/17/2011 9:16 PM, Barry Smith wrote: > > > > > > On Apr 17, 2011, at 8:11 PM, Matthew Knepley wrote: > > > > > >> On Sun, Apr 17, 2011 at 8:05 PM, Barry Smith <bsmith(a)mcs.anl.gov > > <mailto:[email protected]>> wrote: > > >> > > >> I don't understand, what is different and why does it slow > > everything down? > > >> > > >> You introduced a ping-pong between 0 and every process for every > > call to > > >> PetscSynchronizedFlush() in order to "stop a flood of messages to > > 0". As > > >> process count increases, this is going to be horrible. > > > > > > Oh yes, this will do it Is this being used by one of our > > monitor routines? If so we need to fix our monitors to not use > > synchronized printf. So what in the code is calling this. > > > > > > > > >> > > >> This is just the kind of thing that MPI should be solving for us, > > instead of us > > >> putting in bullshit code like this to try and regulate message > > counts. Is there > > >> an MPI construct to manage this? > > > > > > One could use some kind of all to one to get the messages down > > to process zero. > > > > > > Barry > > > > > >> > > >> Matt > > >> > > >> Barry > > >> > > >> On Apr 17, 2011, at 6:50 PM, Sebastian Steiger wrote: > > >> > > >>> Barry, Matt, Satish, > > >>> > > >>> Found it. r16730 is good, r16731 is bad. Matt was right: when I > > turn off > > >>> the petsc messages generated by monitoring the iterations of the > > linear > > >>> solver, then the revisions have comparable speed. When I turn it on, > > >>> then 16731 is 2-3 times slower. > > >>> > > >>> Please fix! > > >>> > > >>> Best > > >>> Sebastian > > >>> > > >>> > > >>> > > >>> On 04/15/2011 07:31 AM, Matthew Knepley wrote: > > >>>> On Thu, Apr 14, 2011 at 11:33 PM, Satish Balay > > <petsc-maint(a)mcs.anl.gov <mailto:[email protected]> > > >>>> <mailto:[email protected] > > <mailto:[email protected]>>> wrote: > > >>>> > > >>>> It would be good to complete the bisection to confirm - but > > could it > > >>>> be this? > > >>>> > > >>>> http://petsc.cs.iit.edu/petsc/petsc-dev/rev/10be2c80ec26 > > >>>> > > >>>> CHKMEMQ is disabled only with --with-errorchecking=0 - but the > > >>>> assumpton for this change was that it would be disabled for > > >>>> --with-debugging=0 ? > > >>>> > > >>>> > > >>>> This is my bet for bad set: > > >>>> > > >>>> http://petsc.cs.iit.edu/petsc/petsc-dev/rev/ee1bdc170b4d > > >>>> > > >>>> It introduces new messages, but only of length 1. Is your code > > printing > > >>>> anything at all > > >>>> during the run? > > >>>> > > >>>> Thanks, > > >>>> > > >>>> Matt > > >>>> > > >>>> > > >>>> Satish > > >>>> > > >>>> > > >>>> On Thu, 14 Apr 2011, Sebastian Steiger wrote: > > >>>> > > >>>>> I'm sorry, actually 16721 is good and 16733 is bad. So it's 12 > > >>>> revisions. > > >>>>> > > >>>>> On 4/14/2011 9:12 PM, Sebastian Steiger wrote: > > >>>>>> Hi, > > >>>>>> > > >>>>>> I've pinned it down to 16 revisions: 16721 is good, 16737 is > > >>>> bad. The > > >>>>>> next bisection step would be 16727, but from the log I see that > > >>>>>> 16727-16736 might have some problems. Should I continue or do > > >>>> you guys > > >>>>>> now have a clue what could cause the factor 2-3 slowdown? > > >>>>>> > > >>>>>> Sebastian > > >>>>>> > > >>>>>> > > >>>>>> > > >>>> > > steiger@jaguarpf-login6:~/src/app-nemo/NEMO/libs/petsc-dev/petsc-dev$ > > hg > > >>>>>> log -r16721:16737 > > >>>>>> changeset: 16721:b788c229de4d > > >>>>>> parent: 16720:9aafffdcf68b > > >>>>>> parent: 16715:e53f46924236 > > >>>>>> user: Jed Brown <jed(a)59A2.org> > > >>>>>> date: Mon Aug 09 17:30:28 2010 -0800 > > >>>>>> summary: merge > > >>>>>> > > >>>>>> changeset: 16722:4ca941c57754 > > >>>>>> user: hong(a)vis-v410v166.mcs.anl-external.org > > <mailto:[email protected]> > > >>>> <mailto:[email protected] > > <mailto:[email protected]>> > > >>>>>> date: Tue Aug 10 11:57:11 2010 -0500 > > >>>>>> summary: add case of np=1 for MatGetMultiProcBlock_MPIAIJ() > > >>>>>> > > >>>>>> changeset: 16723:4d6b54981472 > > >>>>>> user: hong(a)vis-v410v166.mcs.anl-external.org > > <mailto:[email protected]> > > >>>> <mailto:[email protected] > > <mailto:[email protected]>> > > >>>>>> date: Tue Aug 10 12:00:14 2010 -0500 > > >>>>>> summary: fix memory leak in MatHeaderCopy() and > > >>>> MatHeaderReplace() > > >>>>>> > > >>>>>> changeset: 16724:2680bde1c9b8 > > >>>>>> user: Shri(a)mcswl165.mcs.anl.gov > > <mailto:[email protected]> <mailto:[email protected] > > <mailto:[email protected]>> > > >>>>>> date: Thu Aug 12 14:37:43 2010 -0500 > > >>>>>> summary: Started adding semismooth stuff. > > >>>>>> > > >>>>>> changeset: 16725:50dceda7151c > > >>>>>> user: Shri(a)mcswl165.mcs.anl.gov > > <mailto:[email protected]> <mailto:[email protected] > > <mailto:[email protected]>> > > >>>>>> date: Thu Aug 12 14:45:13 2010 -0500 > > >>>>>> summary: Defined Infinity and epsilon > > >>>>>> > > >>>>>> changeset: 16726:c0f1b46d675f > > >>>>>> user: hzhang > > >>>>>> date: Fri Aug 13 11:10:37 2010 -0500 > > >>>>>> summary: update outputfiles for nightly tests > > >>>>>> > > >>>>>> changeset: 16727:67df4fdba5bd > > >>>>>> user: hong(a)hzhangmac.mcs.anl.gov > > <mailto:[email protected]> > > >>>> <mailto:[email protected] > > <mailto:[email protected]>> > > >>>>>> date: Fri Aug 13 13:25:56 2010 -0500 > > >>>>>> summary: add PetscSubcommType in PetscSubcommCreate(); add > > >>>>>> PetscSubcommCreate_contiguous(). > > >>>>>> > > >>>>>> changeset: 16728:10be2c80ec26 > > >>>>>> parent: 16723:4d6b54981472 > > >>>>>> user: Barry Smith bsmith(a)mcs.anl.gov > > <mailto:[email protected]> > > >>>> <mailto:[email protected] <mailto:[email protected]>> > > >>>>>> date: Tue Aug 10 18:15:10 2010 -0500 > > >>>>>> summary: merged CHKMEMQ; into PetscStackPush() and > > >>>> PetscStackPop() > > >>>>>> > > >>>>>> changeset: 16729:ef8e7a4583e6 > > >>>>>> user: Barry Smith bsmith(a)mcs.anl.gov > > <mailto:[email protected]> > > >>>> <mailto:[email protected] <mailto:[email protected]>> > > >>>>>> date: Thu Aug 12 14:16:40 2010 -0500 > > >>>>>> summary: changed MatHeaderCopy() to MatHeaderMerge() and > > >>>> improved > > >>>>>> comment > > >>>>>> > > >>>>>> changeset: 16730:50055ed8b725 > > >>>>>> parent: 16729:ef8e7a4583e6 > > >>>>>> parent: 16725:50dceda7151c > > >>>>>> user: Barry Smith bsmith(a)mcs.anl.gov > > <mailto:[email protected]> > > >>>> <mailto:[email protected] <mailto:[email protected]>> > > >>>>>> date: Thu Aug 12 16:24:48 2010 -0500 > > >>>>>> summary: commit after merge > > >>>>>> > > >>>>>> changeset: 16731:ee1bdc170b4d > > >>>>>> user: Barry Smith bsmith(a)mcs.anl.gov > > <mailto:[email protected]> > > >>>> <mailto:[email protected] <mailto:[email protected]>> > > >>>>>> date: Fri Aug 13 13:50:30 2010 -0500 > > >>>>>> summary: minor formating of short lines > > >>>>>> > > >>>>>> changeset: 16732:a49d9a4b41bc > > >>>>>> parent: 16731:ee1bdc170b4d > > >>>>>> parent: 16727:67df4fdba5bd > > >>>>>> user: Barry Smith bsmith(a)mcs.anl.gov > > <mailto:[email protected]> > > >>>> <mailto:[email protected] <mailto:[email protected]>> > > >>>>>> date: Fri Aug 13 13:50:43 2010 -0500 > > >>>>>> summary: commit after merge > > >>>>>> > > >>>>>> changeset: 16733:804ce43d15c9 > > >>>>>> user: Jed Brown <jed(a)59A2.org> > > >>>>>> date: Fri Aug 13 11:01:24 2010 -0800 > > >>>>>> summary: Update converged reason for Rosenbrock demo to be > > >>>> more portable > > >>>>>> > > >>>>> > > >>>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> -- > > >>>> What most experimenters take for granted before they begin their > > >>>> experiments is infinitely more interesting than any results to > > which > > >>>> their experiments lead. > > >>>> -- Norbert Wiener > > >>> > > >>> > > >>> <bisection-conclusion.txt> > > >> > > >> > > >> > > >> > > >> > > >> -- > > >> What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which > > their experiments lead. > > >> -- Norbert Wiener > > > > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which > > their experiments lead. > > -- Norbert Wiener > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener

1 0

Re: [petsc-dev] Matrix Options shown too many times in -help
by Aron Ahmadia 18 Apr '11

18 Apr '11

Agreed, the overwhelming output from -h is tremendously annoying, and I cannot always grep because I don't know what I'm grepping for :( A On Mon, Apr 18, 2011 at 3:43 PM, Jed Brown <jed(a)59a2.org> wrote: > Currently Matrix Options are printed in MatView_Private which is called > from many places, e.g. > > MatAssemblyEnd > MatZeroRowsColumns > MatZeroRows > MatLUFactorNumeric > MatCholeskyFactorNumeric > > The problem here is that in time-stepping or a multigrid solve, it's common > to have these functions called hundreds or thousands of times which is > really overwhelming to -help output. One solution would be to store on a > per-matrix basis whether help has been displayed and only view it once (this > implies caching all the -mat_view_* options in the Mat, or calling with > PetscOptionsGetBool after the first time). The options could even be moved > to MatSetFromOptions, but that doesn't get called nearly as much as we would > probably like. > > There is also the somewhat deeper problem of multiple objects with the same > prefix (often no prefix). This is most visible for matrices inside of > multigrid and/or fieldsplit. It is easy to have more than 20 matrices with > no prefix so even if the multiple-output problem above was solved, it would > still be fairly overwhelming. I think this is somewhat harder to fix because > it would imply a per-prefix log of whether the option has been printed. I > think that the one special case of no-prefix Mat would already be a > significant usability improvement. >

1 0

Matrix Options shown too many times in -help
by Jed Brown 18 Apr '11

18 Apr '11

Currently Matrix Options are printed in MatView_Private which is called from many places, e.g. MatAssemblyEnd MatZeroRowsColumns MatZeroRows MatLUFactorNumeric MatCholeskyFactorNumeric The problem here is that in time-stepping or a multigrid solve, it's common to have these functions called hundreds or thousands of times which is really overwhelming to -help output. One solution would be to store on a per-matrix basis whether help has been displayed and only view it once (this implies caching all the -mat_view_* options in the Mat, or calling with PetscOptionsGetBool after the first time). The options could even be moved to MatSetFromOptions, but that doesn't get called nearly as much as we would probably like. There is also the somewhat deeper problem of multiple objects with the same prefix (often no prefix). This is most visible for matrices inside of multigrid and/or fieldsplit. It is easy to have more than 20 matrices with no prefix so even if the multiple-output problem above was solved, it would still be fairly overwhelming. I think this is somewhat harder to fix because it would imply a per-prefix log of whether the option has been printed. I think that the one special case of no-prefix Mat would already be a significant usability improvement.

1 0

Fwd: [petsc-maint #70591] Are the petsc4py directions current?
by Barry Smith 17 Apr '11

17 Apr '11

Begin forwarded message: > From: Jed Brown <jed(a)59A2.org> > Date: April 17, 2011 12:14:03 PM CDT > To: petsc-maint <petsc-maint(a)mcs.anl.gov> > Subject: [petsc-maint #70591] Are the petsc4py directions current? > Reply-To: petsc-maint(a)mcs.anl.gov, Jed Brown <jed(a)59A2.org> > > http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#install-petsc4… > > I just had a (Mac) user tell me that this isn't working. He was silently > getting some much older install of petsc4py that he had forgotten about, so > we ended up wasting some time before confirming that he really had not > loaded petsc4py dev. > > When someone follows this procedure, where does petsc4py actually go? > > http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#install-petsc4… > > I just had a (Mac) user tell me that this isn't working. He was silently getting some much older install of petsc4py that he had forgotten about, so we ended up wasting some time before confirming that he really had not loaded petsc4py dev. > > When someone follows this procedure, where does petsc4py actually go?

1 0

Some PETSc developer reminders
by Barry Smith 16 Apr '11

16 Apr '11

Some reminders for people contributing PETSc source code. Please follow the developers guide: http://www.mcs.anl.gov/petsc/petsc-as/developers/developers.pdf NEVER use NULL in PETSc, if you need a null use PETSC_NULL NEVER have checks like a == PETSC_NULL or a != PETSC_NULL these can be done with !a or a NEVER use int, use PetscInt, PetscMPIInt, etc Thanks Barry

1 0

Re: [petsc-dev] PetscOptionsGetString (Barry Smith)
by Jed Brown 16 Apr '11

16 Apr '11

On Sat, Apr 16, 2011 at 23:50, Shiyuan <gshy2014(a)gmail.com> wrote: > It sounds reasonable to make the change but it's better to document it. I > don't see the change in the change log and I am surprised that my code > doesn't work any more. > I agree that it should be mentioned. > Or maybe handle the single character case specially so that there is no > need to change the source code. > On the flip side, if someone writes some generic code that expects NULL termination and determines the buffer size based on some other criterion, they would have to write special code to handle the zero element case. I don't have much preference either way and would suggest PetscOptionsGetChar if the functionality was really important (I don't think it is because a character flag is almost certainly better represented using PetscOptionsGetEList). > Another question is that in the newer version, PetscTruth becomes > PetscBool. I might need to switch between development version and current > released version. But to change the source code back and forth is a > headache. Is there any good way to handle this? Why don't do something like > #define PetscTruth PetscBool in the petsc.h so that we do not need to change > the source code. Any wrong with that approach? > You can certainly do that (typedef PetscBool PetscTruth; would be better), but there are often other more significant changes that make it awkward to support multiple versions at the same time. The reason for the change was confusion between PETSC_TRUE (the enum value of type PetscTruth) and PETSC_TRUTH (the enum value of type PetscDataType, indicates a value of type PetscTruth, this is not used by most users). If one of these was accidentally used in place of another, it would silently produce incorrect results which is the worst possible failure mode. Now there is PETSC_BOOL in PetscDataType so the problem goes away (you won't accidentally type PETSC_BOOL when you mean PETSC_TRUE).

1 0

Re: [petsc-dev] PetscOptionsGetString (Barry Smith)
by Shiyuan 16 Apr '11

16 Apr '11

It sounds reasonable to make the change but it's better to document it. I don't see the change in the change log and I am surprised that my code doesn't work any more. Or maybe handle the single character case specially so that there is no need to change the source code. Another question is that in the newer version, PetscTruth becomes PetscBool. I might need to switch between development version and current released version. But to change the source code back and forth is a headache. Is there any good way to handle this? Why don't do something like #define PetscTruth PetscBool in the petsc.h so that we do not need to change the source code. Any wrong with that approach? Thanks. --------------------------------------------------------------------------------------------------------------------------------------------------------------------- This behavior was changed because we wanted to always make sure that strings obtained were null terminated. When you pass in a string of length 1 there is only room for the null terminator. We could only do the null termination when the the string is larger than one allowing your old code to work. But that is slightly inconsistent handling that case differently. What do you think? You might consider using something like PetscOptionsGetEList() instead of PetscOptionsGetString() also. Barry

1 0

Re: [petsc-dev] PetscOptionsGetString
by Barry Smith 15 Apr '11

15 Apr '11

This behavior was changed because we wanted to always make sure that strings obtained were null terminated. When you pass in a string of length 1 there is only room for the null terminator. We could only do the null termination when the the string is larger than one allowing your old code to work. But that is slightly inconsistent handling that case differently. What do you think? You might consider using something like PetscOptionsGetEList() instead of PetscOptionsGetString() also. Barry On Apr 15, 2011, at 6:44 PM, Shiyuan wrote: > Hi, > In the version of 3.1-p5, PetscOptionsGetString() can be used to get a char input. It seems that it doesn't work any more in development version. > PetscOptionsGetString(PETSC_NULL,"-Mode",&Mode,1,&flag); > This gives me a \0. How can I get the input of only one character in dev? > Thanks. > > >

1 0