February 2013 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] make -j not supported? Why???
by Nystrom, William D 25 Feb '13

25 Feb '13

Is there a way to examine the petsc build logs and determine if I am really getting parallel builds? I have a top level script that I use to the petsc configure script followed by make. For a couple of our test bed systems that stay fairly unloaded, I use the following configure option: --with-make-np=16. But I also invoke my script like this: /usr/bin/time -p makePetscDev_caddy >& makePetscDev_caddy.log. And I'll get a time like the following: real 2160.77 user 1308.21 sys 234.37 So it does not seem like I am getting much if any parallelism in my build. I am also building with lots of external packages and have no idea if parallel builds are being done with them. Thanks, Dave -- Dave Nystrom LANL HPC-5 Phone: 505-667-7913 Email: wdn(a)lanl.gov Smail: Mail Stop B272 Group HPC-5 Los Alamos National Laboratory Los Alamos, NM 87545 ________________________________________ From: petsc-dev-bounces(a)mcs.anl.gov [petsc-dev-bounces(a)mcs.anl.gov] on behalf of Barry Smith [bsmith(a)mcs.anl.gov] Sent: Monday, February 25, 2013 3:25 PM To: For users of the development version of PETSc Subject: Re: [petsc-dev] make -j not supported? Why??? Tim, As usual, a too short error message without enough information. I've changed it so that it will now give you Barrys-MacBook-Pro:petsc-dev barrysmith$ make -j 3 ccmake ********************* ERROR ************************* PETSc compiles are automatically parallel, do not provide the -j option to make ****************************************************** make: *** [chk_makej] Error 1 Barrys-MacBook-Pro:petsc-dev barrysmith$ make -j 3 all-legacy ********************* ERROR ************************* Parallel build with 'make -j' is not supported for PETSc legacy builds. Run without -j <np> or ./configure PETSc with --download-cmake to enable parallel builds with PETSc ****************************************************** make[1]: *** [chk_makej] Error 1 make: *** [all-legacy] Error 2 Barrys-MacBook-Pro:petsc-dev barrysmith$ Explanations: When PETSc is using cmake makefiles from the top level makefiles we pass our own -j <np> down, make doesn't like having two levels of -j and at best prints a confusing warning message about the parallel build. When using PETSc's legacy make system it uses a recursive make that is not able to be parallelized (yes it is bad and hence the cmake version is used by default now). Notes: It would be nice if we could somehow remove the -j <np> at the top level so that it didn't generate the error message and just ran but make doesn't allow this type of thing. It would be nice if we could eliminate the messages like make[1]: *** [chk_makej] Error 1 make: *** [all-legacy] Error 2 that make generates automatically but I cannot figure out how to do that. So the current situation is a compromise where most users get automatically a parallel build and the users who knowingly provide a -j <np> option get slightly peeved. (My guess is 90+% of people who install PETSc don't know about the -j option). At least now it will explain that it is doing a parallel build. I'll check on the -k option next. Barry On Feb 25, 2013, at 2:58 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > This is to support different backends. If configure found CMake, then normal "make" will use multiple jobs, or you can do it explicitly with make -j3 $PETSC_ARCH. > > CMake would be terrible as a hard dependency (it's terrible enough as it is) but it's less bad than some alternatives. > > > On Mon, Feb 25, 2013 at 3:30 PM, Tim Tautges <tautges(a)mcs.anl.gov> wrote: > tautges@chronos:~/code/petsc-dev-moab$ make -j 8 > ********************* ERROR ************************* > Parallel build with 'make -j' is not supported > ****************************************************** > make: *** [chk_makej] Error 1 > tautges@chronos:~/code/petsc-dev-moab$ > > > Why??? I can almost understand petsc's justification for its own autotools-looking python-based build system, but why its need to not support otherwise-standard make options? Similarly, PETSC_MAKE_STOP_ON_ERROR instead of make -k? > > - tim > > -- > ================================================================ > "You will keep in perfect peace him whose mind is > steadfast, because he trusts in you." Isaiah 26:3 > > Tim Tautges Argonne National Laboratory > (tautges(a)mcs.anl.gov) (telecommuting from UW-Madison) > phone (gvoice): (608) 354-1459 1500 Engineering Dr. > fax: (608) 263-4499 Madison, WI 53706 > >

1 0

Re: [petsc-dev] make -j not supported? Why???
by Barry Smith 25 Feb '13

25 Feb '13

On Feb 25, 2013, at 2:30 PM, Tim Tautges <tautges(a)mcs.anl.gov> wrote: > but why its need to not support otherwise-standard make options? Similarly, PETSC_MAKE_STOP_ON_ERROR instead of make -k? Hmm, with the cmake based build -k seems to work fine, it keeps on going compiling everything it can. Without the -k it stops when the first compile fails. So it is supporting the "otherwise-standard make options". Yes, the legacy build which is only a fallback for when cmake fails, is funky. Are you using it? If so, let us know why and we'll try to improve the cmake version so that you don't need the legacy; even I don't use it any more. Barry > > - tim > > -- > ================================================================ > "You will keep in perfect peace him whose mind is > steadfast, because he trusts in you." Isaiah 26:3 > > Tim Tautges Argonne National Laboratory > (tautges(a)mcs.anl.gov) (telecommuting from UW-Madison) > phone (gvoice): (608) 354-1459 1500 Engineering Dr. > fax: (608) 263-4499 Madison, WI 53706 >

1 0

Re: [petsc-dev] make -j not supported? Why???
by Barry Smith 25 Feb '13

25 Feb '13

Tim, As usual, a too short error message without enough information. I've changed it so that it will now give you Barrys-MacBook-Pro:petsc-dev barrysmith$ make -j 3 ccmake ********************* ERROR ************************* PETSc compiles are automatically parallel, do not provide the -j option to make ****************************************************** make: *** [chk_makej] Error 1 Barrys-MacBook-Pro:petsc-dev barrysmith$ make -j 3 all-legacy ********************* ERROR ************************* Parallel build with 'make -j' is not supported for PETSc legacy builds. Run without -j <np> or ./configure PETSc with --download-cmake to enable parallel builds with PETSc ****************************************************** make[1]: *** [chk_makej] Error 1 make: *** [all-legacy] Error 2 Barrys-MacBook-Pro:petsc-dev barrysmith$ Explanations: When PETSc is using cmake makefiles from the top level makefiles we pass our own -j <np> down, make doesn't like having two levels of -j and at best prints a confusing warning message about the parallel build. When using PETSc's legacy make system it uses a recursive make that is not able to be parallelized (yes it is bad and hence the cmake version is used by default now). Notes: It would be nice if we could somehow remove the -j <np> at the top level so that it didn't generate the error message and just ran but make doesn't allow this type of thing. It would be nice if we could eliminate the messages like make[1]: *** [chk_makej] Error 1 make: *** [all-legacy] Error 2 that make generates automatically but I cannot figure out how to do that. So the current situation is a compromise where most users get automatically a parallel build and the users who knowingly provide a -j <np> option get slightly peeved. (My guess is 90+% of people who install PETSc don't know about the -j option). At least now it will explain that it is doing a parallel build. I'll check on the -k option next. Barry On Feb 25, 2013, at 2:58 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > This is to support different backends. If configure found CMake, then normal "make" will use multiple jobs, or you can do it explicitly with make -j3 $PETSC_ARCH. > > CMake would be terrible as a hard dependency (it's terrible enough as it is) but it's less bad than some alternatives. > > > On Mon, Feb 25, 2013 at 3:30 PM, Tim Tautges <tautges(a)mcs.anl.gov> wrote: > tautges@chronos:~/code/petsc-dev-moab$ make -j 8 > ********************* ERROR ************************* > Parallel build with 'make -j' is not supported > ****************************************************** > make: *** [chk_makej] Error 1 > tautges@chronos:~/code/petsc-dev-moab$ > > > Why??? I can almost understand petsc's justification for its own autotools-looking python-based build system, but why its need to not support otherwise-standard make options? Similarly, PETSC_MAKE_STOP_ON_ERROR instead of make -k? > > - tim > > -- > ================================================================ > "You will keep in perfect peace him whose mind is > steadfast, because he trusts in you." Isaiah 26:3 > > Tim Tautges Argonne National Laboratory > (tautges(a)mcs.anl.gov) (telecommuting from UW-Madison) > phone (gvoice): (608) 354-1459 1500 Engineering Dr. > fax: (608) 263-4499 Madison, WI 53706 > >

1 0

Re: [petsc-dev] make -j not supported? Why???
by Satish Balay 25 Feb '13

25 Feb '13

There are 2 build modes - the old one - which is recurrsive make - and does not support parallel build. And there is a new one - based on cmake - which supports parallel make. Our top level makefile supports both - and does not work with '-j'. However if you have a cmake build - it does use paralell make. The '-j' equivalent can be specified to configure via --with-make-np= option or to 'make' with MAKE_NP variable. [Default is configure autodetected value] Satish On Mon, 25 Feb 2013, Tim Tautges wrote: > tautges@chronos:~/code/petsc-dev-moab$ make -j 8 > ********************* ERROR ************************* > Parallel build with 'make -j' is not supported > ****************************************************** > make: *** [chk_makej] Error 1 > tautges@chronos:~/code/petsc-dev-moab$ > > > Why??? I can almost understand petsc's justification for its own > autotools-looking python-based build system, but why its need to not support > otherwise-standard make options? Similarly, PETSC_MAKE_STOP_ON_ERROR instead > of make -k? > > - tim > >

1 0

Re: [petsc-dev] make -j not supported? Why???
by Jed Brown 25 Feb '13

25 Feb '13

This is to support different backends. If configure found CMake, then normal "make" will use multiple jobs, or you can do it explicitly with make -j3 $PETSC_ARCH. CMake would be terrible as a hard dependency (it's terrible enough as it is) but it's less bad than some alternatives. On Mon, Feb 25, 2013 at 3:30 PM, Tim Tautges <tautges(a)mcs.anl.gov> wrote: > tautges@chronos:~/code/petsc-**dev-moab$ make -j 8 > ********************* ERROR ************************* > Parallel build with 'make -j' is not supported > ******************************************************** > make: *** [chk_makej] Error 1 > tautges@chronos:~/code/petsc-**dev-moab$ > > > Why??? I can almost understand petsc's justification for its own > autotools-looking python-based build system, but why its need to not > support otherwise-standard make options? Similarly, > PETSC_MAKE_STOP_ON_ERROR instead of make -k? > > - tim > > -- > ==============================**==============================**==== > "You will keep in perfect peace him whose mind is > steadfast, because he trusts in you." Isaiah 26:3 > > Tim Tautges Argonne National Laboratory > (tautges(a)mcs.anl.gov) (telecommuting from UW-Madison) > phone (gvoice): (608) 354-1459 1500 Engineering Dr. > fax: (608) 263-4499 Madison, WI 53706 > >

1 0

make -j not supported? Why???
by Tim Tautges 25 Feb '13

25 Feb '13

tautges@chronos:~/code/petsc-dev-moab$ make -j 8 ********************* ERROR ************************* Parallel build with 'make -j' is not supported ****************************************************** make: *** [chk_makej] Error 1 tautges@chronos:~/code/petsc-dev-moab$ Why??? I can almost understand petsc's justification for its own autotools-looking python-based build system, but why its need to not support otherwise-standard make options? Similarly, PETSC_MAKE_STOP_ON_ERROR instead of make -k? - tim -- ================================================================ "You will keep in perfect peace him whose mind is steadfast, because he trusts in you." Isaiah 26:3 Tim Tautges Argonne National Laboratory (tautges(a)mcs.anl.gov) (telecommuting from UW-Madison) phone (gvoice): (608) 354-1459 1500 Engineering Dr. fax: (608) 263-4499 Madison, WI 53706

1 0

Re: [petsc-dev] TS problems wasted my time
by Matthew Knepley 24 Feb '13

24 Feb '13

On Sun, Feb 24, 2013 at 3:12 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > Move it out of TSOps. > Yes, it should be handled exactly like we do in SNES. Matt > On Feb 24, 2013 2:10 PM, "Barry Smith" <bsmith(a)mcs.anl.gov> wrote: > >> >> PetscErrorCode TSSetType(TS ts,TSType type) >> { >> PetscErrorCode (*r)(TS); >> PetscBool match; >> PetscErrorCode ierr; >> >> PetscFunctionBegin; >> PetscValidHeaderSpecific(ts, TS_CLASSID,1); >> ierr = PetscObjectTypeCompare((PetscObject) ts, type, >> &match);CHKERRQ(ierr); >> if (match) PetscFunctionReturn(0); >> >> ierr = PetscFunctionListFind(PetscObjectComm((PetscObject)ts),TSList, >> type,PETSC_TRUE, (void (**)(void)) &r);CHKERRQ(ierr); >> if (!r) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_ARG_UNKNOWN_TYPE, "Unknown >> TS type: %s", type); >> if (ts->ops->destroy) { >> ierr = (*(ts)->ops->destroy)(ts);CHKERRQ(ierr); >> >> ts->ops->destroy = NULL; >> } >> ierr = PetscMemzero(ts->ops,sizeof(*ts->ops));CHKERRQ(ierr); >> >> but note that >> >> PetscErrorCode (*snesfunction)(SNES,Vec,Vec,TS); >> PetscErrorCode (*snesjacobian)(SNES,Vec,Mat*,Mat*,MatStructure*,TS); >> PetscErrorCode (*prestep)(TS); >> PetscErrorCode (*prestage)(TS,PetscReal); >> PetscErrorCode (*poststep)(TS); >> PetscErrorCode (*setup)(TS); >> PetscErrorCode (*step)(TS); >> PetscErrorCode (*solve)(TS); >> PetscErrorCode (*interpolate)(TS,PetscReal,Vec); >> PetscErrorCode (*evaluatestep)(TS,PetscInt,Vec,PetscBool*); >> PetscErrorCode (*setfromoptions)(TS); >> PetscErrorCode (*destroy)(TS); >> PetscErrorCode (*view)(TS,PetscViewer); >> PetscErrorCode (*reset)(TS); >> PetscErrorCode >> (*linearstability)(TS,PetscReal,PetscReal,PetscReal*,PetscReal*); >> PetscErrorCode (*load)(TS,PetscViewer); >> >> TSOps is a combination of method specific functions and user provided >> functions. So if one does TSSetPostStep() before TSSetFromOptions() and >> then -ts_type <sometype> the post-step is wiped out even though it should >> not be. >> >> How to fix? Should the memzero() be replaced with something that only >> wipes out certain fields but preserves the pre step, prestage and post step >> ops. Or should those ops be moved out of TSOps and handled differently? >> Needs to be fixed since current behavior is cruel. >> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] TS problems wasted my time
by Jed Brown 24 Feb '13

24 Feb '13

Move it out of TSOps. On Feb 24, 2013 2:10 PM, "Barry Smith" <bsmith(a)mcs.anl.gov> wrote: > > PetscErrorCode TSSetType(TS ts,TSType type) > { > PetscErrorCode (*r)(TS); > PetscBool match; > PetscErrorCode ierr; > > PetscFunctionBegin; > PetscValidHeaderSpecific(ts, TS_CLASSID,1); > ierr = PetscObjectTypeCompare((PetscObject) ts, type, > &match);CHKERRQ(ierr); > if (match) PetscFunctionReturn(0); > > ierr = PetscFunctionListFind(PetscObjectComm((PetscObject)ts),TSList, > type,PETSC_TRUE, (void (**)(void)) &r);CHKERRQ(ierr); > if (!r) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_ARG_UNKNOWN_TYPE, "Unknown TS > type: %s", type); > if (ts->ops->destroy) { > ierr = (*(ts)->ops->destroy)(ts);CHKERRQ(ierr); > > ts->ops->destroy = NULL; > } > ierr = PetscMemzero(ts->ops,sizeof(*ts->ops));CHKERRQ(ierr); > > but note that > > PetscErrorCode (*snesfunction)(SNES,Vec,Vec,TS); > PetscErrorCode (*snesjacobian)(SNES,Vec,Mat*,Mat*,MatStructure*,TS); > PetscErrorCode (*prestep)(TS); > PetscErrorCode (*prestage)(TS,PetscReal); > PetscErrorCode (*poststep)(TS); > PetscErrorCode (*setup)(TS); > PetscErrorCode (*step)(TS); > PetscErrorCode (*solve)(TS); > PetscErrorCode (*interpolate)(TS,PetscReal,Vec); > PetscErrorCode (*evaluatestep)(TS,PetscInt,Vec,PetscBool*); > PetscErrorCode (*setfromoptions)(TS); > PetscErrorCode (*destroy)(TS); > PetscErrorCode (*view)(TS,PetscViewer); > PetscErrorCode (*reset)(TS); > PetscErrorCode > (*linearstability)(TS,PetscReal,PetscReal,PetscReal*,PetscReal*); > PetscErrorCode (*load)(TS,PetscViewer); > > TSOps is a combination of method specific functions and user provided > functions. So if one does TSSetPostStep() before TSSetFromOptions() and > then -ts_type <sometype> the post-step is wiped out even though it should > not be. > > How to fix? Should the memzero() be replaced with something that only > wipes out certain fields but preserves the pre step, prestage and post step > ops. Or should those ops be moved out of TSOps and handled differently? > Needs to be fixed since current behavior is cruel. > >

1 0

TS problems wasted my time
by Barry Smith 24 Feb '13

24 Feb '13

PetscErrorCode TSSetType(TS ts,TSType type) { PetscErrorCode (*r)(TS); PetscBool match; PetscErrorCode ierr; PetscFunctionBegin; PetscValidHeaderSpecific(ts, TS_CLASSID,1); ierr = PetscObjectTypeCompare((PetscObject) ts, type, &match);CHKERRQ(ierr); if (match) PetscFunctionReturn(0); ierr = PetscFunctionListFind(PetscObjectComm((PetscObject)ts),TSList, type,PETSC_TRUE, (void (**)(void)) &r);CHKERRQ(ierr); if (!r) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_ARG_UNKNOWN_TYPE, "Unknown TS type: %s", type); if (ts->ops->destroy) { ierr = (*(ts)->ops->destroy)(ts);CHKERRQ(ierr); ts->ops->destroy = NULL; } ierr = PetscMemzero(ts->ops,sizeof(*ts->ops));CHKERRQ(ierr); but note that PetscErrorCode (*snesfunction)(SNES,Vec,Vec,TS); PetscErrorCode (*snesjacobian)(SNES,Vec,Mat*,Mat*,MatStructure*,TS); PetscErrorCode (*prestep)(TS); PetscErrorCode (*prestage)(TS,PetscReal); PetscErrorCode (*poststep)(TS); PetscErrorCode (*setup)(TS); PetscErrorCode (*step)(TS); PetscErrorCode (*solve)(TS); PetscErrorCode (*interpolate)(TS,PetscReal,Vec); PetscErrorCode (*evaluatestep)(TS,PetscInt,Vec,PetscBool*); PetscErrorCode (*setfromoptions)(TS); PetscErrorCode (*destroy)(TS); PetscErrorCode (*view)(TS,PetscViewer); PetscErrorCode (*reset)(TS); PetscErrorCode (*linearstability)(TS,PetscReal,PetscReal,PetscReal*,PetscReal*); PetscErrorCode (*load)(TS,PetscViewer); TSOps is a combination of method specific functions and user provided functions. So if one does TSSetPostStep() before TSSetFromOptions() and then -ts_type <sometype> the post-step is wiped out even though it should not be. How to fix? Should the memzero() be replaced with something that only wipes out certain fields but preserves the pre step, prestage and post step ops. Or should those ops be moved out of TSOps and handled differently? Needs to be fixed since current behavior is cruel.

1 0

Re: [petsc-dev] [petsc-maint #154101] clarification of FP error thrown in VECNORM in 3.3 PETSc
by Jed Brown 22 Feb '13

22 Feb '13

I would use the same flag, just error earlier so that the trace is more useful. We know that the NaN won't turn into something useful after it is computed. On Feb 22, 2013 9:08 PM, "Barry Smith" <bsmith(a)mcs.anl.gov> wrote: > > On Feb 22, 2013, at 8:40 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > > > > > On Fri, Feb 22, 2013 at 6:19 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > > Ok, I did a search through the code to see where we are checking it. > > > > 1) VecNorm, VecDot, VecMDot -- and generate an error > > > > Yeah, I never liked this. > > > > > > 2) Many of the Krylov solvers and nonlinear solvers check the results > of (most - should be all) norms and dots and either > > > > a) generate a not converged KSP or SNES converged reason (good) > > > > b) generate an error (bad, most do this). > > > > We want the solvers to have an option to raise an error because it helps > users track down where they made a mistake (especially if they're running > in a shitty environment where -fp_trap and gdb doesn't work nicely). I'm > not sure which should be the default. > > Currently KSPSetErrorIfNotConverged() will trigger an error at the end > of the KSPSolve() routine if not converged (including NaN). If this is not > close enough to the problem for you we could use the same flag to stop the > code with a SETERRQ() as soon as the norm, dot, mdot problem is detected? > Or we could add yet another flag? > > Barry > > > > > > > > Note that currently the checks in the solvers are redundant > because an error was already generated in the VecNorm, Dot or MDot so NaN > never > > gets to the iterative solver > > > > 3) Few other irrelevant places. > > > > The simplest thing would be to simply remove the checks in 1). Then, > eventually, update all the solvers to NOT generate an error but instead > return with > > an appropriate diverged reason. The drawback is that other code using > dots and norms will always chug along without generating an error if users > don't > > check the results themselves. > > > > I think this is an event that the caller should deal with since only > they have enough context to return cleanly. Yes, it's more code, but it's > the right thing. > > > > I vote we just make a patch in 3.3 to remove the checks in norm, dot, > mdot and Jed adds another project to the bitbucket list saying upgrade the > checks in the solvers to return with appropriate diverged reason instead of > generating an error. Then the whole flag business goes away. > > > > That's fine with me. > >

1 0