March 2012 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] GAMG hiding MG -help
by Jed Brown 20 Mar '12

20 Mar '12

No, this fails for anyone with more than one PCGAMG. On Mar 20, 2012 9:10 AM, "Mark F. Adams" <mark.adams(a)columbia.edu> wrote: > How about this added to the end of my setoptions: > > ierr = PetscOptionsTail();CHKERRQ(ierr); > > static PetscInt nc=0; > if(!nc++)obj->optionsprinted=PETSC_FALSE; > > PetscFunctionReturn(0); > } > > Mark > > > On Mar 19, 2012, at 11:23 PM, Jed Brown wrote: > > On Mon, Mar 19, 2012 at 22:11, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > >> >> Mark, >> >> The problem is all the code Jed added to prevent the same help message >> from being printed more than once. He added a field to each object, >> optionsprinted, which gets set once an object is used with >> PetscObjectOptionsBegin() and prevents printing the next time that same >> object is used with options. Since the same PC is used for the >> setfromoptions for gamg and mg the second one mg gets ignored. >> >> I cannot see, off hand, the correct fix that respects Jed's desire to >> have the help messages printed exactly once. You can change >> PetscObjectOptionsBegin() to PetscOptionsBegin() to get the MG message >> printed but it will be printed again for each new GAMG setup violating >> Jed's clean output. You could hide a variable in the _GAMG data structure >> indicating the help for MG was printed to prevent multiple prints but that >> is ugly and ideally Jed will point out the correct way to do what you want >> to do. >> > > I don't know anything simple. Best would be to make it possible to call > PCSetFromOptions_MG() before the number of levels has been set. I don't > know if people really want to see the options printed for every level > anyway. > > The easiest thing might be to save and restore obj->optionsprinted. (We > could make it an integer instead of a PetscBool.) > > >> >> >> Barry >> >> It never bothered me that help messages were printed more than once >> ..... >> > > It's still a mess, actually. Try running > > $ ./ex5 -snes_grid_sequence 4 -help > > and tell me you are patient enough to scroll up far enough to see a > meaningful option. > > >

1 0

Re: [petsc-dev] MPIX_Iallreduce()
by Satish Balay 20 Mar '12

20 Mar '12

On Tue, 20 Mar 2012, Jed Brown wrote: > On Tue, Mar 20, 2012 at 00:06, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > > > On Sun, Mar 18, 2012 at 11:55, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > >> Add a glance adding all these new complications to PETSc to chase an > >> impossible overlap of communication and computation sounds fine :-) > > > > > > /Q has a dedicated thread to drive asynchronous comm. I've added this, the > > call to PetscCommSplitReductionBegin() is entirely optional (does not alter > > program semantics), but will allow asynchronous progress to be made. > > > > On conventional systems, there are two choices for driving asynchronous > > progress: > > > > 1. Set the environment variable MPICH_ASYNC_PROGRESS=1. "Setting that > > environment variable will cause a cheesy form of background progress > > wherein the library will spawn an additional background thread per MPI > > process. You'll have to play around with things, but I'd recommend cutting > > your number of processes per node in half to avoid nemesis oversubscription > > badness." -- Dave Goodell > > > > 2. Make any nontrivial calls into the MPI stack. This could specifically > > mean polling a request, but it could also just be your usual communication. > > I suspect that a standard MatMult() and PCApply() will be enough to drive a > > significant amount of progress on the reduction. > > > > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/d2d98894cb5c > > > > And here's some preliminary test output on 32 processes of cg.mcs. I ran a > handful of times with src/vec/vec/examples/tests/ex42.c and these results > seem reproducible. > > VecScatterBegin 300 1.0 8.5473e-04 1.5 0.00e+00 0.0 9.6e+03 4.0e+01 > 0.0e+00 2 0 98 99 0 2 0 98 99 0 0 > VecScatterEnd 300 1.0 1.1136e-02 7.6 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 18 0 0 0 0 18 0 0 0 0 0 > VecReduceArith 100 1.0 1.4210e-04 1.5 9.00e+02 1.0 0.0e+00 0.0e+00 > 0.0e+00 0100 0 0 0 0100 0 0 0 203 > VecReduceComm 100 1.0 2.1405e-02 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 > 1.0e+02 60 0 0 0 89 60 0 0 0 90 0 > > VecScatterBegin 300 1.0 5.9962e-04 1.9 0.00e+00 0.0 9.6e+03 4.0e+01 > 0.0e+00 5 0 98 99 0 5 0 98 99 0 0 > VecScatterEnd 300 1.0 1.6668e-03 4.3 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 12 0 0 0 0 12 0 0 0 0 0 > VecReduceArith 100 1.0 1.1611e-04 2.5 9.00e+02 1.0 0.0e+00 0.0e+00 > 0.0e+00 1100 0 0 0 1100 0 0 0 248 > VecReduceBegin 100 1.0 5.3430e-04 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 4 0 0 0 0 4 0 0 0 0 0 > VecReduceEnd 100 1.0 2.4858e-03 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 21 0 0 0 0 21 0 0 0 0 0 Jed, are you pinning the mpi jobs to specific cores for this tests? Does it make a difference? I'm curious as this machine has assymetric cores wrt L2/FPU. Presumably - using p0,p2,p4 etc should spread out the load - but I don't know if the kernel is doing this automatically. Satish

1 0

Re: [petsc-dev] GAMG hiding MG -help
by Mark F. Adams 20 Mar '12

20 Mar '12

How about this added to the end of my setoptions: ierr = PetscOptionsTail();CHKERRQ(ierr); static PetscInt nc=0; if(!nc++)obj->optionsprinted=PETSC_FALSE; PetscFunctionReturn(0); } Mark On Mar 19, 2012, at 11:23 PM, Jed Brown wrote: > On Mon, Mar 19, 2012 at 22:11, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Mark, > > The problem is all the code Jed added to prevent the same help message from being printed more than once. He added a field to each object, optionsprinted, which gets set once an object is used with PetscObjectOptionsBegin() and prevents printing the next time that same object is used with options. Since the same PC is used for the setfromoptions for gamg and mg the second one mg gets ignored. > > I cannot see, off hand, the correct fix that respects Jed's desire to have the help messages printed exactly once. You can change PetscObjectOptionsBegin() to PetscOptionsBegin() to get the MG message printed but it will be printed again for each new GAMG setup violating Jed's clean output. You could hide a variable in the _GAMG data structure indicating the help for MG was printed to prevent multiple prints but that is ugly and ideally Jed will point out the correct way to do what you want to do. > > I don't know anything simple. Best would be to make it possible to call PCSetFromOptions_MG() before the number of levels has been set. I don't know if people really want to see the options printed for every level anyway. > > The easiest thing might be to save and restore obj->optionsprinted. (We could make it an integer instead of a PetscBool.) > > > > Barry > > It never bothered me that help messages were printed more than once ..... > > It's still a mess, actually. Try running > > $ ./ex5 -snes_grid_sequence 4 -help > > and tell me you are patient enough to scroll up far enough to see a meaningful option.

1 0

Re: [petsc-dev] MPIX_Iallreduce()
by Jed Brown 20 Mar '12

20 Mar '12

On Tue, Mar 20, 2012 at 00:06, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Sun, Mar 18, 2012 at 11:55, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > >> Add a glance adding all these new complications to PETSc to chase an >> impossible overlap of communication and computation sounds fine :-) > > > /Q has a dedicated thread to drive asynchronous comm. I've added this, the > call to PetscCommSplitReductionBegin() is entirely optional (does not alter > program semantics), but will allow asynchronous progress to be made. > > On conventional systems, there are two choices for driving asynchronous > progress: > > 1. Set the environment variable MPICH_ASYNC_PROGRESS=1. "Setting that > environment variable will cause a cheesy form of background progress > wherein the library will spawn an additional background thread per MPI > process. You'll have to play around with things, but I'd recommend cutting > your number of processes per node in half to avoid nemesis oversubscription > badness." -- Dave Goodell > > 2. Make any nontrivial calls into the MPI stack. This could specifically > mean polling a request, but it could also just be your usual communication. > I suspect that a standard MatMult() and PCApply() will be enough to drive a > significant amount of progress on the reduction. > > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/d2d98894cb5c > And here's some preliminary test output on 32 processes of cg.mcs. I ran a handful of times with src/vec/vec/examples/tests/ex42.c and these results seem reproducible. VecScatterBegin 300 1.0 8.5473e-04 1.5 0.00e+00 0.0 9.6e+03 4.0e+01 0.0e+00 2 0 98 99 0 2 0 98 99 0 0 VecScatterEnd 300 1.0 1.1136e-02 7.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 18 0 0 0 0 0 VecReduceArith 100 1.0 1.4210e-04 1.5 9.00e+02 1.0 0.0e+00 0.0e+00 0.0e+00 0100 0 0 0 0100 0 0 0 203 VecReduceComm 100 1.0 2.1405e-02 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+02 60 0 0 0 89 60 0 0 0 90 0 VecScatterBegin 300 1.0 5.9962e-04 1.9 0.00e+00 0.0 9.6e+03 4.0e+01 0.0e+00 5 0 98 99 0 5 0 98 99 0 0 VecScatterEnd 300 1.0 1.6668e-03 4.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 12 0 0 0 0 12 0 0 0 0 0 VecReduceArith 100 1.0 1.1611e-04 2.5 9.00e+02 1.0 0.0e+00 0.0e+00 0.0e+00 1100 0 0 0 1100 0 0 0 248 VecReduceBegin 100 1.0 5.3430e-04 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 4 0 0 0 0 4 0 0 0 0 0 VecReduceEnd 100 1.0 2.4858e-03 2.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 21 0 0 0 0 21 0 0 0 0 0

1 0

Re: [petsc-dev] MPIX_Iallreduce()
by Jed Brown 20 Mar '12

20 Mar '12

On Sun, Mar 18, 2012 at 11:55, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > Add a glance adding all these new complications to PETSc to chase an > impossible overlap of communication and computation sounds fine :-) /Q has a dedicated thread to drive asynchronous comm. I've added this, the call to PetscCommSplitReductionBegin() is entirely optional (does not alter program semantics), but will allow asynchronous progress to be made. On conventional systems, there are two choices for driving asynchronous progress: 1. Set the environment variable MPICH_ASYNC_PROGRESS=1. "Setting that environment variable will cause a cheesy form of background progress wherein the library will spawn an additional background thread per MPI process. You'll have to play around with things, but I'd recommend cutting your number of processes per node in half to avoid nemesis oversubscription badness." -- Dave Goodell 2. Make any nontrivial calls into the MPI stack. This could specifically mean polling a request, but it could also just be your usual communication. I suspect that a standard MatMult() and PCApply() will be enough to drive a significant amount of progress on the reduction. http://petsc.cs.iit.edu/petsc/petsc-dev/rev/d2d98894cb5c

1 0

Re: [petsc-dev] Checking for PETSc Cuda Support
by Jed Brown 19 Mar '12

19 Mar '12

On Mon, Mar 19, 2012 at 22:15, Dave Nystrom <Dave.Nystrom(a)tachyonlogic.com>wrote: > OK. I was thinking of a function but that should work. subroutine does_petsc_have_cuda(does) logical :: does does = .false. #ifdef PETSC_HAVE_CUDA does = .true. #endif end subroutine

1 0

Re: [petsc-dev] GAMG hiding MG -help
by Jed Brown 19 Mar '12

19 Mar '12

On Mon, Mar 19, 2012 at 22:11, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Mark, > > The problem is all the code Jed added to prevent the same help message > from being printed more than once. He added a field to each object, > optionsprinted, which gets set once an object is used with > PetscObjectOptionsBegin() and prevents printing the next time that same > object is used with options. Since the same PC is used for the > setfromoptions for gamg and mg the second one mg gets ignored. > > I cannot see, off hand, the correct fix that respects Jed's desire to > have the help messages printed exactly once. You can change > PetscObjectOptionsBegin() to PetscOptionsBegin() to get the MG message > printed but it will be printed again for each new GAMG setup violating > Jed's clean output. You could hide a variable in the _GAMG data structure > indicating the help for MG was printed to prevent multiple prints but that > is ugly and ideally Jed will point out the correct way to do what you want > to do. > I don't know anything simple. Best would be to make it possible to call PCSetFromOptions_MG() before the number of levels has been set. I don't know if people really want to see the options printed for every level anyway. The easiest thing might be to save and restore obj->optionsprinted. (We could make it an integer instead of a PetscBool.) > > > Barry > > It never bothered me that help messages were printed more than once ..... > It's still a mess, actually. Try running $ ./ex5 -snes_grid_sequence 4 -help and tell me you are patient enough to scroll up far enough to see a meaningful option.

1 0

Re: [petsc-dev] Checking for PETSc Cuda Support
by Dave Nystrom 19 Mar '12

19 Mar '12

Jed Brown writes: > On Mon, Mar 19, 2012 at 21:55, Dave Nystrom <dnystrom1(a)comcast.net> wrote: > > > Is there a way to ask PETSc if it has cuda support compiled in from a > > Fortran > > code? > > > > #ifdef PETSC_HAVE_CUDA > > #endif OK. I was thinking of a function but that should work. Thanks, Dave

1 0

Re: [petsc-dev] GAMG hiding MG -help
by Barry Smith 19 Mar '12

19 Mar '12

Mark, The problem is all the code Jed added to prevent the same help message from being printed more than once. He added a field to each object, optionsprinted, which gets set once an object is used with PetscObjectOptionsBegin() and prevents printing the next time that same object is used with options. Since the same PC is used for the setfromoptions for gamg and mg the second one mg gets ignored. I cannot see, off hand, the correct fix that respects Jed's desire to have the help messages printed exactly once. You can change PetscObjectOptionsBegin() to PetscOptionsBegin() to get the MG message printed but it will be printed again for each new GAMG setup violating Jed's clean output. You could hide a variable in the _GAMG data structure indicating the help for MG was printed to prevent multiple prints but that is ugly and ideally Jed will point out the correct way to do what you want to do. Barry It never bothered me that help messages were printed more than once ..... On Mar 19, 2012, at 6:58 PM, Mark F. Adams wrote: > I just found that GAMG seems to be hiding MG's parameters when -help is used. > > I have this code in my PCSetUp_GAMG: > > /* should be called in PCSetFromOptions_GAMG(), but cannot be called prior to PCMGSetLevels() */ > ierr = PetscObjectOptionsBegin((PetscObject)pc);CHKERRQ(ierr); > ierr = PCSetFromOptions_MG( pc ); CHKERRQ(ierr); > ierr = PetscOptionsEnd();CHKERRQ(ierr); > if (mg->galerkin != 2) SETERRQ(wcomm,PETSC_ERR_USER,"GAMG does Galerkin manually so the -pc_mg_galerkin option must not be used."); > > Which seems funky, but I think I cloned this or had some reason to put this here. And I set: > > pc->ops->setfromoptions = PCSetFromOptions_GAMG; > > in PCCreate_GAMG. > > Any advice on how I should fix this? > > Mark

1 0

Re: [petsc-dev] Checking for PETSc Cuda Support
by Jed Brown 19 Mar '12

19 Mar '12

On Mon, Mar 19, 2012 at 21:55, Dave Nystrom <dnystrom1(a)comcast.net> wrote: > Is there a way to ask PETSc if it has cuda support compiled in from a > Fortran > code? > #ifdef PETSC_HAVE_CUDA #endif

1 0