June 2016 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] asm / gasm
by Barry Smith 24 Jun '16

24 Jun '16

> On Jun 24, 2016, at 1:35 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > On Thu, Jun 23, 2016 at 11:46 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Mark, > > It is not as simple as this to convert to ASM. It will take a little bit of work to use ASM here instead of GASM. > > > Just to be clear: ASM used to work. Did the semantics of ASM change? Show me a commit where ASM worked! Do you mean that the GASM worked? The code has GASM calls in it, not ASM so how could ASM have previously worked? It is possible that something changed in GASM that broke GAMG's usage of GASM. Once you tell me how to reproduce the problem with GASM I can try to track down the problem. > > But before that please please tell me the command line argument and example you use where the GASM crashes so I can get that fixed. Then I will look at using ASM instead after I have the current GASM code running again. > > In branch mark/gamg-agg-asm in ksp ex56, 'make runex56': I don't care about this! This is where you have tried to change from GASM to ASM which I told you is non-trivial. Give me the example and command line where the GASM version in master (or maint) doesn't work where the error message includes ** Max-trans not allowed because matrix is distributed We are not communicating very well, you jumped from stating GASM crashed to monkeying with ASM and now refuse to tell me how to reproduce the GASM crash. We have to start by fixing the current code to work with GASM (if it ever worked) and then move on to using ASM (which is just an optimization of the GASM usage.) Barry > > 14:12 nid00495 ~/petsc/src/ksp/ksp/examples/tutorials$ make runex56 > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Petsc has generated inconsistent data > [0]PETSC ERROR: MPI_Allreduce() called in different locations (code lines) on different processors > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.7.2-633-g4f88208 GIT Date: 2016-06-23 18:53:31 +0200 > [0]PETSC ERROR: /global/u2/m/madams/petsc/src/ksp/ksp/examples/tutorials/./ex56 on a arch-xc30-dbg64-intel named nid00495 by madams Thu Jun 23 14:12:57 2016 > [0]PETSC ERROR: Configure options --COPTFLAGS="-no-ipo -g -O0" --CXXOPTFLAGS="-no-ipo -g -O0" --FOPTFLAGS="-fast -no-ipo -g -O0" --download-parmetis --download-metis --with-ssl=0 --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-debugging=1 --with-fc=0 --with-shared-libraries=0 --with-x=0 --with-mpiexec=srun LIBS=-lstdc++ --with-64-bit-indices PETSC_ARCH=arch-xc30-dbg64-intel > [0]PETSC ERROR: #1 MatGetSubMatrices_MPIAIJ() li > > > > > > > > > > > Barry > > > > On Jun 23, 2016, at 4:19 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > The question boils down to, for empty processors do we: > > > > ierr = ISCreateGeneral(PETSC_COMM_SELF, 0, NULL, PETSC_COPY_VALUES, &is);CHKERRQ(ierr); > > ierr = PCASMSetLocalSubdomains(subpc, 1, &is, NULL);CHKERRQ(ierr); > > ierr = ISDestroy(&is);CHKERRQ(ierr); > > > > or > > > > PCASMSetLocalSubdomains(subpc, 0, NULL, NULL); > > > > The later gives and error that one domain is need and the later gives an error (appended). > > > > I've checked in the code for this second error in ksp (make runex56) > > > > Thanks, > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Petsc has generated inconsistent data > > [0]PETSC ERROR: MPI_Allreduce() called in different locations (code lines) on different processors > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > [0]PETSC ERROR: Petsc Development GIT revision: v3.7.2-633-g4f88208 GIT Date: 2016-06-23 18:53:31 +0200 > > [0]PETSC ERROR: /global/u2/m/madams/petsc/src/ksp/ksp/examples/tutorials/./ex56 on a arch-xc30-dbg64-intel named nid00495 by madams Thu Jun 23 14:12:57 2016 > > [0]PETSC ERROR: Configure options --COPTFLAGS="-no-ipo -g -O0" --CXXOPTFLAGS="-no-ipo -g -O0" --FOPTFLAGS="-fast -no-ipo -g -O0" --download-parmetis --download-metis --with-ssl=0 --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-debugging=1 --with-fc=0 --with-shared-libraries=0 --with-x=0 --with-mpiexec=srun LIBS=-lstdc++ --with-64-bit-indices PETSC_ARCH=arch-xc30-dbg64-intel > > [0]PETSC ERROR: #1 MatGetSubMatrices_MPIAIJ() line 1147 in /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/mpiov.c > > [0]PETSC ERROR: #2 MatGetSubMatrices_MPIAIJ() line 1147 in /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/mpiov.c > > [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > > On Thu, Jun 23, 2016 at 8:05 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > > Where is the command line that generates the error? > > > > > > > On Jun 23, 2016, at 12:08 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > [adding Garth] > > > > > > On Thu, Jun 23, 2016 at 12:52 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > > > > Mark, > > > > > > I think there is a misunderstanding here. With GASM an individual block problem is __solved__ (via a parallel KSP) in parallel by several processes, with ASM each block is "owned" by and solved on a single process. > > > > > > Ah, OK, so this is for multiple processors in a block. Yes, we are looking at small, smother, blocks. > > > > > > > > > With both the "block" can come from any unknowns on any processes. You can have, for example a block that comes from a region snaking across several processes if you like (or it makes sense due to coupling in the matrix). > > > > > > By default if you use ASM it will create one non-overlapping block defined by all unknowns owned by a single process and then extend it by "one level" (defined by the nonzero structure of the matrix) to get overlapping. > > > > > > The default in ASM is one level of overlap? That is new. (OK, I have not looked at ASM in like over 10 years) > > > > > > If you use multiple blocks per process it defines the non-overlapping blocks within a single process's unknowns > > > > > > I assume this still chops the matrix and does not call a partitioner. > > > > > > and extends each of them to have overlap (again by the non-zero structure of the matrix). The default is simple because the user only need indicate the number of blocks per process, the drawback is of course that it does depend on the process layout, number of processes etc and does not take into account particular "coupling information" that the user may know about with their problem. > > > > > > If the user wishes to defined the blocks themselves that is also possible with PCASMSetSubLocalSubdomains(). Each process provides 1 or more index sets for the subdomains it will solve on. Note that the index sets can contain any unknowns in the entire problem so the blocks do not have to "line up" with the parallel decomposition at all. > > > > > > Oh, OK, this is what I want. (I thought this worked). > > > > > > Of course determining and providing good such subdomains may not always be clear. > > > > > > In smoothed aggregation there is an argument that the aggregates are good, but the scale is fixed obviously. On a regular grid smoothed aggregation wants 3^D sized aggregates, which is obviously wonderful for AMS. And for anisotropy you want your ASM blocks to be on strongly connected components, which is what smoothed aggregation wants (not that I do this very well). > > > > > > > > > I see in GAMG you have PCGAMGSetUseASMAggs > > > > > > But the code calls PCGASMSetSubdomains and the command line is -pc_gamg_use_agg_gasm, so this is all messed up. (more below) > > > > > > which sadly does not have an explanation in the users manual and sadly does not have a matching options data base name -pc_gamg_use_agg_gasm following the rule of drop the word set, all lower case, and put _ between words the option should be -pc_gamg_use_asm_aggs. > > > > > > BUT, THIS IS THE WAY IT WAS! It looks like someone hijacked this code and made it gasm. I never did this. > > > > > > Barry: you did this apparently in 2013. > > > > > > > > > In addition to this one you could also have one that uses the aggs but use the PCASM to manage the solves instead of GASM, it would likely be less buggy and more efficient. > > > > > > yes > > > > > > > > > Please tell me exactly what example you tried to run with what options and I will debug it. > > > > > > We got an error message: > > > > > > ** Max-trans not allowed because matrix is distributed > > > > > > Garth: is this from your code perhaps? I don't see it in PETSc. > > > > > > Note that ALL functionality that is included in PETSc should have tests that test that functionality then we will find out immediately when it is broken instead of two years later when it is much harder to debug. If this -pc_gamg_use_agg_gasm had had a test we won't be in this mess now. (Jed's damn code reviews sure don't pick up this stuff). > > > > > > First we need to change gasm to asm. > > > > > > We could add this argument pc_gamg_use_agg_asm to ksp/ex56 (runex56 or make a new test). The SNES version (also ex56) is my current test that I like to refer to as recommended parameters for elasticity. So I'd like to keep that clean, but we can add junk to ksp/ex56. > > > > > > I've done this in a branch mark/gamg-agg-asm. I get an error (appended). It looks like the second coarsest grid, which has 36 dof on one processor has an index 36 in the block on every processor. Strange. I can take a look at it later. > > > > > > Mark > > > > > > > [3]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > > [4]PETSC ERROR: Petsc has generated inconsistent data > > > > [4]PETSC ERROR: ith 0 block entry 36 not owned by any process, upper bound 36 > > > > [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > > > [4]PETSC ERROR: Petsc Development GIT revision: v3.7.2-630-g96e0c40 GIT Date: 2016-06-22 10:03:02 -0500 > > > > [4]PETSC ERROR: ./ex56 on a arch-macosx-gnu-g named MarksMac-3.local by markadams Thu Jun 23 06:53:27 2016 > > > > [4]PETSC ERROR: Configure options COPTFLAGS="-g -O0" CXXOPTFLAGS="-g -O0" FOPTFLAGS="-g -O0" --download-hypre=1 --download-parmetis=1 --download-metis=1 --download-ml=1 --download-p4est=1 --download-exodus=1 --download-triangle=1 --with-hdf5-dir=/Users/markadams/Codes/hdf5 --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g --download-chaco > > > > [4]PETSC ERROR: #1 VecScatterCreate_PtoS() line 2348 in /Users/markadams/Codes/petsc/src/vec/vec/utils/vpscat.c > > > > [4]PETSC ERROR: #2 VecScatterCreate() line 1552 in /Users/markadams/Codes/petsc/src/vec/vec/utils/vscat.c > > > > [4]PETSC ERROR: Petsc has generated inconsistent data > > > > [3]PETSC ERROR: ith 0 block entry 36 not owned by any process, upper bound 36 > > > > [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > > > [3]PETSC ERROR: Petsc Development GIT revision: v3.7.2-630-g96e0c40 GIT Date: 2016-06-22 10:03:02 -0500 > > > > [3]PETSC ERROR: ./ex56 on a arch-macosx-gnu-g named MarksMac-3.local by markadams Thu Jun 23 06:53:27 2016 > > > > [3]PETSC ERROR: Configure options COPTFLAGS="-g -O0" CXXOPTFLAGS="-g -O0" FOPTFLAGS="-g -O0" --download-hypre=1 --download-parmetis=1 --download-metis=1 --download-ml=1 --download-p4est=1 --download-exodus=1 --download-triangle=1 --with-hdf5-dir=/Users/markadams/Codes/hdf5 --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g --download-chaco > > > > [3]PETSC ERROR: #1 VecScatterCreate_PtoS() line 2348 in /Users/markadams/Codes/petsc/src/vec/vec/utils/vpscat.c > > > > [3]PETSC ERROR: #2 VecScatterCreate() line 1552 in /Users/markadams/Codes/petsc/src/vec/vec/utils/vscat.c > > > > [3]PETSC ERROR: #3 PCSetUp_ASM() line 279 in /Users/markadams/Codes/petsc/src/ksp/pc/impls/asm/asm.c > > > > > > > > > > > > > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > > > On Jun 22, 2016, at 5:20 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > > > > > > > > > > > On Wed, Jun 22, 2016 at 8:06 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > > > > > > I suggest focusing on asm. > > > > > > > > OK, I will switch gasm to asm, this does not work anyway. > > > > > > > > Having blocks that span multiple processes seems like over kill for a smoother ? > > > > > > > > No, because it is a pain to have the math convolved with the parallel decompositions strategy (ie, I can't tell an application how to partition their problem). If an aggregate spans processor boundaries, which is fine and needed, and let's say we have a pretty uniform problem, then if the block gets split up, H is small in part of the domain and convergence could suffer along processor boundaries. And having the math change as the parallel decomposition changes is annoying. > > > > > > > > (Major league overkill) in fact doesn't one want multiple blocks per process, ie. pretty small blocks. > > > > > > > > No, it is just doing what would be done in serial. If the cost of moving the data across the processor is a problem then that is a tradeoff to consider. > > > > > > > > And I think you are misunderstanding me. There are lots of blocks per process (the aggregates are say 3^D in size). And many of the aggregates/blocks along the processor boundary will be split between processors, resulting is mall blocks and weak ASM PC on processor boundaries. > > > > > > > > I can understand ASM not being general and not letting blocks span processor boundaries, but I don't think the extra matrix communication costs are a big deal (done just once) and the vector communication costs are not bad, it probably does not include (too many) new processors to communicate with. > > > > > > > > > > > > Barry > > > > > > > > > On Jun 22, 2016, at 7:51 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > > > > > I'm trying to get block smoothers to work for gamg. We (Garth) tried this and got this error: > > > > > > > > > > > > > > > - Another option is use '-pc_gamg_use_agg_gasm true' and use '-mg_levels_pc_type gasm'. > > > > > > > > > > > > > > > Running in parallel, I get > > > > > > > > > > ** Max-trans not allowed because matrix is distributed > > > > > ---- > > > > > > > > > > First, what is the difference between asm and gasm? > > > > > > > > > > Second, I need to fix this to get block smoothers. This used to work. Did we lose the capability to have blocks that span processor subdomains? > > > > > > > > > > gamg only aggregates across processor subdomains within one layer, so maybe I could use one layer of overlap in some way? > > > > > > > > > > Thanks, > > > > > Mark > > > > > > > > > > > > > > > > > > > > > > > > >

1 0

Re: [petsc-dev] MatMFFDSetType
by Barry Smith 24 Jun '16

24 Jun '16

> On Jun 24, 2016, at 12:13 AM, Adrian Croucher <a.croucher(a)auckland.ac.nz> wrote: > > Further to this, I debugged into PetscObjectTypeCompare(). > > It's returning PETSC_FALSE because the type name of the variable mat is "seqbaij" (and I guess would normally be something like "mpibaij" if running in parallel outside gdb), which doesn't match with MATMFFD (= "mffd"). > > Does that mean I should be setting the matrix type to MATMFFD to get this to work? Either you are accessing the "wrong" matrix to set this value or at a later time the given matrix is set to MATMFFD. If you are getting the matrix in your SNES compute Jacobian code make sure you set the value for the first mat argument, not the second. Are you using a command line argument -snes_mf_operator? Or some other command line argument to use mffd? Barry > - Adrian > > On 24/06/16 16:35, Adrian Croucher wrote: >> On 24/06/16 15:58, Adrian Croucher wrote: >>> >>> I will try debugging into MatMFFDSetType() and seeing what it's doing when I pass in "ds". >>> >> >> I did this. In MatMFFDSetType() (mffd.c: 116), it executes >> >> ierr = PetscObjectTypeCompare((PetscObject)mat,MATMFFD,&match);CHKERRQ(ierr); >> >> which returns match = PETSC_FALSE. The next line >> >> if (!match) PetscFunctionReturn(0); >> >> means it just returns out of there without doing anything. >> >> - Adrian > > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > University of Auckland, New Zealand > email: > a.croucher(a)auckland.ac.nz > > tel: +64 (0)9 923 84611 >

1 0

Re: [petsc-dev] Known outdated sections of the User's Manual?
by Matthew Knepley 24 Jun '16

24 Jun '16

On Fri, Jun 24, 2016 at 1:58 AM, Patrick Sanan <patrick.sanan(a)gmail.com> wrote: > I'm working on some groundwork to improve the PETSc documentation, and > the next thing I'd like to look at is the User's Manual, adding > slightly-prettier code listings (as in the update to the dev manual). > > Before doing that, however, it would be very helpful to know if there > are any sections of the manual which are known, by the relevant > experts here, to require deletion or heavy rewrites; it would of > course be a waste of time to format these. > > Specifically, I'm wondering about the following sections, which have > in common that they are things concerned with friendly external tools. > I've seen new users get very frustrated when they expect these sorts > of things to "just work," so it's probably constructive to remove any > outdated information here: > > - Chapter 11: Using MATLAB with PETSc . The support here has changed > quite a lot, so I'm not sure what currently works. Is the MATLAB > Compute Engine still supported? > > - Sections 15.10-15.14: Eclipse/Qt Creator/Developers Studio/XCode > users. This is likely not all current. Is this information helpful > here? > > - Section 15.15 : Graphics. I saw that there were some updates to the > drawing tools recently by Lisandro, so if any of this material is > known to be out of date, that would be helpful to know. > I think graphics is not so much out of date now as incomplete. We really want to be telling people to do things the modern way, but the old ways still work. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] Problem with GAMG
by Mark Adams 24 Jun '16

24 Jun '16

Barry, HEM is experimental and not intended to be fully supported. That said, I don't mind hardening it up. I don't know what PETSc's policy is, if any, on this sort of thing. Anyway, I am trying to pound on HEM, but I get this error: [63]PETSC ERROR: matB must have compressed row usage I have seen this before, but I forget what to do about it. That is getting the B matrix to have compressed row storage. Do you recall? Thanks, Mark On Thu, Jun 23, 2016 at 7:59 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > On Wed, Jun 22, 2016 at 11:27 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > >> >> Mark, >> >> If I use the hem coarsen type I get this error. > > > Does this go away with mis? Don't bother testing, it probably does. > > >> Note that I am setting the -pc_gamg_sym_graph true flag I the debugger >> I do see that PCGAMGFilterGraph() has the >> sym flag set so it seems to me either >> >> 1) the symmetrizer doesn't actually make it symmetric or >> > > It should be symmetric, even with roundoff, I think. > > You have an unsymmetric matrix structurally? > > >> 2) the detector of lack of symmetry thinks it is not symmetric when it is. >> > > The edge weights are used to drop edges and if there is an error then you > can get a structurally unsymmetric graph. Then the parallel graph > algorithms barf (they assume if I know you, you know me, symmetric). > > >> >> Thoughts on how to debug this? >> > > Try with a threshold of 0. I need to take a look at it. Can I run it? > > I have to run for the night (in Vienna) > > > > >> >> Thanks >> >> barry >> >> >> >> $ petscmpiexec -n 2 ./ex19 -pc_type gamg -da_refine 3 -mat_coarsen_type >> hem -pc_gamg_sym_graph true >> lid velocity = 0.0016, prandtl # = 1., grashof # = 1. >> [1]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [1]PETSC ERROR: Petsc has generated inconsistent data >> [1]PETSC ERROR: Have un-symmetric graph (apparently). Use >> '-pc_gamg_sym_graph true' to symetrize the graph or '-pc_gamg_threshold >> -1.0' if the matrix is structurally symmetric. >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Petsc has generated inconsistent data >> [0]PETSC ERROR: Have un-symmetric graph (apparently). Use >> '-pc_gamg_sym_graph true' to symetrize the graph or '-pc_gamg_threshold >> -1.0' if the matrix is structurally symmetric. >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.7.2-596-g05491f7 GIT >> Date: 2016-06-22 13:31:47 -0500 >> [0]PETSC ERROR: ./ex19 on a arch-single named Barrys-MacBook-Pro.local by >> barrysmith Wed Jun 22 16:24:13 2016 >> [0]PETSC ERROR: [1]PETSC ERROR: See >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [1]PETSC ERROR: Petsc Development GIT revision: v3.7.2-596-g05491f7 GIT >> Date: 2016-06-22 13:31:47 -0500 >> [1]PETSC ERROR: ./ex19 on a arch-single named Barrys-MacBook-Pro.local by >> barrysmith Wed Jun 22 16:24:13 2016 >> [1]PETSC ERROR: Configure options >> --with-mpi-dir=/Users/barrysmith/libraries --with-precision=single >> [1]PETSC ERROR: #1 smoothAggs() line 463 in >> /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gamg/agg.c >> Configure options --with-mpi-dir=/Users/barrysmith/libraries >> --with-precision=single >> [0]PETSC ERROR: #1 smoothAggs() line 463 in >> /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gamg/agg.c >> [1]PETSC ERROR: #2 PCGAMGCoarsen_AGG() line 1021 in >> /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gamg/agg.c >> [1]PETSC ERROR: [0]PETSC ERROR: #2 PCGAMGCoarsen_AGG() line 1021 in >> /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gamg/agg.c >> [0]PETSC ERROR: #3 PCSetUp_GAMG() line 524 in >> /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gamg/gamg.c >> #3 PCSetUp_GAMG() line 524 in >> /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gamg/gamg.c >> [1]PETSC ERROR: #4 PCSetUp() line 968 in >> /Users/barrysmith/Src/petsc/src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: #4 PCSetUp() line 968 in >> /Users/barrysmith/Src/petsc/src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: #5 KSPSetUp() line 390 in >> /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: #6 KSPSolve() line 599 in >> /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c >> [1]PETSC ERROR: #5 KSPSetUp() line 390 in >> /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c >> [1]PETSC ERROR: [0]PETSC ERROR: #7 SNESSolve_NEWTONLS() line 230 in >> /Users/barrysmith/Src/petsc/src/snes/impls/ls/ls.c >> [0]PETSC ERROR: #8 SNESSolve() line 4008 in >> /Users/barrysmith/Src/petsc/src/snes/interface/snes.c >> #6 KSPSolve() line 599 in >> /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c >> [1]PETSC ERROR: #7 SNESSolve_NEWTONLS() line 230 in >> /Users/barrysmith/Src/petsc/src/snes/impls/ls/ls.c >> [0]PETSC ERROR: #9 main() line 159 in >> /Users/barrysmith/Src/petsc/src/snes/examples/tutorials/ex19.c >> [0]PETSC ERROR: [1]PETSC ERROR: #8 SNESSolve() line 4008 in >> /Users/barrysmith/Src/petsc/src/snes/interface/snes.c >> [1]PETSC ERROR: PETSc Option Table entries: >> >> >> >

1 0

Known outdated sections of the User's Manual?
by Patrick Sanan 24 Jun '16

24 Jun '16

I'm working on some groundwork to improve the PETSc documentation, and the next thing I'd like to look at is the User's Manual, adding slightly-prettier code listings (as in the update to the dev manual). Before doing that, however, it would be very helpful to know if there are any sections of the manual which are known, by the relevant experts here, to require deletion or heavy rewrites; it would of course be a waste of time to format these. Specifically, I'm wondering about the following sections, which have in common that they are things concerned with friendly external tools. I've seen new users get very frustrated when they expect these sorts of things to "just work," so it's probably constructive to remove any outdated information here: - Chapter 11: Using MATLAB with PETSc . The support here has changed quite a lot, so I'm not sure what currently works. Is the MATLAB Compute Engine still supported? - Sections 15.10-15.14: Eclipse/Qt Creator/Developers Studio/XCode users. This is likely not all current. Is this information helpful here? - Section 15.15 : Graphics. I saw that there were some updates to the drawing tools recently by Lisandro, so if any of this material is known to be out of date, that would be helpful to know.

1 0

Re: [petsc-dev] asm / gasm
by Mark Adams 24 Jun '16

24 Jun '16

On Thu, Jun 23, 2016 at 11:46 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Mark, > > It is not as simple as this to convert to ASM. It will take a little > bit of work to use ASM here instead of GASM. > > Just to be clear: ASM used to work. Did the semantics of ASM change? > But before that please please tell me the command line argument and > example you use where the GASM crashes so I can get that fixed. Then I will > look at using ASM instead after I have the current GASM code running again. > In branch mark/gamg-agg-asm in ksp ex56, 'make runex56': 14:12 nid00495 ~/petsc/src/ksp/ksp/examples/tutorials$ make runex56 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Petsc has generated inconsistent data [0]PETSC ERROR: MPI_Allreduce() called in different locations (code lines) on different processors [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.7.2-633-g4f88208 GIT Date: 2016-06-23 18:53:31 +0200 [0]PETSC ERROR: /global/u2/m/madams/petsc/src/ksp/ksp/examples/tutorials/./ex56 on a arch-xc30-dbg64-intel named nid00495 by madams Thu Jun 23 14:12:57 2016 [0]PETSC ERROR: Configure options --COPTFLAGS="-no-ipo -g -O0" --CXXOPTFLAGS="-no-ipo -g -O0" --FOPTFLAGS="-fast -no-ipo -g -O0" --download-parmetis --download-metis --with-ssl=0 --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-debugging=1 --with-fc=0 --with-shared-libraries=0 --with-x=0 --with-mpiexec=srun LIBS=-lstdc++ --with-64-bit-indices PETSC_ARCH=arch-xc30-dbg64-intel [0]PETSC ERROR: #1 MatGetSubMatrices_MPIAIJ() li > > Barry > > > > On Jun 23, 2016, at 4:19 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > The question boils down to, for empty processors do we: > > > > ierr = ISCreateGeneral(PETSC_COMM_SELF, 0, NULL, PETSC_COPY_VALUES, > &is);CHKERRQ(ierr); > > ierr = PCASMSetLocalSubdomains(subpc, 1, &is, > NULL);CHKERRQ(ierr); > > ierr = ISDestroy(&is);CHKERRQ(ierr); > > > > or > > > > PCASMSetLocalSubdomains(subpc, 0, NULL, NULL); > > > > The later gives and error that one domain is need and the later gives an > error (appended). > > > > I've checked in the code for this second error in ksp (make runex56) > > > > Thanks, > > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Petsc has generated inconsistent data > > [0]PETSC ERROR: MPI_Allreduce() called in different locations (code > lines) on different processors > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Development GIT revision: v3.7.2-633-g4f88208 GIT > Date: 2016-06-23 18:53:31 +0200 > > [0]PETSC ERROR: > /global/u2/m/madams/petsc/src/ksp/ksp/examples/tutorials/./ex56 on a > arch-xc30-dbg64-intel named nid00495 by madams Thu Jun 23 14:12:57 2016 > > [0]PETSC ERROR: Configure options --COPTFLAGS="-no-ipo -g -O0" > --CXXOPTFLAGS="-no-ipo -g -O0" --FOPTFLAGS="-fast -no-ipo -g -O0" > --download-parmetis --download-metis --with-ssl=0 --with-cc=cc > --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 > --with-debugging=1 --with-fc=0 --with-shared-libraries=0 --with-x=0 > --with-mpiexec=srun LIBS=-lstdc++ --with-64-bit-indices > PETSC_ARCH=arch-xc30-dbg64-intel > > [0]PETSC ERROR: #1 MatGetSubMatrices_MPIAIJ() line 1147 in > /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/mpiov.c > > [0]PETSC ERROR: #2 MatGetSubMatrices_MPIAIJ() line 1147 in > /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/mpiov.c > > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > > > On Thu, Jun 23, 2016 at 8:05 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > > Where is the command line that generates the error? > > > > > > > On Jun 23, 2016, at 12:08 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > [adding Garth] > > > > > > On Thu, Jun 23, 2016 at 12:52 AM, Barry Smith <bsmith(a)mcs.anl.gov> > wrote: > > > > > > Mark, > > > > > > I think there is a misunderstanding here. With GASM an individual > block problem is __solved__ (via a parallel KSP) in parallel by several > processes, with ASM each block is "owned" by and solved on a single process. > > > > > > Ah, OK, so this is for multiple processors in a block. Yes, we are > looking at small, smother, blocks. > > > > > > > > > With both the "block" can come from any unknowns on any processes. > You can have, for example a block that comes from a region snaking across > several processes if you like (or it makes sense due to coupling in the > matrix). > > > > > > By default if you use ASM it will create one non-overlapping block > defined by all unknowns owned by a single process and then extend it by > "one level" (defined by the nonzero structure of the matrix) to get > overlapping. > > > > > > The default in ASM is one level of overlap? That is new. (OK, I have > not looked at ASM in like over 10 years) > > > > > > If you use multiple blocks per process it defines the non-overlapping > blocks within a single process's unknowns > > > > > > I assume this still chops the matrix and does not call a partitioner. > > > > > > and extends each of them to have overlap (again by the non-zero > structure of the matrix). The default is simple because the user only need > indicate the number of blocks per process, the drawback is of course that > it does depend on the process layout, number of processes etc and does not > take into account particular "coupling information" that the user may know > about with their problem. > > > > > > If the user wishes to defined the blocks themselves that is also > possible with PCASMSetSubLocalSubdomains(). Each process provides 1 or more > index sets for the subdomains it will solve on. Note that the index sets > can contain any unknowns in the entire problem so the blocks do not have to > "line up" with the parallel decomposition at all. > > > > > > Oh, OK, this is what I want. (I thought this worked). > > > > > > Of course determining and providing good such subdomains may not > always be clear. > > > > > > In smoothed aggregation there is an argument that the aggregates are > good, but the scale is fixed obviously. On a regular grid smoothed > aggregation wants 3^D sized aggregates, which is obviously wonderful for > AMS. And for anisotropy you want your ASM blocks to be on strongly > connected components, which is what smoothed aggregation wants (not that I > do this very well). > > > > > > > > > I see in GAMG you have PCGAMGSetUseASMAggs > > > > > > But the code calls PCGASMSetSubdomains and the command line is > -pc_gamg_use_agg_gasm, so this is all messed up. (more below) > > > > > > which sadly does not have an explanation in the users manual and sadly > does not have a matching options data base name -pc_gamg_use_agg_gasm > following the rule of drop the word set, all lower case, and put _ between > words the option should be -pc_gamg_use_asm_aggs. > > > > > > BUT, THIS IS THE WAY IT WAS! It looks like someone hijacked this code > and made it gasm. I never did this. > > > > > > Barry: you did this apparently in 2013. > > > > > > > > > In addition to this one you could also have one that uses the aggs > but use the PCASM to manage the solves instead of GASM, it would likely be > less buggy and more efficient. > > > > > > yes > > > > > > > > > Please tell me exactly what example you tried to run with what > options and I will debug it. > > > > > > We got an error message: > > > > > > ** Max-trans not allowed because matrix is distributed > > > > > > Garth: is this from your code perhaps? I don't see it in PETSc. > > > > > > Note that ALL functionality that is included in PETSc should have > tests that test that functionality then we will find out immediately when > it is broken instead of two years later when it is much harder to debug. If > this -pc_gamg_use_agg_gasm had had a test we won't be in this mess now. > (Jed's damn code reviews sure don't pick up this stuff). > > > > > > First we need to change gasm to asm. > > > > > > We could add this argument pc_gamg_use_agg_asm to ksp/ex56 (runex56 > or make a new test). The SNES version (also ex56) is my current test that > I like to refer to as recommended parameters for elasticity. So I'd like to > keep that clean, but we can add junk to ksp/ex56. > > > > > > I've done this in a branch mark/gamg-agg-asm. I get an error > (appended). It looks like the second coarsest grid, which has 36 dof on one > processor has an index 36 in the block on every processor. Strange. I can > take a look at it later. > > > > > > Mark > > > > > > > [3]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > > > [4]PETSC ERROR: Petsc has generated inconsistent data > > > > [4]PETSC ERROR: ith 0 block entry 36 not owned by any process, upper > bound 36 > > > > [4]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > > > [4]PETSC ERROR: Petsc Development GIT revision: v3.7.2-630-g96e0c40 > GIT Date: 2016-06-22 10:03:02 -0500 > > > > [4]PETSC ERROR: ./ex56 on a arch-macosx-gnu-g named MarksMac-3.local > by markadams Thu Jun 23 06:53:27 2016 > > > > [4]PETSC ERROR: Configure options COPTFLAGS="-g -O0" CXXOPTFLAGS="-g > -O0" FOPTFLAGS="-g -O0" --download-hypre=1 --download-parmetis=1 > --download-metis=1 --download-ml=1 --download-p4est=1 --download-exodus=1 > --download-triangle=1 --with-hdf5-dir=/Users/markadams/Codes/hdf5 > --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g --download-chaco > > > > [4]PETSC ERROR: #1 VecScatterCreate_PtoS() line 2348 in > /Users/markadams/Codes/petsc/src/vec/vec/utils/vpscat.c > > > > [4]PETSC ERROR: #2 VecScatterCreate() line 1552 in > /Users/markadams/Codes/petsc/src/vec/vec/utils/vscat.c > > > > [4]PETSC ERROR: Petsc has generated inconsistent data > > > > [3]PETSC ERROR: ith 0 block entry 36 not owned by any process, upper > bound 36 > > > > [3]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > > > [3]PETSC ERROR: Petsc Development GIT revision: v3.7.2-630-g96e0c40 > GIT Date: 2016-06-22 10:03:02 -0500 > > > > [3]PETSC ERROR: ./ex56 on a arch-macosx-gnu-g named MarksMac-3.local > by markadams Thu Jun 23 06:53:27 2016 > > > > [3]PETSC ERROR: Configure options COPTFLAGS="-g -O0" CXXOPTFLAGS="-g > -O0" FOPTFLAGS="-g -O0" --download-hypre=1 --download-parmetis=1 > --download-metis=1 --download-ml=1 --download-p4est=1 --download-exodus=1 > --download-triangle=1 --with-hdf5-dir=/Users/markadams/Codes/hdf5 > --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g --download-chaco > > > > [3]PETSC ERROR: #1 VecScatterCreate_PtoS() line 2348 in > /Users/markadams/Codes/petsc/src/vec/vec/utils/vpscat.c > > > > [3]PETSC ERROR: #2 VecScatterCreate() line 1552 in > /Users/markadams/Codes/petsc/src/vec/vec/utils/vscat.c > > > > [3]PETSC ERROR: #3 PCSetUp_ASM() line 279 in > /Users/markadams/Codes/petsc/src/ksp/pc/impls/asm/asm.c > > > > > > > > > > > > > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > > > On Jun 22, 2016, at 5:20 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > > > > > > > > > > > On Wed, Jun 22, 2016 at 8:06 PM, Barry Smith <bsmith(a)mcs.anl.gov> > wrote: > > > > > > > > I suggest focusing on asm. > > > > > > > > OK, I will switch gasm to asm, this does not work anyway. > > > > > > > > Having blocks that span multiple processes seems like over kill for > a smoother ? > > > > > > > > No, because it is a pain to have the math convolved with the > parallel decompositions strategy (ie, I can't tell an application how to > partition their problem). If an aggregate spans processor boundaries, which > is fine and needed, and let's say we have a pretty uniform problem, then if > the block gets split up, H is small in part of the domain and convergence > could suffer along processor boundaries. And having the math change as the > parallel decomposition changes is annoying. > > > > > > > > (Major league overkill) in fact doesn't one want multiple blocks per > process, ie. pretty small blocks. > > > > > > > > No, it is just doing what would be done in serial. If the cost of > moving the data across the processor is a problem then that is a tradeoff > to consider. > > > > > > > > And I think you are misunderstanding me. There are lots of blocks > per process (the aggregates are say 3^D in size). And many of the > aggregates/blocks along the processor boundary will be split between > processors, resulting is mall blocks and weak ASM PC on processor > boundaries. > > > > > > > > I can understand ASM not being general and not letting blocks span > processor boundaries, but I don't think the extra matrix communication > costs are a big deal (done just once) and the vector communication costs > are not bad, it probably does not include (too many) new processors to > communicate with. > > > > > > > > > > > > Barry > > > > > > > > > On Jun 22, 2016, at 7:51 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > > > > > I'm trying to get block smoothers to work for gamg. We (Garth) > tried this and got this error: > > > > > > > > > > > > > > > - Another option is use '-pc_gamg_use_agg_gasm true' and use > '-mg_levels_pc_type gasm'. > > > > > > > > > > > > > > > Running in parallel, I get > > > > > > > > > > ** Max-trans not allowed because matrix is distributed > > > > > ---- > > > > > > > > > > First, what is the difference between asm and gasm? > > > > > > > > > > Second, I need to fix this to get block smoothers. This used to > work. Did we lose the capability to have blocks that span processor > subdomains? > > > > > > > > > > gamg only aggregates across processor subdomains within one layer, > so maybe I could use one layer of overlap in some way? > > > > > > > > > > Thanks, > > > > > Mark > > > > > > > > > > > > > > > > > > > > > > > > >

1 0

Re: [petsc-dev] MatMFFDSetType
by Adrian Croucher 24 Jun '16

24 Jun '16

Further to this, I debugged into PetscObjectTypeCompare(). It's returning PETSC_FALSE because the type name of the variable mat is "seqbaij" (and I guess would normally be something like "mpibaij" if running in parallel outside gdb), which doesn't match with MATMFFD (= "mffd"). Does that mean I should be setting the matrix type to MATMFFD to get this to work? - Adrian On 24/06/16 16:35, Adrian Croucher wrote: > On 24/06/16 15:58, Adrian Croucher wrote: >> >> I will try debugging into MatMFFDSetType() and seeing what it's doing >> when I pass in "ds". >> > > I did this. In MatMFFDSetType() (mffd.c: 116), it executes > > ierr = > PetscObjectTypeCompare((PetscObject)mat,MATMFFD,&match);CHKERRQ(ierr); > > which returns match = PETSC_FALSE. The next line > > if (!match) PetscFunctionReturn(0); > > means it just returns out of there without doing anything. > > - Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher(a)auckland.ac.nz tel: +64 (0)9 923 84611

1 0

Re: [petsc-dev] MatMFFDSetType
by Adrian Croucher 23 Jun '16

23 Jun '16

On 24/06/16 15:58, Adrian Croucher wrote: > > I will try debugging into MatMFFDSetType() and seeing what it's doing > when I pass in "ds". > I did this. In MatMFFDSetType() (mffd.c: 116), it executes ierr = PetscObjectTypeCompare((PetscObject)mat,MATMFFD,&match);CHKERRQ(ierr); which returns match = PETSC_FALSE. The next line if (!match) PetscFunctionReturn(0); means it just returns out of there without doing anything. - Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher(a)auckland.ac.nz tel: +64 (0)9 923 84611

1 0

Re: [petsc-dev] MatMFFDSetType
by Adrian Croucher 23 Jun '16

23 Jun '16

hi, On 24/06/16 15:24, Barry Smith wrote: > If you can send me a simple Fortran program where it ignores the values set in the code I can track down why it isn't working for you, but in all my tests it won't ignore the value in the code. The code I'm using isn't simple, unfortunately. I just tried debugging down into MatFDColoringApply_BAIJ() and checking coloring->htype, and it is definitely still wp, even if I've previously called MatMFFDSetType() with argument "ds". Would it be worth trying to fix the problem preventing me from calling MatMFFDSetType() with argument MATMFFD_DS instead of "ds"? I will try debugging into MatMFFDSetType() and seeing what it's doing when I pass in "ds". - Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher(a)auckland.ac.nz tel: +64 (0)9 923 84611

1 0

Re: [petsc-dev] MatMFFDSetType
by Barry Smith 23 Jun '16

23 Jun '16

Adrian, If you can send me a simple Fortran program where it ignores the values set in the code I can track down why it isn't working for you, but in all my tests it won't ignore the value in the code. Barry > On Jun 23, 2016, at 9:57 PM, Adrian Croucher <a.croucher(a)auckland.ac.nz> wrote: > > > > On 24/06/16 14:56, Barry Smith wrote: >> But using -mat_fd_type ds vs -mat_fd_type wp does produce (slightly) different convergence histories? > Yes, in general ds works better than wp for the problem I'm looking at. That's why I wanted to set it as default in the code. > > - Adrian > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > University of Auckland, New Zealand > email: > a.croucher(a)auckland.ac.nz > > tel: +64 (0)9 923 84611 >

1 0