June 2016 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Problem with GAMG
by Mark Adams 23 Jun '16

23 Jun '16

On Wed, Jun 22, 2016 at 11:27 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Mark, > > If I use the hem coarsen type I get this error. Does this go away with mis? Don't bother testing, it probably does. > Note that I am setting the -pc_gamg_sym_graph true flag I the debugger I > do see that PCGAMGFilterGraph() has the > sym flag set so it seems to me either > > 1) the symmetrizer doesn't actually make it symmetric or > It should be symmetric, even with roundoff, I think. You have an unsymmetric matrix structurally? > 2) the detector of lack of symmetry thinks it is not symmetric when it is. > The edge weights are used to drop edges and if there is an error then you can get a structurally unsymmetric graph. Then the parallel graph algorithms barf (they assume if I know you, you know me, symmetric). > > Thoughts on how to debug this? > Try with a threshold of 0. I need to take a look at it. Can I run it? I have to run for the night (in Vienna) > > Thanks > > barry > > > > $ petscmpiexec -n 2 ./ex19 -pc_type gamg -da_refine 3 -mat_coarsen_type > hem -pc_gamg_sym_graph true > lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: Petsc has generated inconsistent data > [1]PETSC ERROR: Have un-symmetric graph (apparently). Use > '-pc_gamg_sym_graph true' to symetrize the graph or '-pc_gamg_threshold > -1.0' if the matrix is structurally symmetric. > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Petsc has generated inconsistent data > [0]PETSC ERROR: Have un-symmetric graph (apparently). Use > '-pc_gamg_sym_graph true' to symetrize the graph or '-pc_gamg_threshold > -1.0' if the matrix is structurally symmetric. > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.7.2-596-g05491f7 GIT > Date: 2016-06-22 13:31:47 -0500 > [0]PETSC ERROR: ./ex19 on a arch-single named Barrys-MacBook-Pro.local by > barrysmith Wed Jun 22 16:24:13 2016 > [0]PETSC ERROR: [1]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [1]PETSC ERROR: Petsc Development GIT revision: v3.7.2-596-g05491f7 GIT > Date: 2016-06-22 13:31:47 -0500 > [1]PETSC ERROR: ./ex19 on a arch-single named Barrys-MacBook-Pro.local by > barrysmith Wed Jun 22 16:24:13 2016 > [1]PETSC ERROR: Configure options > --with-mpi-dir=/Users/barrysmith/libraries --with-precision=single > [1]PETSC ERROR: #1 smoothAggs() line 463 in > /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gamg/agg.c > Configure options --with-mpi-dir=/Users/barrysmith/libraries > --with-precision=single > [0]PETSC ERROR: #1 smoothAggs() line 463 in > /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gamg/agg.c > [1]PETSC ERROR: #2 PCGAMGCoarsen_AGG() line 1021 in > /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gamg/agg.c > [1]PETSC ERROR: [0]PETSC ERROR: #2 PCGAMGCoarsen_AGG() line 1021 in > /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gamg/agg.c > [0]PETSC ERROR: #3 PCSetUp_GAMG() line 524 in > /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gamg/gamg.c > #3 PCSetUp_GAMG() line 524 in > /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gamg/gamg.c > [1]PETSC ERROR: #4 PCSetUp() line 968 in > /Users/barrysmith/Src/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #4 PCSetUp() line 968 in > /Users/barrysmith/Src/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #5 KSPSetUp() line 390 in > /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #6 KSPSolve() line 599 in > /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: #5 KSPSetUp() line 390 in > /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: [0]PETSC ERROR: #7 SNESSolve_NEWTONLS() line 230 in > /Users/barrysmith/Src/petsc/src/snes/impls/ls/ls.c > [0]PETSC ERROR: #8 SNESSolve() line 4008 in > /Users/barrysmith/Src/petsc/src/snes/interface/snes.c > #6 KSPSolve() line 599 in > /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: #7 SNESSolve_NEWTONLS() line 230 in > /Users/barrysmith/Src/petsc/src/snes/impls/ls/ls.c > [0]PETSC ERROR: #9 main() line 159 in > /Users/barrysmith/Src/petsc/src/snes/examples/tutorials/ex19.c > [0]PETSC ERROR: [1]PETSC ERROR: #8 SNESSolve() line 4008 in > /Users/barrysmith/Src/petsc/src/snes/interface/snes.c > [1]PETSC ERROR: PETSc Option Table entries: > > >

1 0

Re: [petsc-dev] Bug in MatZeroRows_MPIAIJ
by Barry Smith 23 Jun '16

23 Jun '16

More concerned about the complexity of the code. Say someone stumbles on this code in 8 years and starts monkeying with it without fully understanding it. Why not just have the simpler code without this special case, would it be that much slower? Barry > On Jun 23, 2016, at 7:46 AM, Stefano Zampini <stefano.zampini(a)gmail.com> wrote: > > If you are worried about doing memcmp on the ranges, it should be in the order of microseconds. few milliseconds with millions of processes. > On Lisandro workstation, it took 600 microseconds comparing two equal sets of a million of integers. > Since the square case with same row and col layout covers the 90% of the cases (and memcmp need to sweep the entire ranges), > we can always have an int flag for MPIAIJ and its variants which is set to -1 at creation and set up properly the first time MatZeroRow_MPIAIJ is called. > > On Jun 22, 2016, at 8:08 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > >> >>> On Jun 22, 2016, at 1:24 AM, Lisandro Dalcin <dalcinl(a)gmail.com> wrote: >>> >>> On 22 June 2016 at 00:30, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >>>> Why not just remove the first if test (below) and just have the two conditions diag != 0.0 and drag == 0.0 >>>> >>>> if ((diag != 0.0) && (mat->A->rmap->N == mat->A->cmap->N)) { >>>> ierr = MatZeroRows(mat->A, len, lrows, diag, NULL, NULL);CHKERRQ(ierr); >>> >>> What about comparing mat->rmap->N and mat->cmap->N, then compare >>> size(mat->rmap->comm) and size(mat->cmap->comm), and finally compare >>> (memcmp) mat->rmap-range and mat->rmap-range? If everything matches, >>> the row&col layout is square, you know that in all processes so no >>> need to do reductions, and then you can take the fast path zeroing >>> rows on the diagonal block mat->A. >> >> By then it is worth the optimization? >> >>> >>> >>> -- >>> Lisandro Dalcin >>> ============ >>> Research Scientist >>> Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) >>> Extreme Computing Research Center (ECRC) >>> King Abdullah University of Science and Technology (KAUST) >>> http://ecrc.kaust.edu.sa/ >>> >>> 4700 King Abdullah University of Science and Technology >>> al-Khawarizmi Bldg (Bldg 1), Office # 0109 >>> Thuwal 23955-6900, Kingdom of Saudi Arabia >>> http://www.kaust.edu.sa >>> >>> Office Phone: +966 12 808-0459 >> >

1 0

Re: [petsc-dev] DMGetOutputDM hangs on multiple procs
by Matthew Knepley 23 Jun '16

23 Jun '16

On Thu, Jun 23, 2016 at 5:46 AM, Sander Arens <Sander.Arens(a)ugent.be> wrote: > I noticed that DMGetOutputDM hangs when using multiple procs. The culprit > is > https://bitbucket.org/petsc/petsc/commits/c26acbdef0eedda92db6423afb8b38a15…. > > This doesn't work when one or more procs don't have any constraints while > others have, because the ones with constraints call DMClone (collective) > while the other procs don't. > Yep, will have to make the constraint logic global. I can do this at the PETSc Meeting next week in Vienna. Thanks, Matt > Thanks, > Sander > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

DMGetOutputDM hangs on multiple procs
by Sander Arens 23 Jun '16

23 Jun '16

I noticed that DMGetOutputDM hangs when using multiple procs. The culprit is https://bitbucket.org/petsc/petsc/commits/c26acbdef0eedda92db6423afb8b38a15…. This doesn't work when one or more procs don't have any constraints while others have, because the ones with constraints call DMClone (collective) while the other procs don't. Thanks, Sander

1 0

Re: [petsc-dev] Bug in MatZeroRows_MPIAIJ
by Stefano Zampini 23 Jun '16

23 Jun '16

If you are worried about doing memcmp on the ranges, it should be in the order of microseconds. few milliseconds with millions of processes. On Lisandro workstation, it took 600 microseconds comparing two equal sets of a million of integers. Since the square case with same row and col layout covers the 90% of the cases (and memcmp need to sweep the entire ranges), we can always have an int flag for MPIAIJ and its variants which is set to -1 at creation and set up properly the first time MatZeroRow_MPIAIJ is called. On Jun 22, 2016, at 8:08 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > >> On Jun 22, 2016, at 1:24 AM, Lisandro Dalcin <dalcinl(a)gmail.com> wrote: >> >> On 22 June 2016 at 00:30, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >>> Why not just remove the first if test (below) and just have the two conditions diag != 0.0 and drag == 0.0 >>> >>> if ((diag != 0.0) && (mat->A->rmap->N == mat->A->cmap->N)) { >>> ierr = MatZeroRows(mat->A, len, lrows, diag, NULL, NULL);CHKERRQ(ierr); >> >> What about comparing mat->rmap->N and mat->cmap->N, then compare >> size(mat->rmap->comm) and size(mat->cmap->comm), and finally compare >> (memcmp) mat->rmap-range and mat->rmap-range? If everything matches, >> the row&col layout is square, you know that in all processes so no >> need to do reductions, and then you can take the fast path zeroing >> rows on the diagonal block mat->A. > > By then it is worth the optimization? > >> >> >> -- >> Lisandro Dalcin >> ============ >> Research Scientist >> Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) >> Extreme Computing Research Center (ECRC) >> King Abdullah University of Science and Technology (KAUST) >> http://ecrc.kaust.edu.sa/ >> >> 4700 King Abdullah University of Science and Technology >> al-Khawarizmi Bldg (Bldg 1), Office # 0109 >> Thuwal 23955-6900, Kingdom of Saudi Arabia >> http://www.kaust.edu.sa >> >> Office Phone: +966 12 808-0459 >

1 0

Re: [petsc-dev] asm / gasm
by Mark Adams 23 Jun '16

23 Jun '16

I'm not sure how to create an empty subdomain in ASM. I get this error in KSPSetup. I think this is empty subdomains. The old code, that worked once, was giving me the errors that I sent before. Now I do this: if (!sz) { IS is; ierr = ISCreateGeneral(PETSC_COMM_SELF, 0, NULL, PETSC_COPY_VALUES, &is);CHKERRQ(ierr); ierr = PCASMSetLocalSubdomains(subpc, 1, &is, NULL);CHKERRQ(ierr); ierr = ISDestroy(&is);CHKERRQ(ierr); } else { I will keep digging. 14:09 mark/gamg-agg-asm *= ~/Codes/petsc/src/ksp/ksp/examples/tutorials$ make PETSC_ARCH=arch-macosx-gnu-O runex56 [2]PETSC ERROR: #1 VecScatterCreate() line 1492 in /Users/markadams/Codes/petsc/src/vec/vec/utils/vscat.c [2]PETSC ERROR: #2 PCSetUp_ASM() line 279 in /Users/markadams/Codes/petsc/src/ksp/pc/impls/asm/asm.c [2]PETSC ERROR: #3 PCSetUp() line 968 in /Users/markadams/Codes/petsc/src/ksp/pc/interface/precon.c [2]PETSC ERROR: #4 KSPSetUp() line 390 in /Users/markadams/Codes/petsc/src/ksp/ksp/interface/itfunc.c [2]PETSC ERROR: #5 PCSetUp_MG() line 783 in /Users/markadams/Codes/petsc/src/ksp/pc/impls/mg/mg.c [2]PETSC ERROR: #6 PCSetUp_GAMG() line 675 in /Users/markadams/Codes/petsc/src/ksp/pc/impls/gamg/gamg.c [2]PETSC ERROR: #7 PCSetUp() line 968 in /Users/markadams/Codes/petsc/src/ksp/pc/interface/precon.c [2]PETSC ERROR: #8 KSPSetUp() line 390 in /Users/markadams/Codes/petsc/src/ksp/ksp/interface/itfunc.c [2]PETSC ERROR: #9 main() line 279 in /Users/markadams/Codes/petsc/src/ksp/ksp/examples/tutorials/ex56.c On Thu, Jun 23, 2016 at 7:08 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > [adding Garth] > > On Thu, Jun 23, 2016 at 12:52 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > >> >> Mark, >> >> I think there is a misunderstanding here. With GASM an individual >> block problem is __solved__ (via a parallel KSP) in parallel by several >> processes, with ASM each block is "owned" by and solved on a single process. >> > > Ah, OK, so this is for multiple processors in a block. Yes, we are looking > at small, smother, blocks. > > >> >> With both the "block" can come from any unknowns on any processes. You >> can have, for example a block that comes from a region snaking across >> several processes if you like (or it makes sense due to coupling in the >> matrix). >> >> By default if you use ASM it will create one non-overlapping block >> defined by all unknowns owned by a single process and then extend it by >> "one level" (defined by the nonzero structure of the matrix) to get >> overlapping. > > > The default in ASM is one level of overlap? That is new. (OK, I have not > looked at ASM in like over 10 years) > > >> If you use multiple blocks per process it defines the non-overlapping >> blocks within a single process's unknowns > > > I assume this still chops the matrix and does not call a partitioner. > > >> and extends each of them to have overlap (again by the non-zero structure >> of the matrix). The default is simple because the user only need indicate >> the number of blocks per process, the drawback is of course that it does >> depend on the process layout, number of processes etc and does not take >> into account particular "coupling information" that the user may know about >> with their problem. >> >> If the user wishes to defined the blocks themselves that is also >> possible with PCASMSetSubLocalSubdomains(). Each process provides 1 or more >> index sets for the subdomains it will solve on. Note that the index sets >> can contain any unknowns in the entire problem so the blocks do not have to >> "line up" with the parallel decomposition at all. > > > Oh, OK, this is what I want. (I thought this worked). > > >> Of course determining and providing good such subdomains may not always >> be clear. >> > > In smoothed aggregation there is an argument that the aggregates are good, > but the scale is fixed obviously. On a regular grid smoothed aggregation > wants 3^D sized aggregates, which is obviously wonderful for AMS. And for > anisotropy you want your ASM blocks to be on strongly connected components, > which is what smoothed aggregation wants (not that I do this very well). > > >> >> I see in GAMG you have PCGAMGSetUseASMAggs > > > But the code calls PCGASMSetSubdomains and the command line > is -pc_gamg_use_agg_gasm, so this is all messed up. (more below) > > >> which sadly does not have an explanation in the users manual and sadly >> does not have a matching options data base name -pc_gamg_use_agg_gasm >> following the rule of drop the word set, all lower case, and put _ between >> words the option should be -pc_gamg_use_asm_aggs. >> > > BUT, THIS IS THE WAY IT WAS! It looks like someone hijacked this code and > made it gasm. I never did this. > > Barry: you did this apparently in 2013. > > >> In addition to this one you could also have one that uses the aggs but >> use the PCASM to manage the solves instead of GASM, it would likely be less >> buggy and more efficient. > > > yes > > >> >> Please tell me exactly what example you tried to run with what options >> and I will debug it. > > > We got an error message: > > ** Max-trans not allowed because matrix is distributed > > > Garth: is this from your code perhaps? I don't see it in PETSc. > > >> Note that ALL functionality that is included in PETSc should have tests >> that test that functionality then we will find out immediately when it is >> broken instead of two years later when it is much harder to debug. If this >> -pc_gamg_use_agg_gasm had had a test we won't be in this mess now. (Jed's >> damn code reviews sure don't pick up this stuff). >> > > First we need to change gasm to asm. > > We could add this argument pc_gamg_use_agg_asm to ksp/ex56 (runex56 or > make a new test). The SNES version (also ex56) is my current test that I > like to refer to as recommended parameters for elasticity. So I'd like to > keep that clean, but we can add junk to ksp/ex56. > > I've done this in a branch mark/gamg-agg-asm. I get an error (appended). > It looks like the second coarsest grid, which has 36 dof on one processor > has an index 36 in the block on every processor. Strange. I can take a > look at it later. > > Mark > > > [3]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [4]PETSC ERROR: Petsc has generated inconsistent data > > [4]PETSC ERROR: ith 0 block entry 36 not owned by any process, upper > bound 36 > > [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [4]PETSC ERROR: Petsc Development GIT revision: v3.7.2-630-g96e0c40 GIT > Date: 2016-06-22 10:03:02 -0500 > > [4]PETSC ERROR: ./ex56 on a arch-macosx-gnu-g named MarksMac-3.local by > markadams Thu Jun 23 06:53:27 2016 > > [4]PETSC ERROR: Configure options COPTFLAGS="-g -O0" CXXOPTFLAGS="-g > -O0" FOPTFLAGS="-g -O0" --download-hypre=1 --download-parmetis=1 > --download-metis=1 --download-ml=1 --download-p4est=1 --download-exodus=1 > --download-triangle=1 --with-hdf5-dir=/Users/markadams/Codes/hdf5 > --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g --download-chaco > > [4]PETSC ERROR: #1 VecScatterCreate_PtoS() line 2348 in > /Users/markadams/Codes/petsc/src/vec/vec/utils/vpscat.c > > [4]PETSC ERROR: #2 VecScatterCreate() line 1552 in > /Users/markadams/Codes/petsc/src/vec/vec/utils/vscat.c > > [4]PETSC ERROR: Petsc has generated inconsistent data > > [3]PETSC ERROR: ith 0 block entry 36 not owned by any process, upper > bound 36 > > [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [3]PETSC ERROR: Petsc Development GIT revision: v3.7.2-630-g96e0c40 GIT > Date: 2016-06-22 10:03:02 -0500 > > [3]PETSC ERROR: ./ex56 on a arch-macosx-gnu-g named MarksMac-3.local by > markadams Thu Jun 23 06:53:27 2016 > > [3]PETSC ERROR: Configure options COPTFLAGS="-g -O0" CXXOPTFLAGS="-g > -O0" FOPTFLAGS="-g -O0" --download-hypre=1 --download-parmetis=1 > --download-metis=1 --download-ml=1 --download-p4est=1 --download-exodus=1 > --download-triangle=1 --with-hdf5-dir=/Users/markadams/Codes/hdf5 > --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g --download-chaco > > [3]PETSC ERROR: #1 VecScatterCreate_PtoS() line 2348 in > /Users/markadams/Codes/petsc/src/vec/vec/utils/vpscat.c > > [3]PETSC ERROR: #2 VecScatterCreate() line 1552 in > /Users/markadams/Codes/petsc/src/vec/vec/utils/vscat.c > > [3]PETSC ERROR: #3 PCSetUp_ASM() line 279 in > /Users/markadams/Codes/petsc/src/ksp/pc/impls/asm/asm.c > > > > > > > >> >> Barry >> >> >> >> >> >> >> > On Jun 22, 2016, at 5:20 PM, Mark Adams <mfadams(a)lbl.gov> wrote: >> > >> > >> > >> > On Wed, Jun 22, 2016 at 8:06 PM, Barry Smith <bsmith(a)mcs.anl.gov> >> wrote: >> > >> > I suggest focusing on asm. >> > >> > OK, I will switch gasm to asm, this does not work anyway. >> > >> > Having blocks that span multiple processes seems like over kill for a >> smoother ? >> > >> > No, because it is a pain to have the math convolved with the parallel >> decompositions strategy (ie, I can't tell an application how to partition >> their problem). If an aggregate spans processor boundaries, which is fine >> and needed, and let's say we have a pretty uniform problem, then if the >> block gets split up, H is small in part of the domain and convergence could >> suffer along processor boundaries. And having the math change as the >> parallel decomposition changes is annoying. >> > >> > (Major league overkill) in fact doesn't one want multiple blocks per >> process, ie. pretty small blocks. >> > >> > No, it is just doing what would be done in serial. If the cost of >> moving the data across the processor is a problem then that is a tradeoff >> to consider. >> > >> > And I think you are misunderstanding me. There are lots of blocks per >> process (the aggregates are say 3^D in size). And many of the >> aggregates/blocks along the processor boundary will be split between >> processors, resulting is mall blocks and weak ASM PC on processor >> boundaries. >> > >> > I can understand ASM not being general and not letting blocks span >> processor boundaries, but I don't think the extra matrix communication >> costs are a big deal (done just once) and the vector communication costs >> are not bad, it probably does not include (too many) new processors to >> communicate with. >> > >> > >> > Barry >> > >> > > On Jun 22, 2016, at 7:51 AM, Mark Adams <mfadams(a)lbl.gov> wrote: >> > > >> > > I'm trying to get block smoothers to work for gamg. We (Garth) tried >> this and got this error: >> > > >> > > >> > > - Another option is use '-pc_gamg_use_agg_gasm true' and use >> '-mg_levels_pc_type gasm'. >> > > >> > > >> > > Running in parallel, I get >> > > >> > > ** Max-trans not allowed because matrix is distributed >> > > ---- >> > > >> > > First, what is the difference between asm and gasm? >> > > >> > > Second, I need to fix this to get block smoothers. This used to >> work. Did we lose the capability to have blocks that span processor >> subdomains? >> > > >> > > gamg only aggregates across processor subdomains within one layer, so >> maybe I could use one layer of overlap in some way? >> > > >> > > Thanks, >> > > Mark >> > > >> > >> > >> >> >

1 0

Re: [petsc-dev] NEVER put // into PETSc code. PETSc is C89, the only real C.
by Jeff Hammond 23 Jun '16

23 Jun '16

On Thu, Jun 23, 2016 at 6:07 AM, Jed Brown <jed(a)jedbrown.org> wrote: > Barry Smith <bsmith(a)mcs.anl.gov> writes: > > > Hmm, both of these links refer to C99 Standard Library they do not > refer to the C99 standard language. Anywhere that says the language > standard? > > To my knowledge, variable length arrays (which are actually quite useful > for numerical work, particularly as VLA-pointers) are not supported by > MSVC and never will be (which is why they were made optional in C11). > Anyway, that alone means that the language implementation is not > complete. But as far as I know, // comments and for-loop declarations, > for example, should not present a portability issue. > One wonders if it is possible for the PETSc style guide / programmer's manual to permit the subset of C99 that is truly portable at this point in time. Then, in like 2028, PETSc can move to C11, which has useful features like atomics in the language (which are useful not just for threaded programs, but also MPI+shared-memory, which I know the PETSc team likes). I too am a huge fan of VLAs, but VLA for non-POD types are "hard", which is why C++ doesn't support them. Jeff -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [petsc-dev] asm / gasm
by Mark Adams 23 Jun '16

23 Jun '16

[adding Garth] On Thu, Jun 23, 2016 at 12:52 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Mark, > > I think there is a misunderstanding here. With GASM an individual block > problem is __solved__ (via a parallel KSP) in parallel by several > processes, with ASM each block is "owned" by and solved on a single process. > Ah, OK, so this is for multiple processors in a block. Yes, we are looking at small, smother, blocks. > > With both the "block" can come from any unknowns on any processes. You > can have, for example a block that comes from a region snaking across > several processes if you like (or it makes sense due to coupling in the > matrix). > > By default if you use ASM it will create one non-overlapping block > defined by all unknowns owned by a single process and then extend it by > "one level" (defined by the nonzero structure of the matrix) to get > overlapping. The default in ASM is one level of overlap? That is new. (OK, I have not looked at ASM in like over 10 years) > If you use multiple blocks per process it defines the non-overlapping > blocks within a single process's unknowns I assume this still chops the matrix and does not call a partitioner. > and extends each of them to have overlap (again by the non-zero structure > of the matrix). The default is simple because the user only need indicate > the number of blocks per process, the drawback is of course that it does > depend on the process layout, number of processes etc and does not take > into account particular "coupling information" that the user may know about > with their problem. > > If the user wishes to defined the blocks themselves that is also > possible with PCASMSetSubLocalSubdomains(). Each process provides 1 or more > index sets for the subdomains it will solve on. Note that the index sets > can contain any unknowns in the entire problem so the blocks do not have to > "line up" with the parallel decomposition at all. Oh, OK, this is what I want. (I thought this worked). > Of course determining and providing good such subdomains may not always be > clear. > In smoothed aggregation there is an argument that the aggregates are good, but the scale is fixed obviously. On a regular grid smoothed aggregation wants 3^D sized aggregates, which is obviously wonderful for AMS. And for anisotropy you want your ASM blocks to be on strongly connected components, which is what smoothed aggregation wants (not that I do this very well). > > I see in GAMG you have PCGAMGSetUseASMAggs But the code calls PCGASMSetSubdomains and the command line is -pc_gamg_use_agg_gasm, so this is all messed up. (more below) > which sadly does not have an explanation in the users manual and sadly > does not have a matching options data base name -pc_gamg_use_agg_gasm > following the rule of drop the word set, all lower case, and put _ between > words the option should be -pc_gamg_use_asm_aggs. > BUT, THIS IS THE WAY IT WAS! It looks like someone hijacked this code and made it gasm. I never did this. Barry: you did this apparently in 2013. > In addition to this one you could also have one that uses the aggs but > use the PCASM to manage the solves instead of GASM, it would likely be less > buggy and more efficient. yes > > Please tell me exactly what example you tried to run with what options > and I will debug it. We got an error message: ** Max-trans not allowed because matrix is distributed Garth: is this from your code perhaps? I don't see it in PETSc. > Note that ALL functionality that is included in PETSc should have tests > that test that functionality then we will find out immediately when it is > broken instead of two years later when it is much harder to debug. If this > -pc_gamg_use_agg_gasm had had a test we won't be in this mess now. (Jed's > damn code reviews sure don't pick up this stuff). > First we need to change gasm to asm. We could add this argument pc_gamg_use_agg_asm to ksp/ex56 (runex56 or make a new test). The SNES version (also ex56) is my current test that I like to refer to as recommended parameters for elasticity. So I'd like to keep that clean, but we can add junk to ksp/ex56. I've done this in a branch mark/gamg-agg-asm. I get an error (appended). It looks like the second coarsest grid, which has 36 dof on one processor has an index 36 in the block on every processor. Strange. I can take a look at it later. Mark > [3]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [4]PETSC ERROR: Petsc has generated inconsistent data > [4]PETSC ERROR: ith 0 block entry 36 not owned by any process, upper bound 36 > [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [4]PETSC ERROR: Petsc Development GIT revision: v3.7.2-630-g96e0c40 GIT Date: 2016-06-22 10:03:02 -0500 > [4]PETSC ERROR: ./ex56 on a arch-macosx-gnu-g named MarksMac-3.local by markadams Thu Jun 23 06:53:27 2016 > [4]PETSC ERROR: Configure options COPTFLAGS="-g -O0" CXXOPTFLAGS="-g -O0" FOPTFLAGS="-g -O0" --download-hypre=1 --download-parmetis=1 --download-metis=1 --download-ml=1 --download-p4est=1 --download-exodus=1 --download-triangle=1 --with-hdf5-dir=/Users/markadams/Codes/hdf5 --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g --download-chaco > [4]PETSC ERROR: #1 VecScatterCreate_PtoS() line 2348 in /Users/markadams/Codes/petsc/src/vec/vec/utils/vpscat.c > [4]PETSC ERROR: #2 VecScatterCreate() line 1552 in /Users/markadams/Codes/petsc/src/vec/vec/utils/vscat.c > [4]PETSC ERROR: Petsc has generated inconsistent data > [3]PETSC ERROR: ith 0 block entry 36 not owned by any process, upper bound 36 > [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [3]PETSC ERROR: Petsc Development GIT revision: v3.7.2-630-g96e0c40 GIT Date: 2016-06-22 10:03:02 -0500 > [3]PETSC ERROR: ./ex56 on a arch-macosx-gnu-g named MarksMac-3.local by markadams Thu Jun 23 06:53:27 2016 > [3]PETSC ERROR: Configure options COPTFLAGS="-g -O0" CXXOPTFLAGS="-g -O0" FOPTFLAGS="-g -O0" --download-hypre=1 --download-parmetis=1 --download-metis=1 --download-ml=1 --download-p4est=1 --download-exodus=1 --download-triangle=1 --with-hdf5-dir=/Users/markadams/Codes/hdf5 --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g --download-chaco > [3]PETSC ERROR: #1 VecScatterCreate_PtoS() line 2348 in /Users/markadams/Codes/petsc/src/vec/vec/utils/vpscat.c > [3]PETSC ERROR: #2 VecScatterCreate() line 1552 in /Users/markadams/Codes/petsc/src/vec/vec/utils/vscat.c > [3]PETSC ERROR: #3 PCSetUp_ASM() line 279 in /Users/markadams/Codes/petsc/src/ksp/pc/impls/asm/asm.c > > Barry > > > > > > > > On Jun 22, 2016, at 5:20 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > > > On Wed, Jun 22, 2016 at 8:06 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > > I suggest focusing on asm. > > > > OK, I will switch gasm to asm, this does not work anyway. > > > > Having blocks that span multiple processes seems like over kill for a > smoother ? > > > > No, because it is a pain to have the math convolved with the parallel > decompositions strategy (ie, I can't tell an application how to partition > their problem). If an aggregate spans processor boundaries, which is fine > and needed, and let's say we have a pretty uniform problem, then if the > block gets split up, H is small in part of the domain and convergence could > suffer along processor boundaries. And having the math change as the > parallel decomposition changes is annoying. > > > > (Major league overkill) in fact doesn't one want multiple blocks per > process, ie. pretty small blocks. > > > > No, it is just doing what would be done in serial. If the cost of > moving the data across the processor is a problem then that is a tradeoff > to consider. > > > > And I think you are misunderstanding me. There are lots of blocks per > process (the aggregates are say 3^D in size). And many of the > aggregates/blocks along the processor boundary will be split between > processors, resulting is mall blocks and weak ASM PC on processor > boundaries. > > > > I can understand ASM not being general and not letting blocks span > processor boundaries, but I don't think the extra matrix communication > costs are a big deal (done just once) and the vector communication costs > are not bad, it probably does not include (too many) new processors to > communicate with. > > > > > > Barry > > > > > On Jun 22, 2016, at 7:51 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > I'm trying to get block smoothers to work for gamg. We (Garth) tried > this and got this error: > > > > > > > > > - Another option is use '-pc_gamg_use_agg_gasm true' and use > '-mg_levels_pc_type gasm'. > > > > > > > > > Running in parallel, I get > > > > > > ** Max-trans not allowed because matrix is distributed > > > ---- > > > > > > First, what is the difference between asm and gasm? > > > > > > Second, I need to fix this to get block smoothers. This used to work. > Did we lose the capability to have blocks that span processor subdomains? > > > > > > gamg only aggregates across processor subdomains within one layer, so > maybe I could use one layer of overlap in some way? > > > > > > Thanks, > > > Mark > > > > > > > > >

1 0

Re: [petsc-dev] NEVER put // into PETSc code. PETSc is C89, the only real C.
by Jed Brown 22 Jun '16

22 Jun '16

Barry Smith <bsmith(a)mcs.anl.gov> writes: > Hmm, both of these links refer to C99 Standard Library they do not refer to the C99 standard language. Anywhere that says the language standard? To my knowledge, variable length arrays (which are actually quite useful for numerical work, particularly as VLA-pointers) are not supported by MSVC and never will be (which is why they were made optional in C11). Anyway, that alone means that the language implementation is not complete. But as far as I know, // comments and for-loop declarations, for example, should not present a portability issue.

1 0

Re: [petsc-dev] asm / gasm
by Barry Smith 22 Jun '16

22 Jun '16

> On Jun 22, 2016, at 10:20 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > >> >> No joke, Amneet found that PCASM with lots of small subdomains was spending a ton of time in view calls. > > (And, of course, Barry fixed it by adding this configure flag --- thanks Barry!) > > Note that it does not matter whether you use view or not. > > Oh wow. So no view with ASM, > > Garth, you will need to configure with '--with-viewfromoptions=0' Not really, first we make sure it runs correctly and only then do we optimize. Please send me the arguments for the crash case with gasm so I can debug it. Thanks Barry > > > -- Boyce > >> >> -- Boyce >> >>> I guess we should have a flag or not iterate over the blocks in ASMView ... >>> >>> >>> -- Boyce >>> >>>> >>>> Barry >>>> >>>>> On Jun 22, 2016, at 7:51 AM, Mark Adams <mfadams(a)lbl.gov> wrote: >>>>> >>>>> I'm trying to get block smoothers to work for gamg. We (Garth) tried this and got this error: >>>>> >>>>> >>>>> - Another option is use '-pc_gamg_use_agg_gasm true' and use '-mg_levels_pc_type gasm'. >>>>> >>>>> >>>>> Running in parallel, I get >>>>> >>>>> ** Max-trans not allowed because matrix is distributed >>>>> ---- >>>>> >>>>> First, what is the difference between asm and gasm? >>>>> >>>>> Second, I need to fix this to get block smoothers. This used to work. Did we lose the capability to have blocks that span processor subdomains? >>>>> >>>>> gamg only aggregates across processor subdomains within one layer, so maybe I could use one layer of overlap in some way? >>>>> >>>>> Thanks, >>>>> Mark >>>>> >>> >>> > >

1 0