June 2016 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] asm / gasm
by Barry Smith 24 Jun '16

24 Jun '16

Fande, Based on my bisection I got the results There are only 'skip'ped commits left to test. The first bad commit could be any of: 08902ef530872fd742dc4ae06e5f55514aa5a6f4 c14adfe1f4858fc4a331628e0999094c88508aa6 25623ae837111235baaf8310854f127ee4f5d849 f771a2744df1b1e04cbde7267f90af2b8dd270ea bac5b06fd7acf2dbdffb3ef782ccfe7a95cc790d da8d96df7d39c0e17cff0758e470753f324e2e80 4bb724134134efafd385bc6b729da2f75e5ded03 930d09c13c6be24c34efcb20ee858969724c0dd9 b9d0fdaa8b8fd04b16ee28e86ef9a83a5d6857d0 We cannot bisect more! ~/Src/petsc ((6a644ab...)|BISECTING) arch-basic Before these commits src/ksp/ksp/examples/tutorials/ex10 ran fine with petscmpiexec -valgrind -n 2 ./ex10 -f0 ~/Datafiles/matrices/poisson2 -ksp_type cg -ksp_monitor_short -ksp_rtol 1.e-8 -pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 100 -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1 -pc_gamg_threshold 0.0 -ksp_converged_reason -use_mat_nearnullspace true -mg_levels_ksp_max_it 2 -mg_levels_ksp_type chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10 -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type sor -mat_block_size 1 -pc_gamg_use_agg_gasm -mg_levels_pc_type gasm -ksp_view after these commits it ran with a valgrind error $ petscmpiexec -valgrind -n 2 ./ex10 -f0 ~/Datafiles/matrices/poisson2 -ksp_type cg -ksp_monitor_short -ksp_rtol 1.e-8 -pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 100 -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1 -pc_gamg_threshold 0.0 -ksp_converged_reason -use_mat_nearnullspace true -mg_levels_ksp_max_it 2 -mg_levels_ksp_type chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10 -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type sor -mat_block_size 1 -pc_gamg_use_agg_gasm -mg_levels_pc_type gasm -ksp_view ==10897== Invalid write of size 8 ==10897== at 0x100E79764: PCSetUp_GASM (gasm.c:432) ==10897== by 0x100F35D3E: PCSetUp (precon.c:984) ==10897== by 0x101024208: KSPSetUp (itfunc.c:332) ==10897== by 0x100F14059: PCSetUp_MG (mg.c:773) ==10897== by 0x100E9C2C1: PCSetUp_GAMG (gamg.c:734) ==10897== by 0x100F35D3E: PCSetUp (precon.c:984) ==10897== by 0x101024208: KSPSetUp (itfunc.c:332) ==10897== by 0x100005C48: main (in ./ex10) ==10897== Address 0x103d8baa8 is 0 bytes after a block of size 3,880 alloc'd ==10897== at 0x100019EBB: malloc (in /usr/local/Cellar/valgrind/3.11.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so) ==10897== by 0x1000D925C: PetscMallocAlign (mal.c:34) ==10897== by 0x1002BA3D2: VecCreate_MPI_Private (pbvec.c:491) ==10897== by 0x1002BAEAC: VecCreate_MPI (pbvec.c:535) ==10897== by 0x1002ECFBE: VecSetType (vecreg.c:53) ==10897== by 0x1002BB2B0: VecCreate_Standard (pbvec.c:562) ==10897== by 0x1002ECFBE: VecSetType (vecreg.c:53) ==10897== by 0x1003F1A2F: MatCreateVecs (matrix.c:8653) ==10897== by 0x100E78DE9: PCSetUp_GASM (gasm.c:396) ==10897== by 0x100F35D3E: PCSetUp (precon.c:984) ==10897== by 0x101024208: KSPSetUp (itfunc.c:332) ==10897== by 0x100F14059: PCSetUp_MG (mg.c:773) ==10897== by 0x100E9C2C1: PCSetUp_GAMG (gamg.c:734) ==10897== by 0x100F35D3E: PCSetUp (precon.c:984) ==10897== by 0x101024208: KSPSetUp (itfunc.c:332) ==10897== by 0x100005C48: main (in ./ex10) ==10897== [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Nonconforming object sizes [1]PETSC ERROR: Vector wrong size 1078034432 for scatter 950 (scatter forward and vector to != ctx to size) [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Development GIT revision: v3.6.1-1135-ge12b472 GIT Date: 2015-09-26 21:43:36 -0600 [1]PETSC ERROR: ./ex10 on a arch-basic named Barrys-MacBook-Pro.local by barrysmith Fri Jun 24 20:44:37 2016 [1]PETSC ERROR: Configure options --with-fc=0 --download-sowing=0 --with-mpi-dir=/Users/barrysmith/libraries [1]PETSC ERROR: #1 VecScatterBegin() line 1689 in /Users/barrysmith/Src/petsc/src/vec/vec/utils/vscat.c [1]PETSC ERROR: #2 PCSetUp_GASM() line 437 in /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gasm/gasm.c [1]PETSC ERROR: #3 PCSetUp() line 984 in /Users/barrysmith/Src/petsc/src/ksp/pc/interface/precon.c [1]PETSC ERROR: #4 KSPSetUp() line 332 in /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: #5 PCSetUp_MG() line 773 in /Users/barrysmith/Src/petsc/src/ksp/pc/impls/mg/mg.c [1]PETSC ERROR: #6 PCSetUp_GAMG() line 734 in /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gamg/gamg.c [1]PETSC ERROR: #7 PCSetUp() line 984 in /Users/barrysmith/Src/petsc/src/ksp/pc/interface/precon.c [1]PETSC ERROR: #8 KSPSetUp() line 332 in /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: #9 main() line 312 in /Users/barrysmith/Src/petsc/src/ksp/ksp/examples/tutorials/ex10.c [1]PETSC ERROR: PETSc Option Table entries: [1]PETSC ERROR: -f0 /Users/barrysmith/Datafiles/matrices/poisson2 [1]PETSC ERROR: -ksp_converged_reason [1]PETSC ERROR: -ksp_monitor_short [1]PETSC ERROR: -ksp_rtol 1.e-8 [1]PETSC ERROR: -ksp_type cg [1]PETSC ERROR: -ksp_view [1]PETSC ERROR: -malloc_test [1]PETSC ERROR: -mat_block_size 1 [1]PETSC ERROR: -mg_levels_esteig_ksp_max_it 10 [1]PETSC ERROR: -mg_levels_esteig_ksp_type cg [1]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 [1]PETSC ERROR: -mg_levels_ksp_max_it 2 [1]PETSC ERROR: -mg_levels_ksp_type chebyshev [1]PETSC ERROR: -mg_levels_pc_type gasm [1]PETSC ERROR: -pc_gamg_agg_nsmooths 1 [1]PETSC ERROR: -pc_gamg_coarse_eq_limit 100 [1]PETSC ERROR: -pc_gamg_reuse_interpolation true [1]PETSC ERROR: -pc_gamg_square_graph 1 [1]PETSC ERROR: -pc_gamg_threshold 0.0 [1]PETSC ERROR: -pc_gamg_type agg [1]PETSC ERROR: -pc_gamg_use_agg_gasm [1]PETSC ERROR: -pc_type gamg [1]PETSC ERROR: -use_mat_nearnullspace true [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- application called MPI_Abort(MPI_COMM_WORLD, 60) - process 1 ~/Src/petsc/src/ksp/ksp/examples/tutorials ((e12b472...)) arch-basic in commit e12b4729b15ad5de2ed17324d40445da9b32e630 so you either did not make the vector you are accessing large enough or you are accessing its values incorrectly. Could you please see if you can determine the bug so we can get GASM back on track. I urge you to debug in the branch e12b4729b15ad5de2ed17324d40445da9b32e630 or before to find your error not in master since master has so many additional changes. Once you have found the bug we can work together to port the fix up to maint or master Thanks Barry > On Jun 24, 2016, at 9:35 AM, Fande Kong <fdkong.jd(a)gmail.com> wrote: > > > Message: 5 > Date: Fri, 24 Jun 2016 08:59:57 -0500 > From: Barry Smith <bsmith(a)mcs.anl.gov> > To: Mark Adams <mfadams(a)lbl.gov> > Cc: For users of the development version of PETSc > <petsc-dev(a)mcs.anl.gov> > Subject: Re: [petsc-dev] asm / gasm > Message-ID: <55FF6E27-46B0-4729-9A52-37C5ABBCC0CF(a)mcs.anl.gov> > Content-Type: text/plain; charset="us-ascii" > > > > On Jun 24, 2016, at 1:35 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > > > On Thu, Jun 23, 2016 at 11:46 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > > Mark, > > > > It is not as simple as this to convert to ASM. It will take a little bit of work to use ASM here instead of GASM. > > > > > > Just to be clear: ASM used to work. Did the semantics of ASM change? > > Hi Mark, > > Assume that you said GASM used to work, and now is not working any more. > > GASM was originally written by Dmitry. The basic idea is to allow mulit-rank blocks, that is, a mulit-rank subdomain problem could be solved using a small number of processor cores in parallel. This is different from ASM. > > I was involved into the development of GASM last summer. There are some changes: > > (1) Added a function to increase overlap of the multi-rank subdomains. The function is called by GASM in default. > > (2) Added a hierarchical partitioning to optimize data exchange. Ensure that small subdomains in a multi-rank subdomain are geometrically connected. GASM does not use this functionality in default. > > Any way, if you have an example (like Barry asked) showing the broken GASM, I will debug into it (of course, if Barry does not mind). > > Fande Kong, > > > Show me a commit where ASM worked! Do you mean that the GASM worked? The code has GASM calls in it, not ASM so how could ASM have previously worked? It is possible that something changed in GASM that broke GAMG's usage of GASM. Once you tell me how to reproduce the problem with GASM I can try to track down the problem. > > > > > But before that please please tell me the command line argument and example you use where the GASM crashes so I can get that fixed. Then I will look at using ASM instead after I have the current GASM code running again. > > > > In branch mark/gamg-agg-asm in ksp ex56, 'make runex56': > > I don't care about this! This is where you have tried to change from GASM to ASM which I told you is non-trivial. Give me the example and command line where the GASM version in master (or maint) doesn't work where the error message includes ** Max-trans not allowed because matrix is distributed > > We are not communicating very well, you jumped from stating GASM crashed to monkeying with ASM and now refuse to tell me how to reproduce the GASM crash. We have to start by fixing the current code to work with GASM (if it ever worked) and then move on to using ASM (which is just an optimization of the GASM usage.) > > > Barry > > > > > > 14:12 nid00495 ~/petsc/src/ksp/ksp/examples/tutorials$ make runex56 > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Petsc has generated inconsistent data > > [0]PETSC ERROR: MPI_Allreduce() called in different locations (code lines) on different processors > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > [0]PETSC ERROR: Petsc Development GIT revision: v3.7.2-633-g4f88208 GIT Date: 2016-06-23 18:53:31 +0200 > > [0]PETSC ERROR: /global/u2/m/madams/petsc/src/ksp/ksp/examples/tutorials/./ex56 on a arch-xc30-dbg64-intel named nid00495 by madams Thu Jun 23 14:12:57 2016 > > [0]PETSC ERROR: Configure options --COPTFLAGS="-no-ipo -g -O0" --CXXOPTFLAGS="-no-ipo -g -O0" --FOPTFLAGS="-fast -no-ipo -g -O0" --download-parmetis --download-metis --with-ssl=0 --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-debugging=1 --with-fc=0 --with-shared-libraries=0 --with-x=0 --with-mpiexec=srun LIBS=-lstdc++ --with-64-bit-indices PETSC_ARCH=arch-xc30-dbg64-intel > > [0]PETSC ERROR: #1 MatGetSubMatrices_MPIAIJ() li > > > > > > > > > > > > > > > > > > > > > > Barry > > > > > > > On Jun 23, 2016, at 4:19 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > The question boils down to, for empty processors do we: > > > > > > ierr = ISCreateGeneral(PETSC_COMM_SELF, 0, NULL, PETSC_COPY_VALUES, &is);CHKERRQ(ierr); > > > ierr = PCASMSetLocalSubdomains(subpc, 1, &is, NULL);CHKERRQ(ierr); > > > ierr = ISDestroy(&is);CHKERRQ(ierr); > > > > > > or > > > > > > PCASMSetLocalSubdomains(subpc, 0, NULL, NULL); > > > > > > The later gives and error that one domain is need and the later gives an error (appended). > > > > > > I've checked in the code for this second error in ksp (make runex56) > > > > > > Thanks, > > > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > [0]PETSC ERROR: Petsc has generated inconsistent data > > > [0]PETSC ERROR: MPI_Allreduce() called in different locations (code lines) on different processors > > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > > [0]PETSC ERROR: Petsc Development GIT revision: v3.7.2-633-g4f88208 GIT Date: 2016-06-23 18:53:31 +0200 > > > [0]PETSC ERROR: /global/u2/m/madams/petsc/src/ksp/ksp/examples/tutorials/./ex56 on a arch-xc30-dbg64-intel named nid00495 by madams Thu Jun 23 14:12:57 2016 > > > [0]PETSC ERROR: Configure options --COPTFLAGS="-no-ipo -g -O0" --CXXOPTFLAGS="-no-ipo -g -O0" --FOPTFLAGS="-fast -no-ipo -g -O0" --download-parmetis --download-metis --with-ssl=0 --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-debugging=1 --with-fc=0 --with-shared-libraries=0 --with-x=0 --with-mpiexec=srun LIBS=-lstdc++ --with-64-bit-indices PETSC_ARCH=arch-xc30-dbg64-intel > > > [0]PETSC ERROR: #1 MatGetSubMatrices_MPIAIJ() line 1147 in /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/mpiov.c > > > [0]PETSC ERROR: #2 MatGetSubMatrices_MPIAIJ() line 1147 in /global/u2/m/madams/petsc/src/mat/impls/aij/mpi/mpiov.c > > > [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > > > > On Thu, Jun 23, 2016 at 8:05 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > > > > Where is the command line that generates the error? > > > > > > > > > > On Jun 23, 2016, at 12:08 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > > > [adding Garth] > > > > > > > > On Thu, Jun 23, 2016 at 12:52 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > > > > > > Mark, > > > > > > > > I think there is a misunderstanding here. With GASM an individual block problem is __solved__ (via a parallel KSP) in parallel by several processes, with ASM each block is "owned" by and solved on a single process. > > > > > > > > Ah, OK, so this is for multiple processors in a block. Yes, we are looking at small, smother, blocks. > > > > > > > > > > > > With both the "block" can come from any unknowns on any processes. You can have, for example a block that comes from a region snaking across several processes if you like (or it makes sense due to coupling in the matrix). > > > > > > > > By default if you use ASM it will create one non-overlapping block defined by all unknowns owned by a single process and then extend it by "one level" (defined by the nonzero structure of the matrix) to get overlapping. > > > > > > > > The default in ASM is one level of overlap? That is new. (OK, I have not looked at ASM in like over 10 years) > > > > > > > > If you use multiple blocks per process it defines the non-overlapping blocks within a single process's unknowns > > > > > > > > I assume this still chops the matrix and does not call a partitioner. > > > > > > > > and extends each of them to have overlap (again by the non-zero structure of the matrix). The default is simple because the user only need indicate the number of blocks per process, the drawback is of course that it does depend on the process layout, number of processes etc and does not take into account particular "coupling information" that the user may know about with their problem. > > > > > > > > If the user wishes to defined the blocks themselves that is also possible with PCASMSetSubLocalSubdomains(). Each process provides 1 or more index sets for the subdomains it will solve on. Note that the index sets can contain any unknowns in the entire problem so the blocks do not have to "line up" with the parallel decomposition at all. > > > > > > > > Oh, OK, this is what I want. (I thought this worked). > > > > > > > > Of course determining and providing good such subdomains may not always be clear. > > > > > > > > In smoothed aggregation there is an argument that the aggregates are good, but the scale is fixed obviously. On a regular grid smoothed aggregation wants 3^D sized aggregates, which is obviously wonderful for AMS. And for anisotropy you want your ASM blocks to be on strongly connected components, which is what smoothed aggregation wants (not that I do this very well). > > > > > > > > > > > > I see in GAMG you have PCGAMGSetUseASMAggs > > > > > > > > But the code calls PCGASMSetSubdomains and the command line is -pc_gamg_use_agg_gasm, so this is all messed up. (more below) > > > > > > > > which sadly does not have an explanation in the users manual and sadly does not have a matching options data base name -pc_gamg_use_agg_gasm following the rule of drop the word set, all lower case, and put _ between words the option should be -pc_gamg_use_asm_aggs. > > > > > > > > BUT, THIS IS THE WAY IT WAS! It looks like someone hijacked this code and made it gasm. I never did this. > > > > > > > > Barry: you did this apparently in 2013. > > > > > > > > > > > > In addition to this one you could also have one that uses the aggs but use the PCASM to manage the solves instead of GASM, it would likely be less buggy and more efficient. > > > > > > > > yes > > > > > > > > > > > > Please tell me exactly what example you tried to run with what options and I will debug it. > > > > > > > > We got an error message: > > > > > > > > ** Max-trans not allowed because matrix is distributed > > > > > > > > Garth: is this from your code perhaps? I don't see it in PETSc. > > > > > > > > Note that ALL functionality that is included in PETSc should have tests that test that functionality then we will find out immediately when it is broken instead of two years later when it is much harder to debug. If this -pc_gamg_use_agg_gasm had had a test we won't be in this mess now. (Jed's damn code reviews sure don't pick up this stuff). > > > > > > > > First we need to change gasm to asm. > > > > > > > > We could add this argument pc_gamg_use_agg_asm to ksp/ex56 (runex56 or make a new test). The SNES version (also ex56) is my current test that I like to refer to as recommended parameters for elasticity. So I'd like to keep that clean, but we can add junk to ksp/ex56. > > > > > > > > I've done this in a branch mark/gamg-agg-asm. I get an error (appended). It looks like the second coarsest grid, which has 36 dof on one processor has an index 36 in the block on every processor. Strange. I can take a look at it later. > > > > > > > > Mark > > > > > > > > > [3]PETSC ERROR: [4]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > > > [4]PETSC ERROR: Petsc has generated inconsistent data > > > > > [4]PETSC ERROR: ith 0 block entry 36 not owned by any process, upper bound 36 > > > > > [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > > > > [4]PETSC ERROR: Petsc Development GIT revision: v3.7.2-630-g96e0c40 GIT Date: 2016-06-22 10:03:02 -0500 > > > > > [4]PETSC ERROR: ./ex56 on a arch-macosx-gnu-g named MarksMac-3.local by markadams Thu Jun 23 06:53:27 2016 > > > > > [4]PETSC ERROR: Configure options COPTFLAGS="-g -O0" CXXOPTFLAGS="-g -O0" FOPTFLAGS="-g -O0" --download-hypre=1 --download-parmetis=1 --download-metis=1 --download-ml=1 --download-p4est=1 --download-exodus=1 --download-triangle=1 --with-hdf5-dir=/Users/markadams/Codes/hdf5 --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g --download-chaco > > > > > [4]PETSC ERROR: #1 VecScatterCreate_PtoS() line 2348 in /Users/markadams/Codes/petsc/src/vec/vec/utils/vpscat.c > > > > > [4]PETSC ERROR: #2 VecScatterCreate() line 1552 in /Users/markadams/Codes/petsc/src/vec/vec/utils/vscat.c > > > > > [4]PETSC ERROR: Petsc has generated inconsistent data > > > > > [3]PETSC ERROR: ith 0 block entry 36 not owned by any process, upper bound 36 > > > > > [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > > > > [3]PETSC ERROR: Petsc Development GIT revision: v3.7.2-630-g96e0c40 GIT Date: 2016-06-22 10:03:02 -0500 > > > > > [3]PETSC ERROR: ./ex56 on a arch-macosx-gnu-g named MarksMac-3.local by markadams Thu Jun 23 06:53:27 2016 > > > > > [3]PETSC ERROR: Configure options COPTFLAGS="-g -O0" CXXOPTFLAGS="-g -O0" FOPTFLAGS="-g -O0" --download-hypre=1 --download-parmetis=1 --download-metis=1 --download-ml=1 --download-p4est=1 --download-exodus=1 --download-triangle=1 --with-hdf5-dir=/Users/markadams/Codes/hdf5 --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g --download-chaco > > > > > [3]PETSC ERROR: #1 VecScatterCreate_PtoS() line 2348 in /Users/markadams/Codes/petsc/src/vec/vec/utils/vpscat.c > > > > > [3]PETSC ERROR: #2 VecScatterCreate() line 1552 in /Users/markadams/Codes/petsc/src/vec/vec/utils/vscat.c > > > > > [3]PETSC ERROR: #3 PCSetUp_ASM() line 279 in /Users/markadams/Codes/petsc/src/ksp/pc/impls/asm/asm.c > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Jun 22, 2016, at 5:20 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > > > > > > > > > > > > > > > On Wed, Jun 22, 2016 at 8:06 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > > > > > > > > I suggest focusing on asm. > > > > > > > > > > OK, I will switch gasm to asm, this does not work anyway. > > > > > > > > > > Having blocks that span multiple processes seems like over kill for a smoother ? > > > > > > > > > > No, because it is a pain to have the math convolved with the parallel decompositions strategy (ie, I can't tell an application how to partition their problem). If an aggregate spans processor boundaries, which is fine and needed, and let's say we have a pretty uniform problem, then if the block gets split up, H is small in part of the domain and convergence could suffer along processor boundaries. And having the math change as the parallel decomposition changes is annoying. > > > > > > > > > > (Major league overkill) in fact doesn't one want multiple blocks per process, ie. pretty small blocks. > > > > > > > > > > No, it is just doing what would be done in serial. If the cost of moving the data across the processor is a problem then that is a tradeoff to consider. > > > > > > > > > > And I think you are misunderstanding me. There are lots of blocks per process (the aggregates are say 3^D in size). And many of the aggregates/blocks along the processor boundary will be split between processors, resulting is mall blocks and weak ASM PC on processor boundaries. > > > > > > > > > > I can understand ASM not being general and not letting blocks span processor boundaries, but I don't think the extra matrix communication costs are a big deal (done just once) and the vector communication costs are not bad, it probably does not include (too many) new processors to communicate with. > > > > > > > > > > > > > > > Barry > > > > > > > > > > > On Jun 22, 2016, at 7:51 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > > > > > > > I'm trying to get block smoothers to work for gamg. We (Garth) tried this and got this error: > > > > > > > > > > > > > > > > > > - Another option is use '-pc_gamg_use_agg_gasm true' and use '-mg_levels_pc_type gasm'. > > > > > > > > > > > > > > > > > > Running in parallel, I get > > > > > > > > > > > > ** Max-trans not allowed because matrix is distributed > > > > > > ---- > > > > > > > > > > > > First, what is the difference between asm and gasm? > > > > > > > > > > > > Second, I need to fix this to get block smoothers. This used to work. Did we lose the capability to have blocks that span processor subdomains? > > > > > > > > > > > > gamg only aggregates across processor subdomains within one layer, so maybe I could use one layer of overlap in some way? > > > > > > > > > > > > Thanks, > > > > > > Mark > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------ > > _______________________________________________ > petsc-dev mailing list > petsc-dev(a)mcs.anl.gov > https://lists.mcs.anl.gov/mailman/listinfo/petsc-dev > > > End of petsc-dev Digest, Vol 90, Issue 30 > ***************************************** >

1 0

Re: [petsc-dev] Known outdated sections of the User's Manual?
by Matthew Knepley 24 Jun '16

24 Jun '16

On Fri, Jun 24, 2016 at 7:32 AM, Hong <hzhang(a)mcs.anl.gov> wrote: > Matthew: > >> >> b) Use Lisandro's new support for movies, and better support for >> drawing (there is even a Plex draw now) >> > > Any example/instruction on how to use Plex draw? > An undergraduate student is work with me on developing map-drawing (using > saws) for DMNetwork, which is a subclass of DMPlex. > It may have bugs, but in 'next' you should be able to give -dm_view draw to get a wireframe of a 2D Plex, and -vec_view draw to get an unstructured P1 interpolant for that function if the Vec comes from a Plex. Thanks, Matt > Hong > >> >> >>> On Fri, Jun 24, 2016 at 3:45 PM, Matthew Knepley <knepley(a)gmail.com> >>> wrote: >>> > On Fri, Jun 24, 2016 at 1:58 AM, Patrick Sanan < >>> patrick.sanan(a)gmail.com> >>> > wrote: >>> >> >>> >> I'm working on some groundwork to improve the PETSc documentation, and >>> >> the next thing I'd like to look at is the User's Manual, adding >>> >> slightly-prettier code listings (as in the update to the dev manual). >>> >> >>> >> Before doing that, however, it would be very helpful to know if there >>> >> are any sections of the manual which are known, by the relevant >>> >> experts here, to require deletion or heavy rewrites; it would of >>> >> course be a waste of time to format these. >>> >> >>> >> Specifically, I'm wondering about the following sections, which have >>> >> in common that they are things concerned with friendly external tools. >>> >> I've seen new users get very frustrated when they expect these sorts >>> >> of things to "just work," so it's probably constructive to remove any >>> >> outdated information here: >>> >> >>> >> - Chapter 11: Using MATLAB with PETSc . The support here has changed >>> >> quite a lot, so I'm not sure what currently works. Is the MATLAB >>> >> Compute Engine still supported? >>> >> >>> >> - Sections 15.10-15.14: Eclipse/Qt Creator/Developers Studio/XCode >>> >> users. This is likely not all current. Is this information helpful >>> >> here? >>> >> >>> >> - Section 15.15 : Graphics. I saw that there were some updates to the >>> >> drawing tools recently by Lisandro, so if any of this material is >>> >> known to be out of date, that would be helpful to know. >>> > >>> > >>> > I think graphics is not so much out of date now as incomplete. We >>> really >>> > want >>> > to be telling people to do things the modern way, but the old ways >>> still >>> > work. >>> > >>> > Matt >>> > >>> > -- >>> > What most experimenters take for granted before they begin their >>> experiments >>> > is infinitely more interesting than any results to which their >>> experiments >>> > lead. >>> > -- Norbert Wiener >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] asm / gasm
by Mark Adams 24 Jun '16

24 Jun '16

Well, SNES ex56 is completely broken. It is not calling the solvers at all. I will take a look at it. On Fri, Jun 24, 2016 at 6:34 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Ok in master using src/snes/examples/tutorials/ex56.c I get > > $ petscmpiexec -n 2 ./ex56 -ne 2 -alpha 1.e-3 -two_solves true -dm_refine > 1 -petscspace_poly_tensor -petscspace_order 1 -ksp_type cg > -ksp_monitor_short -ksp_rtol 1.e-8 -pc_type gamg -pc_gamg_type agg > -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 100 > -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1 > -pc_gamg_threshold 0.0 -ksp_converged_reason -use_mat_nearnullspace true > -mg_levels_ksp_max_it 2 -mg_levels_ksp_type chebyshev > -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10 > -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type sor > -mat_block_size 3 -petscpartitioner_type parmetis -pc_gamg_use_agg_gasm > -mg_levels_pc_type gasm -ksp_view > [0]main 300/100 eqs in vector > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Nonconforming object sizes > [0]PETSC ERROR: Vector wrong size 120 for scatter 1 (scatter forward and > vector from != ctx from size) > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.7.2-644-gee393a6 GIT > Date: 2016-06-24 09:01:50 -0500 > [0]PETSC ERROR: ./ex56 on a arch-basic named Barrys-MacBook-Pro.local by > barrysmith Fri Jun 24 11:30:00 2016 > [0]PETSC ERROR: Configure options > --with-mpi-dir=/Users/barrysmith/libraries PETSC_ARCH=arch-basic > --download-metis --download-parmetis > [0]PETSC ERROR: #1 VecScatterBegin() line 1773 in > /Users/barrysmith/Src/petsc/src/vec/vec/utils/vscat.c > [0]PETSC ERROR: #2 PCApply_GASM() line 686 in > /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gasm/gasm.c > [0]PETSC ERROR: #3 PCApply() line 482 in > /Users/barrysmith/Src/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #4 KSP_PCApply() line 244 in > /Users/barrysmith/Src/petsc/include/petsc/private/kspimpl.h > [0]PETSC ERROR: #5 KSPSolve_CG() line 137 in > /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cg/cg.c > [0]PETSC ERROR: #6 KSPSolve() line 656 in > /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #7 KSPSolve_Chebyshev() line 390 in > /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cheby/cheby.c > [0]PETSC ERROR: #8 KSPSolve() line 656 in > /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #9 PCMGMCycle_Private() line 21 in > /Users/barrysmith/Src/petsc/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: Nonconforming object sizes > [1]PETSC ERROR: Vector wrong size 180 for scatter 300 (scatter forward and > vector from != ctx from size) > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [1]PETSC ERROR: Petsc Development GIT revision: v3.7.2-644-gee393a6 GIT > Date: 2016-06-24 09:01:50 -0500 > [1]PETSC ERROR: ./ex56 on a arch-basic named Barrys-MacBook-Pro.local by > barrysmith Fri Jun 24 11:30:00 2016 > [1]PETSC ERROR: #10 PCApply_MG() line 343 in > /Users/barrysmith/Src/petsc/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: #11 PCApply() line 482 in > /Users/barrysmith/Src/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #12 KSP_PCApply() line 244 in > /Users/barrysmith/Src/petsc/include/petsc/private/kspimpl.h > [0]PETSC ERROR: #13 KSPSolve_CG() line 137 in > /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cg/cg.c > Configure options --with-mpi-dir=/Users/barrysmith/libraries > PETSC_ARCH=arch-basic --download-metis --download-parmetis > [1]PETSC ERROR: #1 VecScatterBegin() line 1773 in > /Users/barrysmith/Src/petsc/src/vec/vec/utils/vscat.c > [1]PETSC ERROR: #2 PCApply_GASM() line 686 in > /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gasm/gasm.c > [1]PETSC ERROR: #3 PCApply() line 482 in > /Users/barrysmith/Src/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #14 KSPSolve() line 656 in > /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #15 main() line 408 in > /Users/barrysmith/Src/petsc/src/snes/examples/tutorials/ex56.c > [0]PETSC ERROR: PETSc Option Table entries: > [0]PETSC ERROR: -alpha 1.e-3 > [0]PETSC ERROR: -dm_refine 1 > [0]PETSC ERROR: -ksp_converged_reason > [0]PETSC ERROR: -ksp_monitor_short > [0]PETSC ERROR: -ksp_rtol 1.e-8 > [0]PETSC ERROR: -ksp_type cg > [0]PETSC ERROR: [1]PETSC ERROR: #4 KSP_PCApply() line 244 in > /Users/barrysmith/Src/petsc/include/petsc/private/kspimpl.h > [1]PETSC ERROR: #5 KSPSolve_CG() line 137 in > /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cg/cg.c > [1]PETSC ERROR: #6 KSPSolve() line 656 in > /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: #7 KSPSolve_Chebyshev() line 390 in > /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cheby/cheby.c > [1]PETSC ERROR: #8 KSPSolve() line 656 in > /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c > -ksp_view > [0]PETSC ERROR: -malloc_test > [0]PETSC ERROR: -mat_block_size 3 > [0]PETSC ERROR: -mg_levels_esteig_ksp_max_it 10 > [0]PETSC ERROR: -mg_levels_esteig_ksp_type cg > [0]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 > [0]PETSC ERROR: -mg_levels_ksp_max_it 2 > [1]PETSC ERROR: #9 PCMGMCycle_Private() line 21 in > /Users/barrysmith/Src/petsc/src/ksp/pc/impls/mg/mg.c > [1]PETSC ERROR: #10 PCApply_MG() line 343 in > /Users/barrysmith/Src/petsc/src/ksp/pc/impls/mg/mg.c > [0]PETSC ERROR: -mg_levels_ksp_type chebyshev > [0]PETSC ERROR: -mg_levels_pc_type gasm > [0]PETSC ERROR: -ne 2 > [0]PETSC ERROR: -pc_gamg_agg_nsmooths 1 > [0]PETSC ERROR: -pc_gamg_coarse_eq_limit 100 > [0]PETSC ERROR: [1]PETSC ERROR: #11 PCApply() line 482 in > /Users/barrysmith/Src/petsc/src/ksp/pc/interface/precon.c > [1]PETSC ERROR: #12 KSP_PCApply() line 244 in > /Users/barrysmith/Src/petsc/include/petsc/private/kspimpl.h > [1]PETSC ERROR: #13 KSPSolve_CG() line 137 in > /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cg/cg.c > [1]PETSC ERROR: #14 KSPSolve() line 656 in > /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c > -pc_gamg_reuse_interpolation true > [0]PETSC ERROR: -pc_gamg_square_graph 1 > [0]PETSC ERROR: -pc_gamg_threshold 0.0 > [1]PETSC ERROR: #15 main() line 408 in > /Users/barrysmith/Src/petsc/src/snes/examples/tutorials/ex56.c > [1]PETSC ERROR: PETSc Option Table entries: > [1]PETSC ERROR: [0]PETSC ERROR: -pc_gamg_type agg > [0]PETSC ERROR: -pc_gamg_use_agg_gasm > [0]PETSC ERROR: -pc_type gamg > [0]PETSC ERROR: -petscpartitioner_type parmetis > [0]PETSC ERROR: -petscspace_order 1 > [0]PETSC ERROR: -petscspace_poly_tensor > -alpha 1.e-3 > [1]PETSC ERROR: -dm_refine 1 > [1]PETSC ERROR: -ksp_converged_reason > [1]PETSC ERROR: -ksp_monitor_short > [0]PETSC ERROR: -two_solves true > [0]PETSC ERROR: -use_mat_nearnullspace true > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint(a)mcs.anl.gov---------- > [1]PETSC ERROR: -ksp_rtol 1.e-8 > [1]PETSC ERROR: -ksp_type cg > [1]PETSC ERROR: -ksp_view > [1]PETSC ERROR: -malloc_test > [1]PETSC ERROR: -mat_block_size 3 > [1]PETSC ERROR: -mg_levels_esteig_ksp_max_it 10 > [1]PETSC ERROR: -mg_levels_esteig_ksp_type cg > [1]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 > [1]PETSC ERROR: -mg_levels_ksp_max_it 2 > [1]PETSC ERROR: -mg_levels_ksp_type chebyshev > [1]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 60) - process > 0 > -mg_levels_pc_type gasm > [1]PETSC ERROR: -ne 2 > [1]PETSC ERROR: -pc_gamg_agg_nsmooths 1 > [1]PETSC ERROR: -pc_gamg_coarse_eq_limit 100 > [1]PETSC ERROR: -pc_gamg_reuse_interpolation true > [1]PETSC ERROR: -pc_gamg_square_graph 1 > [1]PETSC ERROR: -pc_gamg_threshold 0.0 > [1]PETSC ERROR: -pc_gamg_type agg > [1]PETSC ERROR: -pc_gamg_use_agg_gasm > [1]PETSC ERROR: -pc_type gamg > [1]PETSC ERROR: -petscpartitioner_type parmetis > [1]PETSC ERROR: -petscspace_order 1 > [1]PETSC ERROR: -petscspace_poly_tensor > [1]PETSC ERROR: -two_solves true > [1]PETSC ERROR: -use_mat_nearnullspace true > [1]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint(a)mcs.anl.gov---------- > application called MPI_Abort(MPI_COMM_WORLD, 60) - process 1 > > > I'll have to start poking around to see what is going on. Could be do > to changes in GAMG or in GASM. Bisect here I come. > > > Barry > > > On Jun 24, 2016, at 9:52 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > OK, but how should I proceed? > > > > On Fri, Jun 24, 2016 at 4:47 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > > > On Jun 24, 2016, at 9:35 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > Barry, > > > > > > ASM worked before GASM existed. I've never heard of GASM until > yesterday, other than seeing the directory in the source. > > > > Pretty impressive you never heard of GASM and yet YOU put the GASM > code into GAMG > > > > a2f3521de src/ksp/pc/impls/gamg/gamg.c (Mark Adams > 2012-05-01 12:48:43 -0400 604) > > a2f3521de src/ksp/pc/impls/gamg/gamg.c (Mark Adams > 2012-05-01 12:48:43 -0400 605) /* set defaults */ > > 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith > 2013-01-02 02:05:45 -0600 606) ierr = KSPSetType(smoother, > KSPCHEBYSHEV);CHKERRQ(ierr); > > ffc955d6d src/ksp/pc/impls/gamg/gamg.c (Mark Adams > 2012-03-27 19:09:54 -0400 607) > > 1b18a24a4 src/ksp/pc/impls/gamg/gamg.c (Mark Adams > 2013-11-02 15:56:24 -0400 608) /* set blocks for GASM smoother that > uses the 'aggregates' */ > > 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith > 2013-01-02 02:05:45 -0600 609) if (pc_gamg->use_aggs_in_gasm) { > > 2d3561bba src/ksp/pc/impls/gamg/gamg.c (Satish Balay > 2012-03-29 12:58:42 -0500 610) PetscInt sz; > > 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp > 2013-01-30 22:53:06 -0600 611) IS *is; > > a2f3521de src/ksp/pc/impls/gamg/gamg.c (Mark Adams > 2012-05-01 12:48:43 -0400 612) > > 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp > 2013-01-30 22:53:06 -0600 613) sz = nASMBlocksArr[level]; > > 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp > 2013-01-30 22:53:06 -0600 614) is = ASMLocalIDsArr[level]; > > 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith > 2013-01-02 02:05:45 -0600 615) ierr = PCSetType(subpc, > PCGASM);CHKERRQ(ierr); > > 1b18a24a4 src/ksp/pc/impls/gamg/gamg.c (Mark Adams > 2013-11-02 15:56:24 -0400 616) ierr = PCGASMSetOverlap(subpc, > 0);CHKERRQ(ierr); > > 7f66b68fb src/ksp/pc/impls/gamg/gamg.c (Mark Adams > 2015-08-11 09:52:40 -0400 617) if (!sz) { > > 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp > 2013-01-30 22:53:06 -0600 618) IS is; > > ffc955d6d src/ksp/pc/impls/gamg/gamg.c (Mark Adams > 2012-03-27 19:09:54 -0400 619) PetscInt my0,kk; > > 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith > 2013-01-02 02:05:45 -0600 620) ierr = > MatGetOwnershipRange(Aarr[level], &my0 > > > > I think you put in the GASM but then forgot and thought you put in > ASM. > > > > > It has been so long, I could probably hunt for a working version, but > who cares, we are going to roll back 5 years? > > > > > > > > > > In branch mark/gamg-agg-asm in ksp ex56, 'make runex56': > > > > > > I don't care about this! This is where you have tried to change > from GASM to ASM which I told you is non-trivial. > > > Give me the example and command line where the GASM version in master > (or maint) doesn't work where the error message includes ** Max-trans not > allowed because matrix is distributed > > > > > > Why do you want to debug GASM? I thought we wanted to use ASM. It is > not clear why converting from GASM to ASM is non-trivial, I could have done > this from the command line. This use aggregate-asm stuff just does what a > clever user might do. > > > > > > > > > We are not communicating very well, you jumped from stating GASM > crashed to monkeying with ASM and now refuse to tell me how to reproduce > the GASM crash. > > > > > > I don't think you want to mess with GASM, BUT if you run ksp/ex56 with: > > > > > > -pc_gamg_use_agg_gasm -mg_levels_pc_type gasm > > > > > > and remove the existing -mg_levels_pc_type sor, you should see an > error that Garth and I saw (eg, appended). > > > > > > > > > We have to start by fixing the current code to work with GASM (if it > ever worked) and then move on to using ASM (which is just an optimization > of the GASM usage.) > > > > > > > > > Are you sure? I am deep into debugging asm with DDT. I can switch to > GASM by just calling the GASM set block method and use gasm in the PC, and > continue with DDT debugging. > > > > > > Let me know. > > > > > > > > > 16:29 master *= ~/Codes/petsc/src/ksp/ksp/examples/tutorials$ make > runex56 > > > 1,4c1,85 > > > < Linear solve converged due to CONVERGED_RTOL iterations 8 > > > < Linear solve converged due to CONVERGED_RTOL iterations 8 > > > < Linear solve converged due to CONVERGED_RTOL iterations 8 > > > < [0]main |b-Ax|/|b|=1.940043e-04, |b|=4.969822e+00, emax=9.926090e-01 > > > --- > > > > [1]PETSC ERROR: > ------------------------------------------------------------------------ > > > > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation > Violation, probably memory access out of range > > > > [1]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > > > [1]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > > [1]PETSC ERROR: [2]PETSC ERROR: > ------------------------------------------------------------------------ > > > > [2]PETSC ERROR: Caught signal number 11 SEGV: Segmentation > Violation, probably memory access out of range > > > > [2]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > > > [2]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > > [2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple > Mac OS X to find memory corruption errors > > > > [6]PETSC ERROR: > ------------------------------------------------------------------------ > > > > [6]PETSC ERROR: Caught signal number 11 SEGV: Segmentation > Violation, probably memory access out of range > > > > [6]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > > > [6]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > > [6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple > Mac OS X to find memory corruption errors > > > > [4]PETSC ERROR: > ------------------------------------------------------------------------ > > > > [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation > Violation, probably memory access out of range > > > > [4]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > > > [4]PETSC ERROR: or see > http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > > [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple > Mac OS X to find memory corruption errors > > > > [6]PETSC ERROR: likely location of problem given in stack below > > > > [6]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > > > > [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > > > > [6]PETSC ERROR: INSTEAD the line number of the start of the > function > > > > [6]PETSC ERROR: is given. > > > > [6]PETSC ERROR: [6] MatGetSubMatrices_MPIAIJ_Local line 1228 > /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c > > > > [6]PETSC ERROR: [6] MatGetSubMatrices_MPIAIJ line 1131 > /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c > > > > [6]PETSC ERROR: [6] MatGetSubMatricesMPI_MPIXAIJ line 2163 > /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c > > > > [6]PETSC ERROR: [6] MatGetSubMatricesMPI_MPIAIJ line 2418 > /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c > > > > [6]PETSC ERROR: [6] MatGetSubMatricesMPI line 6805 > /Users/markadams/Codes/petsc/src/mat/interface/matrix.c > > > > [6]PETSC ERROR: [6] PCSetUp_GASM line 357 > /Users/markadams/Codes/petsc/src/ksp/pc/impls/gasm/gasm.c > > > > [6]PETSC ERROR: [6] PCSetUp line 930 > /Users/markadams/Codes/petsc/src/ksp/pc/interface/precon.c > > > > [6]PETSC ERROR: [6] KSPSetUp line 305 > /Users/markadams/Codes/petsc/src/ksp/ksp/interface/itfunc.c > > > > [6]PETSC ERROR: [6] PCSetUp_MG line 561 > /Users/markadams/Codes/petsc/src/ksp/pc/impls/mg/mg.c > > > > [6]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > > > [6]PETSC ERROR: Signal received > > > > > >

1 0

Re: [petsc-dev] asm / gasm
by Mark Adams 24 Jun '16

24 Jun '16

Note, I just pushed a 1 processor example .... and I am using KSP ex56, not SNES.

1 0

Re: [petsc-dev] asm / gasm
by Mark Adams 24 Jun '16

24 Jun '16

I think I found the/a problem. It is failing on the current test when it adds equation 144 to GASM. This test has a 3^3 cell mesh so a 4^3 vertex mesh. There one face with BCs, that means there is a 4x4x3 mesh. With bs==3, there are 144 equations, equation number 144 is the first out-of-bounds equation. This seems like a suspicious coincidence. It has been a long time since I've looked that code but I also see that this test has two aggregates and the IS's that I give GASM have 54 and 90 equations in them. So it looks like I am taking out the BCs. [I just got your new message ...] Mark On Fri, Jun 24, 2016 at 4:52 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > OK, but how should I proceed? > > On Fri, Jun 24, 2016 at 4:47 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > >> >> > On Jun 24, 2016, at 9:35 AM, Mark Adams <mfadams(a)lbl.gov> wrote: >> > >> > Barry, >> > >> > ASM worked before GASM existed. I've never heard of GASM until >> yesterday, other than seeing the directory in the source. >> >> Pretty impressive you never heard of GASM and yet YOU put the GASM >> code into GAMG >> >> a2f3521de src/ksp/pc/impls/gamg/gamg.c (Mark Adams >> 2012-05-01 12:48:43 -0400 604) >> a2f3521de src/ksp/pc/impls/gamg/gamg.c (Mark Adams >> 2012-05-01 12:48:43 -0400 605) /* set defaults */ >> 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith >> 2013-01-02 02:05:45 -0600 606) ierr = KSPSetType(smoother, >> KSPCHEBYSHEV);CHKERRQ(ierr); >> ffc955d6d src/ksp/pc/impls/gamg/gamg.c (Mark Adams >> 2012-03-27 19:09:54 -0400 607) >> 1b18a24a4 src/ksp/pc/impls/gamg/gamg.c (Mark Adams >> 2013-11-02 15:56:24 -0400 608) /* set blocks for GASM smoother that >> uses the 'aggregates' */ >> 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith >> 2013-01-02 02:05:45 -0600 609) if (pc_gamg->use_aggs_in_gasm) { >> 2d3561bba src/ksp/pc/impls/gamg/gamg.c (Satish Balay >> 2012-03-29 12:58:42 -0500 610) PetscInt sz; >> 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp >> 2013-01-30 22:53:06 -0600 611) IS *is; >> a2f3521de src/ksp/pc/impls/gamg/gamg.c (Mark Adams >> 2012-05-01 12:48:43 -0400 612) >> 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp >> 2013-01-30 22:53:06 -0600 613) sz = nASMBlocksArr[level]; >> 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp >> 2013-01-30 22:53:06 -0600 614) is = ASMLocalIDsArr[level]; >> 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith >> 2013-01-02 02:05:45 -0600 615) ierr = PCSetType(subpc, >> PCGASM);CHKERRQ(ierr); >> 1b18a24a4 src/ksp/pc/impls/gamg/gamg.c (Mark Adams >> 2013-11-02 15:56:24 -0400 616) ierr = PCGASMSetOverlap(subpc, >> 0);CHKERRQ(ierr); >> 7f66b68fb src/ksp/pc/impls/gamg/gamg.c (Mark Adams >> 2015-08-11 09:52:40 -0400 617) if (!sz) { >> 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp >> 2013-01-30 22:53:06 -0600 618) IS is; >> ffc955d6d src/ksp/pc/impls/gamg/gamg.c (Mark Adams >> 2012-03-27 19:09:54 -0400 619) PetscInt my0,kk; >> 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith >> 2013-01-02 02:05:45 -0600 620) ierr = >> MatGetOwnershipRange(Aarr[level], &my0 >> >> I think you put in the GASM but then forgot and thought you put in ASM. >> >> > It has been so long, I could probably hunt for a working version, but >> who cares, we are going to roll back 5 years? >> > >> > >> > > In branch mark/gamg-agg-asm in ksp ex56, 'make runex56': >> > >> > I don't care about this! This is where you have tried to change from >> GASM to ASM which I told you is non-trivial. >> > Give me the example and command line where the GASM version in master >> (or maint) doesn't work where the error message includes ** Max-trans not >> allowed because matrix is distributed >> > >> > Why do you want to debug GASM? I thought we wanted to use ASM. It is >> not clear why converting from GASM to ASM is non-trivial, I could have done >> this from the command line. This use aggregate-asm stuff just does what a >> clever user might do. >> > >> > >> > We are not communicating very well, you jumped from stating GASM >> crashed to monkeying with ASM and now refuse to tell me how to reproduce >> the GASM crash. >> > >> > I don't think you want to mess with GASM, BUT if you run ksp/ex56 with: >> > >> > -pc_gamg_use_agg_gasm -mg_levels_pc_type gasm >> > >> > and remove the existing -mg_levels_pc_type sor, you should see an error >> that Garth and I saw (eg, appended). >> > >> > >> > We have to start by fixing the current code to work with GASM (if it >> ever worked) and then move on to using ASM (which is just an optimization >> of the GASM usage.) >> > >> > >> > Are you sure? I am deep into debugging asm with DDT. I can switch to >> GASM by just calling the GASM set block method and use gasm in the PC, and >> continue with DDT debugging. >> > >> > Let me know. >> > >> > >> > 16:29 master *= ~/Codes/petsc/src/ksp/ksp/examples/tutorials$ make >> runex56 >> > 1,4c1,85 >> > < Linear solve converged due to CONVERGED_RTOL iterations 8 >> > < Linear solve converged due to CONVERGED_RTOL iterations 8 >> > < Linear solve converged due to CONVERGED_RTOL iterations 8 >> > < [0]main |b-Ax|/|b|=1.940043e-04, |b|=4.969822e+00, emax=9.926090e-01 >> > --- >> > > [1]PETSC ERROR: >> ------------------------------------------------------------------------ >> > > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> > > [1]PETSC ERROR: Try option -start_in_debugger or >> -on_error_attach_debugger >> > > [1]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> > > [1]PETSC ERROR: [2]PETSC ERROR: >> ------------------------------------------------------------------------ >> > > [2]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> > > [2]PETSC ERROR: Try option -start_in_debugger or >> -on_error_attach_debugger >> > > [2]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> > > [2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple >> Mac OS X to find memory corruption errors >> > > [6]PETSC ERROR: >> ------------------------------------------------------------------------ >> > > [6]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> > > [6]PETSC ERROR: Try option -start_in_debugger or >> -on_error_attach_debugger >> > > [6]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> > > [6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple >> Mac OS X to find memory corruption errors >> > > [4]PETSC ERROR: >> ------------------------------------------------------------------------ >> > > [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> > > [4]PETSC ERROR: Try option -start_in_debugger or >> -on_error_attach_debugger >> > > [4]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> > > [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple >> Mac OS X to find memory corruption errors >> > > [6]PETSC ERROR: likely location of problem given in stack below >> > > [6]PETSC ERROR: --------------------- Stack Frames >> ------------------------------------ >> > > [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not >> available, >> > > [6]PETSC ERROR: INSTEAD the line number of the start of the >> function >> > > [6]PETSC ERROR: is given. >> > > [6]PETSC ERROR: [6] MatGetSubMatrices_MPIAIJ_Local line 1228 >> /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c >> > > [6]PETSC ERROR: [6] MatGetSubMatrices_MPIAIJ line 1131 >> /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c >> > > [6]PETSC ERROR: [6] MatGetSubMatricesMPI_MPIXAIJ line 2163 >> /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c >> > > [6]PETSC ERROR: [6] MatGetSubMatricesMPI_MPIAIJ line 2418 >> /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c >> > > [6]PETSC ERROR: [6] MatGetSubMatricesMPI line 6805 >> /Users/markadams/Codes/petsc/src/mat/interface/matrix.c >> > > [6]PETSC ERROR: [6] PCSetUp_GASM line 357 >> /Users/markadams/Codes/petsc/src/ksp/pc/impls/gasm/gasm.c >> > > [6]PETSC ERROR: [6] PCSetUp line 930 >> /Users/markadams/Codes/petsc/src/ksp/pc/interface/precon.c >> > > [6]PETSC ERROR: [6] KSPSetUp line 305 >> /Users/markadams/Codes/petsc/src/ksp/ksp/interface/itfunc.c >> > > [6]PETSC ERROR: [6] PCSetUp_MG line 561 >> /Users/markadams/Codes/petsc/src/ksp/pc/impls/mg/mg.c >> > > [6]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> > > [6]PETSC ERROR: Signal received >> >> >

1 0

Re: [petsc-dev] asm / gasm
by Barry Smith 24 Jun '16

24 Jun '16

Ok in master using src/snes/examples/tutorials/ex56.c I get $ petscmpiexec -n 2 ./ex56 -ne 2 -alpha 1.e-3 -two_solves true -dm_refine 1 -petscspace_poly_tensor -petscspace_order 1 -ksp_type cg -ksp_monitor_short -ksp_rtol 1.e-8 -pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 100 -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1 -pc_gamg_threshold 0.0 -ksp_converged_reason -use_mat_nearnullspace true -mg_levels_ksp_max_it 2 -mg_levels_ksp_type chebyshev -mg_levels_esteig_ksp_type cg -mg_levels_esteig_ksp_max_it 10 -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -mg_levels_pc_type sor -mat_block_size 3 -petscpartitioner_type parmetis -pc_gamg_use_agg_gasm -mg_levels_pc_type gasm -ksp_view [0]main 300/100 eqs in vector [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Nonconforming object sizes [0]PETSC ERROR: Vector wrong size 120 for scatter 1 (scatter forward and vector from != ctx from size) [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.7.2-644-gee393a6 GIT Date: 2016-06-24 09:01:50 -0500 [0]PETSC ERROR: ./ex56 on a arch-basic named Barrys-MacBook-Pro.local by barrysmith Fri Jun 24 11:30:00 2016 [0]PETSC ERROR: Configure options --with-mpi-dir=/Users/barrysmith/libraries PETSC_ARCH=arch-basic --download-metis --download-parmetis [0]PETSC ERROR: #1 VecScatterBegin() line 1773 in /Users/barrysmith/Src/petsc/src/vec/vec/utils/vscat.c [0]PETSC ERROR: #2 PCApply_GASM() line 686 in /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gasm/gasm.c [0]PETSC ERROR: #3 PCApply() line 482 in /Users/barrysmith/Src/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #4 KSP_PCApply() line 244 in /Users/barrysmith/Src/petsc/include/petsc/private/kspimpl.h [0]PETSC ERROR: #5 KSPSolve_CG() line 137 in /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cg/cg.c [0]PETSC ERROR: #6 KSPSolve() line 656 in /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #7 KSPSolve_Chebyshev() line 390 in /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cheby/cheby.c [0]PETSC ERROR: #8 KSPSolve() line 656 in /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #9 PCMGMCycle_Private() line 21 in /Users/barrysmith/Src/petsc/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Nonconforming object sizes [1]PETSC ERROR: Vector wrong size 180 for scatter 300 (scatter forward and vector from != ctx from size) [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Development GIT revision: v3.7.2-644-gee393a6 GIT Date: 2016-06-24 09:01:50 -0500 [1]PETSC ERROR: ./ex56 on a arch-basic named Barrys-MacBook-Pro.local by barrysmith Fri Jun 24 11:30:00 2016 [1]PETSC ERROR: #10 PCApply_MG() line 343 in /Users/barrysmith/Src/petsc/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: #11 PCApply() line 482 in /Users/barrysmith/Src/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #12 KSP_PCApply() line 244 in /Users/barrysmith/Src/petsc/include/petsc/private/kspimpl.h [0]PETSC ERROR: #13 KSPSolve_CG() line 137 in /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cg/cg.c Configure options --with-mpi-dir=/Users/barrysmith/libraries PETSC_ARCH=arch-basic --download-metis --download-parmetis [1]PETSC ERROR: #1 VecScatterBegin() line 1773 in /Users/barrysmith/Src/petsc/src/vec/vec/utils/vscat.c [1]PETSC ERROR: #2 PCApply_GASM() line 686 in /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gasm/gasm.c [1]PETSC ERROR: #3 PCApply() line 482 in /Users/barrysmith/Src/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #14 KSPSolve() line 656 in /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #15 main() line 408 in /Users/barrysmith/Src/petsc/src/snes/examples/tutorials/ex56.c [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -alpha 1.e-3 [0]PETSC ERROR: -dm_refine 1 [0]PETSC ERROR: -ksp_converged_reason [0]PETSC ERROR: -ksp_monitor_short [0]PETSC ERROR: -ksp_rtol 1.e-8 [0]PETSC ERROR: -ksp_type cg [0]PETSC ERROR: [1]PETSC ERROR: #4 KSP_PCApply() line 244 in /Users/barrysmith/Src/petsc/include/petsc/private/kspimpl.h [1]PETSC ERROR: #5 KSPSolve_CG() line 137 in /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cg/cg.c [1]PETSC ERROR: #6 KSPSolve() line 656 in /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c [1]PETSC ERROR: #7 KSPSolve_Chebyshev() line 390 in /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cheby/cheby.c [1]PETSC ERROR: #8 KSPSolve() line 656 in /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c -ksp_view [0]PETSC ERROR: -malloc_test [0]PETSC ERROR: -mat_block_size 3 [0]PETSC ERROR: -mg_levels_esteig_ksp_max_it 10 [0]PETSC ERROR: -mg_levels_esteig_ksp_type cg [0]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 [0]PETSC ERROR: -mg_levels_ksp_max_it 2 [1]PETSC ERROR: #9 PCMGMCycle_Private() line 21 in /Users/barrysmith/Src/petsc/src/ksp/pc/impls/mg/mg.c [1]PETSC ERROR: #10 PCApply_MG() line 343 in /Users/barrysmith/Src/petsc/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: -mg_levels_ksp_type chebyshev [0]PETSC ERROR: -mg_levels_pc_type gasm [0]PETSC ERROR: -ne 2 [0]PETSC ERROR: -pc_gamg_agg_nsmooths 1 [0]PETSC ERROR: -pc_gamg_coarse_eq_limit 100 [0]PETSC ERROR: [1]PETSC ERROR: #11 PCApply() line 482 in /Users/barrysmith/Src/petsc/src/ksp/pc/interface/precon.c [1]PETSC ERROR: #12 KSP_PCApply() line 244 in /Users/barrysmith/Src/petsc/include/petsc/private/kspimpl.h [1]PETSC ERROR: #13 KSPSolve_CG() line 137 in /Users/barrysmith/Src/petsc/src/ksp/ksp/impls/cg/cg.c [1]PETSC ERROR: #14 KSPSolve() line 656 in /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c -pc_gamg_reuse_interpolation true [0]PETSC ERROR: -pc_gamg_square_graph 1 [0]PETSC ERROR: -pc_gamg_threshold 0.0 [1]PETSC ERROR: #15 main() line 408 in /Users/barrysmith/Src/petsc/src/snes/examples/tutorials/ex56.c [1]PETSC ERROR: PETSc Option Table entries: [1]PETSC ERROR: [0]PETSC ERROR: -pc_gamg_type agg [0]PETSC ERROR: -pc_gamg_use_agg_gasm [0]PETSC ERROR: -pc_type gamg [0]PETSC ERROR: -petscpartitioner_type parmetis [0]PETSC ERROR: -petscspace_order 1 [0]PETSC ERROR: -petscspace_poly_tensor -alpha 1.e-3 [1]PETSC ERROR: -dm_refine 1 [1]PETSC ERROR: -ksp_converged_reason [1]PETSC ERROR: -ksp_monitor_short [0]PETSC ERROR: -two_solves true [0]PETSC ERROR: -use_mat_nearnullspace true [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- [1]PETSC ERROR: -ksp_rtol 1.e-8 [1]PETSC ERROR: -ksp_type cg [1]PETSC ERROR: -ksp_view [1]PETSC ERROR: -malloc_test [1]PETSC ERROR: -mat_block_size 3 [1]PETSC ERROR: -mg_levels_esteig_ksp_max_it 10 [1]PETSC ERROR: -mg_levels_esteig_ksp_type cg [1]PETSC ERROR: -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 [1]PETSC ERROR: -mg_levels_ksp_max_it 2 [1]PETSC ERROR: -mg_levels_ksp_type chebyshev [1]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 60) - process 0 -mg_levels_pc_type gasm [1]PETSC ERROR: -ne 2 [1]PETSC ERROR: -pc_gamg_agg_nsmooths 1 [1]PETSC ERROR: -pc_gamg_coarse_eq_limit 100 [1]PETSC ERROR: -pc_gamg_reuse_interpolation true [1]PETSC ERROR: -pc_gamg_square_graph 1 [1]PETSC ERROR: -pc_gamg_threshold 0.0 [1]PETSC ERROR: -pc_gamg_type agg [1]PETSC ERROR: -pc_gamg_use_agg_gasm [1]PETSC ERROR: -pc_type gamg [1]PETSC ERROR: -petscpartitioner_type parmetis [1]PETSC ERROR: -petscspace_order 1 [1]PETSC ERROR: -petscspace_poly_tensor [1]PETSC ERROR: -two_solves true [1]PETSC ERROR: -use_mat_nearnullspace true [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- application called MPI_Abort(MPI_COMM_WORLD, 60) - process 1 I'll have to start poking around to see what is going on. Could be do to changes in GAMG or in GASM. Bisect here I come. Barry > On Jun 24, 2016, at 9:52 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > OK, but how should I proceed? > > On Fri, Jun 24, 2016 at 4:47 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > On Jun 24, 2016, at 9:35 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > Barry, > > > > ASM worked before GASM existed. I've never heard of GASM until yesterday, other than seeing the directory in the source. > > Pretty impressive you never heard of GASM and yet YOU put the GASM code into GAMG > > a2f3521de src/ksp/pc/impls/gamg/gamg.c (Mark Adams 2012-05-01 12:48:43 -0400 604) > a2f3521de src/ksp/pc/impls/gamg/gamg.c (Mark Adams 2012-05-01 12:48:43 -0400 605) /* set defaults */ > 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith 2013-01-02 02:05:45 -0600 606) ierr = KSPSetType(smoother, KSPCHEBYSHEV);CHKERRQ(ierr); > ffc955d6d src/ksp/pc/impls/gamg/gamg.c (Mark Adams 2012-03-27 19:09:54 -0400 607) > 1b18a24a4 src/ksp/pc/impls/gamg/gamg.c (Mark Adams 2013-11-02 15:56:24 -0400 608) /* set blocks for GASM smoother that uses the 'aggregates' */ > 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith 2013-01-02 02:05:45 -0600 609) if (pc_gamg->use_aggs_in_gasm) { > 2d3561bba src/ksp/pc/impls/gamg/gamg.c (Satish Balay 2012-03-29 12:58:42 -0500 610) PetscInt sz; > 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp 2013-01-30 22:53:06 -0600 611) IS *is; > a2f3521de src/ksp/pc/impls/gamg/gamg.c (Mark Adams 2012-05-01 12:48:43 -0400 612) > 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp 2013-01-30 22:53:06 -0600 613) sz = nASMBlocksArr[level]; > 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp 2013-01-30 22:53:06 -0600 614) is = ASMLocalIDsArr[level]; > 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith 2013-01-02 02:05:45 -0600 615) ierr = PCSetType(subpc, PCGASM);CHKERRQ(ierr); > 1b18a24a4 src/ksp/pc/impls/gamg/gamg.c (Mark Adams 2013-11-02 15:56:24 -0400 616) ierr = PCGASMSetOverlap(subpc, 0);CHKERRQ(ierr); > 7f66b68fb src/ksp/pc/impls/gamg/gamg.c (Mark Adams 2015-08-11 09:52:40 -0400 617) if (!sz) { > 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp 2013-01-30 22:53:06 -0600 618) IS is; > ffc955d6d src/ksp/pc/impls/gamg/gamg.c (Mark Adams 2012-03-27 19:09:54 -0400 619) PetscInt my0,kk; > 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith 2013-01-02 02:05:45 -0600 620) ierr = MatGetOwnershipRange(Aarr[level], &my0 > > I think you put in the GASM but then forgot and thought you put in ASM. > > > It has been so long, I could probably hunt for a working version, but who cares, we are going to roll back 5 years? > > > > > > > In branch mark/gamg-agg-asm in ksp ex56, 'make runex56': > > > > I don't care about this! This is where you have tried to change from GASM to ASM which I told you is non-trivial. > > Give me the example and command line where the GASM version in master (or maint) doesn't work where the error message includes ** Max-trans not allowed because matrix is distributed > > > > Why do you want to debug GASM? I thought we wanted to use ASM. It is not clear why converting from GASM to ASM is non-trivial, I could have done this from the command line. This use aggregate-asm stuff just does what a clever user might do. > > > > > > We are not communicating very well, you jumped from stating GASM crashed to monkeying with ASM and now refuse to tell me how to reproduce the GASM crash. > > > > I don't think you want to mess with GASM, BUT if you run ksp/ex56 with: > > > > -pc_gamg_use_agg_gasm -mg_levels_pc_type gasm > > > > and remove the existing -mg_levels_pc_type sor, you should see an error that Garth and I saw (eg, appended). > > > > > > We have to start by fixing the current code to work with GASM (if it ever worked) and then move on to using ASM (which is just an optimization of the GASM usage.) > > > > > > Are you sure? I am deep into debugging asm with DDT. I can switch to GASM by just calling the GASM set block method and use gasm in the PC, and continue with DDT debugging. > > > > Let me know. > > > > > > 16:29 master *= ~/Codes/petsc/src/ksp/ksp/examples/tutorials$ make runex56 > > 1,4c1,85 > > < Linear solve converged due to CONVERGED_RTOL iterations 8 > > < Linear solve converged due to CONVERGED_RTOL iterations 8 > > < Linear solve converged due to CONVERGED_RTOL iterations 8 > > < [0]main |b-Ax|/|b|=1.940043e-04, |b|=4.969822e+00, emax=9.926090e-01 > > --- > > > [1]PETSC ERROR: ------------------------------------------------------------------------ > > > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > > > [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > > [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > [1]PETSC ERROR: [2]PETSC ERROR: ------------------------------------------------------------------------ > > > [2]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > > > [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > > [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > [2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > > > [6]PETSC ERROR: ------------------------------------------------------------------------ > > > [6]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > > > [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > > [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > [6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > > > [4]PETSC ERROR: ------------------------------------------------------------------------ > > > [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > > > [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > > > [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > > > [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > > > [6]PETSC ERROR: likely location of problem given in stack below > > > [6]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > > > [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > > > [6]PETSC ERROR: INSTEAD the line number of the start of the function > > > [6]PETSC ERROR: is given. > > > [6]PETSC ERROR: [6] MatGetSubMatrices_MPIAIJ_Local line 1228 /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c > > > [6]PETSC ERROR: [6] MatGetSubMatrices_MPIAIJ line 1131 /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c > > > [6]PETSC ERROR: [6] MatGetSubMatricesMPI_MPIXAIJ line 2163 /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c > > > [6]PETSC ERROR: [6] MatGetSubMatricesMPI_MPIAIJ line 2418 /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c > > > [6]PETSC ERROR: [6] MatGetSubMatricesMPI line 6805 /Users/markadams/Codes/petsc/src/mat/interface/matrix.c > > > [6]PETSC ERROR: [6] PCSetUp_GASM line 357 /Users/markadams/Codes/petsc/src/ksp/pc/impls/gasm/gasm.c > > > [6]PETSC ERROR: [6] PCSetUp line 930 /Users/markadams/Codes/petsc/src/ksp/pc/interface/precon.c > > > [6]PETSC ERROR: [6] KSPSetUp line 305 /Users/markadams/Codes/petsc/src/ksp/ksp/interface/itfunc.c > > > [6]PETSC ERROR: [6] PCSetUp_MG line 561 /Users/markadams/Codes/petsc/src/ksp/pc/impls/mg/mg.c > > > [6]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > > [6]PETSC ERROR: Signal received > >

1 0

Re: [petsc-dev] PETSc blame digest (next) 2016-06-24
by Satish Balay 24 Jun '16

24 Jun '16

On Fri, 24 Jun 2016, Dave May wrote: > On Friday, 24 June 2016, Satish Balay <balay(a)mcs.anl.gov> wrote: > > > for (p=0; p<n_bbox_recv; p++) { > > printf("[rank %d]: box from %d : > > range[%+1.4e,%+1.4e]x[%+1.4e,%+1.4e]\n",rank,recv_bbox[p].owner_rank, > > > > recv_bbox[p].min[0],recv_bbox[p].max[0],recv_bbox[p].min[1],recv_bbox[p].max[1]); > > } > > > > > > Is this printf for debugging purposes - and perhaps should be removed? > > > Yeah - absolutely. > > I've been moving house(s) the last days and have been unable to clean up > all the annoying business like this. > Matt unexpectedly / abruptly merged my stuff without warning me :D > > I'll take care of it tonight. This branch is merged into 'knepley/feature-plex-particle-example' - and some of the fixes are in this branch [bad fix workflow] - so you should first check here if some issue is fixes or not. Satish

1 0

Re: [petsc-dev] PETSc blame digest (next) 2016-06-24
by Dave May 24 Jun '16

24 Jun '16

On Friday, 24 June 2016, Satish Balay <balay(a)mcs.anl.gov> wrote: > for (p=0; p<n_bbox_recv; p++) { > printf("[rank %d]: box from %d : > range[%+1.4e,%+1.4e]x[%+1.4e,%+1.4e]\n",rank,recv_bbox[p].owner_rank, > > recv_bbox[p].min[0],recv_bbox[p].max[0],recv_bbox[p].min[1],recv_bbox[p].max[1]); > } > > > Is this printf for debugging purposes - and perhaps should be removed? Yeah - absolutely. I've been moving house(s) the last days and have been unable to clean up all the annoying business like this. Matt unexpectedly / abruptly merged my stuff without warning me :D I'll take care of it tonight. Cheers, Dave > > Satish > > On Fri, 24 Jun 2016, PETSc checkBuilds wrote: > > > > > > > Dear PETSc developer, > > > > This email contains listings of contributions attributed to you by > > `git blame` that caused compiler errors or warnings in PETSc automated > > testing. Follow the links to see the full log files. Please attempt to > fix > > the issues promptly or let us know at petsc-dev(a)mcs.anl.gov > <javascript:;> if you are unable > > to resolve the issues. > > > > Thanks, > > The PETSc development team > > > > ---- > > > > warnings attributed to commit > https://bitbucket.org/petsc/petsc/commits/b16650c > > DMSwarm collect: > > > > src/dm/impls/swarm/swarm_migrate.c:481 > > [ > http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/06/24/build_next_… > ] > > > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:91: > warning: format '%e' expects argument of type 'double', but argument 7 has > type 'PetscReal {aka __float128}' [-Wformat=] > > [ > http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/06/24/build_next_… > ] > > > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:12: > warning: format '%e' expects argument of type 'double', but argument 5 has > type 'PetscReal' [-Wformat=] > > [ > http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/06/24/build_next_… > ] > > > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:91: > warning: format '%e' expects argument of type 'double', but argument 5 has > type 'PetscReal {aka __float128}' [-Wformat=] > > [ > http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/06/24/build_next_… > ] > > > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:91: > warning: format '%e' expects argument of type 'double', but argument 4 has > type 'PetscReal {aka __float128}' [-Wformat=] > > [ > http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/06/24/build_next_… > ] > > > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:12: > warning: format '%e' expects argument of type 'double', but argument 7 has > type 'PetscReal' [-Wformat=] > > [ > http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/06/24/build_next_… > ] > > > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:91: > warning: format '%e' expects argument of type 'double', but argument 6 has > type 'PetscReal {aka __float128}' [-Wformat=] > > [ > http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/06/24/build_next_… > ] > > > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:12: > warning: format '%e' expects argument of type 'double', but argument 4 has > type 'PetscReal' [-Wformat=] > > [ > http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/06/24/build_next_… > ] > > > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:12: > warning: format '%e' expects argument of type 'double', but argument 6 has > type 'PetscReal' [-Wformat=] > > > > ---- > > To opt-out from receiving these messages - send a request to > petsc-dev(a)mcs.anl.gov <javascript:;>. > > > >

1 0

Re: [petsc-dev] [petsc-checkbuilds] PETSc blame digest (next) 2016-06-24
by Satish Balay 24 Jun '16

24 Jun '16

for (p=0; p<n_bbox_recv; p++) { printf("[rank %d]: box from %d : range[%+1.4e,%+1.4e]x[%+1.4e,%+1.4e]\n",rank,recv_bbox[p].owner_rank, recv_bbox[p].min[0],recv_bbox[p].max[0],recv_bbox[p].min[1],recv_bbox[p].max[1]); } Is this printf for debugging purposes - and perhaps should be removed? Satish On Fri, 24 Jun 2016, PETSc checkBuilds wrote: > > > Dear PETSc developer, > > This email contains listings of contributions attributed to you by > `git blame` that caused compiler errors or warnings in PETSc automated > testing. Follow the links to see the full log files. Please attempt to fix > the issues promptly or let us know at petsc-dev(a)mcs.anl.gov if you are unable > to resolve the issues. > > Thanks, > The PETSc development team > > ---- > > warnings attributed to commit https://bitbucket.org/petsc/petsc/commits/b16650c > DMSwarm collect: > > src/dm/impls/swarm/swarm_migrate.c:481 > [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/06/24/build_next_…] > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:91: warning: format '%e' expects argument of type 'double', but argument 7 has type 'PetscReal {aka __float128}' [-Wformat=] > [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/06/24/build_next_…] > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:12: warning: format '%e' expects argument of type 'double', but argument 5 has type 'PetscReal' [-Wformat=] > [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/06/24/build_next_…] > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:91: warning: format '%e' expects argument of type 'double', but argument 5 has type 'PetscReal {aka __float128}' [-Wformat=] > [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/06/24/build_next_…] > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:91: warning: format '%e' expects argument of type 'double', but argument 4 has type 'PetscReal {aka __float128}' [-Wformat=] > [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/06/24/build_next_…] > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:12: warning: format '%e' expects argument of type 'double', but argument 7 has type 'PetscReal' [-Wformat=] > [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/06/24/build_next_…] > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:91: warning: format '%e' expects argument of type 'double', but argument 6 has type 'PetscReal {aka __float128}' [-Wformat=] > [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/06/24/build_next_…] > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:12: warning: format '%e' expects argument of type 'double', but argument 4 has type 'PetscReal' [-Wformat=] > [http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/06/24/build_next_…] > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:12: warning: format '%e' expects argument of type 'double', but argument 6 has type 'PetscReal' [-Wformat=] > > ---- > To opt-out from receiving these messages - send a request to petsc-dev(a)mcs.anl.gov. >

1 0

Re: [petsc-dev] Known outdated sections of the User's Manual?
by Patrick Sanan 24 Jun '16

24 Jun '16

Thanks! I've added these to some running notes and will address these further once I get to those sections in the manual. On Fri, Jun 24, 2016 at 4:37 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > >> On Jun 24, 2016, at 3:58 AM, Patrick Sanan <patrick.sanan(a)gmail.com> wrote: >> >> I'm working on some groundwork to improve the PETSc documentation, and >> the next thing I'd like to look at is the User's Manual, adding >> slightly-prettier code listings (as in the update to the dev manual). >> >> Before doing that, however, it would be very helpful to know if there >> are any sections of the manual which are known, by the relevant >> experts here, to require deletion or heavy rewrites; it would of >> course be a waste of time to format these. >> >> Specifically, I'm wondering about the following sections, which have >> in common that they are things concerned with friendly external tools. >> I've seen new users get very frustrated when they expect these sorts >> of things to "just work," so it's probably constructive to remove any >> outdated information here: >> > > Take out chapter 10 Using ADIFOR with PETSc we have no support for this. > >> - Chapter 11: Using MATLAB with PETSc . The support here has changed >> quite a lot, so I'm not sure what currently works. Is the MATLAB >> Compute Engine still supported? > > Everything here is supported. What is missing is that the section 11.1 Dumping Data for MATLAB should have two parts: > > 1) Dumping small amounts of data with ASCII files and (what is there now) and > > 2) Dumping larger data with the binary viewer and loading it with PetscBinaryRead.m > > under 2 it should also mention PetscViewerPushFormat(viewer,PETSC_VIEWER_BINARY_MATLAB) this causes a .m script to be generated that reads the all the data objects from the binary file and formats them appropriately. For example DMDA vectors get automatically properly shaped in Matlab. See for example src/dm/examples/tutorials/ex7.c > >> >> - Sections 15.10-15.14: Eclipse/Qt Creator/Developers Studio/XCode >> users. This is likely not all current. Is this information helpful >> here? > > Though not a lot of people use this stuff we need to keep this information somewhere. Better in the users manual then lost in some file? > >> >> - Section 15.15 : Graphics. I saw that there were some updates to the >> drawing tools recently by Lisandro, so if any of this material is >> known to be out of date, that would be helpful to know. > > Nothing wrong with the data here. It is just missing some stuff like tikz, simple savings of movies and images >

1 0