May 2012 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Error during KSPDestroy
by Satish Balay 07 May '12

07 May '12

On Mon, 7 May 2012, Barry Smith wrote: > > I am also running complex. > > Look in the file dlasq2.f (it will be in the externalpackages subdirectory of the PETSc directory. Look at line 215, this is where valgrind has a problem. In my copy > > END IF > * > * Check for negative data and compute sums of q's and e's. > * <------ this is line 215 > Z( 2*N ) = ZERO > > it is a comment, which is not good. Is lione 215 also a comment in your copy of dlasq2.f? > > There are two possible causes I can think of for your problem > > 1) PETSc does not allocate enough work space for zgesvd() or > 2) the BLAS/LAPACK routines have a bug where they sometimes access out of their work space. > > > Satish, > > Can you try the same build options on a Linux machine as close to Alexander as we have and see if you can reproduce this? I can reproduce on my linux laptop. [also on petsc.mcs] Satish

1 0

Re: [petsc-dev] Error during KSPDestroy
by Alexander Grayver 07 May '12

07 May '12

On 07.05.2012 15:04, Barry Smith wrote: > I am also running complex. > > Look in the file dlasq2.f (it will be in the externalpackages subdirectory of the PETSc directory. Look at line 215, this is where valgrind has a problem. In my copy > > END IF > * > * Check for negative data and compute sums of q's and e's. > *<------ this is line 215 > Z( 2*N ) = ZERO > > it is a comment, which is not good. Is lione 215 also a comment in your copy of dlasq2.f? Barry, * * Rearrange data for locality: Z=(q1,qq1,e1,ee1,q2,qq2,e2,ee2,...). * DO 30 K = 2*N, 2, -2 Z( 2*K ) = ZERO ! <----------- LINE 215 Z( 2*K-1 ) = Z( K ) Z( 2*K-2 ) = ZERO Z( 2*K-3 ) = Z( K-1 ) 30 CONTINUE In valgrind log you can see that it complaints about following lines as well: ==9009== Invalid write of size 8 ==9009== at 0x10651D5: dlasq2_ (dlasq2.f:215) ==9009== by 0x1064683: dlasq1_ (dlasq1.f:135) ==9009== by 0x104EB3F: zbdsqr_ (zbdsqr.f:225) ==9009== by 0x1023B74: zgesvd_ (zgesvd.f:2040) ==9009== by 0xD38725: KSPComputeExtremeSingularValues_GMRES (gmreig.c:46) ==9009== by 0xCB3CC7: KSPComputeExtremeSingularValues (itfunc.c:47) ==9009== by 0x406DF2: main (solveTest.c:47) ==9009== Address 0x6ef5d88 is 8 bytes before a block of size 832 alloc'd ==9009== at 0x4C2786E: memalign (vg_replace_malloc.c:581) ==9009== by 0x47E3CB: PetscMallocAlign (mal.c:30) ==9009== by 0xD2E286: KSPSetUp_GMRES (gmres.c:73) ==9009== by 0xCB5464: KSPSetUp (itfunc.c:239) ==9009== by 0xCB6E56: KSPSolve (itfunc.c:402) ==9009== by 0x406DDB: main (solveTest.c:46) ==9009== ==9009== Invalid write of size 8 ==9009== at 0x1065204: dlasq2_ (dlasq2.f:216) .... ==9009== ==9009== Invalid write of size 8 ==9009== at 0x1065223: dlasq2_ (dlasq2.f:217) .... ==9009== ==9009== Invalid write of size 8 ==9009== at 0x1065255: dlasq2_ (dlasq2.f:218) .... All further output is also related to the Z array. Hard to believe this is a LAPACK problem... I tried 3 implementations over 2 machines. I have bad feeling it's my stupid mistake somewhere... :) Just in case, I run ubuntu 11.1 and PETSc is configured like this with default gcc compiler: ./configure --with-petsc-arch=mpich-gcc-complex-debug-c --download-f-blas-lapack --with-precision=double --with-scalar-type=complex --download-mpich > There are two possible causes I can think of for your problem > > 1) PETSc does not allocate enough work space for zgesvd() or > 2) the BLAS/LAPACK routines have a bug where they sometimes access out of their work space. > > > Satish, > > Can you try the same build options on a Linux machine as close to Alexander as we have and see if you can reproduce this? > > > Barry > > > > On May 7, 2012, at 2:16 AM, Alexander Grayver wrote: > >> On 06.05.2012 22:24, Barry Smith wrote: >>> Alexander, >>> >>> I cannot reproduce this on my mac with 3 different blas/lapack. >> Barry, >> >> I'm surprised. I ran it on my home PC with ubuntu and PETSc configured from scratch as following: >> --download-mpich --with-fortran-interfaces=1 --download-scalapack --download-blacs --with-scalar-type=complex --download-blas-lapack --with-precision=double >> >> And it's still there. >> Please note that all my numbers are complex. >> >>> Could you please run the case below but with --download-f-blas-lapack (you forgot the -f last time)? Send us the valgrind results. This will tell use the exact line number in dlasq3() that is triggering the bad read. >> I did: >> ./configure --with-petsc-arch=openmpi-intel-complex-debug-c --download-scalapack --download-blacs --download-f-blas-lapack --with-precision=double --with-scalar-type=complex >> >> And then valgrind program. The first message from log: >> >> ==27656== Invalid write of size 8 >> ==27656== at 0x15A8E9E: dlasq2_ (dlasq2.f:215) >> ==27656== by 0x15A83A4: dlasq1_ (dlasq1.f:135) >> ==27656== by 0x158ACEC: zbdsqr_ (zbdsqr.f:225) >> ==27656== by 0x154EC27: zgesvd_ (zgesvd.f:2038) >> ==27656== by 0x695DD3: KSPComputeExtremeSingularValues_GMRES (gmreig.c:46) >> ==27656== by 0x69DD76: KSPComputeExtremeSingularValues (itfunc.c:47) >> ==27656== by 0x44E98C: main (solveTest.c:62) >> ==27656== Address 0xfad2d98 is 8 bytes before a block of size 832 alloc'd >> ==27656== at 0x4C25D66: memalign (vg_replace_malloc.c:694) >> ==27656== by 0x4B642B: PetscMallocAlign (mal.c:30) >> ==27656== by 0x687775: KSPSetUp_GMRES (gmres.c:73) >> ==27656== by 0x69FE4A: KSPSetUp (itfunc.c:239) >> ==27656== by 0x6A2058: KSPSolve (itfunc.c:402) >> ==27656== by 0x44E969: main (solveTest.c:61) >> >> Please find full log attached. >> >>> Barry >>> >>> >>> On May 6, 2012, at 9:16 AM, Alexander Grayver wrote: >>> >>>> On 06.05.2012 15:34, Matthew Knepley wrote: >>>>> On Sun, May 6, 2012 at 9:24 AM, Alexander Grayver<agrayver(a)gfz-potsdam.de> wrote: >>>>> Hm, valgrind gives a lot of output like that (see full log in previous message): >>>>> >>>>> Can you run this with --download-f-blas-lapack? This sounds much more like an MKL bug. >>>> I did: >>>> --download-scalapack --download-blacs --download-blas-lapack --with-precision=double --with-scalar-type=complex >>>> >>>> The error is still there. I checked "ldd solveTest", mkl is not used for sure. This is not an MKL problem I guess: >>>> >>>> ==13600== Invalid read of size 8 >>>> ==13600== at 0x58636AF: dlasq3_ (in /usr/local/lib/liblapack.so.3.2.2) >>>> ==13600== by 0x5862C84: dlasq2_ (in /usr/local/lib/liblapack.so.3.2.2) >>>> ==13600== by 0x5861F2C: dlasq1_ (in /usr/local/lib/liblapack.so.3.2.2) >>>> ==13600== by 0x571A479: zbdsqr_ (in /usr/local/lib/liblapack.so.3.2.2) >>>> ==13600== by 0x57466A7: zgesvd_ (in /usr/local/lib/liblapack.so.3.2.2) >>>> ==13600== by 0x694687: KSPComputeExtremeSingularValues_GMRES (gmreig.c:46) >>>> ==13600== by 0x69C62A: KSPComputeExtremeSingularValues (itfunc.c:47) >>>> ==13600== by 0x44E02C: main (solveTest.c:62) >>>> ==13600== Address 0x10826b90 is 16 bytes before a block of size 832 alloc'd >>>> ==13600== at 0x4C25D66: memalign (vg_replace_malloc.c:694) >>>> ==13600== by 0x4B5ACB: PetscMallocAlign (mal.c:30) >>>> ==13600== by 0x686181: KSPSetUp_GMRES (gmres.c:73) >>>> ==13600== by 0x69E6FE: KSPSetUp (itfunc.c:239) >>>> ==13600== by 0x6A090C: KSPSolve (itfunc.c:402) >>>> ==13600== by 0x44E009: main (solveTest.c:61) >>>> >>>> The weird thing is that the it gives correct result, so zgesvd works fine. >>>> >>>> And also running this program with 10 iterations in valgrind doesn't produce error. The low above is with 100 iterations. >>>> Without valgrind the error is always there. >>>> >>>> -- >>>> Regards, >>>> Alexander >>>> >> >> -- >> Regards, >> Alexander >> >> <valgrind.zip> -- Regards, Alexander

1 0

Re: [petsc-dev] Error during KSPDestroy
by Barry Smith 07 May '12

07 May '12

I am also running complex. Look in the file dlasq2.f (it will be in the externalpackages subdirectory of the PETSc directory. Look at line 215, this is where valgrind has a problem. In my copy END IF * * Check for negative data and compute sums of q's and e's. * <------ this is line 215 Z( 2*N ) = ZERO it is a comment, which is not good. Is lione 215 also a comment in your copy of dlasq2.f? There are two possible causes I can think of for your problem 1) PETSc does not allocate enough work space for zgesvd() or 2) the BLAS/LAPACK routines have a bug where they sometimes access out of their work space. Satish, Can you try the same build options on a Linux machine as close to Alexander as we have and see if you can reproduce this? Barry On May 7, 2012, at 2:16 AM, Alexander Grayver wrote: > On 06.05.2012 22:24, Barry Smith wrote: >> Alexander, >> >> I cannot reproduce this on my mac with 3 different blas/lapack. > > Barry, > > I'm surprised. I ran it on my home PC with ubuntu and PETSc configured from scratch as following: > --download-mpich --with-fortran-interfaces=1 --download-scalapack --download-blacs --with-scalar-type=complex --download-blas-lapack --with-precision=double > > And it's still there. > Please note that all my numbers are complex. > >> Could you please run the case below but with --download-f-blas-lapack (you forgot the -f last time)? Send us the valgrind results. This will tell use the exact line number in dlasq3() that is triggering the bad read. > > I did: > ./configure --with-petsc-arch=openmpi-intel-complex-debug-c --download-scalapack --download-blacs --download-f-blas-lapack --with-precision=double --with-scalar-type=complex > > And then valgrind program. The first message from log: > > ==27656== Invalid write of size 8 > ==27656== at 0x15A8E9E: dlasq2_ (dlasq2.f:215) > ==27656== by 0x15A83A4: dlasq1_ (dlasq1.f:135) > ==27656== by 0x158ACEC: zbdsqr_ (zbdsqr.f:225) > ==27656== by 0x154EC27: zgesvd_ (zgesvd.f:2038) > ==27656== by 0x695DD3: KSPComputeExtremeSingularValues_GMRES (gmreig.c:46) > ==27656== by 0x69DD76: KSPComputeExtremeSingularValues (itfunc.c:47) > ==27656== by 0x44E98C: main (solveTest.c:62) > ==27656== Address 0xfad2d98 is 8 bytes before a block of size 832 alloc'd > ==27656== at 0x4C25D66: memalign (vg_replace_malloc.c:694) > ==27656== by 0x4B642B: PetscMallocAlign (mal.c:30) > ==27656== by 0x687775: KSPSetUp_GMRES (gmres.c:73) > ==27656== by 0x69FE4A: KSPSetUp (itfunc.c:239) > ==27656== by 0x6A2058: KSPSolve (itfunc.c:402) > ==27656== by 0x44E969: main (solveTest.c:61) > > Please find full log attached. > >> Barry >> >> >> On May 6, 2012, at 9:16 AM, Alexander Grayver wrote: >> >>> On 06.05.2012 15:34, Matthew Knepley wrote: >>>> On Sun, May 6, 2012 at 9:24 AM, Alexander Grayver<agrayver(a)gfz-potsdam.de> wrote: >>>> Hm, valgrind gives a lot of output like that (see full log in previous message): >>>> >>>> Can you run this with --download-f-blas-lapack? This sounds much more like an MKL bug. >>> I did: >>> --download-scalapack --download-blacs --download-blas-lapack --with-precision=double --with-scalar-type=complex >>> >>> The error is still there. I checked "ldd solveTest", mkl is not used for sure. This is not an MKL problem I guess: >>> >>> ==13600== Invalid read of size 8 >>> ==13600== at 0x58636AF: dlasq3_ (in /usr/local/lib/liblapack.so.3.2.2) >>> ==13600== by 0x5862C84: dlasq2_ (in /usr/local/lib/liblapack.so.3.2.2) >>> ==13600== by 0x5861F2C: dlasq1_ (in /usr/local/lib/liblapack.so.3.2.2) >>> ==13600== by 0x571A479: zbdsqr_ (in /usr/local/lib/liblapack.so.3.2.2) >>> ==13600== by 0x57466A7: zgesvd_ (in /usr/local/lib/liblapack.so.3.2.2) >>> ==13600== by 0x694687: KSPComputeExtremeSingularValues_GMRES (gmreig.c:46) >>> ==13600== by 0x69C62A: KSPComputeExtremeSingularValues (itfunc.c:47) >>> ==13600== by 0x44E02C: main (solveTest.c:62) >>> ==13600== Address 0x10826b90 is 16 bytes before a block of size 832 alloc'd >>> ==13600== at 0x4C25D66: memalign (vg_replace_malloc.c:694) >>> ==13600== by 0x4B5ACB: PetscMallocAlign (mal.c:30) >>> ==13600== by 0x686181: KSPSetUp_GMRES (gmres.c:73) >>> ==13600== by 0x69E6FE: KSPSetUp (itfunc.c:239) >>> ==13600== by 0x6A090C: KSPSolve (itfunc.c:402) >>> ==13600== by 0x44E009: main (solveTest.c:61) >>> >>> The weird thing is that the it gives correct result, so zgesvd works fine. >>> >>> And also running this program with 10 iterations in valgrind doesn't produce error. The low above is with 100 iterations. >>> Without valgrind the error is always there. >>> >>> -- >>> Regards, >>> Alexander >>> > > > -- > Regards, > Alexander > > <valgrind.zip>

1 0

Re: [petsc-dev] Error during KSPDestroy
by Matthew Knepley 07 May '12

07 May '12

On Mon, May 7, 2012 at 3:16 AM, Alexander Grayver <agrayver(a)gfz-potsdam.de>wrote: > On 06.05.2012 22:24, Barry Smith wrote: > >> Alexander, >> >> I cannot reproduce this on my mac with 3 different blas/lapack. >> > > Barry, > > I'm surprised. I ran it on my home PC with ubuntu and PETSc configured > from scratch as following: > --download-mpich --with-fortran-interfaces=1 --download-scalapack > --download-blacs --with-scalar-type=complex --download-blas-lapack > --with-precision=double > > And it's still there. > Please note that all my numbers are complex. > I just ran this with complex instead of real. I get sh: 1.688716e+00 and no crash. Matt > Could you please run the case below but with >> --download-f-blas-lapack (you forgot the -f last time)? Send us the >> valgrind results. This will tell use the exact line number in dlasq3() that >> is triggering the bad read. >> > > I did: > ./configure --with-petsc-arch=openmpi-**intel-complex-debug-c > --download-scalapack --download-blacs --download-f-blas-lapack > --with-precision=double --with-scalar-type=complex > > And then valgrind program. The first message from log: > > ==27656== Invalid write of size 8 > ==27656== at 0x15A8E9E: dlasq2_ (dlasq2.f:215) > ==27656== by 0x15A83A4: dlasq1_ (dlasq1.f:135) > ==27656== by 0x158ACEC: zbdsqr_ (zbdsqr.f:225) > ==27656== by 0x154EC27: zgesvd_ (zgesvd.f:2038) > ==27656== by 0x695DD3: KSPComputeExtremeSingularValue**s_GMRES > (gmreig.c:46) > ==27656== by 0x69DD76: KSPComputeExtremeSingularValue**s (itfunc.c:47) > ==27656== by 0x44E98C: main (solveTest.c:62) > ==27656== Address 0xfad2d98 is 8 bytes before a block of size 832 alloc'd > ==27656== at 0x4C25D66: memalign (vg_replace_malloc.c:694) > ==27656== by 0x4B642B: PetscMallocAlign (mal.c:30) > ==27656== by 0x687775: KSPSetUp_GMRES (gmres.c:73) > ==27656== by 0x69FE4A: KSPSetUp (itfunc.c:239) > ==27656== by 0x6A2058: KSPSolve (itfunc.c:402) > ==27656== by 0x44E969: main (solveTest.c:61) > > Please find full log attached. > > Barry >> >> >> On May 6, 2012, at 9:16 AM, Alexander Grayver wrote: >> >> On 06.05.2012 15:34, Matthew Knepley wrote: >>> >>>> On Sun, May 6, 2012 at 9:24 AM, Alexander Grayver<agrayver@gfz-potsdam. >>>> **de <agrayver(a)gfz-potsdam.de>> wrote: >>>> Hm, valgrind gives a lot of output like that (see full log in previous >>>> message): >>>> >>>> Can you run this with --download-f-blas-lapack? This sounds much more >>>> like an MKL bug. >>>> >>> I did: >>> --download-scalapack --download-blacs --download-blas-lapack >>> --with-precision=double --with-scalar-type=complex >>> >>> The error is still there. I checked "ldd solveTest", mkl is not used for >>> sure. This is not an MKL problem I guess: >>> >>> ==13600== Invalid read of size 8 >>> ==13600== at 0x58636AF: dlasq3_ (in /usr/local/lib/liblapack.so.3.** >>> 2.2) >>> ==13600== by 0x5862C84: dlasq2_ (in /usr/local/lib/liblapack.so.3.** >>> 2.2) >>> ==13600== by 0x5861F2C: dlasq1_ (in /usr/local/lib/liblapack.so.3.** >>> 2.2) >>> ==13600== by 0x571A479: zbdsqr_ (in /usr/local/lib/liblapack.so.3.** >>> 2.2) >>> ==13600== by 0x57466A7: zgesvd_ (in /usr/local/lib/liblapack.so.3.** >>> 2.2) >>> ==13600== by 0x694687: KSPComputeExtremeSingularValue**s_GMRES >>> (gmreig.c:46) >>> ==13600== by 0x69C62A: KSPComputeExtremeSingularValue**s >>> (itfunc.c:47) >>> ==13600== by 0x44E02C: main (solveTest.c:62) >>> ==13600== Address 0x10826b90 is 16 bytes before a block of size 832 >>> alloc'd >>> ==13600== at 0x4C25D66: memalign (vg_replace_malloc.c:694) >>> ==13600== by 0x4B5ACB: PetscMallocAlign (mal.c:30) >>> ==13600== by 0x686181: KSPSetUp_GMRES (gmres.c:73) >>> ==13600== by 0x69E6FE: KSPSetUp (itfunc.c:239) >>> ==13600== by 0x6A090C: KSPSolve (itfunc.c:402) >>> ==13600== by 0x44E009: main (solveTest.c:61) >>> >>> The weird thing is that the it gives correct result, so zgesvd works >>> fine. >>> >>> And also running this program with 10 iterations in valgrind doesn't >>> produce error. The low above is with 100 iterations. >>> Without valgrind the error is always there. >>> >>> -- >>> Regards, >>> Alexander >>> >>> > > -- > Regards, > Alexander > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] Error during KSPDestroy
by Alexander Grayver 07 May '12

07 May '12

On 06.05.2012 22:24, Barry Smith wrote: > Alexander, > > I cannot reproduce this on my mac with 3 different blas/lapack. Barry, I'm surprised. I ran it on my home PC with ubuntu and PETSc configured from scratch as following: --download-mpich --with-fortran-interfaces=1 --download-scalapack --download-blacs --with-scalar-type=complex --download-blas-lapack --with-precision=double And it's still there. Please note that all my numbers are complex. > Could you please run the case below but with --download-f-blas-lapack (you forgot the -f last time)? Send us the valgrind results. This will tell use the exact line number in dlasq3() that is triggering the bad read. I did: ./configure --with-petsc-arch=openmpi-intel-complex-debug-c --download-scalapack --download-blacs --download-f-blas-lapack --with-precision=double --with-scalar-type=complex And then valgrind program. The first message from log: ==27656== Invalid write of size 8 ==27656== at 0x15A8E9E: dlasq2_ (dlasq2.f:215) ==27656== by 0x15A83A4: dlasq1_ (dlasq1.f:135) ==27656== by 0x158ACEC: zbdsqr_ (zbdsqr.f:225) ==27656== by 0x154EC27: zgesvd_ (zgesvd.f:2038) ==27656== by 0x695DD3: KSPComputeExtremeSingularValues_GMRES (gmreig.c:46) ==27656== by 0x69DD76: KSPComputeExtremeSingularValues (itfunc.c:47) ==27656== by 0x44E98C: main (solveTest.c:62) ==27656== Address 0xfad2d98 is 8 bytes before a block of size 832 alloc'd ==27656== at 0x4C25D66: memalign (vg_replace_malloc.c:694) ==27656== by 0x4B642B: PetscMallocAlign (mal.c:30) ==27656== by 0x687775: KSPSetUp_GMRES (gmres.c:73) ==27656== by 0x69FE4A: KSPSetUp (itfunc.c:239) ==27656== by 0x6A2058: KSPSolve (itfunc.c:402) ==27656== by 0x44E969: main (solveTest.c:61) Please find full log attached. > Barry > > > On May 6, 2012, at 9:16 AM, Alexander Grayver wrote: > >> On 06.05.2012 15:34, Matthew Knepley wrote: >>> On Sun, May 6, 2012 at 9:24 AM, Alexander Grayver<agrayver(a)gfz-potsdam.de> wrote: >>> Hm, valgrind gives a lot of output like that (see full log in previous message): >>> >>> Can you run this with --download-f-blas-lapack? This sounds much more like an MKL bug. >> I did: >> --download-scalapack --download-blacs --download-blas-lapack --with-precision=double --with-scalar-type=complex >> >> The error is still there. I checked "ldd solveTest", mkl is not used for sure. This is not an MKL problem I guess: >> >> ==13600== Invalid read of size 8 >> ==13600== at 0x58636AF: dlasq3_ (in /usr/local/lib/liblapack.so.3.2.2) >> ==13600== by 0x5862C84: dlasq2_ (in /usr/local/lib/liblapack.so.3.2.2) >> ==13600== by 0x5861F2C: dlasq1_ (in /usr/local/lib/liblapack.so.3.2.2) >> ==13600== by 0x571A479: zbdsqr_ (in /usr/local/lib/liblapack.so.3.2.2) >> ==13600== by 0x57466A7: zgesvd_ (in /usr/local/lib/liblapack.so.3.2.2) >> ==13600== by 0x694687: KSPComputeExtremeSingularValues_GMRES (gmreig.c:46) >> ==13600== by 0x69C62A: KSPComputeExtremeSingularValues (itfunc.c:47) >> ==13600== by 0x44E02C: main (solveTest.c:62) >> ==13600== Address 0x10826b90 is 16 bytes before a block of size 832 alloc'd >> ==13600== at 0x4C25D66: memalign (vg_replace_malloc.c:694) >> ==13600== by 0x4B5ACB: PetscMallocAlign (mal.c:30) >> ==13600== by 0x686181: KSPSetUp_GMRES (gmres.c:73) >> ==13600== by 0x69E6FE: KSPSetUp (itfunc.c:239) >> ==13600== by 0x6A090C: KSPSolve (itfunc.c:402) >> ==13600== by 0x44E009: main (solveTest.c:61) >> >> The weird thing is that the it gives correct result, so zgesvd works fine. >> >> And also running this program with 10 iterations in valgrind doesn't produce error. The low above is with 100 iterations. >> Without valgrind the error is always there. >> >> -- >> Regards, >> Alexander >> -- Regards, Alexander

1 0

Re: [petsc-dev] DM RefineLevel and CoarsenLevel
by Jed Brown 06 May '12

06 May '12

The associated ugly hackery is here. I'd welcome suggestions for a better way to do this. http://petsc.cs.iit.edu/petsc/petsc-dev/rev/9943253a5ac2 On Sun, May 6, 2012 at 3:57 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > Now the next round: > > For semi-coarsening, we used to have stuff like -da_refine_hierarchy_x > 1,1,3 -da_refine_hierarchy_y 2,2,1 -da_refine_hierarchy_z 2,2,1. Two > changes make this harder now: > > 1. You essentially got rid of DMRefineHierarchy (it's not called any > more), so each call to DMRefine and DMCoarsen have to figure out where they > are. > > 2. Since the coarse DMs are not reused by PCMG, but instead created again > using DMCoarsen, we have to figure out how to reverse the refinement > process so that the same coarse grids get reconstructed again. > > I added a DMRefineHook so that we can port data the other way and I > modified DMCoarsen_DA and DMRefine_DA to not call DMDACreate{1,2,3}d > because it eagerly calls DMSetFromOptions before we can set the > refinement/coarsen level. Unless someone stops me, I'm also going to add > coarsen_{x,y,z} fields to DM_DA because the refinement ratio may have > nothing to do with the coarsening ratio. > > I have no idea how to expose semi-coarsening through a C API other than to > hold the refinement/coarsening path arrays in each DM_DA so that > refinement/coarsening steps can be retraced. >

1 0

Re: [petsc-dev] DM RefineLevel and CoarsenLevel
by Jed Brown 06 May '12

06 May '12

http://petsc.cs.iit.edu/petsc/petsc-dev/rev/aa929236ce51 On Sun, May 6, 2012 at 1:57 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Fine > > On May 6, 2012, at 2:44 PM, Jed Brown wrote: > > > Should the refinement level be copied over by DMCoarsen (and the coarsen > level be copied by DMRefine)? > > > > It's useful for diagnostics to be able to define a universal level. If I > use PCMG and -snes_grid_sequence, there is effectively a sequence like > > > > DMCreate(,&dm0); // r=0,c=0 > > DMRefine(dm0,&dmf); // r=1,c=0 > > DMCoarsen(dmf,&dmc); // r=0,c=1 > > > > > > I would like a way to identify dmc as being on the "same level" as dm0. > >

1 0

PetscViewerGetSubcomm_ASCII() broken?
by Dmitry Karpeev 06 May '12

06 May '12

More accurately, the PetscViewerASCII subcomm viewer mechanism appears broken. I'm ready to push a fix, but since this is a heavily used and sensitive part of the code I wanted to make sure I'm not missing anything this close to a release. The problem seems to start here: http://petsc.cs.iit.edu/petsc/petsc-dev/file/dc597ff7b29f/src/sys/viewer/im… In particular, 854 <http://petsc.cs.iit.edu/petsc/petsc-dev/file/dc597ff7b29f/src/sys/viewer/im…> (*outviewer)->ops->destroy = PetscViewerDestroy_ASCII_Singleton; 855 <http://petsc.cs.iit.edu/petsc/petsc-dev/file/dc597ff7b29f/src/sys/viewer/im…> /* following might not be correct??? */ 856 <http://petsc.cs.iit.edu/petsc/petsc-dev/file/dc597ff7b29f/src/sys/viewer/im…> if (rank) { 857 <http://petsc.cs.iit.edu/petsc/petsc-dev/file/dc597ff7b29f/src/sys/viewer/im…> (*outviewer)->ops->flush = 0; 858 <http://petsc.cs.iit.edu/petsc/petsc-dev/file/dc597ff7b29f/src/sys/viewer/im…> } else { 859 <http://petsc.cs.iit.edu/petsc/petsc-dev/file/dc597ff7b29f/src/sys/viewer/im…> (*outviewer)->ops->flush = PetscViewerFlush_ASCII_Singleton_0; 860 <http://petsc.cs.iit.edu/petsc/petsc-dev/file/dc597ff7b29f/src/sys/viewer/im…> } I totally agree with the comment in the above snippet. In particular, if rank 0 of the original comm isn't in the subcomm, no flushing will occur. And if rank 0 makes it into the subcomm, it will be the only proc writing to the file. I actually experience both of these scenarios when viewing GASM subdomain ISs, which can live on arbitrary subcomms (including subcomm==comm). The fix here seems to be relatively simple -- just leave the original function pointer in the vtable. There are other things to navigate around -- for example, we can't just copy the fd, since only the original rank 0 has it, and we have to be careful about reopening the file when subcomm==comm, but those are easily taken care of. I've run some tests, and it seems to work okay. I'm sure it will be caught in the nighlies, if this commit breaks something. Otherwise, as currently implemented PetscViewerGetSubcomm appears to be useless (unless subcomm=PETSC_COMM_SELF; by the way, why don't we just use PetscViewerGetSubcomm(view, PETSC_COMM_SELF, &sviewer) instead of the odd-sounding PetscViewerGetSingleton(view, &sviewer)?) Does anybody object to my pushing a fix to this? Dmitry.

1 0

Re: [petsc-dev] [petsc-users] petsc-users Digest, Vol 38, Issue 41
by Jed Brown 06 May '12

06 May '12

On Sun, May 6, 2012 at 6:37 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > Perhaps this should be a property of PCMG, if in the one level case it > is a direct solve or an iterative. Then when you create the PCMG for grid > sequencing you call something like PCMGSetOneLevelDefaultDirectSolver() or > some similar ugly name. > PCMGSetGridSequencingCoarseLevel() ?

1 0

Re: [petsc-dev] Attaching a near null space to an IS
by Matthew Knepley 06 May '12

06 May '12

On Sun, May 6, 2012 at 8:34 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > I think it belongs in the DM (or the matrix if it was put in the matrix). > When you select a subpart of the DM that new DM should contain the near > null space for the subpart. The DM associated with the field defining IS > is where all properties should go, not the IS itself. Of course, we really > have not yet solidified the getting of the new DM for the field yet, maybe > after the release. I do not necessarily disagree in principle, but right now we don;t give smaller DMs, we give ISes. So I think my proposal is the one that works now. Second, I think we should be very careful using opaque objects for this. I consider this a failure when I did it for parts of the system for preconditioning, which was later changed to ISes in FieldSplit. Talking to PETSc, the Lord of Linear Algebra, in sets of integers is the right thing to do. It allows other people to stick their own stuff in place of ours easily, without junking it up with a lots of wrapper objects. Matt > > Barry > > On May 6, 2012, at 6:30 PM, Matthew Knepley wrote: > > > On Sun, May 6, 2012 at 7:15 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > > Matt introduced this concept, he says the IS is a better place to attach > things. > > > > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/2ad289ac99e0 > > > > I don't understand why the IS is better (because it's mostly > immutable?). I'm worried that putting it there is going to be fragile > because the near null space is not a property of an IS at all. > > > > Its not a property of your matrix either, or you would not need me to > tell you. Its a property of the operator. The operator > > is defined by the DM. The near null space is actually a property of a > suboperator, defined by the DM using a field (we are > > not allowing arbitrary divisions). The representation of a field in > PETSc is an IS ( > http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/DM/DMCreateFieldIS.…) > so it makes sense to attach field information to the IS. Moreover, it > > makes a hell of a lat more sense to attach an auxiliary operator (like > L_p) to this IS than to a matrix. > > > > Furthermore, this scheme is completely workable in a nested context. The > user can specify the IS, or pull out the DM IS > > and play with it, without a bunch of cumbersome copies hanging around > that we do not want and can't destroy. That is what > > would happen with persistent submatrices. Lastly, I am running this for > PyLith and it works great. > > > > Matt > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0