February 2017 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] [petsc-users] Multigrid with defect correction
by Jed Brown 25 Feb '17

25 Feb '17

Barry Smith <bsmith(a)mcs.anl.gov> writes: > Do you think this is a reasonable approach or am I missing > something fundamental? I am assuming generally for the "higher > order" DM the Mat it returns is a MATSHELL or a new matrix class > built on "tensor contractions" and that kind of nonsense. I don't > want to do all the coding and then have it turn out that it is > totally useless for CEED etc. Well, this is exactly what we do in pTatin. I would ask, why just two discretizations? I've always thought a better interface would be for Mats (and any other operators/functions) to have optional approximations or supplementary data attached. We can do this with PetscObjectCompose, but that's hard to work with in a structured way. Anyway, I think I would rather just have the Amat with ability to attach one or more Pmats.

1 0

git 'maint' users - reset branch
by Satish Balay 23 Feb '17

23 Feb '17

PETSc git repo users (and developers), We had a bad 'push' to maint branch - that is reverted and fixed now. However if you did a pull of this branch [say between 12:50pm and 1:05pm CST] - and received the commit 'e625f44' in your maint branch - please delete and recreate your maint branch. i.e <in maint branch> git fetch git reset --hard origin/maint Satish

1 0

Re: [petsc-dev] Scatter a single local value to multiple global values
by Chris Eldred 22 Feb '17

22 Feb '17

Ok, it is apparent I will have to think about this some more. Thanks for the help! -Chris On Wed, Feb 22, 2017 at 6:36 PM, Jed Brown <jed(a)jedbrown.org> wrote: > Chris Eldred <chris.eldred(a)gmail.com> writes: > >> Hey Jed, >> >> That makes sense now, thanks! It sounds like the correct way to do things is: >> >> For matrices: >> 1) for each set of coupled dofs, select 1 to be the master and modify >> the local to global maps such that all the local indices that >> previously pointed to the slave dofs now point to the master dof >> -> assembly doesn't produce anything for the slave rows >> 2) fill the rows for the slave dofs with a -1 in the master column and >> 1 in the slave column (since we are just enforcing slave = master). > > It's better to not store slave dofs in the global matrix. Just use the > local-to-global mapping so that you get it in your local spaces. What > you describe makes the matrix nonsymmetric and possibly ill-conditioned > (depends on scaling of the other dofs). > >> For residuals, just do 1) and then set slave dofs equal to the master dof. > > What you describe above with +1 and -1 requires a right-hand side of 0. -- Chris Eldred https://www.math.univ-paris13.fr/~eldred/ Postdoctoral Fellow, LAGA, University of Paris 13 PhD, Atmospheric Science, Colorado State University, 2015 DOE Computational Science Graduate Fellow (Alumni) B.S. Applied Computational Physics, Carnegie Mellon University, 2009 chris.eldred(a)gmail.com

1 0

Re: [petsc-dev] Compilation with ESSL SMP
by Pierre Jolivet 22 Feb '17

22 Feb '17

This seems to do the trick, thank you all for your comments. PS: if you know of an easy way like MKL_VERBOSE=1 to make sure that the threaded ESSL functions are indeed being used, I’d appreciate a pointer. Otherwise, I’ll just have to trust the PETSc build system and IBM compilers. > On Feb 21, 2017, at 4:59 PM, Satish Balay <balay(a)mcs.anl.gov> wrote: > > Ok I see --with-blas-lapack-lib was changed to --with-blaslapack-lib in petsc master. > > I pushed a fix to master so that '--with-blas-lapack-lib' is also accepted by configure [for backward compatibility]. > > Satish > > On Tue, 21 Feb 2017, Satish Balay wrote: > >> >> For one the option to use is --with-blaslapack-lib [not --with-blas-lapack-lib] >> >> And you need to use the following notation: (as mentioned previously) >> >> --with-blaslapack-lib="-lessl -llapack -lessl" i.e >> >> --with-blaslapack-lib="/bgsys/ibm_essl/prod/opt/ibmmath/essl/5.1/lib64/libesslsmpbg.a /bglocal/cn/pub/LAPACK/3.7.0/lib/liblapack.a /bglocal/cn/pub/LAPACK/3.7.0/lib/liblapack.a" >> >> If you still have failures - please resend configure.log >> >> Satish >> >> On Tue, 21 Feb 2017, Pierre Jolivet wrote: >> >>> On Mon, 20 Feb 2017 09:19:32 -0700, Jed Brown wrote: >>>> Pierre Jolivet <Pierre.Jolivet(a)enseeiht.fr> writes: >>>> >>>>> Hello, >>>>> It looks like the configure does not directly handle libesslsmpbg >>>> >>>> What did you pass to configure? LIBS=-lesslsmpbg or similar? >>> >>> Here are my options: >>> --known-sdot-returns-double=0 --known-snrm2-returns-double=0 >>> --known-has-attribute-aligned=1 --CXXOPTFLAGS="-O3 -qnohot -qsimd=noauto >>> -qsmp=omp:noauto" --COPTFLAGS="-O3 -qnohot -qsimd=noauto -qsmp=omp:noauto" >>> --FOPTFLAGS="-O3 -qnohot -qsimd=noauto -qsmp=omp:noauto" >>> --prefix=build/complex --known-bits-per-byte=8 --known-level1-dcache-assoc=0 >>> --known-level1-dcache-linesize=32 --known-level1-dcache-size=32768 >>> --known-memcmp-ok=1 --known-mpi-c-double-complex=1 --known-mpi-int64_t=1 >>> --known-mpi-long-double=1 --known-mpi-shared-libraries=0 >>> --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-sizeof-char=1 >>> --known-sizeof-double=8 --known-sizeof-float=4 --known-sizeof-int=4 >>> --known-sizeof-long-long=8 --known-sizeof-long=8 --known-sizeof-short=2 >>> --known-sizeof-size_t=8 --known-sizeof-void-p=8 --with-batch >>> --with-blas-lapack-lib="[/bgsys/ibm_essl/prod/opt/ibmmath/essl/5.1/lib64/libesslsmpbg.a,/bglocal/cn/pub/LAPACK/3.7.0/lib/liblapack.a]" >>> --with-scalapack-lib="[/bglocal/cn/pub/ScaLAPACK/2.0.2/lib/libscalapack.a]" >>> --with-cc=mpixlc_r --with-clanguage=c --with-cpp="mpixlc_r -C -E" >>> --with-cxx=mpixlcxx_r --with-debugging=0 --with-endian=big >>> --with-fc=mpixlf90_r --with-fortran --with-make=gmake --with-mpi=yes >>> --with-scalar-type=complex --with-x=0 >>> --download-sowing=../externalpackages/sowing-1.1.23-p1.tar.gz >>> --download-metis=../externalpackages/metis-5.1.0-p3.tar.gz >>> --download-mumps=../externalpackages/MUMPS_5.0.2.tar.gz >>> --download-parmetis=../externalpackages/parmetis-4.0.3-p3.tar.gz >>> --download-ptscotch=../externalpackages/scotch_6.0.4-p1.tar.gz >>> --download-pastix=../externalpackages/pastix_5.2.3.tar.bz2 >>> PETSC_ARCH=arch-linux2-c-opt >>> >>> But once the configure stage is complete, the following lib is printed: >>> BLAS/LAPACK: -lesslbg >>> >>> Is it possible to force the use of libesslsmpbg? Do I need to force that >>> through the LIB argument as Jed suggested? >>> >>> Thanks for all the input. >>> >>>>> instead of libesslbg. On the other hand, it handles both >>>>> libmkl_intel_thread and libmkl_sequential. >>>>> Is there a reason for such a limitation? (I'd like to have direct >>>>> solvers linked with threaded BLAS). >>>> >>>> People do this by accident or without understanding the implications far >>>> more often than they do it intentionally. >>> >>> >>> >> >> >

1 0

Re: [petsc-dev] Scatter a single local value to multiple global values
by Jed Brown 22 Feb '17

22 Feb '17

Chris Eldred <chris.eldred(a)gmail.com> writes: > Hey Jed, > > That makes sense now, thanks! It sounds like the correct way to do things is: > > For matrices: > 1) for each set of coupled dofs, select 1 to be the master and modify > the local to global maps such that all the local indices that > previously pointed to the slave dofs now point to the master dof > -> assembly doesn't produce anything for the slave rows > 2) fill the rows for the slave dofs with a -1 in the master column and > 1 in the slave column (since we are just enforcing slave = master). It's better to not store slave dofs in the global matrix. Just use the local-to-global mapping so that you get it in your local spaces. What you describe makes the matrix nonsymmetric and possibly ill-conditioned (depends on scaling of the other dofs). > For residuals, just do 1) and then set slave dofs equal to the master dof. What you describe above with +1 and -1 requires a right-hand side of 0.

1 0

Re: [petsc-dev] Scatter a single local value to multiple global values
by Chris Eldred 22 Feb '17

22 Feb '17

Hey Jed, That makes sense now, thanks! It sounds like the correct way to do things is: For matrices: 1) for each set of coupled dofs, select 1 to be the master and modify the local to global maps such that all the local indices that previously pointed to the slave dofs now point to the master dof -> assembly doesn't produce anything for the slave rows 2) fill the rows for the slave dofs with a -1 in the master column and 1 in the slave column (since we are just enforcing slave = master). For residuals, just do 1) and then set slave dofs equal to the master dof. The star forest write up is exactly what I was looking for, thanks! Regards, Chris On Wed, Feb 22, 2017 at 5:37 PM, Jed Brown <jed(a)jedbrown.org> wrote: > "Kong, Fande" <fande.kong(a)inl.gov> writes: >> How about the inverse of SF? The original leaf becomes a new root, and the >> original leaf will map to multiple new leafs? > > Chris has multiple local indices mapping to the same global index (this > is the usual local-to-global map). He's eager to build a singular > matrix and irritated that the tooling intentionally makes that harder to > do by accident. ;-) -- Chris Eldred https://www.math.univ-paris13.fr/~eldred/ Postdoctoral Fellow, LAGA, University of Paris 13 PhD, Atmospheric Science, Colorado State University, 2015 DOE Computational Science Graduate Fellow (Alumni) B.S. Applied Computational Physics, Carnegie Mellon University, 2009 chris.eldred(a)gmail.com

1 0

Re: [petsc-dev] Scatter a single local value to multiple global values
by Jed Brown 22 Feb '17

22 Feb '17

"Kong, Fande" <fande.kong(a)inl.gov> writes: > How about the inverse of SF? The original leaf becomes a new root, and the > original leaf will map to multiple new leafs? Chris has multiple local indices mapping to the same global index (this is the usual local-to-global map). He's eager to build a singular matrix and irritated that the tooling intentionally makes that harder to do by accident. ;-)

1 0

Re: [petsc-dev] Scatter a single local value to multiple global values
by Kong, Fande 22 Feb '17

22 Feb '17

On Wed, Feb 22, 2017 at 9:24 AM, Jed Brown <jed(a)jedbrown.org> wrote: > Chris Eldred <chris.eldred(a)gmail.com> writes: > > > Hey PETSc-dev, > > > > Is it possible to create a LocalToGlobal mapping (or another PETSc > > construct) such that a single local value is associated with multiple > > global values? > > > > For example, let's say I am in 1D with a DMDA with 4 elements using > > Ghosted boundaries. > > The global space looks like: > > 1 - 2 - 3 - 4 > > and the local space looks like: > > 1 - 2 - 3 - 4 - 5 - 6 > > and the default local to global mapping is: > > 0 -> -1 > > 1 -> 0 > > 2 -> 1 > > 3 -> 2 > > 4 -> 3 > > 5 -> -1 > > Can this be modified to something like: > > 0 -> -1 > > 1 -> 0 > > 2 -> 1 > > 3 -> 2 > > 4 -> 3 , 2 > > 5 -> -1 > > where now the value in index 4 of the local vector gets scattered to > > both index 2 and 3 in the global vector when using VecScatter with the > > LocalToGlobal mapping set to the above? > > > > What about when using MatSetValuesLocal? Is it possible to have a > > single entry in the local matrix get put into multiple locations in > > the global matrix? When the row and column LocalToGlobal maps are > > different? > > > > If so, are there any example or writeups of how to do either of these? > > No, you can have different local indices map to the same global index, > but you can't have one local index map to multiple global indices. If > such a semantic were supported, what would the result of a > Global-to-Local update even mean? > > > It's seems like star forests are the way to do this, but I don't > > really understand how they work. Is there a writeup for them? > > Slightly dated, but mostly accurate. > > https://jedbrown.org/files/StarForest.pdf > > In SF terminology, you can't have a leaf map to multiple roots. > How about the inverse of SF? The original leaf becomes a new root, and the original leaf will map to multiple new leafs? > > > The context for this question is a multipatch finite element code, > > where each patch is treated using a DMDA and we would like to couple > > degrees of freedom along patch boundaries, such that when the > > contribution of the elements along a boundary to the global matrix are > > computed the result goes into the rows for the dofs on each patch (and > > similarly for residual calculations). > > That's a good way to produce a singular global matrix. Either eliminate > the duplicated degrees of freedom (this is the standard method, slave > dofs are written as a linear combination of master dofs) or introduce a > Lagrange multiplier to enforce continuity. >

1 0

Re: [petsc-dev] Scatter a single local value to multiple global values
by Jed Brown 22 Feb '17

22 Feb '17

Chris Eldred <chris.eldred(a)gmail.com> writes: > Hey PETSc-dev, > > Is it possible to create a LocalToGlobal mapping (or another PETSc > construct) such that a single local value is associated with multiple > global values? > > For example, let's say I am in 1D with a DMDA with 4 elements using > Ghosted boundaries. > The global space looks like: > 1 - 2 - 3 - 4 > and the local space looks like: > 1 - 2 - 3 - 4 - 5 - 6 > and the default local to global mapping is: > 0 -> -1 > 1 -> 0 > 2 -> 1 > 3 -> 2 > 4 -> 3 > 5 -> -1 > Can this be modified to something like: > 0 -> -1 > 1 -> 0 > 2 -> 1 > 3 -> 2 > 4 -> 3 , 2 > 5 -> -1 > where now the value in index 4 of the local vector gets scattered to > both index 2 and 3 in the global vector when using VecScatter with the > LocalToGlobal mapping set to the above? > > What about when using MatSetValuesLocal? Is it possible to have a > single entry in the local matrix get put into multiple locations in > the global matrix? When the row and column LocalToGlobal maps are > different? > > If so, are there any example or writeups of how to do either of these? No, you can have different local indices map to the same global index, but you can't have one local index map to multiple global indices. If such a semantic were supported, what would the result of a Global-to-Local update even mean? > It's seems like star forests are the way to do this, but I don't > really understand how they work. Is there a writeup for them? Slightly dated, but mostly accurate. https://jedbrown.org/files/StarForest.pdf In SF terminology, you can't have a leaf map to multiple roots. > The context for this question is a multipatch finite element code, > where each patch is treated using a DMDA and we would like to couple > degrees of freedom along patch boundaries, such that when the > contribution of the elements along a boundary to the global matrix are > computed the result goes into the rows for the dofs on each patch (and > similarly for residual calculations). That's a good way to produce a singular global matrix. Either eliminate the duplicated degrees of freedom (this is the standard method, slave dofs are written as a linear combination of master dofs) or introduce a Lagrange multiplier to enforce continuity.

1 0

Scatter a single local value to multiple global values
by Chris Eldred 22 Feb '17

22 Feb '17

Hey PETSc-dev, Is it possible to create a LocalToGlobal mapping (or another PETSc construct) such that a single local value is associated with multiple global values? For example, let's say I am in 1D with a DMDA with 4 elements using Ghosted boundaries. The global space looks like: 1 - 2 - 3 - 4 and the local space looks like: 1 - 2 - 3 - 4 - 5 - 6 and the default local to global mapping is: 0 -> -1 1 -> 0 2 -> 1 3 -> 2 4 -> 3 5 -> -1 Can this be modified to something like: 0 -> -1 1 -> 0 2 -> 1 3 -> 2 4 -> 3 , 2 5 -> -1 where now the value in index 4 of the local vector gets scattered to both index 2 and 3 in the global vector when using VecScatter with the LocalToGlobal mapping set to the above? What about when using MatSetValuesLocal? Is it possible to have a single entry in the local matrix get put into multiple locations in the global matrix? When the row and column LocalToGlobal maps are different? If so, are there any example or writeups of how to do either of these? It's seems like star forests are the way to do this, but I don't really understand how they work. Is there a writeup for them? The context for this question is a multipatch finite element code, where each patch is treated using a DMDA and we would like to couple degrees of freedom along patch boundaries, such that when the contribution of the elements along a boundary to the global matrix are computed the result goes into the rows for the dofs on each patch (and similarly for residual calculations). Regards, Chris Eldred -- Chris Eldred https://www.math.univ-paris13.fr/~eldred/ Postdoctoral Fellow, LAGA, University of Paris 13 PhD, Atmospheric Science, Colorado State University, 2015 DOE Computational Science Graduate Fellow (Alumni) B.S. Applied Computational Physics, Carnegie Mellon University, 2009 chris.eldred(a)gmail.com

1 0