Re: [petsc-users] [EXTERNAL] Re: PCFIELDSPLIT with BoomerAMG preconditioning on a Schur Complement

27 Apr 2021

      Hello everyone,

Further to what Qi said, one main difficulty we have when we try to solve the second-order problem using dmstag and AMS is that AMS requires the coordinates vector and the discrete gradient operator.

However, in DMStag, our dm has zero degrees of freedom per vertex (dof0 = 0). And that causes us trouble because DMStag won't allow us to get vertex coordinates or define that discrete gradient matrix which is rectangular and takes as input a vertex vector and returns as ouput an edge vector.

For example, this part of the code sets the gradient matrix coefficients for the down left edge of a cell (er,ephi,ez):

/*-------------------------------------------------------------*/

row.i = er ; row.j = ephi ; row.k = ez ; row.loc = DMSTAG_DOWN_LEFT; row.c = 0; nEntries = 2;

col[0].i = er; col[0].j = ephi; col[0].k = ez; col[0].loc = DMSTAG_FRONT_DOWN_LEFT; col[0].c = 0; valG[0] = 1.0 / edgelength;

col[1].i = er; col[1].j = ephi; col[1].k = ez; col[1].loc = DMSTAG_BACK_DOWN_LEFT; col[1].c = 0; valG[1] = - 1.0 / edgelength;

        DMStagMatSetValuesStencil(da,G,1,&row,nEntries,col,valG,INSERT_VALUES);

/*-------------------------------------------------------------*/

Upon execution of the code, DMStagMatSetValuesStencil() would generate the following error:

[0]PETSC ERROR: Argument out of range

[0]PETSC ERROR: Location DMSTAG_BACK_UP_LEFT has no dof attached

What should we do in this case?

Many thanks for your help.

Best regards,

Zakariae

________________________________
From: Tang, Qi
Sent: Thursday, April 22, 2021 12:46:05 PM
To: Barry Smith; Matthew Knepley
Cc: Jorti, Zakariae; [email protected]; Tang, Xianzhu; Wimmer, Golo Albert
Subject: Re: [EXTERNAL] Re: [petsc-users] PCFIELDSPLIT with BoomerAMG preconditioning on a Schur Complement

For our case, the Schur complement becomes a curl curl problem by designed. So we expect it is going to be scalable. We are still experimenting boomeramg, which seems be working, but would like to switch to AMS later if possible.

The question is, can we use AMS for dmstag where dofs in our case lives on edges? One thing we are not so sure is the discrete gradient operator. We were told by our local expert that it needs to be scaled properly instead of 1 and -1. We are wondering if there are some examples using AMS beyond dmda, maybe in firedrake or dmstag? (Actually, in firedrake, we recently got AMS working with the lowest order RT space for a different project). Any example along the line would be useful.

Thanks,

Qi

________________________________
From: Barry Smith <[email protected]>
Sent: Thursday, April 22, 2021 12:12:56 PM
To: Matthew Knepley
Cc: Jorti, Zakariae; Tang, Qi; [email protected]; Tang, Xianzhu
Subject: [EXTERNAL] Re: [petsc-users] PCFIELDSPLIT with BoomerAMG preconditioning on a Schur Complement

  Schur complements are generally dense, unless your original matrix has specific structure that you know of, it is unlikely hypre AMG will work well on the Schur complement.

  Barry

On Apr 20, 2021, at 5:26 PM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote:

On Tue, Apr 20, 2021 at 4:33 PM Jorti, Zakariae via petsc-users <[email protected]<mailto:[email protected]>> wrote:

Hello,

I have a block matrix A,

[A_{00}   A_{01}]

[A_{10}   A_{11}] ,

and am trying to use PCFIELDSPLIT as a preconditioner on it. The preconditioner has this shape:

[inv(A_{00})       0     ]

[  0                -ksp(S)] ;

where S is the Schur Complement matrix:

S = A_{11} - A_{10} * inv(A_{00}) * A_{01},

that I get by calling:

MatGetSchurComplement(A, isrow0, iscol0, isrow1, iscol1, MAT_INITIAL_MATRIX, &S, MAT_SCHUR_COMPLEMENT_AINV_DIAG, MAT_IGNORE_MATRIX, NULL);

I would like to set BoomerAMG preconditioner for ksp(S). So, I am calling:

PCFieldSplitGetSubKSP(pc, &n, &subksp);

KSPSetOperators(subksp[1], S, S);

KSPGetPC(subksp[1], subpc);

PCSetType(subpc,"boomeramg");

but I end up getting the following error:

[0]PETSC ERROR: No support for this operation for this object type

[0]PETSC ERROR: Mat type schurcomplement

[0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.

[0]PETSC ERROR: Petsc Release Version 3.14.2, Dec 03, 2020

[0]PETSC ERROR: ./mimeticcurleuler2 on a arch-linux-c-debug named nid00140 by zjorti Tue Apr 20 12:23:34 2021

[0]PETSC ERROR: Configure options CC=cc CXX=CC FC=ftn COPTFLAGS=-g CXXOPTFLAGS=-g FOPTFLAGS=-g --download-hypre --with-debugging=1 --with-cxx-dialect=C++11

[0]PETSC ERROR: #1 MatGetRow() line 561 in /global/u1/z/zjorti/Software/petsc-3.14.2/src/mat/interface/matrix.c

[0]PETSC ERROR: #2 MatConvert_Basic() line 53 in /global/u1/z/zjorti/Software/petsc-3.14.2/src/mat/utils/convert.c

[0]PETSC ERROR: #3 MatConvert() line 4254 in /global/u1/z/zjorti/Software/petsc-3.14.2/src/mat/interface/matrix.c

[0]PETSC ERROR: #4 PCSetUp_HYPRE() line 231 in /global/u1/z/zjorti/Software/petsc-3.14.2/src/ksp/pc/impls/hypre/hypre.c

[0]PETSC ERROR: #5 PCSetUp() line 1009 in /global/u1/z/zjorti/Software/petsc-3.14.2/src/ksp/pc/interface/precon.c

[0]PETSC ERROR: #6 KSPSetUp() line 406 in /global/u1/z/zjorti/Software/petsc-3.14.2/src/ksp/ksp/interface/itfunc.c

[0]PETSC ERROR: #7 KSPSolve_Private() line 658 in /global/u1/z/zjorti/Software/petsc-3.14.2/src/ksp/ksp/interface/itfunc.c

[0]PETSC ERROR: #8 KSPSolve() line 889 in /global/u1/z/zjorti/Software/petsc-3.14.2/src/ksp/ksp/interface/itfunc.c

[0]PETSC ERROR: #9 PCApply_FieldSplit_Schur() line 1133 in /global/u1/z/zjorti/Software/petsc-3.14.2/src/ksp/pc/impls/fieldsplit/fieldsplit.c

[0]PETSC ERROR: #10 PCApply() line 444 in /global/u1/z/zjorti/Software/petsc-3.14.2/src/ksp/pc/interface/precon.c

[0]PETSC ERROR: #11 KSP_PCApply() line 283 in /global/u1/z/zjorti/Software/petsc-3.14.2/include/petsc/private/kspimpl.h

[0]PETSC ERROR: #12 KSPInitialResidual() line 65 in /global/u1/z/zjorti/Software/petsc-3.14.2/src/ksp/ksp/interface/itres.c

[0]PETSC ERROR: #13 KSPSolve_GMRES() line 245 in /global/u1/z/zjorti/Software/petsc-3.14.2/src/ksp/ksp/impls/gmres/gmres.c

[0]PETSC ERROR: #14 KSPSolve_Private() line 719 in /global/u1/z/zjorti/Software/petsc-3.14.2/src/ksp/ksp/interface/itfunc.c

[0]PETSC ERROR: #15 KSPSolve() line 889 in /global/u1/z/zjorti/Software/petsc-3.14.2/src/ksp/ksp/interface/itfunc.c

[0]PETSC ERROR: #16 SNESSolve_KSPONLY() line 51 in /global/u1/z/zjorti/Software/petsc-3.14.2/src/snes/impls/ksponly/ksponly.c

[0]PETSC ERROR: #17 SNESSolve() line 4569 in /global/u1/z/zjorti/Software/petsc-3.14.2/src/snes/interface/snes.c

[0]PETSC ERROR: #18 TSTheta_SNESSolve() line 185 in /global/u1/z/zjorti/Software/petsc-3.14.2/src/ts/impls/implicit/theta/theta.c

[0]PETSC ERROR: #19 TSStep_Theta() line 223 in /global/u1/z/zjorti/Software/petsc-3.14.2/src/ts/impls/implicit/theta/theta.c

[0]PETSC ERROR: #20 TSStep() line 3757 in /global/u1/z/zjorti/Software/petsc-3.14.2/src/ts/interface/ts.c

[0]PETSC ERROR: #21 TSSolve() line 4154 in /global/u1/z/zjorti/Software/petsc-3.14.2/src/ts/interface/ts.c

Am I doing something wrong here? How can I fix this?

It really is best to do this with options. It will take a lot of programming to replicate what you can do.

You are trying to precondition an implicit matrix (S) with Boomeramg, which needs explicit entries. You can tell PCFIELDSPLIT to build an
explicit matrix for the Schur complement:

  https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplit...

  Thanks,

     Matt

Many thanks.

Best regards,

Zakariae Jorti

--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>