This gives [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Matrix must be set first The PC is from a SNESGetKSP, and I have called SNESSetJacobian before this. Thanks, Sander On Tue, Jun 30, 2015 at 2:09 PM, Matthew Knepley <[email protected]> wrote:

On Tue, Jun 30, 2015 at 11:43 AM, Sander Land <[email protected]> wrote:

...

I am trying to use the Schur complement preconditioner in petsc, but am encountering a null argument error calling PCFieldSplitGetSubKSP. This only happens on PC_COMPOSITE_SCHUR, the multiplicative/additive options do return a KSP array.

Error message:

[0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Null argument, when expecting valid pointer [0]PETSC ERROR: Null Object: Parameter # 1 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.0, Jun, 09, 2015 [0]PETSC ERROR: ./elecmech on a arch-linux2-cxx-debug [0]PETSC ERROR: Configure options --with-shared-libraries=1 --with-debugging=1 --download-openmpi=1 --with-clanguage=c++ --download-fblaslapack=1 --download-scalapack=1 --download-blacs=1 --download-suitesparse=1 --download-pastix=1 --download-superlu=1 --dowload-essl=1 --download-ptscotch=1 --download-mumps=1 --download-lusol=1 [0]PETSC ERROR: #1 MatSchurComplementGetKSP() line 317 in petsc-3.6/src/ksp/ksp/utils/schurm.c [0]PETSC ERROR: #2 PCFieldSplitGetSubKSP_FieldSplit_Schur() line 1264 in petsc-3.6/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #3 PCFieldSplitGetSubKSP() line 1668 in petsc-3.6/src/ksp/pc/impls/fieldsplit/fieldsplit.c

Code snippet:

ISCreateGeneral(PETSC_COMM_SELF,xp_dofs,&ii[0],PETSC_COPY_VALUES,&is_xp); ISCreateGeneral(PETSC_COMM_SELF,all_dofs - xp_dofs,&ii[xp_dofs],PETSC_COPY_VALUES,&is_wk);

PCSetType(pc,PCFIELDSPLIT); PCFieldSplitSetType(pc,PC_COMPOSITE_SCHUR); PCFieldSplitSetIS(pc,"xp",is_xp); PCFieldSplitSetIS(pc,"wk",is_wk); int n; KSP* ksps; PC subpc; PCFieldSplitGetSubKSP(pc,&n,&ksps);

ii here is simply an array with ii[i] = i. There is probably a better way to simply indicate two blocks of different size, but I couldn't find it.

It looks like the PC is not setup. Can you try calling PCSetUp(pc) before GetSubKSP()?

Thanks,

Matt

...

Thanks, Sander

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

Hi Matt, Doing things to the PC like this has worked for me for a long time. It just happened to be formerly KSPPREONLY/PCLU. Setting things like GMRES/PCNONE works as well with no errors, though it converges far too slowly. The actual preconditioner I am trying to form is one from recent literature and uses a slightly modified Schur complement split embedded in another Schur complement split with multiple solves of the inner system and very specific choices for sub-preconditioners. I don't think command line options will work very well here. I've since gone the way of just using PCSHELL and setting things up more explicitly. This seems to work, but i'm not quite sure how to set up boomeramg for one of the sub-problems. boomeramg is used on the matrix A derived from: MatGetSubMatrix(J,is_x ,is_x ,MAT_INITIAL_MATRIX,&ctxp->A); The degrees of freedom for J are xxxyyyzzzpw, with the size of xxxyyyzzz around 20k, p around 1k, and w 8 for my 'small' test problem. I can see that HYPRE_BoomerAMGSetNumFunctions takes its value from the matrix block size. This is 1 for J as it doesn't have a consistent block size. However, the number of functions for boomeramg should be 3. Changing the block size is not allowed by petsc, and there doesn't seem to be another way to reach HYPRE_BoomerAMGSetNumFunctions. What is the best way to handle this, and does it matter that the dofs are ordererd xxxyyyzzz rather than xyzxyzxyz ? Thanks, Sander On Fri, Jul 10, 2015 at 7:44 PM, Matthew Knepley <[email protected]> wrote:

On Tue, Jun 30, 2015 at 3:10 PM, Sander Land <[email protected]> wrote:

...

This gives [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Matrix must be set first

The PC is from a SNESGetKSP, and I have called SNESSetJacobian before this.

If this is embedded in a SNES, it is very messy to yank it out and do thing manually. However, if you set the matrix in the PC manually, it might work.

However, I would encourage you to try an alternate scheme and see if you think it is easier. We have started to use the DM object to hold this kind of information. For FieldSplit, I think it is easier to tell a DMShell about your field division, and then have it pass this on to PCFIELDSPLIT. So you would

a) Create a DMSHELL

b) Create a PetscSection and call DMSetDefaultSection()

c) Call SNESSetDM()

d) use command line options to configure the split -pc_fieldsplit_0_fields 0 -pc_fieldsplit_1_fields 1

Then everything should work, and can work in any embedded context. Does this make sense?

Thanks,

Matt

Thanks,

...

Sander

On Tue, Jun 30, 2015 at 2:09 PM, Matthew Knepley <[email protected]> wrote:

...

On Tue, Jun 30, 2015 at 11:43 AM, Sander Land <[email protected]> wrote:

...

I am trying to use the Schur complement preconditioner in petsc, but am encountering a null argument error calling PCFieldSplitGetSubKSP. This only happens on PC_COMPOSITE_SCHUR, the multiplicative/additive options do return a KSP array.

Error message:

[0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Null argument, when expecting valid pointer [0]PETSC ERROR: Null Object: Parameter # 1 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.0, Jun, 09, 2015 [0]PETSC ERROR: ./elecmech on a arch-linux2-cxx-debug [0]PETSC ERROR: Configure options --with-shared-libraries=1 --with-debugging=1 --download-openmpi=1 --with-clanguage=c++ --download-fblaslapack=1 --download-scalapack=1 --download-blacs=1 --download-suitesparse=1 --download-pastix=1 --download-superlu=1 --dowload-essl=1 --download-ptscotch=1 --download-mumps=1 --download-lusol=1 [0]PETSC ERROR: #1 MatSchurComplementGetKSP() line 317 in petsc-3.6/src/ksp/ksp/utils/schurm.c [0]PETSC ERROR: #2 PCFieldSplitGetSubKSP_FieldSplit_Schur() line 1264 in petsc-3.6/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #3 PCFieldSplitGetSubKSP() line 1668 in petsc-3.6/src/ksp/pc/impls/fieldsplit/fieldsplit.c

Code snippet:

ISCreateGeneral(PETSC_COMM_SELF,xp_dofs,&ii[0],PETSC_COPY_VALUES,&is_xp); ISCreateGeneral(PETSC_COMM_SELF,all_dofs - xp_dofs,&ii[xp_dofs],PETSC_COPY_VALUES,&is_wk);

PCSetType(pc,PCFIELDSPLIT); PCFieldSplitSetType(pc,PC_COMPOSITE_SCHUR); PCFieldSplitSetIS(pc,"xp",is_xp); PCFieldSplitSetIS(pc,"wk",is_wk); int n; KSP* ksps; PC subpc; PCFieldSplitGetSubKSP(pc,&n,&ksps);

ii here is simply an array with ii[i] = i. There is probably a better way to simply indicate two blocks of different size, but I couldn't find it.

It looks like the PC is not setup. Can you try calling PCSetUp(pc) before GetSubKSP()?

Thanks,

Matt

...

Thanks, Sander

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

On Jul 14, 2015, at 11:37 AM, Sander Land <[email protected]> wrote:

Hi Matt,

Doing things to the PC like this has worked for me for a long time. It just happened to be formerly KSPPREONLY/PCLU. Setting things like GMRES/PCNONE works as well with no errors, though it converges far too slowly.

The actual preconditioner I am trying to form is one from recent literature and uses a slightly modified Schur complement split embedded in another Schur complement split with multiple solves of the inner system and very specific choices for sub-preconditioners. I don't think command line options will work very well here.

I've since gone the way of just using PCSHELL and setting things up more explicitly. This seems to work, but i'm not quite sure how to set up boomeramg for one of the sub-problems.

boomeramg is used on the matrix A derived from: MatGetSubMatrix(J,is_x ,is_x ,MAT_INITIAL_MATRIX,&ctxp->A); The degrees of freedom for J are xxxyyyzzzpw, with the size of xxxyyyzzz around 20k, p around 1k, and w 8 for my 'small' test problem.

I can see that HYPRE_BoomerAMGSetNumFunctions takes its value from the matrix block size. This is 1 for J as it doesn't have a consistent block size. However, the number of functions for boomeramg should be 3. Changing the block size is not allowed by petsc, and there doesn't seem to be another way to reach HYPRE_BoomerAMGSetNumFunctions. What is the best way to handle this, and does it matter that the dofs are ordererd xxxyyyzzz rather than xyzxyzxyz ?

I don't think BoomerAMG supports xxxyyyzzz, I think it will just get very confused trying to do algebraic multigrid with this form. Read up on it but my guess is you need to rewrite the code that prepares the system to do the interlacing. Barry

Thanks, Sander

On Fri, Jul 10, 2015 at 7:44 PM, Matthew Knepley <[email protected]> wrote: On Tue, Jun 30, 2015 at 3:10 PM, Sander Land <[email protected]> wrote: This gives [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Matrix must be set first

The PC is from a SNESGetKSP, and I have called SNESSetJacobian before this.

If this is embedded in a SNES, it is very messy to yank it out and do thing manually. However, if you set the matrix in the PC manually, it might work.

However, I would encourage you to try an alternate scheme and see if you think it is easier. We have started to use the DM object to hold this kind of information. For FieldSplit, I think it is easier to tell a DMShell about your field division, and then have it pass this on to PCFIELDSPLIT. So you would

a) Create a DMSHELL

b) Create a PetscSection and call DMSetDefaultSection()

c) Call SNESSetDM()

d) use command line options to configure the split -pc_fieldsplit_0_fields 0 -pc_fieldsplit_1_fields 1

Then everything should work, and can work in any embedded context. Does this make sense?

Thanks,

Matt

Thanks, Sander

On Tue, Jun 30, 2015 at 2:09 PM, Matthew Knepley <[email protected]> wrote: On Tue, Jun 30, 2015 at 11:43 AM, Sander Land <[email protected]> wrote: I am trying to use the Schur complement preconditioner in petsc, but am encountering a null argument error calling PCFieldSplitGetSubKSP. This only happens on PC_COMPOSITE_SCHUR, the multiplicative/additive options do return a KSP array.

Error message:

[0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Null argument, when expecting valid pointer [0]PETSC ERROR: Null Object: Parameter # 1 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.0, Jun, 09, 2015 [0]PETSC ERROR: ./elecmech on a arch-linux2-cxx-debug [0]PETSC ERROR: Configure options --with-shared-libraries=1 --with-debugging=1 --download-openmpi=1 --with-clanguage=c++ --download-fblaslapack=1 --download-scalapack=1 --download-blacs=1 --download-suitesparse=1 --download-pastix=1 --download-superlu=1 --dowload-essl=1 --download-ptscotch=1 --download-mumps=1 --download-lusol=1 [0]PETSC ERROR: #1 MatSchurComplementGetKSP() line 317 in petsc-3.6/src/ksp/ksp/utils/schurm.c [0]PETSC ERROR: #2 PCFieldSplitGetSubKSP_FieldSplit_Schur() line 1264 in petsc-3.6/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #3 PCFieldSplitGetSubKSP() line 1668 in petsc-3.6/src/ksp/pc/impls/fieldsplit/fieldsplit.c

Code snippet:

ISCreateGeneral(PETSC_COMM_SELF,xp_dofs,&ii[0],PETSC_COPY_VALUES,&is_xp); ISCreateGeneral(PETSC_COMM_SELF,all_dofs - xp_dofs,&ii[xp_dofs],PETSC_COPY_VALUES,&is_wk);

PCSetType(pc,PCFIELDSPLIT); PCFieldSplitSetType(pc,PC_COMPOSITE_SCHUR); PCFieldSplitSetIS(pc,"xp",is_xp); PCFieldSplitSetIS(pc,"wk",is_wk); int n; KSP* ksps; PC subpc; PCFieldSplitGetSubKSP(pc,&n,&ksps);

ii here is simply an array with ii[i] = i. There is probably a better way to simply indicate two blocks of different size, but I couldn't find it.

It looks like the PC is not setup. Can you try calling PCSetUp(pc) before GetSubKSP()?

Thanks,

Matt

Thanks, Sander

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

On Jul 15, 2015, at 5:43 AM, Sander Land <[email protected]> wrote:

Hi Barry,

I have some structures set up that map dofs to their respective matrix indices, and it wasn't difficult to change them.

However, now this call: MatMultAdd(ctxp->B,xp,bx,xxtmp); is directed to "MatMultAdd_SeqAIJ_Inode" and gives a vector lock error at line 601 of mat/impls/aij/seq/inode.c ierr = VecGetArray(zz,&z); I think this is a bug and should be VecGetArrayRead as it doesn't need to be written to.

Oh yes, this is a bug. I have fixed it in maint, master and next

In the old xxxyyyzzz ordering this call went to MatMultAdd_SeqAIJ, which (via VecGetArrayPair) does use VecGetArrayRead. I'm not sure what is causing the different matrix formats.

When the values are interlaced the matrix has "identical nodes" (i-nodes) which means certain rows have identical nonzero structure with there neighbors and the AIJ matrix code automatically switches to the more efficient Inode operations. This is why you only saw the bug with interlacing.

I'm also still not sure at what point I can change the matrix block size to get the correct BoomerAMGSetNumFunctions.

Thanks, Sander

On Tue, Jul 14, 2015 at 6:34 PM, Barry Smith <[email protected]> wrote:

...

On Jul 14, 2015, at 11:37 AM, Sander Land <[email protected]> wrote:

Hi Matt,

Doing things to the PC like this has worked for me for a long time. It just happened to be formerly KSPPREONLY/PCLU. Setting things like GMRES/PCNONE works as well with no errors, though it converges far too slowly.

The actual preconditioner I am trying to form is one from recent literature and uses a slightly modified Schur complement split embedded in another Schur complement split with multiple solves of the inner system and very specific choices for sub-preconditioners. I don't think command line options will work very well here.

I've since gone the way of just using PCSHELL and setting things up more explicitly. This seems to work, but i'm not quite sure how to set up boomeramg for one of the sub-problems.

boomeramg is used on the matrix A derived from: MatGetSubMatrix(J,is_x ,is_x ,MAT_INITIAL_MATRIX,&ctxp->A); The degrees of freedom for J are xxxyyyzzzpw, with the size of xxxyyyzzz around 20k, p around 1k, and w 8 for my 'small' test problem.

I can see that HYPRE_BoomerAMGSetNumFunctions takes its value from the matrix block size. This is 1 for J as it doesn't have a consistent block size. However, the number of functions for boomeramg should be 3. Changing the block size is not allowed by petsc, and there doesn't seem to be another way to reach HYPRE_BoomerAMGSetNumFunctions. What is the best way to handle this, and does it matter that the dofs are ordererd xxxyyyzzz rather than xyzxyzxyz ?

I don't think BoomerAMG supports xxxyyyzzz, I think it will just get very confused trying to do algebraic multigrid with this form. Read up on it but my guess is you need to rewrite the code that prepares the system to do the interlacing.

Barry

...

Thanks, Sander

On Fri, Jul 10, 2015 at 7:44 PM, Matthew Knepley <[email protected]> wrote: On Tue, Jun 30, 2015 at 3:10 PM, Sander Land <[email protected]> wrote: This gives [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Matrix must be set first

The PC is from a SNESGetKSP, and I have called SNESSetJacobian before this.

If this is embedded in a SNES, it is very messy to yank it out and do thing manually. However, if you set the matrix in the PC manually, it might work.

However, I would encourage you to try an alternate scheme and see if you think it is easier. We have started to use the DM object to hold this kind of information. For FieldSplit, I think it is easier to tell a DMShell about your field division, and then have it pass this on to PCFIELDSPLIT. So you would

a) Create a DMSHELL

b) Create a PetscSection and call DMSetDefaultSection()

c) Call SNESSetDM()

d) use command line options to configure the split -pc_fieldsplit_0_fields 0 -pc_fieldsplit_1_fields 1

Then everything should work, and can work in any embedded context. Does this make sense?

Thanks,

Matt

Thanks, Sander

On Tue, Jun 30, 2015 at 2:09 PM, Matthew Knepley <[email protected]> wrote: On Tue, Jun 30, 2015 at 11:43 AM, Sander Land <[email protected]> wrote: I am trying to use the Schur complement preconditioner in petsc, but am encountering a null argument error calling PCFieldSplitGetSubKSP. This only happens on PC_COMPOSITE_SCHUR, the multiplicative/additive options do return a KSP array.

Error message:

[0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Null argument, when expecting valid pointer [0]PETSC ERROR: Null Object: Parameter # 1 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.0, Jun, 09, 2015 [0]PETSC ERROR: ./elecmech on a arch-linux2-cxx-debug [0]PETSC ERROR: Configure options --with-shared-libraries=1 --with-debugging=1 --download-openmpi=1 --with-clanguage=c++ --download-fblaslapack=1 --download-scalapack=1 --download-blacs=1 --download-suitesparse=1 --download-pastix=1 --download-superlu=1 --dowload-essl=1 --download-ptscotch=1 --download-mumps=1 --download-lusol=1 [0]PETSC ERROR: #1 MatSchurComplementGetKSP() line 317 in petsc-3.6/src/ksp/ksp/utils/schurm.c [0]PETSC ERROR: #2 PCFieldSplitGetSubKSP_FieldSplit_Schur() line 1264 in petsc-3.6/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #3 PCFieldSplitGetSubKSP() line 1668 in petsc-3.6/src/ksp/pc/impls/fieldsplit/fieldsplit.c

Code snippet:

ISCreateGeneral(PETSC_COMM_SELF,xp_dofs,&ii[0],PETSC_COPY_VALUES,&is_xp); ISCreateGeneral(PETSC_COMM_SELF,all_dofs - xp_dofs,&ii[xp_dofs],PETSC_COPY_VALUES,&is_wk);

PCSetType(pc,PCFIELDSPLIT); PCFieldSplitSetType(pc,PC_COMPOSITE_SCHUR); PCFieldSplitSetIS(pc,"xp",is_xp); PCFieldSplitSetIS(pc,"wk",is_wk); int n; KSP* ksps; PC subpc; PCFieldSplitGetSubKSP(pc,&n,&ksps);

ii here is simply an array with ii[i] = i. There is probably a better way to simply indicate two blocks of different size, but I couldn't find it.

It looks like the PC is not setup. Can you try calling PCSetUp(pc) before GetSubKSP()?

Thanks,

Matt

Thanks, Sander

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

Yes, I figured I'd need to rewrite my dirichlet bc implementation to zero out the row rather than remove the dof completely. For now I'm just avoiding partial bcs. Using MatSetBlockSize is a problem because the matrix has stuff after the xyzxyz (hydrostatic pressure dofs and a few coupled things). MatGetSubMatrix inherits the block size of 1, and I can't change it afterwards. Sander On Thu, Jul 16, 2015 at 12:26 AM, Mark Adams <[email protected]> wrote:

Note, you want to tell the matrix that it has these blocks with MatSetBlockSize, and do not remove a single DOF (like for BCs). That is, xyzxyxyz is not allowed. The block size is critical for GAMG and would think HYPRE uses it also because it does OK on elasticity.

On Wed, Jul 15, 2015 at 2:52 PM, Barry Smith <[email protected]> wrote:

...

...

On Jul 15, 2015, at 5:43 AM, Sander Land <[email protected]> wrote:

Hi Barry,

I have some structures set up that map dofs to their respective matrix indices, and it wasn't difficult to change them.

However, now this call: MatMultAdd(ctxp->B,xp,bx,xxtmp); is directed to "MatMultAdd_SeqAIJ_Inode" and gives a vector lock error at line 601 of mat/impls/aij/seq/inode.c ierr = VecGetArray(zz,&z); I think this is a bug and should be VecGetArrayRead as it doesn't need to be written to.

Oh yes, this is a bug. I have fixed it in maint, master and next

...

In the old xxxyyyzzz ordering this call went to MatMultAdd_SeqAIJ,

which (via VecGetArrayPair) does use VecGetArrayRead. I'm not sure what is causing the different matrix formats.

When the values are interlaced the matrix has "identical nodes" (i-nodes) which means certain rows have identical nonzero structure with there neighbors and the AIJ matrix code automatically switches to the more efficient Inode operations. This is why you only saw the bug with interlacing.

...

I'm also still not sure at what point I can change the matrix block

size to get the correct BoomerAMGSetNumFunctions.

...

Thanks, Sander

On Tue, Jul 14, 2015 at 6:34 PM, Barry Smith <[email protected]>

wrote:

...

...

On Jul 14, 2015, at 11:37 AM, Sander Land <[email protected]>

wrote:

...

...

Hi Matt,

Doing things to the PC like this has worked for me for a long time.

It just happened to be formerly KSPPREONLY/PCLU. Setting things like GMRES/PCNONE works as well with no errors, though it converges far too slowly.

...

The actual preconditioner I am trying to form is one from recent

literature and uses a slightly modified Schur complement split embedded in another Schur complement split with multiple solves of the inner system and very specific choices for sub-preconditioners. I don't think command line options will work very well here.

...

I've since gone the way of just using PCSHELL and setting things up

more explicitly. This seems to work, but i'm not quite sure how to set up boomeramg for one of the sub-problems.

...

boomeramg is used on the matrix A derived from: MatGetSubMatrix(J,is_x ,is_x ,MAT_INITIAL_MATRIX,&ctxp->A); The degrees of freedom for J are xxxyyyzzzpw, with the size of

xxxyyyzzz around 20k, p around 1k, and w 8 for my 'small' test problem.

...

I can see that HYPRE_BoomerAMGSetNumFunctions takes its value from

the matrix block size. This is 1 for J as it doesn't have a consistent block size. However, the number of functions for boomeramg should be 3.

...

Changing the block size is not allowed by petsc, and there doesn't seem to be another way to reach HYPRE_BoomerAMGSetNumFunctions. What is the best way to handle this, and does it matter that the dofs are ordererd xxxyyyzzz rather than xyzxyzxyz ?

I don't think BoomerAMG supports xxxyyyzzz, I think it will just get very confused trying to do algebraic multigrid with this form. Read up on it but my guess is you need to rewrite the code that prepares the system to do the interlacing.

Barry

...

Thanks, Sander

On Fri, Jul 10, 2015 at 7:44 PM, Matthew Knepley <[email protected]>

wrote:

...

On Tue, Jun 30, 2015 at 3:10 PM, Sander Land <[email protected]> wrote: This gives [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Matrix must be set first

The PC is from a SNESGetKSP, and I have called SNESSetJacobian before this.

If this is embedded in a SNES, it is very messy to yank it out and do thing manually. However, if you set the matrix in the PC manually, it might work.

However, I would encourage you to try an alternate scheme and see if you think it is easier. We have started to use the DM object to hold this kind of information. For FieldSplit, I think it is easier to tell a DMShell about your field division, and then have it pass this on to PCFIELDSPLIT. So you would

a) Create a DMSHELL

b) Create a PetscSection and call DMSetDefaultSection()

c) Call SNESSetDM()

d) use command line options to configure the split -pc_fieldsplit_0_fields 0 -pc_fieldsplit_1_fields 1

Then everything should work, and can work in any embedded context. Does this make sense?

Thanks,

Matt

Thanks, Sander

On Tue, Jun 30, 2015 at 2:09 PM, Matthew Knepley <[email protected]> wrote: On Tue, Jun 30, 2015 at 11:43 AM, Sander Land <[email protected]> wrote: I am trying to use the Schur complement preconditioner in petsc, but am encountering a null argument error calling PCFieldSplitGetSubKSP. This only happens on PC_COMPOSITE_SCHUR, the multiplicative/additive options do return a KSP array.

Error message:

[0]PETSC ERROR: --------------------- Error Message

---

...

...

[0]PETSC ERROR: Null argument, when expecting valid pointer [0]PETSC ERROR: Null Object: Parameter # 1 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.0, Jun, 09, 2015 [0]PETSC ERROR: ./elecmech on a arch-linux2-cxx-debug [0]PETSC ERROR: Configure options --with-shared-libraries=1 --with-debugging=1 --download-openmpi=1 --with-clanguage=c++ --download-fblaslapack=1 --download-scalapack=1 --download-blacs=1 --download-suitesparse=1 --download-pastix=1 --download-superlu=1 --dowload-essl=1 --download-ptscotch=1 --download-mumps=1 --download-lusol=1 [0]PETSC ERROR: #1 MatSchurComplementGetKSP() line 317 in petsc-3.6/src/ksp/ksp/utils/schurm.c [0]PETSC ERROR: #2 PCFieldSplitGetSubKSP_FieldSplit_Schur() line 1264 in petsc-3.6/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #3 PCFieldSplitGetSubKSP() line 1668 in petsc-3.6/src/ksp/pc/impls/fieldsplit/fieldsplit.c

Code snippet:

ISCreateGeneral(PETSC_COMM_SELF,xp_dofs,&ii[0],PETSC_COPY_VALUES,&is_xp);

...

ISCreateGeneral(PETSC_COMM_SELF,all_dofs - xp_dofs,&ii[xp_dofs],PETSC_COPY_VALUES,&is_wk);

PCSetType(pc,PCFIELDSPLIT); PCFieldSplitSetType(pc,PC_COMPOSITE_SCHUR); PCFieldSplitSetIS(pc,"xp",is_xp); PCFieldSplitSetIS(pc,"wk",is_wk); int n; KSP* ksps; PC subpc; PCFieldSplitGetSubKSP(pc,&n,&ksps);

ii here is simply an array with ii[i] = i. There is probably a better way to simply indicate two blocks of different size, but I couldn't find it.

It looks like the PC is not setup. Can you try calling PCSetUp(pc) before GetSubKSP()?

Thanks,

Matt

Thanks, Sander

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

Hadn't even thought of IS being the place to look for this, I see there is ISBlockGetSize and will give that a try tomorrow. Thanks, Sander On Thu, Jul 16, 2015 at 5:10 PM, Mark Adams <[email protected]> wrote:

On Thu, Jul 16, 2015 at 6:33 AM, Sander Land <[email protected]> wrote:

...

Yes, I figured I'd need to rewrite my dirichlet bc implementation to zero out the row rather than remove the dof completely. For now I'm just avoiding partial bcs. Using MatSetBlockSize is a problem because the matrix has stuff after the xyzxyz (hydrostatic pressure dofs and a few coupled things). MatGetSubMatrix inherits the block size of 1, and I can't change it afterwards.

Humm, this needs to work. Are you saying you get the (sub) matrix with an IS of the non-pressure part? You should be able to set that block size to 3. If this get submatrix happens inside of FieldSplit then I think there is a solution for this.

...

Sander

On Thu, Jul 16, 2015 at 12:26 AM, Mark Adams <[email protected]> wrote:

...

Note, you want to tell the matrix that it has these blocks with MatSetBlockSize, and do not remove a single DOF (like for BCs). That is, xyzxyxyz is not allowed. The block size is critical for GAMG and would think HYPRE uses it also because it does OK on elasticity.

On Wed, Jul 15, 2015 at 2:52 PM, Barry Smith <[email protected]> wrote:

...

...

On Jul 15, 2015, at 5:43 AM, Sander Land <[email protected]> wrote:

Hi Barry,

I have some structures set up that map dofs to their respective matrix indices, and it wasn't difficult to change them.

However, now this call: MatMultAdd(ctxp->B,xp,bx,xxtmp); is directed to "MatMultAdd_SeqAIJ_Inode" and gives a vector lock error at line 601 of mat/impls/aij/seq/inode.c ierr = VecGetArray(zz,&z); I think this is a bug and should be VecGetArrayRead as it doesn't need to be written to.

Oh yes, this is a bug. I have fixed it in maint, master and next

...

In the old xxxyyyzzz ordering this call went to MatMultAdd_SeqAIJ,

which (via VecGetArrayPair) does use VecGetArrayRead. I'm not sure what is causing the different matrix formats.

When the values are interlaced the matrix has "identical nodes" (i-nodes) which means certain rows have identical nonzero structure with there neighbors and the AIJ matrix code automatically switches to the more efficient Inode operations. This is why you only saw the bug with interlacing.

...

I'm also still not sure at what point I can change the matrix block

size to get the correct BoomerAMGSetNumFunctions.

...

Thanks, Sander

On Tue, Jul 14, 2015 at 6:34 PM, Barry Smith <[email protected]>

wrote:

...

...

On Jul 14, 2015, at 11:37 AM, Sander Land <[email protected]>

wrote:

...

...

Hi Matt,

Doing things to the PC like this has worked for me for a long time.

It just happened to be formerly KSPPREONLY/PCLU. Setting things like GMRES/PCNONE works as well with no errors, though it converges far too slowly.

...

The actual preconditioner I am trying to form is one from recent

literature and uses a slightly modified Schur complement split embedded in another Schur complement split with multiple solves of the inner system and very specific choices for sub-preconditioners. I don't think command line options will work very well here.

...

I've since gone the way of just using PCSHELL and setting things up

more explicitly. This seems to work, but i'm not quite sure how to set up boomeramg for one of the sub-problems.

...

boomeramg is used on the matrix A derived from: MatGetSubMatrix(J,is_x ,is_x ,MAT_INITIAL_MATRIX,&ctxp->A); The degrees of freedom for J are xxxyyyzzzpw, with the size of

xxxyyyzzz around 20k, p around 1k, and w 8 for my 'small' test problem.

...

I can see that HYPRE_BoomerAMGSetNumFunctions takes its value from

the matrix block size. This is 1 for J as it doesn't have a consistent block size. However, the number of functions for boomeramg should be 3.

...

Changing the block size is not allowed by petsc, and there doesn't seem to be another way to reach HYPRE_BoomerAMGSetNumFunctions. What is the best way to handle this, and does it matter that the dofs are ordererd xxxyyyzzz rather than xyzxyzxyz ?

I don't think BoomerAMG supports xxxyyyzzz, I think it will just get very confused trying to do algebraic multigrid with this form. Read up on it but my guess is you need to rewrite the code that prepares the system to do the interlacing.

Barry

...

Thanks, Sander

On Fri, Jul 10, 2015 at 7:44 PM, Matthew Knepley <[email protected]>

wrote:

...

On Tue, Jun 30, 2015 at 3:10 PM, Sander Land <[email protected]> wrote: This gives [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Matrix must be set first

The PC is from a SNESGetKSP, and I have called SNESSetJacobian before this.

If this is embedded in a SNES, it is very messy to yank it out and do thing manually. However, if you set the matrix in the PC manually, it might work.

However, I would encourage you to try an alternate scheme and see if you think it is easier. We have started to use the DM object to hold this kind of information. For FieldSplit, I think it is easier to tell a DMShell about your field division, and then have it pass this on to PCFIELDSPLIT. So you would

a) Create a DMSHELL

b) Create a PetscSection and call DMSetDefaultSection()

c) Call SNESSetDM()

d) use command line options to configure the split -pc_fieldsplit_0_fields 0 -pc_fieldsplit_1_fields 1

Then everything should work, and can work in any embedded context. Does this make sense?

Thanks,

Matt

Thanks, Sander

On Tue, Jun 30, 2015 at 2:09 PM, Matthew Knepley <[email protected]> wrote: On Tue, Jun 30, 2015 at 11:43 AM, Sander Land < [email protected]> wrote: I am trying to use the Schur complement preconditioner in petsc, but am encountering a null argument error calling PCFieldSplitGetSubKSP. This only happens on PC_COMPOSITE_SCHUR, the multiplicative/additive options do return a KSP array.

Error message:

[0]PETSC ERROR: --------------------- Error Message

---

...

...

[0]PETSC ERROR: Null argument, when expecting valid pointer [0]PETSC ERROR: Null Object: Parameter # 1 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.0, Jun, 09, 2015 [0]PETSC ERROR: ./elecmech on a arch-linux2-cxx-debug [0]PETSC ERROR: Configure options --with-shared-libraries=1 --with-debugging=1 --download-openmpi=1 --with-clanguage=c++ --download-fblaslapack=1 --download-scalapack=1 --download-blacs=1 --download-suitesparse=1 --download-pastix=1 --download-superlu=1 --dowload-essl=1 --download-ptscotch=1 --download-mumps=1 --download-lusol=1 [0]PETSC ERROR: #1 MatSchurComplementGetKSP() line 317 in petsc-3.6/src/ksp/ksp/utils/schurm.c [0]PETSC ERROR: #2 PCFieldSplitGetSubKSP_FieldSplit_Schur() line 1264 in petsc-3.6/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #3 PCFieldSplitGetSubKSP() line 1668 in petsc-3.6/src/ksp/pc/impls/fieldsplit/fieldsplit.c

Code snippet:

ISCreateGeneral(PETSC_COMM_SELF,xp_dofs,&ii[0],PETSC_COPY_VALUES,&is_xp);

...

ISCreateGeneral(PETSC_COMM_SELF,all_dofs - xp_dofs,&ii[xp_dofs],PETSC_COPY_VALUES,&is_wk);

PCSetType(pc,PCFIELDSPLIT); PCFieldSplitSetType(pc,PC_COMPOSITE_SCHUR); PCFieldSplitSetIS(pc,"xp",is_xp); PCFieldSplitSetIS(pc,"wk",is_wk); int n; KSP* ksps; PC subpc; PCFieldSplitGetSubKSP(pc,&n,&ksps);

ii here is simply an array with ii[i] = i. There is probably a better way to simply indicate two blocks of different size, but I couldn't find it.

It looks like the PC is not setup. Can you try calling PCSetUp(pc) before GetSubKSP()?

Thanks,

Matt

Thanks, Sander

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener