March 2020 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Configure --download-eigen hangs
by Satish Balay 24 Mar '20

24 Mar '20

I just attempted a build on osx - and didn't see any such hang. I'm not sure why your build is hanging.. balay@kpro petsc % ./configure --download-eigen --download-pragmatic --download-metis --download-mpich Satish On Tue, 24 Mar 2020, Jacob Faibussowitsch wrote: > Hello all, > > Trying to install pragmatic using petsc, which requires Eigen package. I am installing these packages using arch/lib/petsc/conf/reconfigure —download-eigen —download-pragmatic. This is as far as it gets: > > /reconfigure-64-arch-darwin-c-debug.py --download-eigen > =============================================================================== > Configuring PETSc to compile on your system > =============================================================================== > =============================================================================== > Trying to download git://https://gitlab.com/libeigen/eigen for EIGEN =============================================================================== > =============================================================================== > Configuring EIGEN with cmake, this may take several minutes =============================================================================== > > I have let this run for ~30 minute periods 3 times and it never progresses. I think that it does finish configure, but never progresses to the make stage. > > I have checked htop (brew version of top) and the only relevant processes running are bash with ~5% CPU. This is less robust but on my setup the window bar (where the close, minimize, etc buttons are) of the terminal window also reports the currently running process and this also only shows bash. For reference when configure is running normally, the top bar switches between clang, ld, expr, cmake, g++, etc. System resources used usually shoots up to ~40-90% as it gets into the different configures and makes for each package. > > I have tried this on 2 different petsc configurations; --with-64-bit-indices=0/1, since this originally didn’t work with 64 bit indices so I figured that may be the issue. I have also cleared my ccache completely using ccache -c -C and rerun both configures. > > I have attached both configure.logs, note configure.log —> —with-64-bit-indices=1, configure.log.bkp —> —with-64-bit-indices=0 > > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > Cell: (312) 694-3391 > >

1 0

Configure --download-eigen hangs
by Jacob Faibussowitsch 24 Mar '20

24 Mar '20

Hello all, Trying to install pragmatic using petsc, which requires Eigen package. I am installing these packages using arch/lib/petsc/conf/reconfigure —download-eigen —download-pragmatic. This is as far as it gets: /reconfigure-64-arch-darwin-c-debug.py --download-eigen =============================================================================== Configuring PETSc to compile on your system =============================================================================== =============================================================================== Trying to download git://https://gitlab.com/libeigen/eigen for EIGEN =============================================================================== =============================================================================== Configuring EIGEN with cmake, this may take several minutes =============================================================================== I have let this run for ~30 minute periods 3 times and it never progresses. I think that it does finish configure, but never progresses to the make stage. I have checked htop (brew version of top) and the only relevant processes running are bash with ~5% CPU. This is less robust but on my setup the window bar (where the close, minimize, etc buttons are) of the terminal window also reports the currently running process and this also only shows bash. For reference when configure is running normally, the top bar switches between clang, ld, expr, cmake, g++, etc. System resources used usually shoots up to ~40-90% as it gets into the different configures and makes for each package. I have tried this on 2 different petsc configurations; --with-64-bit-indices=0/1, since this originally didn’t work with 64 bit indices so I figured that may be the issue. I have also cleared my ccache completely using ccache -c -C and rerun both configures. I have attached both configure.logs, note configure.log —> —with-64-bit-indices=1, configure.log.bkp —> —with-64-bit-indices=0 Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) Cell: (312) 694-3391

1 0

Re: [petsc-dev] [petsc-users] About the interpolation and restriction matrix for cell-centered multigrid.
by Jed Brown 24 Mar '20

24 Mar '20

ex18 has the same issue, but this aspect can probably be fixed by setting DMDA_Q0. Patrick Sanan <patrick.sanan(a)gmail.com> writes: > This sort of feedback is great in terms of learning what can move out of > "tutorials" and into "tests". > https://gitlab.com/petsc/petsc/-/merge_requests/2629 > > > Am Di., 24. März 2020 um 16:22 Uhr schrieb Jed Brown <jed(a)jedbrown.org>: > >> Mark Adams <mfadams(a)lbl.gov> writes: >> >> > Good question. It does look like there is Q1: >> > >> > src/dm/impls/da/da.c:- ctype - DMDA_Q1 and DMDA_Q0 are currently the >> only >> > supported forms >> > >> > And in looking at a cell centered >> > example src/snes/examples/tutorials/ex20.c, it looks like only DMDA_Q1 >> > works. I get an error when I set it to DMDA_Q0 (DMDA_Q1 is the default). >> > This is puzzling, Q0 is natural in cell centered. >> >> The comments in those examples are kinda wrong -- they never told the DM >> it was cell-centered so it uses a multigrid that isn't compatible with >> the boundary conditions. The interpolation is Q1 on the dual grid, not >> conservative Q1 on cells. >> >> > I am not familiar with DMDA and I don't understand why, from ex20, that >> you >> > have an odd number of points on a cell centered grid and an even number >> for >> > vertex centered (eg, ex14). I would think that it should be the opposite. >> >> The example is bad. >>

1 0

Re: [petsc-dev] Unions of IS and MatNest
by Lawrence Mitchell 24 Mar '20

24 Mar '20

> On 24 Mar 2020, at 16:02, Pierre Jolivet <pierre.jolivet(a)enseeiht.fr> wrote: > > Thanks! > >> On 24 Mar 2020, at 4:50 PM, Lawrence Mitchell <wence(a)gmx.li> wrote: >> >> >> >>> On 24 Mar 2020, at 15:39, Pierre Jolivet <pierre.jolivet(a)enseeiht.fr> wrote: >>> >>> Hello Lawrence, >>> Any luck finding the appropriate patch? >> >> It's pretty old > > I can see that, any reason why you didn’t try to get this into master? > Do you think it’s best for the caller to do this? Never got enough roundtoits to ensure it works properly! Lawrence

1 0

Re: [petsc-dev] Unions of IS and MatNest
by Pierre Jolivet 24 Mar '20

24 Mar '20

Thanks! > On 24 Mar 2020, at 4:50 PM, Lawrence Mitchell <wence(a)gmx.li> wrote: > > > >> On 24 Mar 2020, at 15:39, Pierre Jolivet <pierre.jolivet(a)enseeiht.fr> wrote: >> >> Hello Lawrence, >> Any luck finding the appropriate patch? > > It's pretty old I can see that, any reason why you didn’t try to get this into master? Do you think it’s best for the caller to do this? Thanks, Pierre > and doesn't apply cleanly, but see attached. > > We use a similar idea (working) in Firedrake (see https://github.com/firedrakeproject/firedrake/blob/master/firedrake/matrix_…) > > That latter algorithm has had more testing. > > Thanks, > > Lawrence<0001-WIP-Support-pulling-arbitrary-subblocks-out-of-MATNE.patch>

1 0

Re: [petsc-dev] [petsc-users] About the interpolation and restriction matrix for cell-centered multigrid.
by Patrick Sanan 24 Mar '20

24 Mar '20

This sort of feedback is great in terms of learning what can move out of "tutorials" and into "tests". https://gitlab.com/petsc/petsc/-/merge_requests/2629 Am Di., 24. März 2020 um 16:22 Uhr schrieb Jed Brown <jed(a)jedbrown.org>: > Mark Adams <mfadams(a)lbl.gov> writes: > > > Good question. It does look like there is Q1: > > > > src/dm/impls/da/da.c:- ctype - DMDA_Q1 and DMDA_Q0 are currently the > only > > supported forms > > > > And in looking at a cell centered > > example src/snes/examples/tutorials/ex20.c, it looks like only DMDA_Q1 > > works. I get an error when I set it to DMDA_Q0 (DMDA_Q1 is the default). > > This is puzzling, Q0 is natural in cell centered. > > The comments in those examples are kinda wrong -- they never told the DM > it was cell-centered so it uses a multigrid that isn't compatible with > the boundary conditions. The interpolation is Q1 on the dual grid, not > conservative Q1 on cells. > > > I am not familiar with DMDA and I don't understand why, from ex20, that > you > > have an odd number of points on a cell centered grid and an even number > for > > vertex centered (eg, ex14). I would think that it should be the opposite. > > The example is bad. >

1 0

Re: [petsc-dev] Unions of IS and MatNest
by Lawrence Mitchell 24 Mar '20

24 Mar '20

> On 24 Mar 2020, at 15:39, Pierre Jolivet <pierre.jolivet(a)enseeiht.fr> wrote: > > Hello Lawrence, > Any luck finding the appropriate patch? It's pretty old and doesn't apply cleanly, but see attached. We use a similar idea (working) in Firedrake (see https://github.com/firedrakeproject/firedrake/blob/master/firedrake/matrix_…) That latter algorithm has had more testing. Thanks, Lawrence

1 0

Re: [petsc-dev] Unions of IS and MatNest
by Pierre Jolivet 24 Mar '20

24 Mar '20

Hello Lawrence, Any luck finding the appropriate patch? Thanks, Pierre > On 20 Mar 2020, at 10:53 PM, Lawrence Mitchell <wence(a)gmx.li> wrote: > > > On Fri, 20 Mar 2020 at 21:07, Pierre Jolivet <pierre.jolivet(a)enseeiht.fr <mailto:[email protected]>> wrote: > > >> On 19 Mar 2020, at 7:56 PM, Matthew Knepley <knepley(a)gmail.com <mailto:[email protected]>> wrote: >> >> On Fri, Mar 13, 2020 at 11:20 AM Pierre Jolivet <pierre.jolivet(a)enseeiht.fr <mailto:[email protected]>> wrote: >> >> >>> On 13 Mar 2020, at 2:36 PM, Matthew Knepley <knepley(a)gmail.com <mailto:[email protected]>> wrote: >>> >>> On Fri, Mar 13, 2020 at 3:19 AM Pierre Jolivet <pierre.jolivet(a)enseeiht.fr <mailto:[email protected]>> wrote: >>> >>> >>>> On 12 Mar 2020, at 11:40 PM, Matthew Knepley <knepley(a)gmail.com <mailto:[email protected]>> wrote: >>>> >>>> On Thu, Mar 12, 2020 at 5:59 PM Jed Brown <jed(a)jedbrown.org <mailto:[email protected]>> wrote: >>>> Pierre Jolivet <pierre.jolivet(a)enseeiht.fr <mailto:[email protected]>> writes: >>>> >>>> > Hello, >>>> > Has there been any follow-up on this https://lists.mcs.anl.gov/pipermail/petsc-users/2015-January/024020.html <https://lists.mcs.anl.gov/pipermail/petsc-users/2015-January/024020.html> <https://lists.mcs.anl.gov/pipermail/petsc-users/2015-January/024020.html <https://lists.mcs.anl.gov/pipermail/petsc-users/2015-January/024020.html>>? >>>> > Given a 3x3 MatNest A = [A_00,0,0 ; 0,A_11,0 ; 0,0,A_22], I’d like to setup a two-way fieldsplit coupling [A_00,0 ; 0,A_11] and [A_22] but I can’t figure out the proper options. >>>> >>>> Are you looking for a Schur split or additive/multiplicative? >>> >>> Don’t know yet which will perform best, do you have a specific solution in mind for one scenario or the other? >>> I was mostly wondering if it was possible in a general context, not taking -pc_fieldsplit_type into account. >>> >>>> -pc_fieldsplit_field_0 0,1 -pc_fieldsplit_field_1 2 -pc_fieldsplit_type schur >>> >>> These flags, used with my .cpp, yield: >>> [0]PETSC ERROR: Arguments are incompatible >>> [0]PETSC ERROR: To use Schur complement preconditioner you must have exactly 2 fields >>> If I use -pc_fieldsplit_%d_fields <a,b,..> as advocated in the manual (instead of -pc_fieldsplit_field_%d as you suggested), I get the same error. >>> >>> Okay it is -pc_fieldsplit_%d_fields. When you use this, how many fields does it think you have? >> >> Three. >> In case it’s not clear, the MWE is at the bottom of my first email (nest.cpp). >> $ mpicxx nest.cpp -I$PETSC_DIR/$PETSC_ARCH/include -I$PETSC_DIR/include -L$PETSC_DIR/$PETSC_ARCH/lib -lpetsc >> $ ./a.out -pc_type fieldsplit -pc_fieldsplit_0_fields 0,1 -pc_fieldsplit_1_fields 2 -pc_fieldsplit_type schur >> […] >> [0]PETSC ERROR: Arguments are incompatible >> [0]PETSC ERROR: To use Schur complement preconditioner you must have exactly 2 fields >> […] >> Split info: >> Split number 0 Defined by IS >> Split number 1 Defined by IS >> Split number 2 Defined by IS >> […] >> >> I see now. If you call PCFieldsplitSetIS(), this overrides anything else. We do not even try to discover the split. If you want >> a split determined by command line arguments, you have to defer the split. It looks like it can get splits from the MatNest, >> so just do not call SetIS(). > > So, I’ve commented the three PCFieldSplitSetIS. > $ ./a.out -pc_type fieldsplit -pc_fieldsplit_0_fields 0,1 -pc_fieldsplit_1_fields 2 -pc_fieldsplit_block_size 3 > is starting to give me something meaningful in the KSPView, i.e.: > […] > Split number 0 Fields 0, 1 > […] > Split number 1 Fields 2 > But it’s surrounded by a bunch of errors such as: > [0]PETSC ERROR: Arguments are incompatible > [0]PETSC ERROR: Could not find index set > > Do I need some other command-line options? > > If you have a flat 3x3 matnest, the index set matching will not find a subset that is anything other than a 1-field split. I have a patch lying around that i think fixes this. I can try and dig it out on Monday. > > Lawrence

1 0

Re: [petsc-dev] Suggestions for MatProductCreate()
by Lisandro Dalcin 24 Mar '20

24 Mar '20

On Tue, 24 Mar 2020 at 05:02, hzhang(a)mcs.anl.gov <hzhang(a)mcs.anl.gov> wrote: > Matt and Lisandro, > Let me explain a little more about new API for mat-mat operations, which > include MatMaMult (AB), MatPtAP (PtAP), MatMatMatMult (ABC), with a total 6 > matproduct types. > > 1) MatProduct is logically similar to MatFactor, not a single Mat. Calling > MatProductCreate(A,B,NULL,&D); > MatProductSetType(D, MATPRODUCT_AB); > MatDestory(&A); // A is gone > MatDestory(&B); // B is gone > MatProductSymbolic(D); > is equivalent to calling MatMatMult(null, null, ..., &D) > or MatGetFactor(null, ...,&F). > If you do this, you'll get segfault (I'll add additional error message for > this use case). > > No, it is not really equivalent to MatMatMult(null, null). Because if you pass NULL to MatMatMult you can check for them. However, in the new MatProduct API, if you desdtroy A,B and next call MatProductSymbolic(D) as in my example, the internal pointers to A and B are not NULL, they are "dangling" pointers, holding the same addresses at MatProductCreate() time. This is way I said that this API is prone to dangling pointer issues if ownership is not properly handled. All I'm asking is you to implement things this way: static PetscErrorCode MatProductCreate_Private(Mat A,Mat B,Mat C,Mat D) { ... PetscObjectReference(A); // take ownership PetscObjectReference(B); // take ownership PetscObjectReference(C); // take ownership product->A = A; product->B = B; product->C = C; } PetscErrorCode MatProductReplaceMats(Mat A,Mat B,Mat C,Mat D) { ... if (A) { if (!product->Areplaced) { PetscObjectReference(A); // take ownership of input first MatDestroy(&product->A); // release old reference product->A = A; } else ... } ... } PetscErrorCode MatDestroy(Mat *A) { ... if ((*A)->product) { Mat_Product *product = (*A)->product; MatDestroy(&product->A); // release reference MatDestroy(&product->B); // release reference MatDestroy(&product->C); // release reference MatDestroy(&product->Dwork); PetscFree(product); } ... } > 2) New API is an incremental improvement over the old API, e.g., > MatMatMultSymbolic(A,B, fill,&C); MatMatMultNumeric(A,B,C) > are replaced by > MatProductCreate(A,B,NULL,&D); > MatProductSetType(D, MATPRODUCT_AB); > MatProductSetFill(D,fill); > MatProductSetFromOptions(D); > MatProductSymbolic(D): > MatProductNumeric(D); > > There are at least two reasons we decided to introduce new API. > a) there are multiple implementations (algorithms) for some product types > and matrix types, e.g., A*B and PtAP for mpiaij_mpiaij matrices. > Developers are adding more and more special implementations, e.g, for gpu > etc. Previously, the options of algorithms are selected in > MatMatMultSymbolic_xxx(...,&D) with random option names. However, some > options need to be selected before D is setup, and attached to D. For this > reason, we introduced new API, and add MatProductSetFromOptions(D) to > handle all the options. Note: the options can only be determined at the > level of producttype and mattype are known, thus the options are enabled in > MatProductSetFromOptions_mattype_producttype(). > As I said in previous emails, I'm not complaining about the API, I said it was a welcome adition. All I'm asking is: 1) Fix the handling of reference counting. This is an implementation detail, not really an API change. 2) Add a simple new routine to the API, MatProductReset(D) [or maybe name it MatProductClear(D) ], that should throw away all of the D->product stuff to release memory related to the product algorithm implementation. After calling MatProductReset(D), you cannot call anymore MatProductNumeric(). > b) Reuse of MatProduct D = A*B means same A and B with updated values. > However, some users change A or B, some use different D without knowing > original D might contain internal data structure that is associated with A > and B. The new API forces reuse same A, B, unless user > calls MatProductReplaceMats(A,B,C,D). > All that is good, no objections. But please consider adding MatProductReset(D) with the behavior I described above. Of course, this is only needed if you implement the changes to reference counting that I requested at the beginning. > > 3) For MatProduct, I consider > MatProductSetFromOptions() is logically equivalent to MatSetFromOptions(), > MatProductSymbolic() is logically equivalent to MatSetUp(), > and > MatProductNumeric() assembles the matproduct D repeatedly when A or B are > updated. > > OK, so... your point it that what looks like "setup" code in MatProductSetFromOptions() is just some minimal stuff required to actually dispatch the SetFromOptions_typeA_typeB routine, right? In that case, you are right, no changes/refactors needed here. -- Lisandro Dalcin ============ Research Scientist Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/

1 0

Re: [petsc-dev] Suggestions for MatProductCreate()
by hzhang＠mcs.anl.gov 23 Mar '20

23 Mar '20

Matt and Lisandro, Let me explain a little more about new API for mat-mat operations, which include MatMaMult (AB), MatPtAP (PtAP), MatMatMatMult (ABC), with a total 6 matproduct types. 1) MatProduct is logically similar to MatFactor, not a single Mat. Calling MatProductCreate(A,B,NULL,&D); MatProductSetType(D, MATPRODUCT_AB); MatDestory(&A); // A is gone MatDestory(&B); // B is gone MatProductSymbolic(D); is equivalent to calling MatMatMult(null, null, ..., &D) or MatGetFactor(null, ...,&F). If you do this, you'll get segfault (I'll add additional error message for this use case). 2) New API is an incremental improvement over the old API, e.g., MatMatMultSymbolic(A,B, fill,&C); MatMatMultNumeric(A,B,C) are replaced by MatProductCreate(A,B,NULL,&D); MatProductSetType(D, MATPRODUCT_AB); MatProductSetFill(D,fill); MatProductSetFromOptions(D); MatProductSymbolic(D): MatProductNumeric(D); There are at least two reasons we decided to introduce new API. a) there are multiple implementations (algorithms) for some product types and matrix types, e.g., A*B and PtAP for mpiaij_mpiaij matrices. Developers are adding more and more special implementations, e.g, for gpu etc. Previously, the options of algorithms are selected in MatMatMultSymbolic_xxx(...,&D) with random option names. However, some options need to be selected before D is setup, and attached to D. For this reason, we introduced new API, and add MatProductSetFromOptions(D) to handle all the options. Note: the options can only be determined at the level of producttype and mattype are known, thus the options are enabled in MatProductSetFromOptions_mattype_producttype(). b) Reuse of MatProduct D = A*B means same A and B with updated values. However, some users change A or B, some use different D without knowing original D might contain internal data structure that is associated with A and B. The new API forces reuse same A, B, unless user calls MatProductReplaceMats(A,B,C,D). 3) For MatProduct, I consider MatProductSetFromOptions() is logically equivalent to MatSetFromOptions(), MatProductSymbolic() is logically equivalent to MatSetUp(), and MatProductNumeric() assembles the matproduct D repeatedly when A or B are updated. I'm willing to incorporate your thoughts and comments into this new API. Hong On Mon, Mar 23, 2020 at 4:29 PM Matthew Knepley <knepley(a)gmail.com> wrote: > On Mon, Mar 23, 2020 at 5:12 PM Lisandro Dalcin <dalcinl(a)gmail.com> wrote: > >> >> >> On Mon, 23 Mar 2020 at 17:57, hzhang(a)mcs.anl.gov <hzhang(a)mcs.anl.gov> >> wrote: >> >>> Lisandro: >>> >>>> MatProductSetFromOptions() is not really setting any options. It looks >>>> like the correct name for this API is MatProductSetUp(). In any case, it >>>> looks like both SetFromOptions() and SetUp() should be provided. >>>> >>> >>> Not all product and matrix types enable algorithmic options. For those >>> who do have options, >>> MatProductSetFromOptions_mattype_producttype() enables it. >>> See MatProductSetFromOptions_MPIAIJ_AB(). >>> >> >> I have to insist, you should separate SetFromOptions() from SetUp(). For >> many types, SetFromOptions may be empty (you can even use a NULL pointer to >> handle these cases), but the rest of the code that does setup stuff (that >> is, preparing the data structures) should go SetUp() routines. >> This design pattern is used everywhere in PETSc. That's the proper way to >> do it, IMHO. This way is the one that gives maximum flexibility to power >> users/developers, because some times developers may want to configure >> objects with some parameters, and do a SetUp() without calling >> SetFromOptions() to prevent users to mess up with the parameters the >> developer carefully chose. >> > > I agree here. We should not be using options in SetUp or setting up in > SetFromOptions() since it might not be called. I am tying to fix all this > in DM. > > Matt > > >> -- >> Lisandro Dalcin >> ============ >> Research Scientist >> Extreme Computing Research Center (ECRC) >> King Abdullah University of Science and Technology (KAUST) >> http://ecrc.kaust.edu.sa/ >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> >

1 0