July 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] flushed member in _p_Mat
by Barry Smith 15 Jul '21

15 Jul '21

I think it would be helpful if you could describe exact examples of your workflows in building matrices that are troublesome so we can see how to improve the assembly process. It is not easy to simply "keep memory around" in sparse matrices since operations performed on the matrix may not be friendly to having "holes" in the matrix data structure. Barry > On Jul 15, 2021, at 12:30 PM, Fande Kong <fdkong.jd(a)gmail.com> wrote: > > We (Alex and Fande) have had a discussion again on the concerned issue. An alternative way to resolve our problem could be an extra optional flag (MAT_NO_SHRINK_MEMORY) to tell PETSc not to shrink memory at certain stages. > > The typical usage would be that we turn this flag on at the beginning of our FormJacobian, and then turn this flag off at the very end of our FormJacobian right before we call MatAssembly(final). We may argue that you should call MatAssembly(final) only once, and then you will be fine. "Call MatAssembly(final) only once" only works in an ideal world. In the real world, we have to manipulate the matrix in a very complicated way to formulate physics. > > As always, we are willing to contribute but want to make sure our idea makes sense for you guys before spending time on that. > > Thanks, > > Fande > > On Thu, Jul 15, 2021 at 11:03 AM Alexander Lindsay <alexlindsay239(a)gmail.com <mailto:[email protected]>> wrote: > As per https://gitlab.com/petsc/petsc/-/issues/852 <https://gitlab.com/petsc/petsc/-/issues/852> we have some fairly complex functions in MOOSE that sometimes require the matrix to be "assembled" at an intermediate stage of our Jacobian evaluation in order to perform multi-body Constraints. In an ideal world I would like to assemble with FLUSH because as per my post today in petsc-users FINAL may shrink my allocation before I'm ready for it to. > > We cannot currently use FLUSH at intermediate stages, however, because there is a late call to MatZeroRows which we leverage before applying our last set of objects (which are Dirichlet boundary conditions). MatZeroRows checks the `assembled` flag which only gets set during FINAL assembly and errors out if `assembled` is not true. From my understanding, MatZeroRows should be a safe operation even if the matrix has just been FLUSHed. MatZeroRows is just one example; I would guess there are other operations that could potentially be done with a FLUSHed matrix as opposed to a fully assembled one. Consequently, I'm wondering if it makes sense to add something like a `flushed` data member to _p_Mat that would allow for some less strict checking in operations like MatZeroRows. > > Alex

1 0

Re: [petsc-dev] Sync scotch package to ANL mirror sites
by Fande Kong 15 Jul '21

15 Jul '21

Thanks, again, Satish Fande On Thu, Jul 15, 2021 at 10:20 AM Satish Balay <balay(a)mcs.anl.gov> wrote: > Yes - I've added the tarball to ANL sites. > > Satish > > On Thu, 15 Jul 2021, Fande Kong wrote: > > > Thanks Satish, > > > > https://gitlab.inria.fr/scotch/scotch is back up now. Could you upload > the > > package to ANL mirrors? > > > > Thanks, > > > > Fande > > > > On Wed, Jul 14, 2021 at 4:44 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > > > > > Will do - once gitlab.inria.fr is back up - and I can download the > tarball > > > > > > Satish > > > > > > On Wed, 14 Jul 2021, Fande Kong wrote: > > > > > > > Hi Team, > > > > > > > > >From PETSc-3.15.x, we started to use scotch-v6.1.0. We always take > > > > advantage of the "--download" capability to manage external packages > for > > > > MOOSE. Recently, we hit something like this frequently: > > > > > > > > > > > > ============================================================================================= > > > > Trying to download git:// > > > > https://gitlab.inria.fr/scotch/scotch.git for PTSCOTCH > > > > > > > > > > > > ============================================================================================= > > > > > > > > > > > > ============================================================================================= > > > > Trying to download git:// > > > > https://gitlab.inria.fr/scotch/scotch.git for PTSCOTCH > > > > > > > > > > > > ============================================================================================= > > > > > > > > > > > > ============================================================================================= > > > > Trying to download git:// > > > > https://gitlab.inria.fr/scotch/scotch.git for PTSCOTCH > > > > > > > > > > > > ============================================================================================= > > > > > > > > > > > > ============================================================================================= > > > > Trying to download > > > > > > > > https://gitlab.inria.fr/scotch/scotch/-/archive/v6.1.0/scotch-v6.1.0.tar.gz > > > > for PTSCOTCH > > > > > > > > > > > > ============================================================================================= > > > > > > > > > > > > ============================================================================================= > > > > Trying to download > > > > > http://ftp.mcs.anl.gov/pub/petsc/externalpackages/scotch-v6.1.0.tar.gz > > > for > > > > PTSCOTCH > > > > > > > > > > > > ============================================================================================= > > > > > > > > > > > > ============================================================================================= > > > > Trying to download > > > > > ftp://ftp.mcs.anl.gov/pub/petsc/externalpackages/scotch-v6.1.0.tar.gz > > > for > > > > PTSCOTCH > > > > > > > > > > > > ============================================================================================= > > > > > > > > > > > > ============================================================================================= > > > > Trying to download > > > > > > > > https://www.mcs.anl.gov/petsc/mirror/externalpackages/scotch-v6.1.0.tar.gz > > > > for PTSCOTCH > > > > > > > > > > > > ============================================================================================= > > > > > > > > ******************************************************************************* > > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see > configure.log for > > > > details): > > > > > > > > ------------------------------------------------------------------------------- > > > > > > > > I would not be surprised when we could not download PTSCOTCH from > > > upstream > > > > websites in France. But it seemed that we could not even download > scotch > > > > from ANL mirrors either. I navigated through the ANL mirrors ( > > > > https://www.mcs.anl.gov/petsc/mirror/externalpackages and > > > > ftp.mcs.anl.gov/pub/petsc/externalpackages) I found we did not have > > > > scotch-v6.1.0.tar.gz yet. > > > > > > > > Could you please upload scotch-v6.1.0.tar.gz to mirror sites? > > > > > > > > > > > > Thanks so much! > > > > > > > > Fande > > > > > > > > > > > > > >

1 0

Re: [petsc-dev] flushed member in _p_Mat
by Fande Kong 15 Jul '21

15 Jul '21

We (Alex and Fande) have had a discussion again on the concerned issue. An alternative way to resolve our problem could be an extra optional flag (MAT_NO_SHRINK_MEMORY) to tell PETSc not to shrink memory at certain stages. The typical usage would be that we turn this flag on at the beginning of our FormJacobian, and then turn this flag off at the very end of our FormJacobian right before we call MatAssembly(final). We may argue that you should call MatAssembly(final) only once, and then you will be fine. "Call MatAssembly(final) only once" only works in an ideal world. In the real world, we have to manipulate the matrix in a very complicated way to formulate physics. As always, we are willing to contribute but want to make sure our idea makes sense for you guys before spending time on that. Thanks, Fande On Thu, Jul 15, 2021 at 11:03 AM Alexander Lindsay <alexlindsay239(a)gmail.com> wrote: > As per https://gitlab.com/petsc/petsc/-/issues/852 we have some fairly > complex functions in MOOSE that sometimes require the matrix to be > "assembled" at an intermediate stage of our Jacobian evaluation in order to > perform multi-body Constraints. In an ideal world I would like to assemble > with FLUSH because as per my post today in petsc-users FINAL may shrink my > allocation before I'm ready for it to. > > We cannot currently use FLUSH at intermediate stages, however, because > there is a late call to MatZeroRows which we leverage before applying our > last set of objects (which are Dirichlet boundary conditions). MatZeroRows > checks the `assembled` flag which only gets set during FINAL assembly and > errors out if `assembled` is not true. From my understanding, MatZeroRows > should be a safe operation even if the matrix has just been FLUSHed. > MatZeroRows is just one example; I would guess there are other operations > that could potentially be done with a FLUSHed matrix as opposed to a fully > assembled one. Consequently, I'm wondering if it makes sense to add > something like a `flushed` data member to _p_Mat that would allow for some > less strict checking in operations like MatZeroRows. > > Alex >

1 0

flushed member in _p_Mat
by Alexander Lindsay 15 Jul '21

15 Jul '21

As per https://gitlab.com/petsc/petsc/-/issues/852 we have some fairly complex functions in MOOSE that sometimes require the matrix to be "assembled" at an intermediate stage of our Jacobian evaluation in order to perform multi-body Constraints. In an ideal world I would like to assemble with FLUSH because as per my post today in petsc-users FINAL may shrink my allocation before I'm ready for it to. We cannot currently use FLUSH at intermediate stages, however, because there is a late call to MatZeroRows which we leverage before applying our last set of objects (which are Dirichlet boundary conditions). MatZeroRows checks the `assembled` flag which only gets set during FINAL assembly and errors out if `assembled` is not true. From my understanding, MatZeroRows should be a safe operation even if the matrix has just been FLUSHed. MatZeroRows is just one example; I would guess there are other operations that could potentially be done with a FLUSHed matrix as opposed to a fully assembled one. Consequently, I'm wondering if it makes sense to add something like a `flushed` data member to _p_Mat that would allow for some less strict checking in operations like MatZeroRows. Alex

1 0

Re: [petsc-dev] Sync scotch package to ANL mirror sites
by Satish Balay 15 Jul '21

15 Jul '21

Yes - I've added the tarball to ANL sites. Satish On Thu, 15 Jul 2021, Fande Kong wrote: > Thanks Satish, > > https://gitlab.inria.fr/scotch/scotch is back up now. Could you upload the > package to ANL mirrors? > > Thanks, > > Fande > > On Wed, Jul 14, 2021 at 4:44 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > > > Will do - once gitlab.inria.fr is back up - and I can download the tarball > > > > Satish > > > > On Wed, 14 Jul 2021, Fande Kong wrote: > > > > > Hi Team, > > > > > > >From PETSc-3.15.x, we started to use scotch-v6.1.0. We always take > > > advantage of the "--download" capability to manage external packages for > > > MOOSE. Recently, we hit something like this frequently: > > > > > > > > ============================================================================================= > > > Trying to download git:// > > > https://gitlab.inria.fr/scotch/scotch.git for PTSCOTCH > > > > > > > > ============================================================================================= > > > > > > > > ============================================================================================= > > > Trying to download git:// > > > https://gitlab.inria.fr/scotch/scotch.git for PTSCOTCH > > > > > > > > ============================================================================================= > > > > > > > > ============================================================================================= > > > Trying to download git:// > > > https://gitlab.inria.fr/scotch/scotch.git for PTSCOTCH > > > > > > > > ============================================================================================= > > > > > > > > ============================================================================================= > > > Trying to download > > > > > https://gitlab.inria.fr/scotch/scotch/-/archive/v6.1.0/scotch-v6.1.0.tar.gz > > > for PTSCOTCH > > > > > > > > ============================================================================================= > > > > > > > > ============================================================================================= > > > Trying to download > > > http://ftp.mcs.anl.gov/pub/petsc/externalpackages/scotch-v6.1.0.tar.gz > > for > > > PTSCOTCH > > > > > > > > ============================================================================================= > > > > > > > > ============================================================================================= > > > Trying to download > > > ftp://ftp.mcs.anl.gov/pub/petsc/externalpackages/scotch-v6.1.0.tar.gz > > for > > > PTSCOTCH > > > > > > > > ============================================================================================= > > > > > > > > ============================================================================================= > > > Trying to download > > > > > https://www.mcs.anl.gov/petsc/mirror/externalpackages/scotch-v6.1.0.tar.gz > > > for PTSCOTCH > > > > > > > > ============================================================================================= > > > > > ******************************************************************************* > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > > > details): > > > > > ------------------------------------------------------------------------------- > > > > > > I would not be surprised when we could not download PTSCOTCH from > > upstream > > > websites in France. But it seemed that we could not even download scotch > > > from ANL mirrors either. I navigated through the ANL mirrors ( > > > https://www.mcs.anl.gov/petsc/mirror/externalpackages and > > > ftp.mcs.anl.gov/pub/petsc/externalpackages) I found we did not have > > > scotch-v6.1.0.tar.gz yet. > > > > > > Could you please upload scotch-v6.1.0.tar.gz to mirror sites? > > > > > > > > > Thanks so much! > > > > > > Fande > > > > > > > >

1 0

Re: [petsc-dev] Sync scotch package to ANL mirror sites
by Fande Kong 15 Jul '21

15 Jul '21

Thanks Satish, https://gitlab.inria.fr/scotch/scotch is back up now. Could you upload the package to ANL mirrors? Thanks, Fande On Wed, Jul 14, 2021 at 4:44 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > Will do - once gitlab.inria.fr is back up - and I can download the tarball > > Satish > > On Wed, 14 Jul 2021, Fande Kong wrote: > > > Hi Team, > > > > >From PETSc-3.15.x, we started to use scotch-v6.1.0. We always take > > advantage of the "--download" capability to manage external packages for > > MOOSE. Recently, we hit something like this frequently: > > > > > ============================================================================================= > > Trying to download git:// > > https://gitlab.inria.fr/scotch/scotch.git for PTSCOTCH > > > > > ============================================================================================= > > > > > ============================================================================================= > > Trying to download git:// > > https://gitlab.inria.fr/scotch/scotch.git for PTSCOTCH > > > > > ============================================================================================= > > > > > ============================================================================================= > > Trying to download git:// > > https://gitlab.inria.fr/scotch/scotch.git for PTSCOTCH > > > > > ============================================================================================= > > > > > ============================================================================================= > > Trying to download > > > https://gitlab.inria.fr/scotch/scotch/-/archive/v6.1.0/scotch-v6.1.0.tar.gz > > for PTSCOTCH > > > > > ============================================================================================= > > > > > ============================================================================================= > > Trying to download > > http://ftp.mcs.anl.gov/pub/petsc/externalpackages/scotch-v6.1.0.tar.gz > for > > PTSCOTCH > > > > > ============================================================================================= > > > > > ============================================================================================= > > Trying to download > > ftp://ftp.mcs.anl.gov/pub/petsc/externalpackages/scotch-v6.1.0.tar.gz > for > > PTSCOTCH > > > > > ============================================================================================= > > > > > ============================================================================================= > > Trying to download > > > https://www.mcs.anl.gov/petsc/mirror/externalpackages/scotch-v6.1.0.tar.gz > > for PTSCOTCH > > > > > ============================================================================================= > > > ******************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > > details): > > > ------------------------------------------------------------------------------- > > > > I would not be surprised when we could not download PTSCOTCH from > upstream > > websites in France. But it seemed that we could not even download scotch > > from ANL mirrors either. I navigated through the ANL mirrors ( > > https://www.mcs.anl.gov/petsc/mirror/externalpackages and > > ftp.mcs.anl.gov/pub/petsc/externalpackages) I found we did not have > > scotch-v6.1.0.tar.gz yet. > > > > Could you please upload scotch-v6.1.0.tar.gz to mirror sites? > > > > > > Thanks so much! > > > > Fande > > > >

1 0

Re: [petsc-dev] Sync scotch package to ANL mirror sites
by Satish Balay 14 Jul '21

14 Jul '21

Will do - once gitlab.inria.fr is back up - and I can download the tarball Satish On Wed, 14 Jul 2021, Fande Kong wrote: > Hi Team, > > >From PETSc-3.15.x, we started to use scotch-v6.1.0. We always take > advantage of the "--download" capability to manage external packages for > MOOSE. Recently, we hit something like this frequently: > > ============================================================================================= > Trying to download git:// > https://gitlab.inria.fr/scotch/scotch.git for PTSCOTCH > > ============================================================================================= > > ============================================================================================= > Trying to download git:// > https://gitlab.inria.fr/scotch/scotch.git for PTSCOTCH > > ============================================================================================= > > ============================================================================================= > Trying to download git:// > https://gitlab.inria.fr/scotch/scotch.git for PTSCOTCH > > ============================================================================================= > > ============================================================================================= > Trying to download > https://gitlab.inria.fr/scotch/scotch/-/archive/v6.1.0/scotch-v6.1.0.tar.gz > for PTSCOTCH > > ============================================================================================= > > ============================================================================================= > Trying to download > http://ftp.mcs.anl.gov/pub/petsc/externalpackages/scotch-v6.1.0.tar.gz for > PTSCOTCH > > ============================================================================================= > > ============================================================================================= > Trying to download > ftp://ftp.mcs.anl.gov/pub/petsc/externalpackages/scotch-v6.1.0.tar.gz for > PTSCOTCH > > ============================================================================================= > > ============================================================================================= > Trying to download > https://www.mcs.anl.gov/petsc/mirror/externalpackages/scotch-v6.1.0.tar.gz > for PTSCOTCH > > ============================================================================================= > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > ------------------------------------------------------------------------------- > > I would not be surprised when we could not download PTSCOTCH from upstream > websites in France. But it seemed that we could not even download scotch > from ANL mirrors either. I navigated through the ANL mirrors ( > https://www.mcs.anl.gov/petsc/mirror/externalpackages and > ftp.mcs.anl.gov/pub/petsc/externalpackages) I found we did not have > scotch-v6.1.0.tar.gz yet. > > Could you please upload scotch-v6.1.0.tar.gz to mirror sites? > > > Thanks so much! > > Fande >

1 0

Sync scotch package to ANL mirror sites
by Fande Kong 14 Jul '21

14 Jul '21

Hi Team, >From PETSc-3.15.x, we started to use scotch-v6.1.0. We always take advantage of the "--download" capability to manage external packages for MOOSE. Recently, we hit something like this frequently: ============================================================================================= Trying to download git:// https://gitlab.inria.fr/scotch/scotch.git for PTSCOTCH ============================================================================================= ============================================================================================= Trying to download git:// https://gitlab.inria.fr/scotch/scotch.git for PTSCOTCH ============================================================================================= ============================================================================================= Trying to download git:// https://gitlab.inria.fr/scotch/scotch.git for PTSCOTCH ============================================================================================= ============================================================================================= Trying to download https://gitlab.inria.fr/scotch/scotch/-/archive/v6.1.0/scotch-v6.1.0.tar.gz for PTSCOTCH ============================================================================================= ============================================================================================= Trying to download http://ftp.mcs.anl.gov/pub/petsc/externalpackages/scotch-v6.1.0.tar.gz for PTSCOTCH ============================================================================================= ============================================================================================= Trying to download ftp://ftp.mcs.anl.gov/pub/petsc/externalpackages/scotch-v6.1.0.tar.gz for PTSCOTCH ============================================================================================= ============================================================================================= Trying to download https://www.mcs.anl.gov/petsc/mirror/externalpackages/scotch-v6.1.0.tar.gz for PTSCOTCH ============================================================================================= ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- I would not be surprised when we could not download PTSCOTCH from upstream websites in France. But it seemed that we could not even download scotch from ANL mirrors either. I navigated through the ANL mirrors ( https://www.mcs.anl.gov/petsc/mirror/externalpackages and ftp.mcs.anl.gov/pub/petsc/externalpackages) I found we did not have scotch-v6.1.0.tar.gz yet. Could you please upload scotch-v6.1.0.tar.gz to mirror sites? Thanks so much! Fande

1 0

Notice: CI outage July 18-20th
by Satish Balay 14 Jul '21

14 Jul '21

A heads-up, PETSc CI will be down during this outage - and it can take some time (after this outage) to bring back all the CI machines. Satish ---------- Forwarded message ---------- Date: Mon, 12 Jul 2021 22:11:41 +0000 From: "Murphy-Olson, Daniel E." <d(a)anl.gov> To: "systems-announce(a)cels.anl.gov" <systems-announce(a)cels.anl.gov> Subject: [Systems Announce] Computing outage, July 18-20th Due to the construction in the data center related to the Aurora project, the power to building 240 (including the data center) will be interrupted for most of the day on Monday, July 19. For CELS Computing, this will affect the various computing clusters in the room (you should get a separate communication on those), as well as all file servers for the MCS Legacy computing environment and the CELS General Compute Environment (GCE) will be offline beginning at 5PM on Sunday, July 18. As such, all workstations and login nodes will be down. Account management for both (accounts.mcs.anl.gov, accounts.cels.anl.gov) will also be offline. Affected services will be down until Tuesday morning. We will send updates as we determine when services can be restored and/or when they become available again. BIO divisional users, your Y drive and Z drive will be down during this period. Web services hosted and provided by CELS Systems (Wordpress, Confluence, Jira, Gitlab, svn, repocafe), mailing lists, and divisional websites will remain up and online, though some aspects that rely on components in building 240 may be unresponsive. This does not affect general email services, nor any other services provided by BIS. If power is restored sooner than the expected window, we will bring things back online and send announcements accordingly. We know this is not convenient. Thanks for your understanding. -Dan

1 0

Missing petsc4py bindings
by Pierre Jolivet 13 Jul '21

13 Jul '21

Hello, I’d like to repartition a MATMPIAIJ in petsc4py, is there an equivalent of: MatPartitioning mpart; ierr = MatPartitioningCreate(PETSC_COMM_WORLD,&mpart);CHKERRQ(ierr); ierr = MatPartitioningSetAdjacency(mpart,B);CHKERRQ(ierr); ierr = MatPartitioningSetFromOptions(mpart);CHKERRQ(ierr); ierr = MatPartitioningApply(mpart,&is);CHKERRQ(ierr); ierr = MatPartitioningDestroy(&mpart);CHKERRQ(ierr); ierr = ISBuildTwoSided(is,NULL,&rows);CHKERRQ(ierr); I can’t find any reference to MatPartitioning or ISBuildTwoSided in src/binding/petsc4py Thanks, Pierre

1 0