September 2017 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] MatAssembly removes all preallocation info
by Barry Smith 25 Sep '17

25 Sep '17

> On Sep 25, 2017, at 1:17 PM, Kong, Fande <fande.kong(a)inl.gov> wrote: > > > > On Mon, Sep 25, 2017 at 11:41 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > On Sep 25, 2017, at 11:22 AM, Kong, Fande <fande.kong(a)inl.gov> wrote: > > > > > > > > On Mon, Sep 25, 2017 at 10:05 AM, Stefano Zampini <stefano.zampini(a)gmail.com> wrote: > > If you know the union of the different sparsity patterns, after you preallocate you can set all zeros to use all the entries. This way PETSc will not complain about new nonzeros in successive assemblies. > > > > > > Thanks, Stefano, > > > > Yes, we are actually using this way right now. We are explicitly setting zeros to the matrix to stop petsc shrinking the memory. > > This > > But we want to know why we free the extra memory? Yes, we shrink the memory by moving all data together to have an efficiency computation. But it is still possible to not throw away our preallocated memory space if we want to use latter. We don't actually "throw out" the extra memory we just don't use it and it becomes unaccessible. Yes, one could actually keep the preallocation information. This would essentially require 1) an additional array of length number of rows that is allocated and filled with the user provide nonzero counts when the user sets the preallocation (currently this information is lost when the matrix is "shrunk") and 2) A API say MatResetPreallocation() that used this saved information to fill in the ilen and other integer arrays in the matrix as if the user had called MatPreallocateXXX() again. Actually now that I say this I realize it is pretty easy. You could make a pull request and introduce this functionality. Barry > > contradicts this. > > If at the beginning you put in all the zeros you will ever need then PETSc will not "throw away" preallocated space. Most likely you are not actually setting the initial nonzero structure using the union of all future nonzero structures. > > We are explicitly setting zeros, and the code is working. But setting zeros is not a free function. In this email, I am trying to understand why we have to do in this way. If we could not throw away the extra memory when we do not explicitly set zeros, it would be nice. > > > If the nonzero structure changes dramatically over time, for instance if the mesh is moving around removing lots of old connections and introducing lots of new ones then you do not want to just preallocate. Instead you need to preallocate for each totally new nonzero structure. > > There is no way to "preserve" the initial preallocation information through later time except by explicitly putting zero entries into the matrix. > > OK. Got it. > > Thanks, > > Fande, > > > Barry > > > > > > > Fande, > > > > > > Il 25 Set 2017 7:01 PM, "Kong, Fande" <fande.kong(a)inl.gov> ha scritto: > > Hi Matt, > > > > Thanks for your reply. > > > > The sparsity pattern is slightly different from one Newton iteration to another. We preallocate enough memory at the beginning, and want to use that memory for the following iterations. > > > > Does PETSc accutally free the preallocated (extra) memory? I so cannot use it during the second iteration. > > > > > > > > Fande, > > > > On Mon, Sep 25, 2017 at 9:43 AM, Matthew Knepley <knepley(a)gmail.com> wrote: > > On Mon, Sep 25, 2017 at 11:39 AM, Kong, Fande <fande.kong(a)inl.gov> wrote: > > Hi All, > > > > A matrix is created with the right preallocation, and then MatAssembly is called. The preallocation info will be removed. We insert any values then, and will encounter an malloc error. > > > > My question is that we was intending to design like this way? Attached simple example demonstrates what I am talking about. > > > > Yes, this is the intent. Why are you assembling? Could you use MAT_ASSEMBLY_FLUSH? > > > > Matt > > > > > > Fande, > > > > > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cse.buffalo.edu_-7… > > > >

1 0

Re: [petsc-dev] MatAssembly removes all preallocation info
by Kong, Fande 25 Sep '17

25 Sep '17

On Mon, Sep 25, 2017 at 11:41 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > On Sep 25, 2017, at 11:22 AM, Kong, Fande <fande.kong(a)inl.gov> wrote: > > > > > > > > On Mon, Sep 25, 2017 at 10:05 AM, Stefano Zampini < > stefano.zampini(a)gmail.com> wrote: > > If you know the union of the different sparsity patterns, after you > preallocate you can set all zeros to use all the entries. This way PETSc > will not complain about new nonzeros in successive assemblies. > > > > > > Thanks, Stefano, > > > > Yes, we are actually using this way right now. We are explicitly setting > zeros to the matrix to stop petsc shrinking the memory. > > This > > But we want to know why we free the extra memory? Yes, we shrink the > memory by moving all data together to have an efficiency computation. But > it is still possible to not throw away our preallocated memory space if > we want to use latter. > > contradicts this. > > If at the beginning you put in all the zeros you will ever need then > PETSc will not "throw away" preallocated space. Most likely you are not > actually setting the initial nonzero structure using the union of all > future nonzero structures. > We are explicitly setting zeros, and the code is working. But setting zeros is not a free function. In this email, I am trying to understand why we have to do in this way. If we could not throw away the extra memory when we do not explicitly set zeros, it would be nice. > If the nonzero structure changes dramatically over time, for instance > if the mesh is moving around removing lots of old connections and > introducing lots of new ones then you do not want to just preallocate. > Instead you need to preallocate for each totally new nonzero structure. > > There is no way to "preserve" the initial preallocation information > through later time except by explicitly putting zero entries into the > matrix. > OK. Got it. Thanks, Fande, > > Barry > > > > > > > Fande, > > > > > > Il 25 Set 2017 7:01 PM, "Kong, Fande" <fande.kong(a)inl.gov> ha scritto: > > Hi Matt, > > > > Thanks for your reply. > > > > The sparsity pattern is slightly different from one Newton iteration to > another. We preallocate enough memory at the beginning, and want to use > that memory for the following iterations. > > > > Does PETSc accutally free the preallocated (extra) memory? I so cannot > use it during the second iteration. > > > > > > > > Fande, > > > > On Mon, Sep 25, 2017 at 9:43 AM, Matthew Knepley <knepley(a)gmail.com> > wrote: > > On Mon, Sep 25, 2017 at 11:39 AM, Kong, Fande <fande.kong(a)inl.gov> > wrote: > > Hi All, > > > > A matrix is created with the right preallocation, and then MatAssembly > is called. The preallocation info will be removed. We insert any values > then, and will encounter an malloc error. > > > > My question is that we was intending to design like this way? Attached > simple example demonstrates what I am talking about. > > > > Yes, this is the intent. Why are you assembling? Could you use > MAT_ASSEMBLY_FLUSH? > > > > Matt > > > > > > Fande, > > > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__www. > cse.buffalo.edu_-7Eknepley_&d=DwIFAg&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB_ > _aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=U9L00IdHyC- > 4OTIJws8B4BeQTX1DBhNdzqB-NGUI3Es&s=ZakQrYVOAFg6ShY9i0MSYHkgGWvqwR > fACulWOiIX07o&e= > > > > > >

1 0

Re: [petsc-dev] MatAssembly removes all preallocation info
by Barry Smith 25 Sep '17

25 Sep '17

> On Sep 25, 2017, at 11:22 AM, Kong, Fande <fande.kong(a)inl.gov> wrote: > > > > On Mon, Sep 25, 2017 at 10:05 AM, Stefano Zampini <stefano.zampini(a)gmail.com> wrote: > If you know the union of the different sparsity patterns, after you preallocate you can set all zeros to use all the entries. This way PETSc will not complain about new nonzeros in successive assemblies. > > > Thanks, Stefano, > > Yes, we are actually using this way right now. We are explicitly setting zeros to the matrix to stop petsc shrinking the memory. This > But we want to know why we free the extra memory? Yes, we shrink the memory by moving all data together to have an efficiency computation. But it is still possible to not throw away our preallocated memory space if we want to use latter. contradicts this. If at the beginning you put in all the zeros you will ever need then PETSc will not "throw away" preallocated space. Most likely you are not actually setting the initial nonzero structure using the union of all future nonzero structures. If the nonzero structure changes dramatically over time, for instance if the mesh is moving around removing lots of old connections and introducing lots of new ones then you do not want to just preallocate. Instead you need to preallocate for each totally new nonzero structure. There is no way to "preserve" the initial preallocation information through later time except by explicitly putting zero entries into the matrix. Barry > > > Fande, > > > Il 25 Set 2017 7:01 PM, "Kong, Fande" <fande.kong(a)inl.gov> ha scritto: > Hi Matt, > > Thanks for your reply. > > The sparsity pattern is slightly different from one Newton iteration to another. We preallocate enough memory at the beginning, and want to use that memory for the following iterations. > > Does PETSc accutally free the preallocated (extra) memory? I so cannot use it during the second iteration. > > > > Fande, > > On Mon, Sep 25, 2017 at 9:43 AM, Matthew Knepley <knepley(a)gmail.com> wrote: > On Mon, Sep 25, 2017 at 11:39 AM, Kong, Fande <fande.kong(a)inl.gov> wrote: > Hi All, > > A matrix is created with the right preallocation, and then MatAssembly is called. The preallocation info will be removed. We insert any values then, and will encounter an malloc error. > > My question is that we was intending to design like this way? Attached simple example demonstrates what I am talking about. > > Yes, this is the intent. Why are you assembling? Could you use MAT_ASSEMBLY_FLUSH? > > Matt > > > Fande, > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > >

1 0

Re: [petsc-dev] MatAssembly removes all preallocation info
by Kong, Fande 25 Sep '17

25 Sep '17

On Mon, Sep 25, 2017 at 10:05 AM, Stefano Zampini <stefano.zampini(a)gmail.com > wrote: > If you know the union of the different sparsity patterns, after you > preallocate you can set all zeros to use all the entries. This way PETSc > will not complain about new nonzeros in successive assemblies. > > Thanks, Stefano, Yes, we are actually using this way right now. We are explicitly setting zeros to the matrix to stop petsc shrinking the memory. But we want to know why we free the extra memory? Yes, we shrink the memory by moving all data together to have an efficiency computation. But it is still possible to not throw away our preallocated memory space if we want to use latter. Fande, > Il 25 Set 2017 7:01 PM, "Kong, Fande" <fande.kong(a)inl.gov> ha scritto: > >> Hi Matt, >> >> Thanks for your reply. >> >> The sparsity pattern is slightly different from one Newton iteration to >> another. We preallocate enough memory at the beginning, and want to use >> that memory for the following iterations. >> >> Does PETSc accutally free the preallocated (extra) memory? I so cannot >> use it during the second iteration. >> >> >> >> Fande, >> >> On Mon, Sep 25, 2017 at 9:43 AM, Matthew Knepley <knepley(a)gmail.com> >> wrote: >> >>> On Mon, Sep 25, 2017 at 11:39 AM, Kong, Fande <fande.kong(a)inl.gov> >>> wrote: >>> >>>> Hi All, >>>> >>>> A matrix is created with the right preallocation, and then MatAssembly >>>> is called. The preallocation info will be removed. We insert any values >>>> then, and will encounter an malloc error. >>>> >>>> My question is that we was intending to design like this way? Attached >>>> simple example demonstrates what I am talking about. >>>> >>> >>> Yes, this is the intent. Why are you assembling? Could you use >>> MAT_ASSEMBLY_FLUSH? >>> >>> Matt >>> >>> >>>> >>>> Fande, >>>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.caam.rice.edu_-7Emk…> >>> >> >>

1 0

Re: [petsc-dev] MatAssembly removes all preallocation info
by Stefano Zampini 25 Sep '17

25 Sep '17

If you know the union of the different sparsity patterns, after you preallocate you can set all zeros to use all the entries. This way PETSc will not complain about new nonzeros in successive assemblies. Il 25 Set 2017 7:01 PM, "Kong, Fande" <fande.kong(a)inl.gov> ha scritto: > Hi Matt, > > Thanks for your reply. > > The sparsity pattern is slightly different from one Newton iteration to > another. We preallocate enough memory at the beginning, and want to use > that memory for the following iterations. > > Does PETSc accutally free the preallocated (extra) memory? I so cannot use > it during the second iteration. > > > > Fande, > > On Mon, Sep 25, 2017 at 9:43 AM, Matthew Knepley <knepley(a)gmail.com> > wrote: > >> On Mon, Sep 25, 2017 at 11:39 AM, Kong, Fande <fande.kong(a)inl.gov> wrote: >> >>> Hi All, >>> >>> A matrix is created with the right preallocation, and then MatAssembly >>> is called. The preallocation info will be removed. We insert any values >>> then, and will encounter an malloc error. >>> >>> My question is that we was intending to design like this way? Attached >>> simple example demonstrates what I am talking about. >>> >> >> Yes, this is the intent. Why are you assembling? Could you use >> MAT_ASSEMBLY_FLUSH? >> >> Matt >> >> >>> >>> Fande, >>> >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.caam.rice.edu_-7Emk…> >> > >

1 0

Re: [petsc-dev] MatAssembly removes all preallocation info
by Kong, Fande 25 Sep '17

25 Sep '17

Hi Matt, Thanks for your reply. The sparsity pattern is slightly different from one Newton iteration to another. We preallocate enough memory at the beginning, and want to use that memory for the following iterations. Does PETSc accutally free the preallocated (extra) memory? I so cannot use it during the second iteration. Fande, On Mon, Sep 25, 2017 at 9:43 AM, Matthew Knepley <knepley(a)gmail.com> wrote: > On Mon, Sep 25, 2017 at 11:39 AM, Kong, Fande <fande.kong(a)inl.gov> wrote: > >> Hi All, >> >> A matrix is created with the right preallocation, and then MatAssembly is >> called. The preallocation info will be removed. We insert any values then, >> and will encounter an malloc error. >> >> My question is that we was intending to design like this way? Attached >> simple example demonstrates what I am talking about. >> > > Yes, this is the intent. Why are you assembling? Could you use > MAT_ASSEMBLY_FLUSH? > > Matt > > >> >> Fande, >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.caam.rice.edu_-7Emk…> >

1 0

Re: [petsc-dev] MatAssembly removes all preallocation info
by Matthew Knepley 25 Sep '17

25 Sep '17

On Mon, Sep 25, 2017 at 11:39 AM, Kong, Fande <fande.kong(a)inl.gov> wrote: > Hi All, > > A matrix is created with the right preallocation, and then MatAssembly is > called. The preallocation info will be removed. We insert any values then, > and will encounter an malloc error. > > My question is that we was intending to design like this way? Attached > simple example demonstrates what I am talking about. > Yes, this is the intent. Why are you assembling? Could you use MAT_ASSEMBLY_FLUSH? Matt > > Fande, > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/>

1 0

MatAssembly removes all preallocation info
by Kong, Fande 25 Sep '17

25 Sep '17

Hi All, A matrix is created with the right preallocation, and then MatAssembly is called. The preallocation info will be removed. We insert any values then, and will encounter an malloc error. My question is that we was intending to design like this way? Attached simple example demonstrates what I am talking about. Fande,

1 0

DMCoarsenHookAdd in SNESSetUpMatrices
by Stefano Zampini 25 Sep '17

25 Sep '17

Jed, Today I fought for several hours against a strange behaviour occuring when reusing a TS for multiple solves. You can reproduce it by applying the attached patch. The issue is with DMCoarsenHookAdd called in SNESSetUpMatrices, and git blames you ;-). Every time you call TSGetRHSMats_Private, TSGetIJacobian gets called, which in turns calls SNESSetUpMatrices, that adds the entry in the linked list of hooks via DMCoarsenHookAdd. This causes a considerable slow down when using the RHS interface with an implicit solver, as you can see from running the patched code. can you please take a look? -- Stefano

1 0

Re: [petsc-dev] What is the difference between shift and target in SLEPc ?
by Jose E. Roman 25 Sep '17

25 Sep '17

Yes. > El 25 sept 2017, a las 15:37, Franck Houssen <franck.houssen(a)inria.fr> escribió: > > OK, thanks, this is helpful. > > If I got you correctly: beforehand, there is no way to know exactly what the eigen values are. If it turns out that an eigen value makes A-sigma*I or A-sigma*B singular, then the solve may break. If so, afterwards, it's possible to change slightly the shift to avoid solve break down (but there is no way to know that beforehand). > > Franck > > ----- Mail original ----- >> De: "Jose E. Roman" <jroman(a)dsic.upv.es> >> À: "Franck Houssen" <franck.houssen(a)inria.fr> >> Cc: "For users of the development version of PETSc" <petsc-dev(a)mcs.anl.gov> >> Envoyé: Lundi 25 Septembre 2017 14:50:48 >> Objet: Re: [petsc-dev] What is the difference between shift and target in SLEPc ? >> >> >>> El 25 sept 2017, a las 13:21, Franck Houssen <franck.houssen(a)inria.fr> >>> escribió: >>> >>> What is the difference between shift and target in SLEPc ? Shift >>> (STSetShift) is clear to me, but, target (EPSSetTarget) is not. >>> Can somebody give an example where one want/need to have a target which >>> would be different from the shift ? >>> >>> Franck >> >> In shift-and-invert the shift is equal to the target by default. The target >> is what you use to indicate where you want the eigenvalues to be sought (it >> can be used without shift-and-invert). Normal usage is having both values >> equal. If the target is exactly equal to an eigenvalue, then you may want to >> perturb the shift (change it to a slightly different value) in order to >> avoid a singular matrix A-sigma*I in the linear solves. (Some solvers such >> as MUMPS do not have problems with singular matrices, so this is not >> necessary in that case). >> >> Jose >> >>

1 0