February 2012 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] petsc-dev on bitbucket
by Barry Smith 08 Feb '12

08 Feb '12

On Feb 8, 2012, at 9:06 AM, Matthew Knepley wrote: > On Wed, Feb 8, 2012 at 7:54 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > On Feb 8, 2012, at 6:43 AM, Satish Balay wrote: > > > On Tue, 7 Feb 2012, Barry Smith wrote: > > > >> > >> On Feb 7, 2012, at 9:09 PM, Sean Farley wrote: > >> > >>> I'm sure Jed (or Matt in his prime) could have run over to IIT and restarted the machine in less time than this :-) > >>> > >>> Sure, and like everybody else they would have had to wait outside until they had keys :-) > >> > >> Those guys are very resourceful; I cannot image a simple locked door would be an issue for them. > >> > >> Barry > >> > >> Besides who the heck set up the machine so it cannot be started remotely? Should have used an Apple machine :-) > > > > > > It was a human error [when you tell something to shutdown - it should not automatically restart]. > > > > yeah - if we installed server infrastructure with remote admin feature > > - then it could have been powered up remotely [from the remote > > management console or something like that..] > > Isn't that a basic Linux thing, start on LANS signal. > > > > > looks like folks [Sean,Matt,Barry] are happy with bitbucket. > > Not me. I'm not happy with it. I prefer the PETSc machine, bitbucket is just a back up when the PETSc machine goes down. If the PETSc machine is back up then we switch the master repository back. > > What is wrong? Not enough freedom to mess up the machine? I don't feel like pushing 2 places. Push two places manually? WTF, presumably Mecurial is feature rich enough that you could automate the whole process of "pushing to 2 places"? Ok, I need to understand more how bitbucket handles a hierarchy of different repositories with different permissions in different parts and have a hierarchy of managers of the repositories and adding new repositories. I don't want to just have haphazard creation of new repositories without a proper relationship between them. Barry Barry > > Matt > > > Barry > > > > > Sean - you'll have to transfer all repos and keys to the new site. > > > > For now - I've removed petsc-dev and BuildSystem from petsc.cs.iit - > > and will plan a phased shutdown of the machine - as soon as you can > > find new home for all repos. > > > > Satish > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener

1 0

Re: [petsc-dev] petsc-dev on bitbucket
by Matthew Knepley 08 Feb '12

08 Feb '12

On Wed, Feb 8, 2012 at 7:54 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > On Feb 8, 2012, at 6:43 AM, Satish Balay wrote: > > > On Tue, 7 Feb 2012, Barry Smith wrote: > > > >> > >> On Feb 7, 2012, at 9:09 PM, Sean Farley wrote: > >> > >>> I'm sure Jed (or Matt in his prime) could have run over to IIT and > restarted the machine in less time than this :-) > >>> > >>> Sure, and like everybody else they would have had to wait outside > until they had keys :-) > >> > >> Those guys are very resourceful; I cannot image a simple locked door > would be an issue for them. > >> > >> Barry > >> > >> Besides who the heck set up the machine so it cannot be started > remotely? Should have used an Apple machine :-) > > > > > > It was a human error [when you tell something to shutdown - it should > not automatically restart]. > > > > yeah - if we installed server infrastructure with remote admin feature > > - then it could have been powered up remotely [from the remote > > management console or something like that..] > > Isn't that a basic Linux thing, start on LANS signal. > > > > > looks like folks [Sean,Matt,Barry] are happy with bitbucket. > > Not me. I'm not happy with it. I prefer the PETSc machine, bitbucket > is just a back up when the PETSc machine goes down. If the PETSc machine is > back up then we switch the master repository back. What is wrong? Not enough freedom to mess up the machine? I don't feel like pushing 2 places. Matt > > Barry > > > > > Sean - you'll have to transfer all repos and keys to the new site. > > > > For now - I've removed petsc-dev and BuildSystem from petsc.cs.iit - > > and will plan a phased shutdown of the machine - as soon as you can > > find new home for all repos. > > > > Satish > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] petsc-dev on bitbucket
by Satish Balay 08 Feb '12

08 Feb '12

On Wed, 8 Feb 2012, Barry Smith wrote: > >> Besides who the heck set up the machine so it cannot be started remotely? Should have used an Apple machine :-) > > > > > > It was a human error [when you tell something to shutdown - it should not automatically restart]. > > > > yeah - if we installed server infrastructure with remote admin feature > > - then it could have been powered up remotely [from the remote > > management console or something like that..] > > Isn't that a basic Linux thing, start on LANS signal. I guess you mean 'wake up on lan'. I don't know enough about it. https://en.wikipedia.org/wiki/Wake-on-LAN Sounds like one needs to: - enable this feature in bios [if its supported]. - have some clinet that can send the magic packet for WOL. [usually hardware on the same LAN] - and WOL might not work from all types of shutdowns [perhaps it will work from software initiated shutdown]. Needs some experimentation to figure this out. Also there is a BIOS feature: restart ofter power failure - and this is enabled on this machine. > > > > looks like folks [Sean,Matt,Barry] are happy with bitbucket. > > Not me. I'm not happy with it. I prefer the PETSc machine, bitbucket is just a back up when the PETSc machine goes down. If the PETSc machine is back up then we switch the master repository back. Ok - I restored the repos on petsc.cs.iit.edu - but I think the debate will continue [about permanant switch]. Satish

1 0

Re: [petsc-dev] petsc-dev on bitbucket
by Barry Smith 08 Feb '12

08 Feb '12

On Feb 8, 2012, at 6:43 AM, Satish Balay wrote: > On Tue, 7 Feb 2012, Barry Smith wrote: > >> >> On Feb 7, 2012, at 9:09 PM, Sean Farley wrote: >> >>> I'm sure Jed (or Matt in his prime) could have run over to IIT and restarted the machine in less time than this :-) >>> >>> Sure, and like everybody else they would have had to wait outside until they had keys :-) >> >> Those guys are very resourceful; I cannot image a simple locked door would be an issue for them. >> >> Barry >> >> Besides who the heck set up the machine so it cannot be started remotely? Should have used an Apple machine :-) > > > It was a human error [when you tell something to shutdown - it should not automatically restart]. > > yeah - if we installed server infrastructure with remote admin feature > - then it could have been powered up remotely [from the remote > management console or something like that..] Isn't that a basic Linux thing, start on LANS signal. > > looks like folks [Sean,Matt,Barry] are happy with bitbucket. Not me. I'm not happy with it. I prefer the PETSc machine, bitbucket is just a back up when the PETSc machine goes down. If the PETSc machine is back up then we switch the master repository back. Barry > > Sean - you'll have to transfer all repos and keys to the new site. > > For now - I've removed petsc-dev and BuildSystem from petsc.cs.iit - > and will plan a phased shutdown of the machine - as soon as you can > find new home for all repos. > > Satish

1 0

Re: [petsc-dev] petsc-dev on bitbucket
by Satish Balay 08 Feb '12

08 Feb '12

On Tue, 7 Feb 2012, Barry Smith wrote: > > On Feb 7, 2012, at 9:09 PM, Sean Farley wrote: > > > I'm sure Jed (or Matt in his prime) could have run over to IIT and restarted the machine in less time than this :-) > > > > Sure, and like everybody else they would have had to wait outside until they had keys :-) > > Those guys are very resourceful; I cannot image a simple locked door would be an issue for them. > > Barry > > Besides who the heck set up the machine so it cannot be started remotely? Should have used an Apple machine :-) It was a human error [when you tell something to shutdown - it should not automatically restart]. yeah - if we installed server infrastructure with remote admin feature - then it could have been powered up remotely [from the remote management console or something like that..] looks like folks [Sean,Matt,Barry] are happy with bitbucket. Sean - you'll have to transfer all repos and keys to the new site. For now - I've removed petsc-dev and BuildSystem from petsc.cs.iit - and will plan a phased shutdown of the machine - as soon as you can find new home for all repos. Satish

1 0

Re: [petsc-dev] MatMatMult gives different results
by Alexander Grayver 08 Feb '12

08 Feb '12

Hong, It seems now I know why I used MAT_REUSE_MATRIX and preallocated matrix. I removed MatCreateMPIAIJ() and use just MatTranspose with MAT_INITIAL_MATRIX. As a consequence I get hundreds of errors like: [30]PETSC ERROR: MatSetValues_MPIAIJ() line 538 in /home/mt/agrayver/lib/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c [30]PETSC ERROR: MatAssemblyEnd_MPIAIJ() line 653 in /home/mt/agrayver/lib/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c [30]PETSC ERROR: MatAssemblyEnd() line 4978 in /home/mt/agrayver/lib/petsc-dev/src/mat/interface/matrix.c [30]PETSC ERROR: MatTranspose_MPIAIJ() line 2061 in /home/mt/agrayver/lib/petsc-dev/src/mat/impls/aij/mpi/mpiaij.c [30]PETSC ERROR: MatTranspose() line 4397 in /home/mt/agrayver/lib/petsc-dev/src/mat/interface/matrix.c [31]PETSC ERROR: --------------------- Error Message ------------------------------------ [31]PETSC ERROR: Argument out of range! [31]PETSC ERROR: New nonzero at (1659,53337) caused a malloc! Ans this is the lastest petsc-dev revision. I know there were some changes in petsc-dev concerning this issue. Can you give me a hint how to avoid this? As for why I need C=A^T*B. This product is used further as a system matrix for LSQR solver. The largest dimension is on the order of 10^6 Since I form A and B myself I can, of course, form A^T explicitly, but I thought I would first implement everything as it is written down on paper and then optimize once it works (I thought it's easier way). On 07.02.2012 20:44, Hong Zhang wrote: > Alexander, > I'm curious about why do you need parallel C=A^T*B? > How large your matrices are? > > In petsc-dev, we have MatTransposeMatMult() for mpiaij and mpiaij, but > not mpiaij and mpidense. > We may add support of MatTransposeMatMult_MPIAIJ_MPIDense() if there > is such need. > > Hong > > > On Tue, Feb 7, 2012 at 1:18 PM, agrayver(a)gfz-potsdam.de > <mailto:[email protected]> <agrayver(a)gfz-potsdam.de > <mailto:[email protected]>> wrote: > > Hong, > > Thanks for explanation. I will try this tomorrow. Good to have > this stuff in the help now. > > And sorry for misleading you initially. > > Regards, > Alexander > > > ----- Reply message ----- > From: "Hong Zhang" <hzhang(a)mcs.anl.gov <mailto:[email protected]>> > To: "For users of the development version of PETSc" > <petsc-dev(a)mcs.anl.gov <mailto:[email protected]>> > Subject: [petsc-dev] MatMatMult gives different results > Date: Tue, Feb 7, 2012 19:09 > > > Alexander : > > > There is something I didn't get yet, I hope you could clarify it. > > So, when I use flag MAT_INITIAL_MATRIX in test program it > works fine. > > Good to know :-) > > If I put this flag in my original program I get dozens of > exceptions like: > [42]PETSC ERROR: Argument out of range! > [42]PETSC ERROR: New nonzero at (1336,153341) caused a malloc! > > You cannot do > MatCreateMPIAIJ() > MatTranspose(A,MAT_INITIAL_MATRIX,&AT); > > MatCreateMPIAIJ() creates AT and preallocates approximate > nonzeros, which does not match exactly the nonzeros in > MatTranspose(A,MAT_INITIAL_MATRIX,&AT); > MatTranspose(A,MAT_INITIAL_MATRIX,&AT) creates matrix AT and sets > correct nonzero pattern and values in AT. > MatTranspose() only takes in "MAT_INITIAL_MATRIX" - for a new AT, > and "MAT_REUSE_MATRIX" when AT is created with > MatTranspose(A,MAT_INITIAL_MATRIX,&AT) > and reuse for updating its values (not nonzero patten). > > I'm updating petsc help menu on MatTranspose(). Thanks for the report. > > Hong > > > I changed this flag to MAT_REUSE_MATRIX and exceptions > disappeared, but result is incorrect again (same as for > MAT_IGNORE_MATRIX) > I tried test program with MAT_REUSE_MATRIX and it also gives > different matrix product. > > Since there is no description of MatReuse structure for > MatTranspose it's a bit confusing what to expect from it. > >>> >>> Do you mean 'Cm = A'*B;'? >>> 'Cm = A.'*B;' gives component-wise matrix product, not >>> matrix product. >> >> .' operator means non-Hermitian transpose. That is what I >> get with MatTranspose (in contrast with >> MatHermitianTranspose) >> component-wise matrix product would be .* >> >> You are correct. >> >> Hong >> >> >> >>> >>> Hong >>> >>> C = PetscBinaryRead('C.dat','complex',true); >>> >>> Matrix C is different depending on number of cores I >>> use. >>> My PETSc is: >>> Using Petsc Development HG revision: >>> 876c894d95f4fa6561d0a91310ca914592527960 HG Date: >>> Tue Jan 10 19:27:14 2012 +0100 >>> >>> >>> On 06.02.2012 17:13, Hong Zhang wrote: >>>> MatMatMult() in petsc is not well-tested for >>>> complex - could be buggy. >>>> Can you send us the matrices A and B in petsc >>>> binary format for investigation? >>>> >>>> Hong >>>> >>>> On Mon, Feb 6, 2012 at 5:55 AM, Alexander Grayver >>>> <agrayver(a)gfz-potsdam.de >>>> <mailto:[email protected]>> wrote: >>>> >>>> Dear PETSc team, >>>> >>>> I try to use: >>>> call >>>> MatMatMult(A,B,MAT_INITIAL_MATRIX,PETSC_DEFAULT_DOUBLE_PRECISION,C,ierr);CHKERRQ(ierr) >>>> >>>> Where both A and B are rectangular, but A is >>>> sparse and B is dense. Both are double complex >>>> and distributed. >>>> The product PETSc gives me contains some errors >>>> in some part of the matrix. >>>> I output A, B and C then computed product in >>>> matlab. >>>> >>>> Attached you see figure plotted as: >>>> imagesc(log10(abs(C-Cm))) >>>> >>>> Where Cm -- product computed in matlab. >>>> >>>> The pattern and amplitude vary depending on the >>>> number of cores I use. This picture is obtained >>>> for 48 cores (I've tried 12, 64 cores as well). >>>> >>>> Where should I look for possible explanation? >>>> >>>> -- >>>> Regards, >>>> Alexander >>>> >>>> >>> >>> >>> -- >>> Regards, >>> Alexander >>> >>> >> >> >> -- >> Regards, >> Alexander >> >> > > > -- > Regards, > Alexander > > > -- Regards, Alexander

1 0

Re: [petsc-dev] FAS with TS
by Jed Brown 08 Feb '12

08 Feb '12

On Wed, Feb 8, 2012 at 09:45, Matthew Knepley <knepley(a)gmail.com> wrote: > So you would have some weird call to that DM that says "make another DM > and share the internal state"? That sounds > error prone and hard to inspect from outside. Things we complain about > when other people do them. > DMRedimension() for simple cases of changing the number of dofs per node (if you had constant), something else for moving around the CW complex, changing stencil width, etc. This would be better than creating new ones from scratch anyway, but doesn't require that the DM manage its potentially sharable state in a particular way (e.g. it might share different information between a coarse-fine relation, between a vertex-center relation, and between a simple change in dofs/node).

1 0

Re: [petsc-dev] FAS with TS
by Matthew Knepley 08 Feb '12

08 Feb '12

On Wed, Feb 8, 2012 at 12:35 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Wed, Feb 8, 2012 at 08:57, Matthew Knepley <knepley(a)gmail.com> wrote: > >> I just want to point out that Jed envisions that coefficients (and maybe >> subproblems, etc) cannot be accommodated on the >> same DM. I agree. However, this silly idea that we can make DMs all over >> the place with no cost, like DAs, if they contain >> all the mesh information, is just wrong. I think this is a good argument >> for having both a topology object and a DM handling >> layout/solver information. What is the counter-argument? >> > > Why can't we have multiple DMs that internally share topology? Then each > implementation can share or not share as much as they like. Some DMs might > also share topological information between levels. I don't think it makes > sense to encode a specific sharing model into the type system. > So you would have some weird call to that DM that says "make another DM and share the internal state"? That sounds error prone and hard to inspect from outside. Things we complain about when other people do them. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] FAS with TS
by Jed Brown 08 Feb '12

08 Feb '12

On Wed, Feb 8, 2012 at 08:57, Matthew Knepley <knepley(a)gmail.com> wrote: > I just want to point out that Jed envisions that coefficients (and maybe > subproblems, etc) cannot be accommodated on the > same DM. I agree. However, this silly idea that we can make DMs all over > the place with no cost, like DAs, if they contain > all the mesh information, is just wrong. I think this is a good argument > for having both a topology object and a DM handling > layout/solver information. What is the counter-argument? > Why can't we have multiple DMs that internally share topology? Then each implementation can share or not share as much as they like. Some DMs might also share topological information between levels. I don't think it makes sense to encode a specific sharing model into the type system.

1 0

Re: [petsc-dev] FAS with TS
by Matthew Knepley 07 Feb '12

07 Feb '12

On Tue, Feb 7, 2012 at 11:50 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Wed, Feb 8, 2012 at 04:19, Peter Brune <prbrune(a)gmail.com> wrote: > >> >> >> On Tue, Feb 7, 2012 at 6:40 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: >> >>> Suppose I want to solve a finite time step problem using FAS. The time >>> step will be defined something like >>> >>> static PetscErrorCode SNESTSFormFunction_ARKIMEX(SNES snes,Vec X,Vec >>> F,TS ts) >>> { >>> TS_ARKIMEX *ark = (TS_ARKIMEX*)ts->data; >>> PetscErrorCode ierr; >>> >>> PetscFunctionBegin; >>> ierr = >>> VecAXPBYPCZ(ark->Ydot,-ark->shift,ark->shift,0,ark->Z,X);CHKERRQ(ierr); /* >>> Ydot = shift*(X-Z) */ >>> ierr = >>> TSComputeIFunction(ts,ark->stage_time,X,ark->Ydot,F,ark->imex);CHKERRQ(ierr); >>> PetscFunctionReturn(0); >>> } >>> >>> >>> where ark->Z was determined from previous stages and ark->Ydot is a work >>> vector. So FAS creates some hierarchy, but it isn't very forthcoming about >>> showing the hierarchy to the user. But I need a way to >>> >>> 1. Restrict ark->Z to the coarse level and obtain the work vector >>> ark->Ydot somewhere that I can reuse it. The user might also have to >>> restrict "auxiliary" variables to coarse grids. It's conceivable that they >>> would want to do this in a different way from the restriction/prolongation >>> for the solution variables. For example, if you were using upwinded R and P >>> for a hyperbolic problem, but you needed to restrict coefficients (like >>> bathymetry). Or they might want to sample differently out of some file. So >>> I think we need an auxiliary transfer callback that is at least called >>> every top-level SNESSolve() and optionally also called in every transfer >>> (because it could be homogenizing coefficient structure that is a nonlinear >>> function of state). >>> >>> >> The restriction (injection, etc) might be one step too hard at this >> point. We have to define the problem at each level in FAS and grid >> sequencing so we might as well make it easy. Is it possible to have the >> callback for the SNES point to things that have to be prolonged or >> restricted? This could be called both before FAS and before grid >> sequencing to get the coefficients up to the next level. We could provide >> some sort of default that goes through a number of user-provided vectors on >> a level given vectors on another level, and inject them. A user with >> user-created multilevel coefficients could provide their own similar >> callbacks. >> > > So coefficients usually won't live on the same DM as the solution. This > case with TS is easier in that regard, but it can't be the sole driver of > design. There needs to be a general function to perform level setup. > ^^^^^^^^^^^^^^^^^^^^^^^^ I just want to point out that Jed envisions that coefficients (and maybe subproblems, etc) cannot be accommodated on the same DM. I agree. However, this silly idea that we can make DMs all over the place with no cost, like DAs, if they contain all the mesh information, is just wrong. I think this is a good argument for having both a topology object and a DM handling layout/solver information. What is the counter-argument? Matt > > /* called by SNESSetUp_FAS() before restricting a nonlinear solution to a > coarser level (but usually only used the first time unless this is doing > solution-dependent homogenization) */ > typedef PetscErrorCode (*SNESFASRestrictHook)(SNES fine,Vec Xfine,SNES > coarse); > PetscErrorCode SNESFASSetRestrictHook(SNES fine,SNESFASRestrictHook > hook,void *ctx); > > /* called in SNESSolve() each time a new solution is prolongated */ > typedef PetscErrorCode (*SNESGridSequenceInterpolateHook)(SNES coarse,Vec > Xcoarse,SNES fine); > PetscErrorCode SNESGridSequenceSetInterpolateHook(SNES > coarse,SNESGridSequenceInterpolateHook hook,void *ctx); > > > I think we also need to add state to DMCreateInterpolation() so that we > can make it nonlinear when appropriate. > > >> >>> 2. Put the DM from this SNES into the TS that calls the user's >>> IFunction. The easiest way is to have TSPushDM(TS,DM) that just jams it in >>> there, caching the old DM (so the user's call to TSGetDM() works >>> correctly), and TSPopDM(TS). This is ugly and is scary if the user does >>> something weird like TSGetSolution() (returning the state at the beginning >>> of the step, not the current value at the stage). This is something that >>> doesn't make semantic sense except maybe if they have some weird >>> diagnostics, but TSGetIJacobian() might be used for caching, and I'm scared >>> of the semantics it would involve to support this sort of caching. >>> >>> >> This option seems like it would be scary complex. I don't like it. >> > > I think it's likely simpler than the others, at least as a first > implementation. I'm not wild about it either, but I'm not sure the > alternatives are better. > > >> >> >>> The alternative is to make different TSs for each level and somehow >>> locate the correct TS (maybe cache it in the SNES hierarchy). I think this >>> could be quite confusing to restrict all the members to the coarse grid. >>> But having a multi-level TS object may eventually be sensible because >>> multilevel time integration schemes exist. Spectral deferred correction is >>> one example where coarse grids (in space and time) can be used to >>> accelerate convergence to a high-order solution. But these somewhat >>> naturally expose additional parallelism in time (usually more >>> communication, but also more concurrency), so it likely needs a more >>> complete solution that includes ways to instantiate entire models on >>> sub-communicators (we'll also want an interface of this type for UQ). >>> >>> >> I'm not sure this would be harmonious with FAS given the coarse problem >> is redefined with a defect from the fine problem (making a FAS cycle on a >> actual fine solution do nothing) at each stage. As I understand it, we'd >> need a "real" time derivative rather than a derivative of the defect >> problem. Therefore, we would be stuck solving at each level and using each >> level for FAS independently. >> > > Multilevel time integration would involve having a fully functional TS on > each level, each of which may or may not use FAS. A conventional time > integration scheme solved by FAS need these things with type TS to pass to > the user, that implement TSGetDM() and perhaps some similar functionality. > They also need the affine vectors to define the residual. > > Maybe it would be better to make a little container that was basically > just a struct, but masqueraded as a TS. > > >> However, having a multilevel time integration scheme outside of FAS >> accessible to FAS to accelerate solution might be an interesting project; >> are there references? Would this look like grid sequencing + FAS with the >> grid sequence solutions moved forward in time? >> > > Hmm, I haven't seen anything really like that, but note that the stability > threshold for explicit methods is larger on coarse levels. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0