May 2015 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] new P^1.5 algorithm in VecAssembleBegin?
by Jed Brown 29 May '15

29 May '15

Barry Smith <bsmith(a)mcs.anl.gov> writes: > I cannot explain why the load balance would be 1.0 unless, by > unlikely coincidence on the 248 different calls to the function > different processes are the ones waiting so that the sum of the > waits on different processes matches over the 248 calls. Possible > but Uh, it's the same reason VecNorm often shows significant load imbalance. >> I've added a barrier in the code. > > You don't need a barrier. If you do not have a barrier you should > see all the "wait time" now accumulate somewhere later in the code > at the next reduction after the VecAssemblyBegin/End. Presumably he added a barrier *before* calling the function. The function does a small amount of work (basically none because he has no off-process entries) and synchronizes (PetscMaxSum). If there was load imbalance before calling VecAssemblyBegin, the timer would start at different times on each process, but end at about the same time.

1 0

Re: [petsc-dev] new P^1.5 algorithm in VecAssembleBegin?
by Barry Smith 29 May '15

29 May '15

> On May 29, 2015, at 7:55 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > I am suspecting that it is catching load imbalance and just not reporting it correctly. The code is trivial and exactly the same as in many other places where the load balance of not 1.0 is reported so something is funky: PetscErrorCode VecAssemblyBegin(Vec vec) { PetscErrorCode ierr; PetscFunctionBegin; PetscValidHeaderSpecific(vec,VEC_CLASSID,1); PetscValidType(vec,1); ierr = VecStashViewFromOptions(vec,NULL,"-vec_view_stash");CHKERRQ(ierr); ierr = PetscLogEventBegin(VEC_AssemblyBegin,vec,0,0,0);CHKERRQ(ierr); if (vec->ops->assemblybegin) { ierr = (*vec->ops->assemblybegin)(vec);CHKERRQ(ierr); } ierr = PetscLogEventEnd(VEC_AssemblyBegin,vec,0,0,0);CHKERRQ(ierr); ierr = PetscObjectStateIncrease((PetscObject)vec);CHKERRQ(ierr); PetscFunctionReturn(0); } I cannot explain why the load balance would be 1.0 unless, by unlikely coincidence on the 248 different calls to the function different processes are the ones waiting so that the sum of the waits on different processes matches over the 248 calls. Possible but > I've added a barrier in the code. You don't need a barrier. If you do not have a barrier you should see all the "wait time" now accumulate somewhere later in the code at the next reduction after the VecAssemblyBegin/End. Barry > > Here are the two log files. > > On Thu, May 28, 2015 at 7:48 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > VecAssemblyBegin() serves as a barrier unless you set the vector option VEC_IGNORE_OFF_PROC_ENTRIES so I am not surprised that it "appears" to take a lot of time. BUT the balance between the fastest and slowest is listed in your table below is 1.0 which is very surprising; indicating every process supposedly spent the same amount of time within the VecAssemblyBegin(). Note that for VecAssemblyEnd() the balance is 2.3 which is what I commonly would expect. Please send me ALL the output for -log_summary for these cases. Version of PETSc shouldn't matter for this issue. > > > On May 28, 2015, at 4:59 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > We are seeing some large times spent in VecAssemblyBegin: > > > > VecAssemblyBegin 242 1.0 7.9796e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 7.3e+02 12 0 0 0 5 76 0 0 0 10 0 > > VecAssemblyEnd 242 1.0 5.6624e-04 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > > > This is with 64K cores on Edison. On 128K cores (weak speedup) we see: > > > > VecAssemblyBegin 248 1.0 2.3615e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 7.4e+02 17 0 0 0 4 87 0 0 0 10 0 > > VecAssemblyEnd 248 1.0 6.8855e-04 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > > > We are working on using older versions of PETSc to make sure this is a PETSc issue but does anyone have any thoughts on this? > > > > Mark > > > <log_64K><log_128K>

1 0

Re: [petsc-dev] FieldSplit error
by Barry Smith 29 May '15

29 May '15

I will fix this. Barry > On May 29, 2015, at 10:40 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > I have a 2 x 2 MatComposite and PCSetup is hanging because this branch below does "NEED" to be checked. This is in matrix.c around line 7500. Barry added this a year ago (and the code has been working so I am puzzled). I commented this code out and it works. > > The 2nd block here is very small and is all on one processor, so all but one proc have (n == rend-rstart). > > This code seems to work in maint, so I am puzzled. Perhaps this code was merged recently? > Mark > > if (!iscol || isrow == iscol) { > > PetscBool stride; > > ierr = PetscObjectTypeCompare((PetscObject)isrow,ISSTRIDE,&stride);CHKERRQ(ierr); > > if (stride) { > > PetscInt first,step,n,rstart,rend; > > ierr = ISStrideGetInfo(isrow,&first,&step);CHKERRQ(ierr); > > if (step == 1) { > > ierr = MatGetOwnershipRange(mat,&rstart,&rend);CHKERRQ(ierr); > > if (rstart == first) { > > ierr = ISGetLocalSize(isrow,&n);CHKERRQ(ierr); > > if (n == rend-rstart) { > > /* special case grabbing all rows; NEED to do a global reduction to make sure all processes are doing this */ > > if (cll == MAT_INITIAL_MATRIX) { > > *newmat = mat; > > ierr = PetscObjectReference((PetscObject)mat);CHKERRQ(ierr); > > } > > PetscFunctionReturn(0); > > } > > } > > } > > } > > } >

1 0

Re: [petsc-dev] Errors with DM & PCSHELL
by Barry Smith 29 May '15

29 May '15

> On May 29, 2015, at 10:32 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > I am getting errors in code that I thought was working ... > > I create a PC shell like so: > > ! set shell PC > call SNESGetKSP(solver%snes,innerksp,ierr) > call KSPGetPC(innerksp,spc,ierr) > call PCSetType(spc,PCSHELL,ierr) ! overriding opions !!! > call PCShellSetApply(spc,PCShermanMorrisonApply,ierr) > call PCSetApplicationContext(spc,solver,ierr) > > and in PCShermanMorrisonApply I have: > > call PCGetApplicationContext(pc,solver,ierr) > call DMCompositeGetAccessArray(solver%da,xin,itwo,PETSC_NULL_INTEGER,Xsub,ierr) > call DMCompositeGetAccessArray(solver%da,yout,itwo,PETSC_NULL_INTEGER,Ysub,ierr) > call PCFieldSplitGetSubKSP(solver%pc,PETSC_NULL_INTEGER,subksp,ierr) > call KSPSolve(subksp(1),Xsub(1),Ysub(1),ierr) > call MatMult(solver%Cmat,Ysub(1),Xsub(2),ierr) ! ignoring <> input > > I get an error on this last statement about a vector being locked. What vector? the Xsub(2) vector? Based on the names of the variable I am guessing that xin is the input vector to your Apply() hence it is read only. Thus Xsub is read only, but you are trying to put the results of a MatMult() int Xsub(2) which is completely wrong. You cannot use Xsub for work space because it will corrupt the outer solver. > There does not seem to be: > > call DMCompositeGetAccessArrayRead( ... There doesn't need to be. Internally the DMCompositeGetAccessArray() will make the subvectors read only if the large vector is read only. > > I thought I fixed this problem ... or at least one like it. > > Mark

1 0

Re: [petsc-dev] new P^1.5 algorithm in VecAssembleBegin?
by David Trebotich 29 May '15

29 May '15

We won't know until 64k and 128k jobs get thru the queue. Will keep u posted. Thanks for your help. David Trebotich sent from mobile (510) 384-6868 On May 29, 2015 8:32 AM, "Jed Brown" <jed(a)jedbrown.org> wrote: > Mark Adams <mfadams(a)lbl.gov> writes: > > > I don't need your branch because 1) we are not doing any communication in > > this VecAssembly > > There is still synchronization cost in determining that nothing needs to > be done. Moreover, using a size P MPI_Allreduce for that (in > PetscMaxSum) is non-scalable. My branch uses MPI_Reduce_scatter_block > (scalable) when available (MPI-2.2). > > > and I added the IGNORE stuff > > So it takes no time now and everything is groovy? >

1 0

Re: [petsc-dev] new P^1.5 algorithm in VecAssembleBegin?
by Mark Adams 29 May '15

29 May '15

On Fri, May 29, 2015 at 11:32 AM, Jed Brown <jed(a)jedbrown.org> wrote: > Mark Adams <mfadams(a)lbl.gov> writes: > > > I don't need your branch because 1) we are not doing any communication in > > this VecAssembly > > There is still synchronization cost in determining that nothing needs to > be done. I added: VecSetOption(m_xx,VEC_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE); Doesn't this remove the synchronization? > Moreover, using a size P MPI_Allreduce for that (in > PetscMaxSum) is non-scalable. My branch uses MPI_Reduce_scatter_block > (scalable) when available (MPI-2.2). > > > and I added the IGNORE stuff > > So it takes no time now and everything is groovy? >

1 0

FieldSplit error
by Mark Adams 29 May '15

29 May '15

I have a 2 x 2 MatComposite and PCSetup is hanging because this branch below does "NEED" to be checked. This is in matrix.c around line 7500. Barry added this a year ago (and the code has been working so I am puzzled). I commented this code out and it works. The 2nd block here is very small and is all on one processor, so all but one proc have (n == rend-rstart). This code seems to work in maint, so I am puzzled. Perhaps this code was merged recently? Mark if (!iscol || isrow == iscol) { PetscBool stride; ierr = PetscObjectTypeCompare((PetscObject)isrow,ISSTRIDE,&stride);CHKERRQ(ierr); if (stride) { PetscInt first,step,n,rstart,rend; ierr = ISStrideGetInfo(isrow,&first,&step);CHKERRQ(ierr); if (step == 1) { ierr = MatGetOwnershipRange(mat,&rstart,&rend);CHKERRQ(ierr); if (rstart == first) { ierr = ISGetLocalSize(isrow,&n);CHKERRQ(ierr); if (n == rend-rstart) { /* special case grabbing all rows; NEED to do a global reduction to make sure all processes are doing this */ if (cll == MAT_INITIAL_MATRIX) { *newmat = mat; ierr = PetscObjectReference((PetscObject)mat);CHKERRQ(ierr); } PetscFunctionReturn(0); } } } } }

1 0

Re: [petsc-dev] new P^1.5 algorithm in VecAssembleBegin?
by Jed Brown 29 May '15

29 May '15

Mark Adams <mfadams(a)lbl.gov> writes: > I don't need your branch because 1) we are not doing any communication in > this VecAssembly There is still synchronization cost in determining that nothing needs to be done. Moreover, using a size P MPI_Allreduce for that (in PetscMaxSum) is non-scalable. My branch uses MPI_Reduce_scatter_block (scalable) when available (MPI-2.2). > and I added the IGNORE stuff So it takes no time now and everything is groovy?

1 0

Errors with DM & PCSHELL
by Mark Adams 29 May '15

29 May '15

I am getting errors in code that I thought was working ... I create a PC shell like so: ! set shell PC call SNESGetKSP(solver%snes,innerksp,ierr) call KSPGetPC(innerksp,spc,ierr) call PCSetType(spc,PCSHELL,ierr) ! overriding opions !!! call PCShellSetApply(spc,PCShermanMorrisonApply,ierr) call PCSetApplicationContext(spc,solver,ierr) and in PCShermanMorrisonApply I have: call PCGetApplicationContext(pc,solver,ierr) call DMCompositeGetAccessArray(solver%da,xin,itwo,PETSC_NULL_INTEGER,Xsub,ierr) call DMCompositeGetAccessArray(solver%da,yout,itwo,PETSC_NULL_INTEGER,Ysub,ierr) call PCFieldSplitGetSubKSP(solver%pc,PETSC_NULL_INTEGER,subksp,ierr) call KSPSolve(subksp(1),Xsub(1),Ysub(1),ierr) call MatMult(solver%Cmat,Ysub(1),Xsub(2),ierr) ! ignoring <> input I get an error on this last statement about a vector being locked. There does not seem to be: call DMCompositeGetAccessArrayRead( ... I thought I fixed this problem ... or at least one like it. Mark

1 0

Re: [petsc-dev] new P^1.5 algorithm in VecAssembleBegin?
by Mark Adams 29 May '15

29 May '15

I don't need your branch because 1) we are not doing any communication in this VecAssembly and I added the IGNORE stuff just to make sure and 2) we are catching load imbalance from another part of the code, I am now sure. I can now see that VecAssemblyBegin's timers do not report this type of load imbalance because it starts with a synch. This is what I suspected but I was wondering why the timers did not report this load imbalance. On Fri, May 29, 2015 at 11:12 AM, Jed Brown <jed(a)jedbrown.org> wrote: > Mark Adams <mfadams(a)lbl.gov> writes: > > > Thanks, this code does not have any communication in VecAssembly so I > added > > the INGNORE stuff and added a barrier before this section of code. I am > > suspecting that VecAssembly is catching load imbalance but not reporting > it > > for some reason. > > If you put a barrier before, it starts at about the same time on all > processes. The operation contains synchronization followed by > (scalable) point-to-point messaging. The slowness is almost certainly > caused by the non-scalable algorithm -- I've seen it with other > applications. So try my branch. >

1 0