May 2015 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Errors with DM & PCSHELL
by Mark Adams 29 May '15

29 May '15

Woops, sorry, I see now. Thanks, On Fri, May 29, 2015 at 4:12 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > On May 29, 2015, at 3:01 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > > > On Fri, May 29, 2015 at 2:03 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > > > On May 29, 2015, at 10:32 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > I am getting errors in code that I thought was working ... > > > > > > I create a PC shell like so: > > > > > > ! set shell PC > > > call SNESGetKSP(solver%snes,innerksp,ierr) > > > call KSPGetPC(innerksp,spc,ierr) > > > call PCSetType(spc,PCSHELL,ierr) ! overriding opions !!! > > > call PCShellSetApply(spc,PCShermanMorrisonApply,ierr) > > > call PCSetApplicationContext(spc,solver,ierr) > > > > > > and in PCShermanMorrisonApply I have: > > > > > > call PCGetApplicationContext(pc,solver,ierr) > > > call > DMCompositeGetAccessArray(solver%da,xin,itwo,PETSC_NULL_INTEGER,Xsub,ierr) > > > call > DMCompositeGetAccessArray(solver%da,yout,itwo,PETSC_NULL_INTEGER,Ysub,ierr) > > > call PCFieldSplitGetSubKSP(solver%pc,PETSC_NULL_INTEGER,subksp,ierr) > > > call KSPSolve(subksp(1),Xsub(1),Ysub(1),ierr) > > > call MatMult(solver%Cmat,Ysub(1),Xsub(2),ierr) ! ignoring <> input > > > > > > I get an error on this last statement about a vector being locked. > > > > What vector? the Xsub(2) vector? > > > > Yes, arg #3 > > > > > > Based on the names of the variable I am guessing that xin is the > input vector to your Apply() hence it is read only. Thus Xsub is read only, > but you are trying to put the results of a MatMult() int Xsub(2) which is > completely wrong. You cannot use Xsub for work space because it will > corrupt the outer solver. > > Mark, > > I just told you above. Since xin is readonly input to the shell apply > and xsub points into xin you cannot use xsub as workspace in your > computations. You need to get another vector to use for workspace. If it > worked in that past it was wrong in the past because it would have > corrupted the xin vector. > > Barry > > > > > > There does not seem to be: > > > > > > call DMCompositeGetAccessArrayRead( ... > > > > There doesn't need to be. Internally the DMCompositeGetAccessArray() > will make the subvectors read only if the large vector is read only. > > > > OK, but I am getting an error. This is it. Line 2218 here is (not clear > what is going on here, a macro?): > > > > VecLocked(y,3); > > > > [8]PETSC ERROR: Vec is locked read only, argument # 3 > > [8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [8]PETSC ERROR: Petsc Development GIT revision: v3.5.4-3050-g7d7ead0 > GIT Date: 2015-05-28 23:28:47 -0500 > > [8]PETSC ERROR: /global/homes/m/madams/epsi/XGC1_3/xgc-es on a > arch-xc30-dbg-intel named nid05966 by madams Fri May 29 12:56:27 2015 > > [8]PETSC ERROR: Configure options --COPTFLAGS="-no-ipo -g -O0" > --CXXOPTFLAGS="-no-ipo -g -O0" --FOPTFLAGS="-no-ipo -g -O0" > --download-parmetis --download-metis --with-cc=cc --with-clib-autodetect=0 > --with-cxx=CC --with-cxxlib-autodetect=0 --with-debugging=1 --with-fc=ftn > --with-fortranlib-autodetect=0 > --with-hdf5-dir=/opt/cray/hdf5-parallel/1.8.13/intel/140/ > --with-shared-libraries=0 --with-x=0 --with-mpiexec=aprun LIBS=-lstdc++ > PETSC_ARCH=arch-xc30-dbg-intel > > [8]PETSC ERROR: #1 MatMult() line 2218 in > /global/u2/m/madams/petsc/src/mat/interface/matrix.c > > > > > > > > > > > > I thought I fixed this problem ... or at least one like it. > > > > > > Mark > >

1 0

Re: [petsc-dev] new P^1.5 algorithm in VecAssembleBegin?
by Barry Smith 29 May '15

29 May '15

> On May 29, 2015, at 2:56 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > Mark Adams <mfadams(a)lbl.gov> writes: >> Yea, I realized that VecAssembly should see this load imbalance unless it >> had a barrier before its timer. So I'm not sure what is going on. > > Since you don't have any outgoing entries, I think the huge > MPI_Allreduce is expensive. No surprise. This explanation matches the known information.

1 0

Re: [petsc-dev] Errors with DM & PCSHELL
by Barry Smith 29 May '15

29 May '15

> On May 29, 2015, at 3:01 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > On Fri, May 29, 2015 at 2:03 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > On May 29, 2015, at 10:32 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > I am getting errors in code that I thought was working ... > > > > I create a PC shell like so: > > > > ! set shell PC > > call SNESGetKSP(solver%snes,innerksp,ierr) > > call KSPGetPC(innerksp,spc,ierr) > > call PCSetType(spc,PCSHELL,ierr) ! overriding opions !!! > > call PCShellSetApply(spc,PCShermanMorrisonApply,ierr) > > call PCSetApplicationContext(spc,solver,ierr) > > > > and in PCShermanMorrisonApply I have: > > > > call PCGetApplicationContext(pc,solver,ierr) > > call DMCompositeGetAccessArray(solver%da,xin,itwo,PETSC_NULL_INTEGER,Xsub,ierr) > > call DMCompositeGetAccessArray(solver%da,yout,itwo,PETSC_NULL_INTEGER,Ysub,ierr) > > call PCFieldSplitGetSubKSP(solver%pc,PETSC_NULL_INTEGER,subksp,ierr) > > call KSPSolve(subksp(1),Xsub(1),Ysub(1),ierr) > > call MatMult(solver%Cmat,Ysub(1),Xsub(2),ierr) ! ignoring <> input > > > > I get an error on this last statement about a vector being locked. > > What vector? the Xsub(2) vector? > > Yes, arg #3 > > > Based on the names of the variable I am guessing that xin is the input vector to your Apply() hence it is read only. Thus Xsub is read only, but you are trying to put the results of a MatMult() int Xsub(2) which is completely wrong. You cannot use Xsub for work space because it will corrupt the outer solver. Mark, I just told you above. Since xin is readonly input to the shell apply and xsub points into xin you cannot use xsub as workspace in your computations. You need to get another vector to use for workspace. If it worked in that past it was wrong in the past because it would have corrupted the xin vector. Barry > > > There does not seem to be: > > > > call DMCompositeGetAccessArrayRead( ... > > There doesn't need to be. Internally the DMCompositeGetAccessArray() will make the subvectors read only if the large vector is read only. > > OK, but I am getting an error. This is it. Line 2218 here is (not clear what is going on here, a macro?): > > VecLocked(y,3); > > [8]PETSC ERROR: Vec is locked read only, argument # 3 > [8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [8]PETSC ERROR: Petsc Development GIT revision: v3.5.4-3050-g7d7ead0 GIT Date: 2015-05-28 23:28:47 -0500 > [8]PETSC ERROR: /global/homes/m/madams/epsi/XGC1_3/xgc-es on a arch-xc30-dbg-intel named nid05966 by madams Fri May 29 12:56:27 2015 > [8]PETSC ERROR: Configure options --COPTFLAGS="-no-ipo -g -O0" --CXXOPTFLAGS="-no-ipo -g -O0" --FOPTFLAGS="-no-ipo -g -O0" --download-parmetis --download-metis --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-debugging=1 --with-fc=ftn --with-fortranlib-autodetect=0 --with-hdf5-dir=/opt/cray/hdf5-parallel/1.8.13/intel/140/ --with-shared-libraries=0 --with-x=0 --with-mpiexec=aprun LIBS=-lstdc++ PETSC_ARCH=arch-xc30-dbg-intel > [8]PETSC ERROR: #1 MatMult() line 2218 in /global/u2/m/madams/petsc/src/mat/interface/matrix.c > > > > > > > I thought I fixed this problem ... or at least one like it. > > > > Mark

1 0

Re: [petsc-dev] new P^1.5 algorithm in VecAssembleBegin?
by Jed Brown 29 May '15

29 May '15

Mark Adams <mfadams(a)lbl.gov> writes: > This is on Edison and this was not a problem a few months ago. Edison > could have changed however. Could be effected by a software upgrade. Do you have an old profile?

1 0

Re: [petsc-dev] new P^1.5 algorithm in VecAssembleBegin?
by Mark Adams 29 May '15

29 May '15

On Fri, May 29, 2015 at 3:56 PM, Jed Brown <jed(a)jedbrown.org> wrote: > Mark Adams <mfadams(a)lbl.gov> writes: > > Yea, I realized that VecAssembly should see this load imbalance unless it > > had a barrier before its timer. So I'm not sure what is going on. > > Since you don't have any outgoing entries, I think the huge > MPI_Allreduce is expensive. No surprise. > This is on Edison and this was not a problem a few months ago. Edison could have changed however.

1 0

Re: [petsc-dev] Errors with DM & PCSHELL
by Mark Adams 29 May '15

29 May '15

On Fri, May 29, 2015 at 2:03 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > On May 29, 2015, at 10:32 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > I am getting errors in code that I thought was working ... > > > > I create a PC shell like so: > > > > ! set shell PC > > call SNESGetKSP(solver%snes,innerksp,ierr) > > call KSPGetPC(innerksp,spc,ierr) > > call PCSetType(spc,PCSHELL,ierr) ! overriding opions !!! > > call PCShellSetApply(spc,PCShermanMorrisonApply,ierr) > > call PCSetApplicationContext(spc,solver,ierr) > > > > and in PCShermanMorrisonApply I have: > > > > call PCGetApplicationContext(pc,solver,ierr) > > call > DMCompositeGetAccessArray(solver%da,xin,itwo,PETSC_NULL_INTEGER,Xsub,ierr) > > call > DMCompositeGetAccessArray(solver%da,yout,itwo,PETSC_NULL_INTEGER,Ysub,ierr) > > call PCFieldSplitGetSubKSP(solver%pc,PETSC_NULL_INTEGER,subksp,ierr) > > call KSPSolve(subksp(1),Xsub(1),Ysub(1),ierr) > > call MatMult(solver%Cmat,Ysub(1),Xsub(2),ierr) ! ignoring <> input > > > > I get an error on this last statement about a vector being locked. > > What vector? the Xsub(2) vector? > Yes, arg #3 > Based on the names of the variable I am guessing that xin is the input > vector to your Apply() hence it is read only. Thus Xsub is read only, but > you are trying to put the results of a MatMult() int Xsub(2) which is > completely wrong. You cannot use Xsub for work space because it will > corrupt the outer solver. > > > There does not seem to be: > > > > call DMCompositeGetAccessArrayRead( ... > > There doesn't need to be. Internally the DMCompositeGetAccessArray() > will make the subvectors read only if the large vector is read only. > OK, but I am getting an error. This is it. Line 2218 here is (not clear what is going on here, a macro?): VecLocked(y,3); [8]PETSC ERROR: Vec is locked read only, argument # 3 [8]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [8]PETSC ERROR: Petsc Development GIT revision: v3.5.4-3050-g7d7ead0 GIT Date: 2015-05-28 23:28:47 -0500 [8]PETSC ERROR: /global/homes/m/madams/epsi/XGC1_3/xgc-es on a arch-xc30-dbg-intel named nid05966 by madams Fri May 29 12:56:27 2015 [8]PETSC ERROR: Configure options --COPTFLAGS="-no-ipo -g -O0" --CXXOPTFLAGS="-no-ipo -g -O0" --FOPTFLAGS="-no-ipo -g -O0" --download-parmetis --download-metis --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-debugging=1 --with-fc=ftn --with-fortranlib-autodetect=0 --with-hdf5-dir=/opt/cray/hdf5-parallel/1.8.13/intel/140/ --with-shared-libraries=0 --with-x=0 --with-mpiexec=aprun LIBS=-lstdc++ PETSC_ARCH=arch-xc30-dbg-intel [8]PETSC ERROR: #1 MatMult() line 2218 in /global/u2/m/madams/petsc/src/mat/interface/matrix.c > > > > > I thought I fixed this problem ... or at least one like it. > > > > Mark > >

1 0

Re: [petsc-dev] new P^1.5 algorithm in VecAssembleBegin?
by Jed Brown 29 May '15

29 May '15

Mark Adams <mfadams(a)lbl.gov> writes: > Yea, I realized that VecAssembly should see this load imbalance unless it > had a barrier before its timer. So I'm not sure what is going on. Since you don't have any outgoing entries, I think the huge MPI_Allreduce is expensive. No surprise.

1 0

Re: [petsc-dev] new P^1.5 algorithm in VecAssembleBegin?
by Jed Brown 29 May '15

29 May '15

Barry Smith <bsmith(a)mcs.anl.gov> writes: >> On May 29, 2015, at 2:29 PM, Jed Brown <jed(a)jedbrown.org> wrote: >> >> Barry Smith <bsmith(a)mcs.anl.gov> writes: >> >>> I cannot explain why the load balance would be 1.0 unless, by >>> unlikely coincidence on the 248 different calls to the function >>> different processes are the ones waiting so that the sum of the >>> waits on different processes matches over the 248 calls. Possible >>> but >> >> Uh, it's the same reason VecNorm often shows significant load imbalance. > > Uh, I don't understand. It shows NO imbalance but huge > times. Normally I would expect a large imbalance and huge times. So > I cannot explain why it has no imbalance. 1.0 means no imbalance. Sorry, I mixed two comments. There are two non-scalable operations, determining ownership for outgoing entries (the loop I showed) and the huge MPI_Allreduce. Our timers can never observe load imbalance in the ownership determination because all that work is done after the timer has started and the timer can't end until after the MPI_Allreduce. The MPI_Allreduce is really expensive because it involves 1-2 MB from each of 128k cores. (MPI_Reduce_scatter_block is much better.) If incoming load imbalance is small relative to the ownership determination plus MPI_Allreduce, then we see 1.0. Putting a barrier before (sort of) guarantees that, but if it was already the case, the barrier won't change anything.

1 0

Re: [petsc-dev] new P^1.5 algorithm in VecAssembleBegin?
by Mark Adams 29 May '15

29 May '15

On Fri, May 29, 2015 at 3:29 PM, Jed Brown <jed(a)jedbrown.org> wrote: > Barry Smith <bsmith(a)mcs.anl.gov> writes: > > > I cannot explain why the load balance would be 1.0 unless, by > > unlikely coincidence on the 248 different calls to the function > > different processes are the ones waiting so that the sum of the > > waits on different processes matches over the 248 calls. Possible > > but > > Uh, it's the same reason VecNorm often shows significant load imbalance. > > >> I've added a barrier in the code. > > > > You don't need a barrier. If you do not have a barrier you should > > see all the "wait time" now accumulate somewhere later in the code > > at the next reduction after the VecAssemblyBegin/End. > > Presumably he added a barrier *before* calling the function. The > function does a small amount of work (basically none because he has no > off-process entries) and synchronizes (PetscMaxSum). If there was load > imbalance before calling VecAssemblyBegin, the timer would start at > different times on each process, but end at about the same time. > Yea, I realized that VecAssembly should see this load imbalance unless it had a barrier before its timer. So I'm not sure what is going on.

1 0

Re: [petsc-dev] new P^1.5 algorithm in VecAssembleBegin?
by Barry Smith 29 May '15

29 May '15

> On May 29, 2015, at 2:29 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > Barry Smith <bsmith(a)mcs.anl.gov> writes: > >> I cannot explain why the load balance would be 1.0 unless, by >> unlikely coincidence on the 248 different calls to the function >> different processes are the ones waiting so that the sum of the >> waits on different processes matches over the 248 calls. Possible >> but > > Uh, it's the same reason VecNorm often shows significant load imbalance. Uh, I don't understand. It shows NO imbalance but huge times. Normally I would expect a large imbalance and huge times. So I cannot explain why it has no imbalance. 1.0 means no imbalance. > >>> I've added a barrier in the code. >> >> You don't need a barrier. If you do not have a barrier you should >> see all the "wait time" now accumulate somewhere later in the code >> at the next reduction after the VecAssemblyBegin/End. > > Presumably he added a barrier *before* calling the function. The > function does a small amount of work (basically none because he has no > off-process entries) and synchronizes (PetscMaxSum). If there was load > imbalance before calling VecAssemblyBegin, the timer would start at > different times on each process, but end at about the same time. Yes, but then it would show a large imbalance, while it in fact shows no imbalance.

1 0