March 2012 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] PCFieldSplitSetFields() API change
by Jed Brown 23 Mar '12

23 Mar '12

But what does using B as 00 block actually mean? It seems to me that you pivot, producing P' P J where P J has B in the 00 location. On Mar 23, 2012 6:13 PM, "Barry Smith" <bsmith(a)mcs.anl.gov> wrote: > > On Mar 23, 2012, at 6:00 PM, Jed Brown wrote: > > > On Fri, Mar 23, 2012 at 17:48, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > I'm confused. I am NOT injecting it into the original definition, I > am injecting it into the PCFieldSplit in the same way that your > "permutation" would inject it just into the PCFieldSplit, it is just (in my > opinion) a more user friendly way of the user providing this permutation. > Note that this "two fields" business is NOT going in the DM (and hence into > the original definition of the problem) and would normally only be used > when the DM definition was "bad" (due to zero block for example) just like > your permutation would only be used when things were bad. > > > > 1. The PCFieldSplitSetIS() interface does not have this. > > > > 2. The reason for pivoting is not that the DM ordering was "bad", it's > that a symmetric ordering without the 0 block does not exist (factorization > does not work with _any_ symmetric ordering). So you pivot for > factorization and restore. > > I don't agree. Consider [A B ; C D] where all four matrices are > square and non-singular. It may be that using B as the 00 block is just > better for convergence etc > > > > > 3. You now have one name for a set of rows and columns that may not "go > together". > > That name is only for setting fieldsplit options etc. It is not > fieldnames in the way that DM fieldnames are names of fields. > > > > > Somehow I think this is just more confusing and less run-time-friendly, > but maybe that's my fault. In any case, the change has to be consistent > (PCFieldSplitSetIS) and we have to decide how the DM can specify when > pivoting is needed. > > You may be right. Spec out your alternative Functions I would call and > command line options I would use to get the same effect as the affect > Jungho and I were trying to achieve. I was bothered by the separation in > your original proposal between the > selection of the blocks the "pivoting" of your model. I don't really see > using a different block in 00 as necessarily a "pivoting" that is why you > proposal turned me off. > > > Barry > > >

1 0

Re: [petsc-dev] PCFieldSplitSetFields() API change
by Barry Smith 23 Mar '12

23 Mar '12

On Mar 23, 2012, at 6:00 PM, Jed Brown wrote: > On Fri, Mar 23, 2012 at 17:48, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > I'm confused. I am NOT injecting it into the original definition, I am injecting it into the PCFieldSplit in the same way that your "permutation" would inject it just into the PCFieldSplit, it is just (in my opinion) a more user friendly way of the user providing this permutation. Note that this "two fields" business is NOT going in the DM (and hence into the original definition of the problem) and would normally only be used when the DM definition was "bad" (due to zero block for example) just like your permutation would only be used when things were bad. > > 1. The PCFieldSplitSetIS() interface does not have this. > > 2. The reason for pivoting is not that the DM ordering was "bad", it's that a symmetric ordering without the 0 block does not exist (factorization does not work with _any_ symmetric ordering). So you pivot for factorization and restore. I don't agree. Consider [A B ; C D] where all four matrices are square and non-singular. It may be that using B as the 00 block is just better for convergence etc > > 3. You now have one name for a set of rows and columns that may not "go together". That name is only for setting fieldsplit options etc. It is not fieldnames in the way that DM fieldnames are names of fields. > > Somehow I think this is just more confusing and less run-time-friendly, but maybe that's my fault. In any case, the change has to be consistent (PCFieldSplitSetIS) and we have to decide how the DM can specify when pivoting is needed. You may be right. Spec out your alternative Functions I would call and command line options I would use to get the same effect as the affect Jungho and I were trying to achieve. I was bothered by the separation in your original proposal between the selection of the blocks the "pivoting" of your model. I don't really see using a different block in 00 as necessarily a "pivoting" that is why you proposal turned me off. Barry

1 0

Re: [petsc-dev] PCFieldSplitSetFields() API change
by Jed Brown 23 Mar '12

23 Mar '12

On Fri, Mar 23, 2012 at 17:48, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > I'm confused. I am NOT injecting it into the original definition, I am > injecting it into the PCFieldSplit in the same way that your "permutation" > would inject it just into the PCFieldSplit, it is just (in my opinion) a > more user friendly way of the user providing this permutation. Note that > this "two fields" business is NOT going in the DM (and hence into the > original definition of the problem) and would normally only be used when > the DM definition was "bad" (due to zero block for example) just like your > permutation would only be used when things were bad. 1. The PCFieldSplitSetIS() interface does not have this. 2. The reason for pivoting is not that the DM ordering was "bad", it's that a symmetric ordering without the 0 block does not exist (factorization does not work with _any_ symmetric ordering). So you pivot for factorization and restore. 3. You now have one name for a set of rows and columns that may not "go together". Somehow I think this is just more confusing and less run-time-friendly, but maybe that's my fault. In any case, the change has to be consistent (PCFieldSplitSetIS) and we have to decide how the DM can specify when pivoting is needed.

1 0

Re: [petsc-dev] PCFieldSplitSetFields() API change
by Barry Smith 23 Mar '12

23 Mar '12

On Mar 23, 2012, at 5:41 PM, Jed Brown wrote: > On Fri, Mar 23, 2012 at 17:14, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > But you are implicitly assuming that for example the first component of input space is "somehow" tied to the first component of the output space etc. I submit this is only true if the user has chosen first component of the input space and output space "correctly" and the user may not have done this. Giving this additional freedom to the PCFieldSplit allows the person defining the components (first etc in the input and output spaces) to not be concerned with how those definitions affect the solver, since the PCFiledSplit stage can correct for dumb choices. > > Suppose I have a permutation matrix P, P^T P = P P^T = I. The usual point of pivoting is > > A^{-1} = (P^T P A)^{-1} = (P A)^{-1} P^{-T} = (P A)^{-1} P > > A typical reason for this is that A has 0 in the first diagonal block, but P A is nice. Now you're telling me that you want to inject the permutation into the original definition of the problem. I'm confused. I am NOT injecting it into the original definition, I am injecting it into the PCFieldSplit in the same way that your "permutation" would inject it just into the PCFieldSplit, it is just (in my opinion) a more user friendly way of the user providing this permutation. Note that this "two fields" business is NOT going in the DM (and hence into the original definition of the problem) and would normally only be used when the DM definition was "bad" (due to zero block for example) just like your permutation would only be used when things were bad. Barry > So I guess the user would only give B = P A and they would define their problem as B x = (P b), never telling me about P anywhere. > > Okay, but what needs to be written to change P? It sounds like I would need to change the way assembly is done and also the way residuals are evaluated. That just seems more intrusive and definitely less amenable to automatically detecting the need for pivoting. I wanted to be able to experiment with different elimination orders at run-time.

1 0

Re: [petsc-dev] PCFieldSplitSetFields() API change
by Jed Brown 23 Mar '12

23 Mar '12

On Fri, Mar 23, 2012 at 17:14, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > But you are implicitly assuming that for example the first component of > input space is "somehow" tied to the first component of the output space > etc. I submit this is only true if the user has chosen first component of > the input space and output space "correctly" and the user may not have done > this. Giving this additional freedom to the PCFieldSplit allows the person > defining the components (first etc in the input and output spaces) to not > be concerned with how those definitions affect the solver, since the > PCFiledSplit stage can correct for dumb choices. > Suppose I have a permutation matrix P, P^T P = P P^T = I. The usual point of pivoting is A^{-1} = (P^T P A)^{-1} = (P A)^{-1} P^{-T} = (P A)^{-1} P A typical reason for this is that A has 0 in the first diagonal block, but P A is nice. Now you're telling me that you want to inject the permutation into the original definition of the problem. So I guess the user would only give B = P A and they would define their problem as B x = (P b), never telling me about P anywhere. Okay, but what needs to be written to change P? It sounds like I would need to change the way assembly is done and also the way residuals are evaluated. That just seems more intrusive and definitely less amenable to automatically detecting the need for pivoting. I wanted to be able to experiment with different elimination orders at run-time.

1 0

Re: [petsc-dev] PCFieldSplitSetFields() API change
by Barry Smith 23 Mar '12

23 Mar '12

On Mar 23, 2012, at 4:56 PM, Jed Brown wrote: > This changed the API for PCFieldSplitSetFields(), but didn't update the man page, src/docs/website/documentation/changes/dev.html, or the examples that called the function (one of the two examples was updated later). > > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/55721edbdaa2 > > But is this even the right thing to do? I always thought that when we talked about non-symmetric pivoting, we were identifying fields in the unpivoted system and that we would implement pivoting internally (restoring the original ordering at the end of preconditioner application). That was your proposal :-) > Is there an example that uses this flexibility so I can understand what's happening? > > I'm very nervous to make the definition of a field a thing that involves both input and output spaces. But you are implicitly assuming that for example the first component of input space is "somehow" tied to the first component of the output space etc. I submit this is only true if the user has chosen first component of the input space and output space "correctly" and the user may not have done this. Giving this additional freedom to the PCFieldSplit allows the person defining the components (first etc in the input and output spaces) to not be concerned with how those definitions affect the solver, since the PCFiledSplit stage can correct for dumb choices. In all cases the "original ordering" will appear outside of the PCFieldsplit in the same way that your "pivoting internal" proposal does. I happen to think allowing setting both input and output fields is much more natural then having the user indicate some kind of pivoting that you proposed. Barry

1 0

PCFieldSplitSetFields() API change
by Jed Brown 23 Mar '12

23 Mar '12

This changed the API for PCFieldSplitSetFields(), but didn't update the man page, src/docs/website/documentation/changes/dev.html, or the examples that called the function (one of the two examples was updated later). http://petsc.cs.iit.edu/petsc/petsc-dev/rev/55721edbdaa2 But is this even the right thing to do? I always thought that when we talked about non-symmetric pivoting, we were identifying fields in the unpivoted system and that we would implement pivoting internally (restoring the original ordering at the end of preconditioner application). Is there an example that uses this flexibility so I can understand what's happening? I'm very nervous to make the definition of a field a thing that involves both input and output spaces.

1 0

Re: [petsc-dev] How does this compare...
by Hong Zhang 23 Mar '12

23 Mar '12

Interesting paper. I'll read it and get back to you on "How does this compare to petsc MatMatMult(). Hong On Wed, Mar 21, 2012 at 2:43 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > to the new scalable MatMatMult() that is done in PETSc? Is it possible to > even compare > the flop rates sensibly? > > http://arxiv.org/abs/1203.1692 > > This is what Jeff is planning on using in his SciDAC. > > Matt > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener >

1 0

Re: [petsc-dev] PCApplyTranspose in BiCG
by Alexander Grayver 23 Mar '12

23 Mar '12

Barry, gmres produces the same result. What does that mean? On 22.03.2012 21:10, Barry Smith wrote: > Ok, first thing to confirm is if other KSP also produce different results. Run both exactly the same but use -ksp_type gmres on both; does it produce different results or the same? > > Barry > > On Mar 22, 2012, at 12:30 PM, Alexander Grayver wrote: > >> Jed, >> >> Ok, I see. Initially I've changed solver because I observed one thing. I have two petsc-dev (one is older than the other) versions compiled. >> Running the same program with both gives different results: >> >> ./solveTest -ksp_view -ksp_converged_reason -ksp_monitor_true_residual -log_summary -mat_type aij -ksp_rtol 1.0e-7 -ksp_type tfqmr -pc_type sor -ksp_max_it 750 >> 0 KSP preconditioned resid norm 9.760858225033e-10 true resid norm 4.108026443997e-08 ||r(i)||/||b|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 9.497939702484e-10 true resid norm 3.591828545356e-08 ||r(i)||/||b|| 8.743440662618e-01 >> 1 KSP preconditioned resid norm 1.341212527158e-09 true resid norm 3.754099340764e-08 ||r(i)||/||b|| 9.138449793207e-01 >> 2 KSP preconditioned resid norm 9.481744679086e-10 true resid norm 3.710424003325e-08 ||r(i)||/||b|| 9.032132713620e-01 >> 2 KSP preconditioned resid norm 1.340878639500e-09 true resid norm 3.730261027312e-08 ||r(i)||/||b|| 9.080421166135e-01 >> 3 KSP preconditioned resid norm 9.481415577317e-10 true resid norm 3.725585425324e-08 ||r(i)||/||b|| 9.069039540306e-01 >> ... >> 750 KSP preconditioned resid norm 1.340874209607e-09 true resid norm 3.726506102951e-08 ||r(i)||/||b|| 9.071280708031e-01 >> Linear solve did not converge due to DIVERGED_ITS iterations 750 >> KSP Object: 1 MPI processes >> type: tfqmr >> maximum iterations=750, initial guess is zero >> tolerances: relative=1e-07, absolute=1e-50, divergence=10000 >> left preconditioning >> using PRECONDITIONED norm type for convergence test >> PC Object: 1 MPI processes >> type: sor >> SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 >> linear system matrix = precond matrix: >> Matrix Object: 1 MPI processes >> type: seqaij >> rows=1375920, cols=1375920 >> total: nonzeros=17671714, allocated nonzeros=17671714 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node routines >> >> Using Petsc Development HG revision: 59ca71b85d784abbe22f12c34ffcb2ccc6dc68b0 HG Date: Tue Mar 20 01:05:13 2012 -0500 >> Configure options: --with-petsc-arch=openmpi-intel-complex-release-c --with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2 --with-scalar-type=complex --with-precision=double --with-blas-lapack-dir=/usr/local/lib --with-x=0 --with-debugging=0 >> >> ####################################### >> And now second one: >> ./solveTest -ksp_view -ksp_converged_reason -ksp_monitor_true_residual -log_summary -mat_type aij -ksp_rtol 1.0e-7 -ksp_type tfqmr -pc_type sor -ksp_max_it 750 >> 0 KSP preconditioned resid norm 9.760858225034e-10 true resid norm 4.108026443997e-08 ||r(i)||/||b|| 1.000000000000e+00 >> 1 KSP preconditioned resid norm 9.497930576113e-10 true resid norm 3.591633653854e-08 ||r(i)||/||b|| 8.742966246242e-01 >> 1 KSP preconditioned resid norm 1.341211098002e-09 true resid norm 3.754049143693e-08 ||r(i)||/||b|| 9.138327600539e-01 >> 2 KSP preconditioned resid norm 9.481662292281e-10 true resid norm 3.709648505700e-08 ||r(i)||/||b|| 9.030244951612e-01 >> 2 KSP preconditioned resid norm 1.340863953080e-09 true resid norm 3.730358317575e-08 ||r(i)||/||b|| 9.080657995827e-01 >> 3 KSP preconditioned resid norm 9.481308142928e-10 true resid norm 3.725036905334e-08 ||r(i)||/||b|| 9.067704300630e-01 >> ... >> 230 KSP preconditioned resid norm 8.875661094394e-17 true resid norm 1.062030983067e-14 ||r(i)||/||b|| 2.585258389996e-07 >> Linear solve converged due to CONVERGED_RTOL iterations 230 >> KSP Object: 1 MPI processes >> type: tfqmr >> maximum iterations=750, initial guess is zero >> tolerances: relative=1e-07, absolute=1e-50, divergence=10000 >> left preconditioning >> using PRECONDITIONED norm type for convergence test >> PC Object: 1 MPI processes >> type: sor >> SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 >> linear system matrix = precond matrix: >> Matrix Object: 1 MPI processes >> type: seqaij >> rows=1375920, cols=1375920 >> total: nonzeros=17671714, allocated nonzeros=17671714 >> total number of mallocs used during MatSetValues calls =0 >> not using I-node routines >> >> Using Petsc Development HG revision: a4dae06b4d05ef5b78a9a9aa2cbc31b2b158834f HG Date: Wed Feb 08 11:29:52 2012 -0600 >> Configure options: --with-petsc-arch=openmpi-intel-complex-release-f-mkl --with-fortran-interfaces=1 --download-superlu --download-superlu_dist --download-mumps --download-parmetis --download-ptscotch --download-metis --with-scalapack-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_scalapack_lp64.a --with-scalapack-include=/opt/intel/Compiler/11.1/072/mkl/include --with-blacs-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_blacs_openmpi_lp64.a --with-blacs-include=/opt/intel/Compiler/11.1/072/mkl/include --with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2 --with-scalar-type=complex --with-blas-lapack-lib="[/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_lp64.a,/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_thread.a,/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_core.a,/opt/intel/Compiler/11.1/072/lib/intel64/libiomp5.a]" --with-precision=double --with-x=0 --with-debugging=0 >> >> >> Well, two version compiled with different lines, but still I don't think this is the reason... >> Where actually to look at? >> >> Thanks. >> >> On 22.03.2012 15:42, Jed Brown wrote: >>> On Thu, Mar 22, 2012 at 04:23, Alexander Grayver<agrayver(a)gfz-potsdam.de> wrote: >>> I've got the following error when applying bicg+sor to my problem: >>> >>> mt-soft/solveTest> ./solveTest -ksp_view -ksp_converged_reason -ksp_monitor_true_residual -log_summary -mat_type aij -ksp_rtol 1.0e-7 -ksp_type bicg -pc_type sor -ksp_max_it 750 >>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> [0]PETSC ERROR: No support for this operation for this object type! >>> [0]PETSC ERROR: PC does not have apply transpose! >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Development HG revision: 59ca71b85d784abbe22f12c34ffcb2ccc6dc68b0 HG Date: Tue Mar 20 01:05:13 2012 -0500 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Configure options --with-petsc-arch=openmpi-intel-complex-release-c --with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2 --with-scalar-type=complex --with-precision=double --with-blas-lapack-dir=/usr/local/lib --with-x=0 --with-debugging=0 >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: PCApplyTranspose() line 515 in /home/mt/agrayver/lib/petsc-dev/src/ksp/pc/interface/precon.c >>> [0]PETSC ERROR: KSPSolve_BiCG() line 55 in /home/mt/agrayver/lib/petsc-dev/src/ksp/ksp/impls/bicg/bicg.c >>> [0]PETSC ERROR: KSPSolve() line 434 in /home/mt/agrayver/lib/petsc-dev/src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: main() line 114 in /home/solveTest.c >>> >>> Any ideas? >>> >>> PCApplyTranspose_SOR is not implemented. It is unfortunately not an operation that is efficient to implement in-place like SOR. >> >> -- >> Regards, >> Alexander >> -- Regards, Alexander

1 0

Re: [petsc-dev] bad BuildSystem output
by Jed Brown 22 Mar '12

22 Mar '12

On Thu, Mar 22, 2012 at 17:20, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > configure output > > sh: mpicc -o /tmp/petsc-jZ4A3c/config.setCompilers/conftest > /tmp/petsc-jZ4A3c/config.setCompilers/conftest.o > Executing: mpicc -o /tmp/petsc-jZ4A3c/config.setCompilers/conftest > /tmp/petsc-jZ4A3c/config.setCompilers/conftest.o > sh: > ERROR while running executable: > /tmp/petsc-jZ4A3c/config.setCompilers/conftest is not executablesh: mpicc > --help > Executing: mpicc --help > sh: Usage: gcc [options] file... > > > The relevant source > > def outputRun(self, includes, body, cleanup = 1, defaultOutputArg = '', > executor = None): > if not self.checkLink(includes, body, cleanup = 0): return ('', 1) > if not os.path.isfile(self.linkerObj) or not os.access(self.linkerObj, > os.X_OK): > self.framework.log.write('ERROR while running executable: > '+self.linkerObj+' is not executable') > return ('', 1) > > > 1) The error is not "while running" the executable. It is while checking > if the executable file exists or has the x bit, error message is bd > > 2) It should produce a different error message if the executable doesn't > exist vs does not have the x bit > > 3) Why TF is :" mpicc --help" on the same line of the very informative > configure log with the error message? > I only fixed this one http://petsc.cs.iit.edu/petsc/BuildSystem/rev/aae8f08a1f67 > > 4) Just before the "ERROR while running exe" line why doesn't it have a > line "Testing executable NAME to see if it can be run" or some similar > message? Reading the configure file I see oh it is compiling something, oh > executable cannot be run, they don't fit well together for an inexperienced > reader. > > 5) The output printed for the end user is not bad > > Cannot run executables created with C. If this machine uses a batch system > to submit jobs you will need to configure using ./configure with the > additional option --with-batch. > Otherwise there is problem with the compilers. Can you compile and run > code with your C/C++ (and maybe Fortran) compilers? > > except it has the bad vague phrasing "with your C/C++ (and maybe Fortran) > compilers?" it should tell the user EXACTLY what compiler it was trying to > run (which may not be the compiler the user thinks is their "user's > compiler". ) > > It is sloppy output like this that makes debugging configure failures more > painful than they need be. Remember people are going to still be using > this stuff in 20 years but we are not going to be here to interpret the > error conditions. > > Barry > > > > > > Begin forwarded message: > > > From: mahesh natarajan <mn1729(a)gmail.com> > > Date: March 22, 2012 4:46:24 PM CDT > > To: Matthew Knepley <petsc-maint(a)mcs.anl.gov> > > Subject: [petsc-maint #110458] Re : Configuring petsc > > Reply-To: petsc-maint(a)mcs.anl.gov, mahesh natarajan <mn1729(a)gmail.com> > > > > Hi, > > > > I am trying to install petsc on my laptop. But I get an error. PFA log > > files. Any fixes ? Thanks. > > > > > > > > -- > > Regards > > Mahesh Natarajan > > > [see attached file: log_summary.dat] [see attached file: configure.log] >

1 0