March 2012 - petsc-dev - mailman3.cels.anl.gov

bad BuildSystem output
by Barry Smith 22 Mar '12

22 Mar '12

configure output sh: mpicc -o /tmp/petsc-jZ4A3c/config.setCompilers/conftest /tmp/petsc-jZ4A3c/config.setCompilers/conftest.o Executing: mpicc -o /tmp/petsc-jZ4A3c/config.setCompilers/conftest /tmp/petsc-jZ4A3c/config.setCompilers/conftest.o sh: ERROR while running executable: /tmp/petsc-jZ4A3c/config.setCompilers/conftest is not executablesh: mpicc --help Executing: mpicc --help sh: Usage: gcc [options] file... The relevant source def outputRun(self, includes, body, cleanup = 1, defaultOutputArg = '', executor = None): if not self.checkLink(includes, body, cleanup = 0): return ('', 1) if not os.path.isfile(self.linkerObj) or not os.access(self.linkerObj, os.X_OK): self.framework.log.write('ERROR while running executable: '+self.linkerObj+' is not executable') return ('', 1) 1) The error is not "while running" the executable. It is while checking if the executable file exists or has the x bit, error message is bd 2) It should produce a different error message if the executable doesn't exist vs does not have the x bit 3) Why TF is :" mpicc --help" on the same line of the very informative configure log with the error message? 4) Just before the "ERROR while running exe" line why doesn't it have a line "Testing executable NAME to see if it can be run" or some similar message? Reading the configure file I see oh it is compiling something, oh executable cannot be run, they don't fit well together for an inexperienced reader. 5) The output printed for the end user is not bad Cannot run executables created with C. If this machine uses a batch system to submit jobs you will need to configure using ./configure with the additional option --with-batch. Otherwise there is problem with the compilers. Can you compile and run code with your C/C++ (and maybe Fortran) compilers? except it has the bad vague phrasing "with your C/C++ (and maybe Fortran) compilers?" it should tell the user EXACTLY what compiler it was trying to run (which may not be the compiler the user thinks is their "user's compiler". ) It is sloppy output like this that makes debugging configure failures more painful than they need be. Remember people are going to still be using this stuff in 20 years but we are not going to be here to interpret the error conditions. Barry Begin forwarded message: > From: mahesh natarajan <mn1729(a)gmail.com> > Date: March 22, 2012 4:46:24 PM CDT > To: Matthew Knepley <petsc-maint(a)mcs.anl.gov> > Subject: [petsc-maint #110458] Re : Configuring petsc > Reply-To: petsc-maint(a)mcs.anl.gov, mahesh natarajan <mn1729(a)gmail.com> > > Hi, > > I am trying to install petsc on my laptop. But I get an error. PFA log > files. Any fixes ? Thanks. > > > > -- > Regards > Mahesh Natarajan > [see attached file: log_summary.dat] [see attached file: configure.log]

1 0

Re: [petsc-dev] PCApplyTranspose in BiCG
by Barry Smith 22 Mar '12

22 Mar '12

Ok, first thing to confirm is if other KSP also produce different results. Run both exactly the same but use -ksp_type gmres on both; does it produce different results or the same? Barry On Mar 22, 2012, at 12:30 PM, Alexander Grayver wrote: > Jed, > > Ok, I see. Initially I've changed solver because I observed one thing. I have two petsc-dev (one is older than the other) versions compiled. > Running the same program with both gives different results: > > ./solveTest -ksp_view -ksp_converged_reason -ksp_monitor_true_residual -log_summary -mat_type aij -ksp_rtol 1.0e-7 -ksp_type tfqmr -pc_type sor -ksp_max_it 750 > 0 KSP preconditioned resid norm 9.760858225033e-10 true resid norm 4.108026443997e-08 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 9.497939702484e-10 true resid norm 3.591828545356e-08 ||r(i)||/||b|| 8.743440662618e-01 > 1 KSP preconditioned resid norm 1.341212527158e-09 true resid norm 3.754099340764e-08 ||r(i)||/||b|| 9.138449793207e-01 > 2 KSP preconditioned resid norm 9.481744679086e-10 true resid norm 3.710424003325e-08 ||r(i)||/||b|| 9.032132713620e-01 > 2 KSP preconditioned resid norm 1.340878639500e-09 true resid norm 3.730261027312e-08 ||r(i)||/||b|| 9.080421166135e-01 > 3 KSP preconditioned resid norm 9.481415577317e-10 true resid norm 3.725585425324e-08 ||r(i)||/||b|| 9.069039540306e-01 > ... > 750 KSP preconditioned resid norm 1.340874209607e-09 true resid norm 3.726506102951e-08 ||r(i)||/||b|| 9.071280708031e-01 > Linear solve did not converge due to DIVERGED_ITS iterations 750 > KSP Object: 1 MPI processes > type: tfqmr > maximum iterations=750, initial guess is zero > tolerances: relative=1e-07, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=1375920, cols=1375920 > total: nonzeros=17671714, allocated nonzeros=17671714 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > > Using Petsc Development HG revision: 59ca71b85d784abbe22f12c34ffcb2ccc6dc68b0 HG Date: Tue Mar 20 01:05:13 2012 -0500 > Configure options: --with-petsc-arch=openmpi-intel-complex-release-c --with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2 --with-scalar-type=complex --with-precision=double --with-blas-lapack-dir=/usr/local/lib --with-x=0 --with-debugging=0 > > ####################################### > And now second one: > ./solveTest -ksp_view -ksp_converged_reason -ksp_monitor_true_residual -log_summary -mat_type aij -ksp_rtol 1.0e-7 -ksp_type tfqmr -pc_type sor -ksp_max_it 750 > 0 KSP preconditioned resid norm 9.760858225034e-10 true resid norm 4.108026443997e-08 ||r(i)||/||b|| 1.000000000000e+00 > 1 KSP preconditioned resid norm 9.497930576113e-10 true resid norm 3.591633653854e-08 ||r(i)||/||b|| 8.742966246242e-01 > 1 KSP preconditioned resid norm 1.341211098002e-09 true resid norm 3.754049143693e-08 ||r(i)||/||b|| 9.138327600539e-01 > 2 KSP preconditioned resid norm 9.481662292281e-10 true resid norm 3.709648505700e-08 ||r(i)||/||b|| 9.030244951612e-01 > 2 KSP preconditioned resid norm 1.340863953080e-09 true resid norm 3.730358317575e-08 ||r(i)||/||b|| 9.080657995827e-01 > 3 KSP preconditioned resid norm 9.481308142928e-10 true resid norm 3.725036905334e-08 ||r(i)||/||b|| 9.067704300630e-01 > ... > 230 KSP preconditioned resid norm 8.875661094394e-17 true resid norm 1.062030983067e-14 ||r(i)||/||b|| 2.585258389996e-07 > Linear solve converged due to CONVERGED_RTOL iterations 230 > KSP Object: 1 MPI processes > type: tfqmr > maximum iterations=750, initial guess is zero > tolerances: relative=1e-07, absolute=1e-50, divergence=10000 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: sor > SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 > linear system matrix = precond matrix: > Matrix Object: 1 MPI processes > type: seqaij > rows=1375920, cols=1375920 > total: nonzeros=17671714, allocated nonzeros=17671714 > total number of mallocs used during MatSetValues calls =0 > not using I-node routines > > Using Petsc Development HG revision: a4dae06b4d05ef5b78a9a9aa2cbc31b2b158834f HG Date: Wed Feb 08 11:29:52 2012 -0600 > Configure options: --with-petsc-arch=openmpi-intel-complex-release-f-mkl --with-fortran-interfaces=1 --download-superlu --download-superlu_dist --download-mumps --download-parmetis --download-ptscotch --download-metis --with-scalapack-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_scalapack_lp64.a --with-scalapack-include=/opt/intel/Compiler/11.1/072/mkl/include --with-blacs-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_blacs_openmpi_lp64.a --with-blacs-include=/opt/intel/Compiler/11.1/072/mkl/include --with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2 --with-scalar-type=complex --with-blas-lapack-lib="[/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_lp64.a,/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_thread.a,/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_core.a,/opt/intel/Compiler/11.1/072/lib/intel64/libiomp5.a]" --with-precision=double --with-x=0 --with-debugging=0 > > > Well, two version compiled with different lines, but still I don't think this is the reason... > Where actually to look at? > > Thanks. > > On 22.03.2012 15:42, Jed Brown wrote: >> On Thu, Mar 22, 2012 at 04:23, Alexander Grayver <agrayver(a)gfz-potsdam.de> wrote: >> I've got the following error when applying bicg+sor to my problem: >> >> mt-soft/solveTest> ./solveTest -ksp_view -ksp_converged_reason -ksp_monitor_true_residual -log_summary -mat_type aij -ksp_rtol 1.0e-7 -ksp_type bicg -pc_type sor -ksp_max_it 750 >> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >> [0]PETSC ERROR: No support for this operation for this object type! >> [0]PETSC ERROR: PC does not have apply transpose! >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Petsc Development HG revision: 59ca71b85d784abbe22f12c34ffcb2ccc6dc68b0 HG Date: Tue Mar 20 01:05:13 2012 -0500 >> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >> [0]PETSC ERROR: See docs/index.html for manual pages. >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: Configure options --with-petsc-arch=openmpi-intel-complex-release-c --with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2 --with-scalar-type=complex --with-precision=double --with-blas-lapack-dir=/usr/local/lib --with-x=0 --with-debugging=0 >> [0]PETSC ERROR: ------------------------------------------------------------------------ >> [0]PETSC ERROR: PCApplyTranspose() line 515 in /home/mt/agrayver/lib/petsc-dev/src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: KSPSolve_BiCG() line 55 in /home/mt/agrayver/lib/petsc-dev/src/ksp/ksp/impls/bicg/bicg.c >> [0]PETSC ERROR: KSPSolve() line 434 in /home/mt/agrayver/lib/petsc-dev/src/ksp/ksp/interface/itfunc.c >> [0]PETSC ERROR: main() line 114 in /home/solveTest.c >> >> Any ideas? >> >> PCApplyTranspose_SOR is not implemented. It is unfortunately not an operation that is efficient to implement in-place like SOR. > > > -- > Regards, > Alexander >

1 0

Re: [petsc-dev] PCApplyTranspose in BiCG
by Alexander Grayver 22 Mar '12

22 Mar '12

Jed, Ok, I see. Initially I've changed solver because I observed one thing. I have two petsc-dev (one is older than the other) versions compiled. Running the same program with both gives different results: ./solveTest -ksp_view -ksp_converged_reason -ksp_monitor_true_residual -log_summary -mat_type aij -ksp_rtol 1.0e-7 -ksp_type tfqmr -pc_type sor -ksp_max_it 750 0 KSP preconditioned resid norm 9.760858225033e-10 true resid norm 4.108026443997e-08 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.497939702484e-10 true resid norm 3.591828545356e-08 ||r(i)||/||b|| 8.743440662618e-01 1 KSP preconditioned resid norm 1.341212527158e-09 true resid norm 3.754099340764e-08 ||r(i)||/||b|| 9.138449793207e-01 2 KSP preconditioned resid norm 9.481744679086e-10 true resid norm 3.710424003325e-08 ||r(i)||/||b|| 9.032132713620e-01 2 KSP preconditioned resid norm 1.340878639500e-09 true resid norm 3.730261027312e-08 ||r(i)||/||b|| 9.080421166135e-01 3 KSP preconditioned resid norm 9.481415577317e-10 true resid norm 3.725585425324e-08 ||r(i)||/||b|| 9.069039540306e-01 ... 750 KSP preconditioned resid norm 1.340874209607e-09 true resid norm 3.726506102951e-08 ||r(i)||/||b|| 9.071280708031e-01 Linear solve did not converge due to DIVERGED_ITS iterations 750 KSP Object: 1 MPI processes type: tfqmr maximum iterations=750, initial guess is zero tolerances: relative=1e-07, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=1375920, cols=1375920 total: nonzeros=17671714, allocated nonzeros=17671714 total number of mallocs used during MatSetValues calls =0 not using I-node routines Using Petsc Development HG revision: 59ca71b85d784abbe22f12c34ffcb2ccc6dc68b0 HG Date: Tue Mar 20 01:05:13 2012 -0500 Configure options: --with-petsc-arch=openmpi-intel-complex-release-c --with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2 --with-scalar-type=complex --with-precision=double --with-blas-lapack-dir=/usr/local/lib --with-x=0 --with-debugging=0 ####################################### And now second one: ./solveTest -ksp_view -ksp_converged_reason -ksp_monitor_true_residual -log_summary -mat_type aij -ksp_rtol 1.0e-7 -ksp_type tfqmr -pc_type sor -ksp_max_it 750 0 KSP preconditioned resid norm 9.760858225034e-10 true resid norm 4.108026443997e-08 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 9.497930576113e-10 true resid norm 3.591633653854e-08 ||r(i)||/||b|| 8.742966246242e-01 1 KSP preconditioned resid norm 1.341211098002e-09 true resid norm 3.754049143693e-08 ||r(i)||/||b|| 9.138327600539e-01 2 KSP preconditioned resid norm 9.481662292281e-10 true resid norm 3.709648505700e-08 ||r(i)||/||b|| 9.030244951612e-01 2 KSP preconditioned resid norm 1.340863953080e-09 true resid norm 3.730358317575e-08 ||r(i)||/||b|| 9.080657995827e-01 3 KSP preconditioned resid norm 9.481308142928e-10 true resid norm 3.725036905334e-08 ||r(i)||/||b|| 9.067704300630e-01 ... 230 KSP preconditioned resid norm 8.875661094394e-17 true resid norm 1.062030983067e-14 ||r(i)||/||b|| 2.585258389996e-07 Linear solve converged due to CONVERGED_RTOL iterations 230 KSP Object: 1 MPI processes type: tfqmr maximum iterations=750, initial guess is zero tolerances: relative=1e-07, absolute=1e-50, divergence=10000 left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: sor SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1 linear system matrix = precond matrix: Matrix Object: 1 MPI processes type: seqaij rows=1375920, cols=1375920 total: nonzeros=17671714, allocated nonzeros=17671714 total number of mallocs used during MatSetValues calls =0 not using I-node routines Using Petsc Development HG revision: a4dae06b4d05ef5b78a9a9aa2cbc31b2b158834f HG Date: Wed Feb 08 11:29:52 2012 -0600 Configure options: --with-petsc-arch=openmpi-intel-complex-release-f-mkl --with-fortran-interfaces=1 --download-superlu --download-superlu_dist --download-mumps --download-parmetis --download-ptscotch --download-metis --with-scalapack-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_scalapack_lp64.a --with-scalapack-include=/opt/intel/Compiler/11.1/072/mkl/include --with-blacs-lib=/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_blacs_openmpi_lp64.a --with-blacs-include=/opt/intel/Compiler/11.1/072/mkl/include --with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2 --with-scalar-type=complex --with-blas-lapack-lib="[/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_lp64.a,/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_thread.a,/opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_core.a,/opt/intel/Compiler/11.1/072/lib/intel64/libiomp5.a]" --with-precision=double --with-x=0 --with-debugging=0 Well, two version compiled with different lines, but still I don't think this is the reason... Where actually to look at? Thanks. On 22.03.2012 15:42, Jed Brown wrote: > On Thu, Mar 22, 2012 at 04:23, Alexander Grayver > <agrayver(a)gfz-potsdam.de <mailto:[email protected]>> wrote: > > I've got the following error when applying bicg+sor to my problem: > > mt-soft/solveTest> ./solveTest -ksp_view -ksp_converged_reason > -ksp_monitor_true_residual -log_summary -mat_type aij -ksp_rtol > 1.0e-7 -ksp_type bicg -pc_type sor -ksp_max_it 750 > [0]PETSC ERROR: --------------------- Error Message > ------------------------------------ > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: PC does not have apply transpose! > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Development HG revision: > 59ca71b85d784abbe22f12c34ffcb2ccc6dc68b0 HG Date: Tue Mar 20 > 01:05:13 2012 -0500 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: Configure options > --with-petsc-arch=openmpi-intel-complex-release-c > --with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2 > --with-scalar-type=complex --with-precision=double > --with-blas-lapack-dir=/usr/local/lib --with-x=0 --with-debugging=0 > [0]PETSC ERROR: > ------------------------------------------------------------------------ > [0]PETSC ERROR: PCApplyTranspose() line 515 in > /home/mt/agrayver/lib/petsc-dev/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPSolve_BiCG() line 55 in > /home/mt/agrayver/lib/petsc-dev/src/ksp/ksp/impls/bicg/bicg.c > [0]PETSC ERROR: KSPSolve() line 434 in > /home/mt/agrayver/lib/petsc-dev/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: main() line 114 in /home/solveTest.c > > Any ideas? > > > PCApplyTranspose_SOR is not implemented. It is unfortunately not an > operation that is efficient to implement in-place like SOR. -- Regards, Alexander

1 0

Re: [petsc-dev] PCApplyTranspose in BiCG
by Jed Brown 22 Mar '12

22 Mar '12

On Thu, Mar 22, 2012 at 04:23, Alexander Grayver <agrayver(a)gfz-potsdam.de>wrote: > I've got the following error when applying bicg+sor to my problem: > > mt-soft/solveTest> ./solveTest -ksp_view -ksp_converged_reason > -ksp_monitor_true_residual -log_summary -mat_type aij -ksp_rtol 1.0e-7 > -ksp_type bicg -pc_type sor -ksp_max_it 750 > [0]PETSC ERROR: --------------------- Error Message > ------------------------------**------ > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: PC does not have apply transpose! > [0]PETSC ERROR: ------------------------------** > ------------------------------**------------ > [0]PETSC ERROR: Petsc Development HG revision: > 59ca71b85d784abbe22f12c34ffcb2**ccc6dc68b0 HG Date: Tue Mar 20 01:05:13 > 2012 -0500 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------** > ------------------------------**------------ > [0]PETSC ERROR: Configure options --with-petsc-arch=openmpi-**intel-complex-release-c > --with-mpi-dir=/opt/mpi/intel/**openmpi-1.4.2 --with-scalar-type=complex > --with-precision=double --with-blas-lapack-dir=/usr/**local/lib > --with-x=0 --with-debugging=0 > [0]PETSC ERROR: ------------------------------** > ------------------------------**------------ > [0]PETSC ERROR: PCApplyTranspose() line 515 in /home/mt/agrayver/lib/petsc- > **dev/src/ksp/pc/interface/**precon.c > [0]PETSC ERROR: KSPSolve_BiCG() line 55 in /home/mt/agrayver/lib/petsc-** > dev/src/ksp/ksp/impls/bicg/**bicg.c > [0]PETSC ERROR: KSPSolve() line 434 in /home/mt/agrayver/lib/petsc-** > dev/src/ksp/ksp/interface/**itfunc.c > [0]PETSC ERROR: main() line 114 in /home/solveTest.c > > Any ideas? > PCApplyTranspose_SOR is not implemented. It is unfortunately not an operation that is efficient to implement in-place like SOR.

1 0

Re: [petsc-dev] PETSc OpenMP benchmarking
by Gerard Gorman 22 Mar '12

22 Mar '12

Hi We have repeated the benchmarks using the Cray compiler. In terms of performance, it is a close call between 2MPI+16OMP and 1MPI+32OMP on the interlagos. As expected from the EPCC OpenMP micro-benchmarks, this is fairing much better than GCC. Also included are the parallel efficiencies when using pure MPI on the node, and pure OpenMP. RCM reordering is used. You can see that the parallel efficiency of the OpenMP implementation starts to increase as cache sharing starts to take effect. You can argue that this effect will diminish for larger problem sizes - but as the ratio of total memory to number of cores becomes more important, these are the kinds of effects you would like to start taking advantage of. Any comments are welcome. Cheers Gerard Gerard Gorman emailed the following on 17/03/12 08:33: > Hi > > We have profiled on Cray compute nodes with two 16-core AMD Opteron > 2.3GHz Interlagos processors, using the same matrix but this time with > -ksp_type cg and -pc_type jacobi. Attached are the logs with the 32 MPI > processes and the 32 OpenMP threads tests. > > Most of the time is in stage 2. As seen previously, MatMult is > performing well, but the overall performance in KSPSolve drops for > OpenMP. I have attached a plot of the (hybrid mpi+openmp time)/(pure > openmp) where all 32 cores are always used. What the graph shows is that > we are always getting better performance in MatMult for pure OpenMP but > there is something additional in KSPSolve that degrades the OpenMP > performance. > > So far we have profiled with oprofile measuring the event > CPU_CLK_UNHALTED, but this has not shown up the bottleneck. So more > digging is required. > > Any suggestions/comments gratefully received. > > Cheers > Gerard > > Gerard Gorman emailed the following on 14/03/12 16:59: >> Hi >> >> Since Vec and most of Mat is now threaded we have started to do more >> detailed profiling. I'm posting these initial tasters from a two socket >> Intel Core Bloomfield processor system (i.e. 8 cores) to stimulate >> discussion. >> >> The matrix comes from a 3D lock exchange problem discretised using a >> continuous Galerkin finite element formulation and has about 450k >> degrees of freedom. >> >> I have configured the simulator (Fluidity - >> http://amcg.ese.ic.ac.uk/Fluidity) to dump out PETSc matrices at each >> solve. These individual matrices are then solved using >> petsc-dev/src/ksp/ksp/examples/tests/ex6 compiled with GCC 4.6.3 >> --with-debugging=0. >> >> The PETSc options are: >> -get_total_flops -pc_type gamg -ksp_type cg -ksp_rtol 1.0e-6 -log_summary >> >> The 3 log files attached are for OMP_NUM_THREADS=1, OMP_NUM_THREADS=8 >> and non-threaded MPI run with 8 processes for comparison. >> >> So the reason this benchmark is interesting is because it is pressure >> which is really stiff , and it uses GAMG as a blackbox. >> >> Using xxdiff to compare the logs I think the interesting points are: >> - Overall OpenMP compares favourably with MPI. >> - OpenMP converged in 2 less iterations than with MPI. Earlier I was >> expecting fewer iterations simply because of the absence of partitions >> to diminish the effectiveness of coarsening. I have not been following >> Mark's GAMG development but it looks repartitioning is being used to get >> around that issue (?). However, the biggest plus is because Chebychev is >> used as a smoother (rather than something difficult to parallelise like >> SSOR), GAMG appears to scale pretty well when threaded with OpenMP. >> - Important operations like MatMult etc perform well. >> - From the summary, "mystage 1" is the main section where OMP appears to >> need more work. We suffer from operations such as MatPtAP and >> MatTrnMatMult for example which we have not got around to looking at yet. >> >> As this is a relatively small and boring UMA machine I have not bothered >> with scaling curves. We are setting the same benchmark up on 32-core >> Interlagos compute nodes at the moment - hopefully these will be ready >> by tomorrow. >> >> Comments welcome. >> >> Cheers >> Gerard >>

1 0

PCApplyTranspose in BiCG
by Alexander Grayver 22 Mar '12

22 Mar '12

I've got the following error when applying bicg+sor to my problem: mt-soft/solveTest> ./solveTest -ksp_view -ksp_converged_reason -ksp_monitor_true_residual -log_summary -mat_type aij -ksp_rtol 1.0e-7 -ksp_type bicg -pc_type sor -ksp_max_it 750 [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: No support for this operation for this object type! [0]PETSC ERROR: PC does not have apply transpose! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development HG revision: 59ca71b85d784abbe22f12c34ffcb2ccc6dc68b0 HG Date: Tue Mar 20 01:05:13 2012 -0500 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Configure options --with-petsc-arch=openmpi-intel-complex-release-c --with-mpi-dir=/opt/mpi/intel/openmpi-1.4.2 --with-scalar-type=complex --with-precision=double --with-blas-lapack-dir=/usr/local/lib --with-x=0 --with-debugging=0 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PCApplyTranspose() line 515 in /home/mt/agrayver/lib/petsc-dev/src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPSolve_BiCG() line 55 in /home/mt/agrayver/lib/petsc-dev/src/ksp/ksp/impls/bicg/bicg.c [0]PETSC ERROR: KSPSolve() line 434 in /home/mt/agrayver/lib/petsc-dev/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: main() line 114 in /home/solveTest.c Any ideas? -- Regards, Alexander

1 0

How does this compare...
by Matthew Knepley 21 Mar '12

21 Mar '12

to the new scalable MatMatMult() that is done in PETSc? Is it possible to even compare the flop rates sensibly? http://arxiv.org/abs/1203.1692 This is what Jeff is planning on using in his SciDAC. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] making BuildSystem hard to debug
by Jed Brown 21 Mar '12

21 Mar '12

On Wed, Mar 21, 2012 at 12:11, Matthew Knepley <knepley(a)gmail.com> wrote: > That was not the problem. It was generated support files. Sure, but all that information can be put into the exception so that it is only printed if it's not caught.

1 0

Re: [petsc-dev] making BuildSystem hard to debug
by Matthew Knepley 21 Mar '12

21 Mar '12

On Wed, Mar 21, 2012 at 12:06 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Wed, Mar 21, 2012 at 12:01, Matthew Knepley <knepley(a)gmail.com> wrote: > >> I am not sure what you mean "wrap the whole test case up". > > > Bundle the environment that was in scope (via your push/pop) with the > exception so that it can be restored when printing diagnostics. > That was not the problem. It was generated support files. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] making BuildSystem hard to debug
by Jed Brown 21 Mar '12

21 Mar '12

On Wed, Mar 21, 2012 at 12:01, Matthew Knepley <knepley(a)gmail.com> wrote: > I am not sure what you mean "wrap the whole test case up". Bundle the environment that was in scope (via your push/pop) with the exception so that it can be restored when printing diagnostics.

1 0