November 2015 - petsc-users - mailman3.cels.anl.gov

Re: [petsc-users] .pc file does not include dependencies
by Arne Morten Kvarving 27 Nov '15

27 Nov '15

On 25/11/15 14:07, Matthew Knepley wrote: > On Wed, Nov 25, 2015 at 6:16 AM, Arne Morten Kvarving > <arne.morten.kvarving(a)sintef.no> wrote: > > hi, > > a while back i was told to use the .pc file for petsc if i wanted > pkgconfig style (duh) configuration. > i used to use a (confusingly named) PETScConfig.cmake file for > this, which was not intended for such use. > > problem now is that the pc file of petsc is a bit broken. > > in particular, i want to link petsc static, but not all the > dependencies. if i build petsc (static or not), > the lib entry is listed as > > Libs: -L<path-to-petsc-configuration>/lib -lpetsc > > now, the dependencies are listed in > > Libs.private: /usr/lib/liblapack.a /usr/lib/libblas.a libX11.a > libpthread.a libssl.a libcrypto.a libhwloc.a libm.a > -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -L/usr/lib/x86_64-linux-gnu > -L/lib/x86_64-linux-gnu -lgfortran -lm -lgfortran -lm -lquadmath > -lm -L/usr/lib/gcc/x86_64-linux-gnu/4.9 > -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lstdc++ > -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -L/usr/lib/x86_64-linux-gnu > -L/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -lgcc_s > -ldl -ldl > > which are for a totally static link (you'd use --static with > pkg-config to obtain it). > > the problem with a totally static link, however, is that > pkg-config returns this: > > pkg-config --static --libs PETSc -> > > /usr/lib/liblapack.a /usr/lib/libblas.a libX11.a libpthread.a > libssl.a libcrypto.a libhwloc.a libm.a > -L/home/akva/kode/petsc/current/linux-gnu-cxx-opt/lib > -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -lpetsc -lgfortran -lquadmath > -lm -lstdc++ -lgcc_s -ldl > > which breaks at linking because petsc won't see any of the symbols > in those .a files before (static link is order dependent). > > > I do not understand this explanation. > > 1) Why does the link fail? Please send the error. it's because with static linking, the order of libraries matter. if you do -lfoo -lbar the linker will see the symbols in the bar library when linking foo, but not the other way around. thus the linker do not see the symbols in e.g. libX11.a when linking petsc and you get a lots of inker errors, random line; xops.c:(.text+0x2a9): undefined reference to `XSetForeground' xops.c:(.text+0x3e0): undefined reference to `XDrawLine' /home/akva/kode/petsc/current/linux-gnu-cxx-opt/lib/libpetsc.a(xops.o): In function `PetscDrawArrow_X(_p_PetscDraw*, double, double, double, double, int)': > > 2) Is pkgconfig changing/filtering the link line? That would seem like > a bug in pkgconfig. yes, pkg-config is behaving very weird. i have no idea of the logic here, it's likely a bug. if I 1) move Libs.private below Libs in the .pc file 2) replace the absolute references to archive foo.a with -lfoo it works. i.e. using http://paste.ubuntu.com/13502223/ (ignore the irrelevant changes, that is using the variables instead of absolute paths), i get /usr/lib/liblapack.a /usr/lib/libblas.a libX11.a libpthread.a libssl.a libcrypto.a libhwloc.a libm.a -L/home/akva/kode/petsc/current/linux-gnu-cxx-opt/lib -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -lpetsc -lgfortran -lquadmath -lm -lstdc++ -lgcc_s -ldl which breaks (so step 1 is not enough). if i however use http://paste.ubuntu.com/13502228/ i get -L/home/akva/kode/petsc/current/linux-gnu-cxx-opt/lib -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -lpetsc -llapack -lblas -lX11 -lpthread -lssl -lcrypto -lhwloc -lm -lstdc++ -lgcc_s -ldl as wanted (missing a few -L's in principle for the last file). this is with pkg-config 0.26 (ubuntu trusty). arnem

2 4

problem compiling
by Randall Mackie 26 Nov '15

26 Nov '15

I was trying to recompile PETSc using superlu_dist on a linux system, and I had configure download the necessary packages, and the configure went fine. Compilation bombed out with the error message: /usr/bin/ld cannot find -ldat this came immediately after the ztaulinesearch compile in the make.log. I thought before I mailed off the make.log to petsc-maint I would see if anyone knew the issue here. Thanks, Randy M.

2 2

.pc file does not include dependencies
by Arne Morten Kvarving 25 Nov '15

25 Nov '15

hi, a while back i was told to use the .pc file for petsc if i wanted pkgconfig style (duh) configuration. i used to use a (confusingly named) PETScConfig.cmake file for this, which was not intended for such use. problem now is that the pc file of petsc is a bit broken. in particular, i want to link petsc static, but not all the dependencies. if i build petsc (static or not), the lib entry is listed as Libs: -L<path-to-petsc-configuration>/lib -lpetsc now, the dependencies are listed in Libs.private: /usr/lib/liblapack.a /usr/lib/libblas.a libX11.a libpthread.a libssl.a libcrypto.a libhwloc.a libm.a -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lgfortran -lm -lgfortran -lm -lquadmath -lm -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lstdc++ -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -L/usr/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -ldl -lgcc_s -ldl -ldl which are for a totally static link (you'd use --static with pkg-config to obtain it). the problem with a totally static link, however, is that pkg-config returns this: pkg-config --static --libs PETSc -> /usr/lib/liblapack.a /usr/lib/libblas.a libX11.a libpthread.a libssl.a libcrypto.a libhwloc.a libm.a -L/home/akva/kode/petsc/current/linux-gnu-cxx-opt/lib -L/usr/lib/gcc/x86_64-linux-gnu/4.9 -lpetsc -lgfortran -lquadmath -lm -lstdc++ -lgcc_s -ldl which breaks at linking because petsc won't see any of the symbols in those .a files before (static link is order dependent). so i cannot use this file to do a fully static build either (unless i add -Wl,--whole-archive or equivalent hacks). my suggested fix would be to always list the link dependencies in the Libs: line (and in the proper order - -lpetsc goes first). this works fine with dynamic links (even though -lpetsc would bring in the dependencies) and it works with petsc static. before i try to grok the petsc buildsystem i wanted to check if this would be welcome, or even better if i there is something i'm missing.. arnem

3 2

Re: [petsc-users] .pc file does not include dependencies
by Arne Morten Kvarving 25 Nov '15

25 Nov '15

actually, it's proper behavior according to pkgconfig documentation. all options which do not start with -l or -L is stuck in the '--libs-only-other' output, and these will be first when it builds up the output string with --static --libs. so the proper fix is indeed for petsc to use -l and -L for those static dependencies i believe. sorry for the dupes matthew, reply-to-all-nub-error on my behalf..

2 1

[SLEPc] GD is not deterministic when using different number of cores
by Denis Davydov 25 Nov '15

25 Nov '15

Dear all, I was trying to get some scaling results for the GD eigensolver as applied to the density functional theory. Interestingly enough, the number of self-consistent iterations (solution of couple eigenvalue problem and poisson equations) depends on the number of MPI cores used. For my case the range of iterations is 19-24 for MPI cores between 2 and 160. That makes the whole scaling check useless as the eigenproblem is solved different number of times. That is **not** the case when I use Krylov-Schur eigensolver with zero shift, which makes me believe that I am missing some settings on GD to make it fully deterministic. The only non-deterministic part I am currently aware of is the initial subspace for the first SC iterations. But that’s the case for both KS and GD. For subsequent iterations I provide previously obtained eigenvectors as initial subspace. Certainly there will be some round-off error due to different partition of DoFs for different number of MPI cores, but i don’t expect it to have such a strong influence. Especially given the fact that I don’t see this problem with KS. Below is the output of -eps-view for GD with -eps_type gd -eps_harmonic -st_pc_type bjacobi -eps_gd_krylov_start -eps_target -10.0 I would appreciate any suggestions on how to address the issue. As a side question, why GD uses KSP pre-only? It could as well be using a proper linear solver to apply K^{-1} in the expansion state -- I assume the Olsen variant is the default in SLEPc? Kind regards, Denis EPS Object: 4 MPI processes type: gd Davidson: search subspace is B-orthogonalized Davidson: block size=1 Davidson: type of the initial subspace: Krylov Davidson: size of the subspace after restarting: 6 Davidson: number of vectors after restarting from the previous iteration: 0 problem type: generalized symmetric eigenvalue problem extraction type: harmonic Ritz selected portion of the spectrum: closest to target: -10 (in magnitude) postprocessing eigenvectors with purification number of eigenvalues (nev): 87 number of column vectors (ncv): 175 maximum dimension of projected problem (mpd): 175 maximum number of iterations: 57575 tolerance: 1e-10 convergence test: absolute dimension of user-provided initial space: 87 BV Object: 4 MPI processes type: svec 175 columns of global length 57575 vector orthogonalization method: classical Gram-Schmidt orthogonalization refinement: if needed (eta: 0.7071) block orthogonalization method: Gram-Schmidt non-standard inner product Mat Object: 4 MPI processes type: mpiaij rows=57575, cols=57575 total: nonzeros=1.51135e+06, allocated nonzeros=1.51135e+06 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines doing matmult as a single matrix-matrix product DS Object: 4 MPI processes type: gnhep ST Object: 4 MPI processes type: precond shift: -10 number of matrices: 2 all matrices have different nonzero pattern KSP Object: (st_) 4 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-08, absolute=1e-50, divergence=10000 left preconditioning using DEFAULT norm type for convergence test PC Object: (st_) 4 MPI processes type: bjacobi block Jacobi: number of blocks = 4 Local solve is same for all blocks, in the following KSP and PC objects: KSP Object: (st_sub_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test PC Object: (st_sub_) 1 MPI processes type: ilu ILU: out-of-place factorization 0 levels of fill tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 1, needed 1 Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=15557, cols=15557 package used to perform factorization: petsc total: nonzeros=388947, allocated nonzeros=388947 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=15557, cols=15557 total: nonzeros=388947, allocated nonzeros=388947 total number of mallocs used during MatSetValues calls =0 not using I-node routines linear system matrix = precond matrix: Mat Object: 4 MPI processes type: mpiaij rows=57575, cols=57575 total: nonzeros=1.51135e+06, allocated nonzeros=1.51135e+06 total number of mallocs used during MatSetValues calls =0 not using I-node (on process 0) routines

2 4

Speaker for your Event
by Nick Scott 24 Nov '15

24 Nov '15

I am a professional speaker and would like to work something out to come and speak. One of my core messages is that overcoming adversity is all about a mindset and I am living proof. To help your Marketing Team make this a successful event I have many high resolutions pictures and other material for them to use, plus I will mail you Free 6x9 Pictures to handout prior to the event. You can see and read more what I can do to help make the event a huge success. www.speakernickscott.com/planning_tools Below is some information of who I am and you can watch me speak live with the demo. I can customize my speech to fit your theme and get across your message in a very powerful way. Looking forward to working something out. On Saturday, September 26, 2015 I was one of the performers for Pope Francis and you can see it here: www.speakernickscott.com/pope All Programs: http://www.speakernickscott.com/all_programs Demo/Speaking: http://www.speakernickscott.com/demo My Biography: http://www.speakernickscott.com/biography Link: Professional Resume & CV <http://speakernickscott.com/i/u/10146476/f/Nick_Scott_s_Resume___CV.pdf> Link: Professional Press Kit <http://speakernickscott.com/i/u/10146476/f/Nick_Scott_s_Press_Kit.pdf> *Entrepreneurism and Education:* - Nick's Movie Perspective is the Overall Best Film Winner at the 2011 Arnold Sports Film Festival - Associate of Science - Bachelor's in Business Administration - Management Programs - Cash Flow Consultant - Certified Fitness Trainer (CFT) - Certified as a Specialist in Performance Nutrition (SPN) *Leadership and Motivation:* - Founder/CEO/President of Wheelchair Athletics, Inc. - Founder/CEO/President of Wheelchair Bodybuilding, Inc. - Published Author - Global Promoter of Wheelchair Bodybuilding - Abilities Expo National Ambassador - Expo's for the Disable Community - Founder/Owner of www.WheelchairBodybuilding.com <http://www.wheelchairbodybuilding.com/> - Professional Speaker - Professional Model - Certified Fitness Trainer - National Spokesperson - Actor *Athletic Achievements:* - Paralypian - Spokes Model - #1 IFBB Professional Wheelchair Bodybuilder in the World - Professional Wheelchair Ballroom Dancer - 2-Time World Powerlifting Champion - Competed and Guest Posed over 200 Bodybuilding Shows/Events around the World - Signed Athlete - Sponsored Athlete Sincerely, *Nick Scott* (785) 418-4191 CEO | President | Founder Wheelchair Athletics Foundation | Wheelchair Bodybuilding, Inc.

1 0

CG+GAMG convergence issues in GHEP Krylov-Schur for some MPI runs
by Denis Davydov 24 Nov '15

24 Nov '15

Dear all, I experience strange convergence problems in SLEPc for GHEP with Krylov-Schur and CG + GAMG. The issue appears to be contingent on the number of MPI cores used. Say for 8 cores there is no issue and for 4 cores there is an issue. When I substitute GAMG with Jacobi for the problematic number of cores -- all works. To be more specific, I solve Ax=\lambda Bx for a sequence of A’s where A is a function of eigenvectors. On each iteration step currently the eigensolver EPS is initialised from scratch. And thus should the underlying ST, KSP, PC objects: -st_ksp_type cg -st_pc_type gamg -st_ksp_rtol 1e-12. For these particular matrices the issue appears on the 4th iteration, even though the matrix to be inverted (mass/overlap matrix) is the same, does not change!!! From my debuging info the A matrix has the same norm for CG + GAMG vs CG + Jacobi cases: DEAL:: frobenius_norm = 365.7 DEAL:: linfty_norm = 19.87 DEAL:: l1_norm = 19.87 Just to be sure that there are no bugs on my side which would result in different mass matrices i check that it has the same norm for CG + GAMG vs CG + Jacobi BEFORE i start iteration: DEAL:: frobenius_norm = 166.4 DEAL:: linfty_norm = 8.342 DEAL:: l1_norm = 8.342 All the dependent scalar quantities I calculate on each iteration are identical for the two cases, which makes me believe that the solution path is the same up to the certain tolerance. The only output which is slightly different are the number iterations for convergence in EPS (e.g. 113 vs 108) and the resulting maxing EPSComputeResidualNorm : 4.1524e-07 vs 2.9639e-08. Any ideas what could be an issue, especially given the fact that it does work for some numbers of cores and does not for other? Perhaps some default settings in GAMG preconditioner? Although that does not explain why it works for the first 3 iterations and does not on 4th as the mass matrix is unchanged... Lastly, i suppose ideally i should keep the eigensolver context between the iterations and just update the matrices by EPSSetOperators. Is it correct to assume that since B matrix does not change between iterations and I use the default shift transformation with zero shift (operator is B^{-1)A ), the GAMG preconditioner will not be re-initialised and thus I should save some time? p.s. the relevant error message is below. I have the same issues on CentOS cluster, so it is not related to OS-X. Kind regards, Denis === [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] KSPSolve line 510 /private/tmp/petsc20151102-50378-1t7b3in/petsc-3.6.2/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [0] STMatSolve line 148 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/sys/classes/st/interface/stsles.c [0]PETSC ERROR: [0] STApply_Shift line 33 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/sys/classes/st/impls/shift/shift.c [0]PETSC ERROR: [0] STApply line 50 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/sys/classes/st/interface/stsolve.c [0]PETSC ERROR: [0] EPSGetStartVector line 726 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/eps/interface/epssolve.c [0]PETSC ERROR: [0] EPSSolve_KrylovSchur_Symm line 41 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/eps/impls/krylov/krylovschur/ks-symm.c [0]PETSC ERROR: [0] EPSSolve line 83 /private/tmp/slepc20151102-3081-1xln4h0/slepc-3.6.1/src/eps/interface/epssolve.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015 [0]PETSC ERROR: /Users/davydden/Desktop/work/C++/deal.ii-dft/build_debug~/dft on a real named MBP-Denis.fritz.box by davydden Tue Nov 3 07:02:47 2015 [0]PETSC ERROR: Configure options CC=/usr/local/bin/mpicc CXX=/usr/local/bin/mpicxx F77=/usr/local/bin/mpif77 FC=/usr/local/bin/mpif90 --with-shared-libraries=1 --with-pthread=0 --with-openmp=0 --with-debugging=1 --with-ssl=0 --with-superlu_dist-include=/usr/local/opt/superlu_dist/include/superlu_dist --with-superlu_dist-lib="-L/usr/local/opt/superlu_dist/lib -lsuperlu_dist" --with-superlu-include=/usr/local/Cellar/superlu43/4.3/include/superlu --with-superlu-lib="-L/usr/local/Cellar/superlu43/4.3/lib -lsuperlu" --with-fftw-dir=/usr/local/opt/fftw --with-netcdf-dir=/usr/local/opt/netcdf --with-suitesparse-dir=/usr/local/opt/suite-sparse --with-hdf5-dir=/usr/local/opt/hdf5 --with-metis-dir=/usr/local/opt/metis --with-parmetis-dir=/usr/local/opt/parmetis --with-scalapack-dir=/usr/local/opt/scalapack --with-mumps-dir=/usr/local/opt/mumps --with-x=0 --prefix=/usr/local/Cellar/petsc/3.6.2/real --with-scalar-type=real --with-hypre-dir=/usr/local/opt/hypre --with-sundials-dir=/usr/local/opt/sundials --with-hwloc-dir=/usr/local/opt/hwloc [0]PETSC ERROR: #8 User provided function() line 0 in unknown file -------------------------------------------------------------------------- mpirun noticed that process rank 3 with PID 96754 on node MBP-Denis exited on signal 6 (Abort trap: 6). --------------------------------------------------------------------------

6 30

Using PFFT within PETSc
by Giuseppe Pitton 24 Nov '15

24 Nov '15

Dear users and developers, I am trying to interface PETSc with the parallel fast Fourier transform library PFFT (https://www-user.tu-chemnitz.de/~potts/workgroup/pippig/software.php.en) based in turn on FFTW. My plan is to build a spectral differentiation code, and in the attached files you can see a simple example. The code works correctly in serial, but in parallel there are some problems regarding the output of the results, I think due to some differences in the way PETSc and PFFT store data, but I'm not sure if this is really the issue. In the attached code, the number of processors used should be specified at compile time in the variable "ncpus". As long as ncpus = 1, everything works fine, but if ncpus = 2 or an higher power of 2, the code terminates correctly but the results show some artifacts, as you can see from the generated hdf5 file, named "output-pfft.h5". In the makefile the variables PFFTINC, PFFTLIB, FFTWINC and FFTWLIB should be set correctly. Thank you, Giuseppe

3 5

Debugging failed solve (what's an acceptable upper bound to the condition number?)
by Alex Lindsay 23 Nov '15

23 Nov '15

Hello, I have an application built on top of the Moose framework, and I'm trying to debug a solve that is not converging. My linear solve converges very nicely. However, my non-linear solve does not, and the problem appears to be in the line search. Reading the PetSc FAQ, I see that the most common cause of poor line searches are bad Jacobians. However, I'm using a finite-differenced Jacobian; if I run -snes_type=test, I get "norm of matrix ratios" < 1e-15. Thus in this case the Jacobian should be accurate. I'm wondering then if my problem might be these (taken from the FAQ page): * The matrix is very ill-conditioned. Check the condition number <http://www.mcs.anl.gov/petsc/documentation/faq.html#conditionnumber>. o Try to improve it by choosing the relative scaling of components/boundary conditions. o Try |-ksp_diagonal_scale -ksp_diagonal_scale_fix|. o Perhaps change the formulation of the problem to produce more friendly algebraic equations. * The matrix is nonlinear (e.g. evaluated using finite differencing of a nonlinear function). Try different differencing parameters, |./configure --with-precision=__float128 --download-f2cblaslapack|, check if it converges in "easier" parameter regimes. I'm almost ashamed to share my condition number because I'm sure it must be absurdly high. Without applying -ksp_diagonal_scale and -ksp_diagonal_scale_fix, the condition number is around 1e25. When I do apply those two parameters, the condition number is reduced to 1e17. Even after scaling all my variable residuals so that they were all on the order of unity (a suggestion on the Moose list), I still have a condition number of 1e12. I have no experience with condition numbers, but knowing that perfect condition number is unity, 1e12 seems unacceptable. What's an acceptable upper limit on the condition number? Is it problem dependent? Having already tried scaling the individual variable residuals, I'm not exactly sure what my next method would be for trying to reduce the condition number. I definitely have a nonlinear problem. Could I be having problems because I'm finite differencing non-linear residuals to form my Jacobian? I can see about using a different differencing parameter. I'm also going to consider trying quad precision. However, my hypothesis is that my condition number is the fundamental problem. Is that a reasonable hypothesis? If it's useful, below is console output with -pc_type=svd Time Step 1, time = 1e-10 dt = 1e-10 |residual|_2 of individual variables: potential: 8.12402e+07 potentialliq: 0.000819748 em: 49.206 emliq: 3.08187e-11 Arp: 2375.94 0 Nonlinear |R| = 8.124020e+07 SVD: condition number 1.457087640207e+12, 0 of 851 singular values are (nearly) zero SVD: smallest singular values: 5.637144317564e-09 9.345415388433e-08 4.106132915572e-05 1.017339655185e-04 1.147649477723e-04 SVD: largest singular values : 1.498505466947e+03 1.577560767570e+03 1.719172328193e+03 2.344218235296e+03 8.213813311188e+03 0 KSP unpreconditioned resid norm 3.185019606208e+05 true resid norm 3.185019606208e+05 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 6.382886902896e-07 true resid norm 6.382761808414e-07 ||r(i)||/||b|| 2.003994511046e-12 Linear solve converged due to CONVERGED_RTOL iterations 1 Line search: Using full step: fnorm 8.124020470169e+07 gnorm 1.097605946684e+01 |residual|_2 of individual variables: potential: 8.60047 potentialliq: 0.335436 em: 2.26472 emliq: 0.642578 Arp: 6.39151 1 Nonlinear |R| = 1.097606e+01 SVD: condition number 1.457473763066e+12, 0 of 851 singular values are (nearly) zero SVD: smallest singular values: 5.637185516434e-09 9.347128557672e-08 1.017339655587e-04 1.146760266781e-04 4.064422034774e-04 SVD: largest singular values : 1.498505466944e+03 1.577544976882e+03 1.718956369043e+03 2.343692402876e+03 8.216049987736e+03 0 KSP unpreconditioned resid norm 2.653715381459e+01 true resid norm 2.653715381459e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 6.031179341420e-05 true resid norm 6.031183387732e-05 ||r(i)||/||b|| 2.272731819648e-06 Linear solve converged due to CONVERGED_RTOL iterations 1 Line search: gnorm after quadratic fit 2.485190757827e+11 Line search: Cubic step no good, shrinking lambda, current gnorm 2.632996340352e+10 lambda=5.0000000000000003e-02 Line search: Cubic step no good, shrinking lambda, current gnorm 4.290675557416e+09 lambda=2.5000000000000001e-02 Line search: Cubic step no good, shrinking lambda, current gnorm 4.332980055153e+08 lambda=1.2500000000000001e-02 Line search: Cubic step no good, shrinking lambda, current gnorm 1.677118626669e+07 lambda=6.2500000000000003e-03 Line search: Cubic step no good, shrinking lambda, current gnorm 1.024469780306e+05 lambda=3.1250000000000002e-03 Line search: Cubic step no good, shrinking lambda, current gnorm 7.011543252988e+03 lambda=1.5625000000000001e-03 Line search: Cubic step no good, shrinking lambda, current gnorm 1.750171277470e+03 lambda=7.8125000000000004e-04 Line search: Cubic step no good, shrinking lambda, current gnorm 3.486970625406e+02 lambda=3.4794637057251714e-04 Line search: Cubic step no good, shrinking lambda, current gnorm 7.830624839582e+01 lambda=1.5977866967992950e-04 Line search: Cubic step no good, shrinking lambda, current gnorm 2.147529381328e+01 lambda=6.8049915671999093e-05 Line search: Cubic step no good, shrinking lambda, current gnorm 1.138950943123e+01 lambda=1.7575203052774536e-05 Line search: Cubically determined step, current gnorm 1.095195976135e+01 lambda=1.7575203052774537e-06 |residual|_2 of individual variables: potential: 8.59984 potentialliq: 0.395753 em: 2.26492 emliq: 0.642578 Arp: 6.34735 2 Nonlinear |R| = 1.095196e+01 SVD: condition number 1.457459214030e+12, 0 of 851 singular values are (nearly) zero SVD: smallest singular values: 5.637295371943e-09 9.347057884198e-08 1.017339655949e-04 1.146738253493e-04 4.064421554132e-04 SVD: largest singular values : 1.498505466946e+03 1.577543742603e+03 1.718948052797e+03 2.343672206864e+03 8.216128082047e+03 0 KSP unpreconditioned resid norm 2.653244141805e+01 true resid norm 2.653244141805e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 4.480869560737e-05 true resid norm 4.480686665183e-05 ||r(i)||/||b|| 1.688757771886e-06 Linear solve converged due to CONVERGED_RTOL iterations 1 Line search: gnorm after quadratic fit 2.481752147885e+11 Line search: Cubic step no good, shrinking lambda, current gnorm 2.631959989642e+10 lambda=5.0000000000000003e-02 Line search: Cubic step no good, shrinking lambda, current gnorm 4.289110800463e+09 lambda=2.5000000000000001e-02 Line search: Cubic step no good, shrinking lambda, current gnorm 4.332043942482e+08 lambda=1.2500000000000001e-02 Line search: Cubic step no good, shrinking lambda, current gnorm 1.677933337886e+07 lambda=6.2500000000000003e-03 Line search: Cubic step no good, shrinking lambda, current gnorm 1.027980597206e+05 lambda=3.1250000000000002e-03 Line search: Cubic step no good, shrinking lambda, current gnorm 7.054113639063e+03 lambda=1.5625000000000001e-03 Line search: Cubic step no good, shrinking lambda, current gnorm 1.771258630210e+03 lambda=7.8125000000000004e-04 Line search: Cubic step no good, shrinking lambda, current gnorm 3.517070127496e+02 lambda=3.4519087020105563e-04 Line search: Cubic step no good, shrinking lambda, current gnorm 7.844350966118e+01 lambda=1.5664532891249369e-04 Line search: Cubic step no good, shrinking lambda, current gnorm 2.114833995101e+01 lambda=6.5367917100814859e-05 Line search: Cubic step no good, shrinking lambda, current gnorm 1.144636844292e+01 lambda=1.6044984646715980e-05 Line search: Cubic step no good, shrinking lambda, current gnorm 1.095640770627e+01 lambda=1.6044984646715980e-06 Line search: Cubic step no good, shrinking lambda, current gnorm 1.095196729511e+01 lambda=1.6044984646715980e-07 Line search: Cubically determined step, current gnorm 1.095195451041e+01 lambda=2.3994454223607641e-08 |residual|_2 of individual variables: potential: 8.59983 potentialliq: 0.396107 em: 2.26492 emliq: 0.642578 Arp: 6.34733 3 Nonlinear |R| = 1.095195e+01 SVD: condition number 1.457474387942e+12, 0 of 851 singular values are (nearly) zero SVD: smallest singular values: 5.637237413167e-09 9.347057670885e-08 1.017339654798e-04 1.146737961973e-04 4.064420550524e-04 SVD: largest singular values : 1.498505466946e+03 1.577543716995e+03 1.718947893048e+03 2.343671853830e+03 8.216129148438e+03 0 KSP unpreconditioned resid norm 2.653237816527e+01 true resid norm 2.653237816527e+01 ||r(i)||/||b|| 1.000000000000e+00 1 KSP unpreconditioned resid norm 8.525213442515e-05 true resid norm 8.527696332776e-05 ||r(i)||/||b|| 3.214071607022e-06 Linear solve converged due to CONVERGED_RTOL iterations 1 Line search: gnorm after quadratic fit 2.481576195523e+11 Line search: Cubic step no good, shrinking lambda, current gnorm 2.632005412624e+10 lambda=5.0000000000000003e-02 Line search: Cubic step no good, shrinking lambda, current gnorm 4.289212002697e+09 lambda=2.5000000000000001e-02 Line search: Cubic step no good, shrinking lambda, current gnorm 4.332196637845e+08 lambda=1.2500000000000001e-02 Line search: Cubic step no good, shrinking lambda, current gnorm 1.678040222943e+07 lambda=6.2500000000000003e-03 Line search: Cubic step no good, shrinking lambda, current gnorm 1.027868984884e+05 lambda=3.1250000000000002e-03 Line search: Cubic step no good, shrinking lambda, current gnorm 7.010733464460e+03 lambda=1.5625000000000001e-03 Line search: Cubic step no good, shrinking lambda, current gnorm 1.751519860441e+03 lambda=7.8125000000000004e-04 Line search: Cubic step no good, shrinking lambda, current gnorm 3.497889916171e+02 lambda=3.4753778542938795e-04 Line search: Cubic step no good, shrinking lambda, current gnorm 7.932631084466e+01 lambda=1.5879606741873878e-04 Line search: Cubic step no good, shrinking lambda, current gnorm 2.194608479634e+01 lambda=6.5716583192912669e-05 Line search: Cubic step no good, shrinking lambda, current gnorm 1.117190149691e+01 lambda=1.1541218569257328e-05 Line search: Cubically determined step, current gnorm 1.093879875464e+01 lambda=1.1541218569257329e-06 |residual|_2 of individual variables: potential: 8.59942 potentialliq: 0.403326 em: 2.26505 emliq: 0.714844 Arp: 6.3169 4 Nonlinear |R| = 1.093880e+01

3 7

Re: [petsc-users] Debugging failed solve (what's an acceptable upper bound to the condition number?)
by Barry Smith 23 Nov '15

23 Nov '15

Always make sure that when you reply it goes to everyone on the mailing list; otherwise you're stuck with only stupid old me trying to understand what is going on. Can you run with everything else the same but use equal permittivities? Do all the huge condition numbers and no convergence of the nonlinear solve go away? Barry > On Nov 20, 2015, at 7:09 PM, Alex Lindsay <adlinds3(a)ncsu.edu> wrote: > > I think I may be honing in on what's causing my problems. I have an interface where I am coupling two different subdomains. Among the physics at the interface is a jump discontinuity in the gradient of the electrical potential (e.g. a jump discontinuity in the electric field), governed by the ratio of the permittivities on either side of the interface. This is implemented in my code like this: > > Real > DGMatDiffusionInt::computeQpResidual(Moose::DGResidualType type) > { > if (_D_neighbor[_qp] < std::numeric_limits<double>::epsilon()) > mooseError("It doesn't appear that DG material properties got passed."); > > Real r = 0; > > switch (type) > { > case Moose::Element: > r += 0.5 * (-_D[_qp] * _grad_u[_qp] * _normals[_qp] + -_D_neighbor[_qp] * _grad_neighbor_value[_qp] * _normals[_qp]) * _test[_i][_qp]; > break; > > case Moose::Neighbor: > r += 0.5 * (_D[_qp] * _grad_u[_qp] * _normals[_qp] + _D_neighbor[_qp] * _grad_neighbor_value[_qp] * _normals[_qp]) * _test_neighbor[_i][_qp]; > break; > } > > return r; > } > > where here _D and _D_neighbor are the permittivities on either side of the interface. Attached are pictures showing the solution using Newton, and the solution using a Jacobin-Free method. Newton's method yields electric fields with opposite signs on either side of the interface, which is physically impossible. The Jacobian-free solution yields electric fields with the same sign, and with the proper ratio (a ratio of 5, equivalent to the ratio of the permittivities). I'm sure if I had the proper numerical analysis background, I might know why Newton's method has a hard time here, but I don't. Could someone explain why? > > Alex > > On 11/20/2015 03:24 PM, Barry Smith wrote: >> Do you really only have 851 variables? >> >> SVD: condition number 1.457087640207e+12, 0 of 851 singular values are (nearly) zero >> >> if so you can use -snes_fd and -ksp_view_pmat binary:filename to save the small matrix and then load it up into >> MATLAB or similar tool to fully analysis its eigenstructure to see the distribution from the tiny values to the large values; Is it just a small number of tiny ones etc. >> >> Note that with such a large condition number the factor the linear system "converges" quickly may be meaningless since a small residual doesn't always mean a small error. The error code still be huge >> >> >> Barry >> >> >> >>> On Nov 20, 2015, at 12:40 PM, Alex Lindsay <adlinds3(a)ncsu.edu> wrote: >>> >>> Hello, >>> >>> I have an application built on top of the Moose framework, and I'm trying to debug a solve that is not converging. My linear solve converges very nicely. However, my non-linear solve does not, and the problem appears to be in the line search. Reading the PetSc FAQ, I see that the most common cause of poor line searches are bad Jacobians. However, I'm using a finite-differenced Jacobian; if I run -snes_type=test, I get "norm of matrix ratios" < 1e-15. Thus in this case the Jacobian should be accurate. I'm wondering then if my problem might be these (taken from the FAQ page): >>> >>> • The matrix is very ill-conditioned. Check the condition number. >>> • Try to improve it by choosing the relative scaling of components/boundary conditions. >>> • Try -ksp_diagonal_scale -ksp_diagonal_scale_fix. >>> • Perhaps change the formulation of the problem to produce more friendly algebraic equations. >>> • The matrix is nonlinear (e.g. evaluated using finite differencing of a nonlinear function). Try different differencing parameters, ./configure --with-precision=__float128 --download-f2cblaslapack, check if it converges in "easier" parameter regimes. >>> I'm almost ashamed to share my condition number because I'm sure it must be absurdly high. Without applying -ksp_diagonal_scale and -ksp_diagonal_scale_fix, the condition number is around 1e25. When I do apply those two parameters, the condition number is reduced to 1e17. Even after scaling all my variable residuals so that they were all on the order of unity (a suggestion on the Moose list), I still have a condition number of 1e12. I have no experience with condition numbers, but knowing that perfect condition number is unity, 1e12 seems unacceptable. What's an acceptable upper limit on the condition number? Is it problem dependent? Having already tried scaling the individual variable residuals, I'm not exactly sure what my next method would be for trying to reduce the condition number. >>> >>> I definitely have a nonlinear problem. Could I be having problems because I'm finite differencing non-linear residuals to form my Jacobian? I can see about using a different differencing parameter. I'm also going to consider trying quad precision. However, my hypothesis is that my condition number is the fundamental problem. Is that a reasonable hypothesis? >>> >>> If it's useful, below is console output with -pc_type=svd >>> >>> Time Step 1, time = 1e-10 >>> dt = 1e-10 >>> |residual|_2 of individual variables: >>> potential: 8.12402e+07 >>> potentialliq: 0.000819748 >>> em: 49.206 >>> emliq: 3.08187e-11 >>> Arp: 2375.94 >>> >>> 0 Nonlinear |R| = 8.124020e+07 >>> SVD: condition number 1.457087640207e+12, 0 of 851 singular values are (nearly) zero >>> SVD: smallest singular values: 5.637144317564e-09 9.345415388433e-08 4.106132915572e-05 1.017339655185e-04 1.147649477723e-04 >>> SVD: largest singular values : 1.498505466947e+03 1.577560767570e+03 1.719172328193e+03 2.344218235296e+03 8.213813311188e+03 >>> 0 KSP unpreconditioned resid norm 3.185019606208e+05 true resid norm 3.185019606208e+05 ||r(i)||/||b|| 1.000000000000e+00 >>> 1 KSP unpreconditioned resid norm 6.382886902896e-07 true resid norm 6.382761808414e-07 ||r(i)||/||b|| 2.003994511046e-12 >>> Linear solve converged due to CONVERGED_RTOL iterations 1 >>> Line search: Using full step: fnorm 8.124020470169e+07 gnorm 1.097605946684e+01 >>> |residual|_2 of individual variables: >>> potential: 8.60047 >>> potentialliq: 0.335436 >>> em: 2.26472 >>> emliq: 0.642578 >>> Arp: 6.39151 >>> >>> 1 Nonlinear |R| = 1.097606e+01 >>> SVD: condition number 1.457473763066e+12, 0 of 851 singular values are (nearly) zero >>> SVD: smallest singular values: 5.637185516434e-09 9.347128557672e-08 1.017339655587e-04 1.146760266781e-04 4.064422034774e-04 >>> SVD: largest singular values : 1.498505466944e+03 1.577544976882e+03 1.718956369043e+03 2.343692402876e+03 8.216049987736e+03 >>> 0 KSP unpreconditioned resid norm 2.653715381459e+01 true resid norm 2.653715381459e+01 ||r(i)||/||b|| 1.000000000000e+00 >>> 1 KSP unpreconditioned resid norm 6.031179341420e-05 true resid norm 6.031183387732e-05 ||r(i)||/||b|| 2.272731819648e-06 >>> Linear solve converged due to CONVERGED_RTOL iterations 1 >>> Line search: gnorm after quadratic fit 2.485190757827e+11 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 2.632996340352e+10 lambda=5.0000000000000003e-02 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 4.290675557416e+09 lambda=2.5000000000000001e-02 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 4.332980055153e+08 lambda=1.2500000000000001e-02 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 1.677118626669e+07 lambda=6.2500000000000003e-03 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 1.024469780306e+05 lambda=3.1250000000000002e-03 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 7.011543252988e+03 lambda=1.5625000000000001e-03 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 1.750171277470e+03 lambda=7.8125000000000004e-04 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 3.486970625406e+02 lambda=3.4794637057251714e-04 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 7.830624839582e+01 lambda=1.5977866967992950e-04 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 2.147529381328e+01 lambda=6.8049915671999093e-05 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 1.138950943123e+01 lambda=1.7575203052774536e-05 >>> Line search: Cubically determined step, current gnorm 1.095195976135e+01 lambda=1.7575203052774537e-06 >>> |residual|_2 of individual variables: >>> potential: 8.59984 >>> potentialliq: 0.395753 >>> em: 2.26492 >>> emliq: 0.642578 >>> Arp: 6.34735 >>> >>> 2 Nonlinear |R| = 1.095196e+01 >>> SVD: condition number 1.457459214030e+12, 0 of 851 singular values are (nearly) zero >>> SVD: smallest singular values: 5.637295371943e-09 9.347057884198e-08 1.017339655949e-04 1.146738253493e-04 4.064421554132e-04 >>> SVD: largest singular values : 1.498505466946e+03 1.577543742603e+03 1.718948052797e+03 2.343672206864e+03 8.216128082047e+03 >>> 0 KSP unpreconditioned resid norm 2.653244141805e+01 true resid norm 2.653244141805e+01 ||r(i)||/||b|| 1.000000000000e+00 >>> 1 KSP unpreconditioned resid norm 4.480869560737e-05 true resid norm 4.480686665183e-05 ||r(i)||/||b|| 1.688757771886e-06 >>> Linear solve converged due to CONVERGED_RTOL iterations 1 >>> Line search: gnorm after quadratic fit 2.481752147885e+11 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 2.631959989642e+10 lambda=5.0000000000000003e-02 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 4.289110800463e+09 lambda=2.5000000000000001e-02 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 4.332043942482e+08 lambda=1.2500000000000001e-02 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 1.677933337886e+07 lambda=6.2500000000000003e-03 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 1.027980597206e+05 lambda=3.1250000000000002e-03 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 7.054113639063e+03 lambda=1.5625000000000001e-03 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 1.771258630210e+03 lambda=7.8125000000000004e-04 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 3.517070127496e+02 lambda=3.4519087020105563e-04 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 7.844350966118e+01 lambda=1.5664532891249369e-04 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 2.114833995101e+01 lambda=6.5367917100814859e-05 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 1.144636844292e+01 lambda=1.6044984646715980e-05 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 1.095640770627e+01 lambda=1.6044984646715980e-06 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 1.095196729511e+01 lambda=1.6044984646715980e-07 >>> Line search: Cubically determined step, current gnorm 1.095195451041e+01 lambda=2.3994454223607641e-08 >>> |residual|_2 of individual variables: >>> potential: 8.59983 >>> potentialliq: 0.396107 >>> em: 2.26492 >>> emliq: 0.642578 >>> Arp: 6.34733 >>> >>> 3 Nonlinear |R| = 1.095195e+01 >>> SVD: condition number 1.457474387942e+12, 0 of 851 singular values are (nearly) zero >>> SVD: smallest singular values: 5.637237413167e-09 9.347057670885e-08 1.017339654798e-04 1.146737961973e-04 4.064420550524e-04 >>> SVD: largest singular values : 1.498505466946e+03 1.577543716995e+03 1.718947893048e+03 2.343671853830e+03 8.216129148438e+03 >>> 0 KSP unpreconditioned resid norm 2.653237816527e+01 true resid norm 2.653237816527e+01 ||r(i)||/||b|| 1.000000000000e+00 >>> 1 KSP unpreconditioned resid norm 8.525213442515e-05 true resid norm 8.527696332776e-05 ||r(i)||/||b|| 3.214071607022e-06 >>> Linear solve converged due to CONVERGED_RTOL iterations 1 >>> Line search: gnorm after quadratic fit 2.481576195523e+11 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 2.632005412624e+10 lambda=5.0000000000000003e-02 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 4.289212002697e+09 lambda=2.5000000000000001e-02 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 4.332196637845e+08 lambda=1.2500000000000001e-02 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 1.678040222943e+07 lambda=6.2500000000000003e-03 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 1.027868984884e+05 lambda=3.1250000000000002e-03 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 7.010733464460e+03 lambda=1.5625000000000001e-03 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 1.751519860441e+03 lambda=7.8125000000000004e-04 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 3.497889916171e+02 lambda=3.4753778542938795e-04 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 7.932631084466e+01 lambda=1.5879606741873878e-04 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 2.194608479634e+01 lambda=6.5716583192912669e-05 >>> Line search: Cubic step no good, shrinking lambda, current gnorm 1.117190149691e+01 lambda=1.1541218569257328e-05 >>> Line search: Cubically determined step, current gnorm 1.093879875464e+01 lambda=1.1541218569257329e-06 >>> |residual|_2 of individual variables: >>> potential: 8.59942 >>> potentialliq: 0.403326 >>> em: 2.26505 >>> emliq: 0.714844 >>> Arp: 6.3169 >>> >>> 4 Nonlinear |R| = 1.093880e+01 >>> > > <EFields_NEWTON.png><EFields_PJFNK.png>

2 1