I fixed it; the prefix fix from last week for PFLOTRAN broke it for composed solvers' default options for the line search type, making the richardson smoothers run with the basic linesearch and diverge ASAP. The reason it took so long to show up in dev is due to it being pushed to 3.3 originally, so noticing it was even harder. - Peter On Mon, Aug 13, 2012 at 9:24 PM, Peter Brune <[email protected]> wrote:
Something broke; I think I see what it is from your logs. I'll fix it.
- Peter
On Mon, Aug 13, 2012 at 9:17 PM, Barry Smith <[email protected]> wrote:
What's up with this stuff? Are they bugs? Bad algorithms? What? It is hard to make a major change to PETSc (and use alltests to check that you haven't broken anything) when this crap comes up constantly.
Is there a plan to deal with this or are we just screwed forever?
Barry
1 SNES Function norm 59.1937
[0]PETSC ERROR: --------------------- Error Message
[0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 83 is not-a-number or infinite at end of function: Parameter number 3! [0]PETSC ERROR:
[0]PETSC ERROR: Petsc Development HG revision: bd03fdcd3cf87c5b29ea28a5f4737ae936d9df04 HG Date: Sat Aug 11 20:22:42 2012 -0500 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR:
[0]PETSC ERROR: ./ex5 on a arch-gnu named Barrys-MacBook-Pro.local by barrysmith Mon Aug 13 21:12:18 2012 [0]PETSC ERROR: Libraries linked from /Users/barrysmith/Src/petsc-dev/arch-gnu/lib [0]PETSC ERROR: Configure run at Sat Aug 11 20:25:02 2012 [0]PETSC ERROR: Configure options --download-blacs --download-fftw --download-hypre --download-libyaml --download-metis --download-ml --download-mpich --download-mumps --download-parmetis --download-ptscotch --download-scalapack --download-superlu --download-superlu_dist --download-triangle --download-yaml --with-ams-dir=/Users/barrysmith/Src/ams-dev --with-java --with-openmp --with-server --with-shared-libraries PETSC_ARCH=arch-gnu [0]PETSC ERROR:
[0]PETSC ERROR: SNESComputeFunction() line 1927 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESLineSearchApply_Basic() line 41 in /Users/barrysmith/Src/petsc-dev/src/snes/linesearch/impls/basic/linesearchbasic.c [0]PETSC ERROR: SNESLineSearchApply() line 506 in /Users/barrysmith/Src/petsc-dev/src/snes/linesearch/interface/linesearch.c [0]PETSC ERROR: SNESSolve_NRichardson() line 181 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/richardson/snesrichardson.c [0]PETSC ERROR: SNESSolve() line 3570 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESSolve_NGMRES() line 259 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/ngmres/snesngmres.c [0]PETSC ERROR: SNESSolve() line 3570 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: main() line 152 in src/snes/examples/tutorials/ex5.c application called MPI_Abort(MPI_COMM_WORLD, 72) - process 0 [cli_0]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 72) - process 0 /Users/barrysmith/Src/petsc-dev/src/snes/examples/tutorials Possible problem with with ex5_5_ngmres_richardson, diffs above ========================================= 2,21c2,44 < 1 SNES Function norm 0.18208 < 2 SNES Function norm 0.0272102 < 3 SNES Function norm 0.00491305 < 4 SNES Function norm 0.000996383 < 5 SNES Function norm 0.000243372 < 6 SNES Function norm 6.53869e-05 < 7 SNES Function norm 1.84573e-05 < 8 SNES Function norm 5.30014e-06 < 9 SNES Function norm 1.5228e-06 < 10 SNES Function norm 4.34313e-07 < 11 SNES Function norm 1.22592e-07 < 12 SNES Function norm 3.42365e-08 < 13 SNES Function norm 9.46937e-09 < 14 SNES Function norm 2.59769e-09 < 15 SNES Function norm 7.078e-10 < 16 SNES Function norm 1.918e-10 < 17 SNES Function norm 5.173e-11 < 18 SNES Function norm 1.390e-11 < 19 SNES Function norm < 1.e-11 < 20 SNES Function norm < 1.e-11
[0]PETSC ERROR: --------------------- Error Message
[0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 14 is not-a-number or infinite at end of function: Parameter number 3! [0]PETSC ERROR:
[0]PETSC ERROR: Petsc Development HG revision: bd03fdcd3cf87c5b29ea28a5f4737ae936d9df04 HG Date: Sat Aug 11 20:22:42 2012 -0500 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR:
[0]PETSC ERROR: ./ex5 on a arch-gnu named Barrys-MacBook-Pro.local by barrysmith Mon Aug 13 21:12:19 2012 [0]PETSC ERROR: Libraries linked from /Users/barrysmith/Src/petsc-dev/arch-gnu/lib [0]PETSC ERROR: Configure run at Sat Aug 11 20:25:02 2012 [0]PETSC ERROR: Configure options --download-blacs --download-fftw --download-hypre --download-libyaml --download-metis --download-ml --download-mpich --download-mumps --download-parmetis --download-ptscotch --download-scalapack --download-superlu --download-superlu_dist --download-triangle --download-yaml --with-ams-dir=/Users/barrysmith/Src/ams-dev --with-java --with-openmp --with-server --with-shared-libraries PETSC_ARCH=arch-gnu [0]PETSC ERROR:
[0]PETSC ERROR: SNESComputeFunction() line 1927 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESLineSearchApply_Basic() line 41 in /Users/barrysmith/Src/petsc-dev/src/snes/linesearch/impls/basic/linesearchbasic.c [0]PETSC ERROR: SNESLineSearchApply() line 506 in /Users/barrysmith/Src/petsc-dev/src/snes/linesearch/interface/linesearch.c [0]PETSC ERROR: SNESSolve_NRichardson() line 181 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/richardson/snesrichardson.c [0]PETSC ERROR: SNESSolve() line 3570 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESFASUpSmooth_Private() line 447 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESFASCycle_Multiplicative() line 695 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve_FAS() line 774 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve() line 3570 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESFASCoarseCorrection() line 567 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESFASCycle_Multiplicative() line 694 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve_FAS() line 774 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve() line 3570 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESFASCoarseCorrection() line 567 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESFASCycle_Multiplicative() line 694 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve_FAS() line 774 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve() line 3570 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESFASCoarseCorrection() line 567 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESFASCycle_Multiplicative() line 694 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve_FAS() line 774 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve() line 3570 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESFASCoarseCorrection() line 567 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESFASCycle_Multiplicative() line 694 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve_FAS() line 774 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve() line 3570 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: main() line 152 in src/snes/examples/tutorials/ex5.c application called MPI_Abort(MPI_COMM_WORLD, 72) - process 0 [cli_0]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 72) - process 0 /Users/barrysmith/Src/petsc-dev/src/snes/examples/tutorials Possible problem with with ex5_5_fas, diffs above ========================================= 2,11c2,46 < 1 SNES Function norm 0.156469 < 2 SNES Function norm 0.0115623 < 3 SNES Function norm 0.00145395 < 4 SNES Function norm 0.000151214 < 5 SNES Function norm 1.97033e-05 < 6 SNES Function norm 2.60149e-06 < 7 SNES Function norm 3.26556e-07 < 8 SNES Function norm 4.31521e-08 < 9 SNES Function norm 5.41224e-09 < 10 SNES Function norm 1.18135e-09
[0]PETSC ERROR: --------------------- Error Message
[0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 14 is not-a-number or infinite at end of function: Parameter number 3! [0]PETSC ERROR:
[0]PETSC ERROR: Petsc Development HG revision: bd03fdcd3cf87c5b29ea28a5f4737ae936d9df04 HG Date: Sat Aug 11 20:22:42 2012 -0500 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR:
[0]PETSC ERROR: ./ex5 on a arch-gnu named Barrys-MacBook-Pro.local by barrysmith Mon Aug 13 21:12:20 2012 [0]PETSC ERROR: Libraries linked from /Users/barrysmith/Src/petsc-dev/arch-gnu/lib [0]PETSC ERROR: Configure run at Sat Aug 11 20:25:02 2012 [0]PETSC ERROR: Configure options --download-blacs --download-fftw --download-hypre --download-libyaml --download-metis --download-ml --download-mpich --download-mumps --download-parmetis --download-ptscotch --download-scalapack --download-superlu --download-superlu_dist --download-triangle --download-yaml --with-ams-dir=/Users/barrysmith/Src/ams-dev --with-java --with-openmp --with-server --with-shared-libraries PETSC_ARCH=arch-gnu [0]PETSC ERROR:
[0]PETSC ERROR: SNESComputeFunction() line 1927 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESLineSearchApply_Basic() line 41 in /Users/barrysmith/Src/petsc-dev/src/snes/linesearch/impls/basic/linesearchbasic.c [0]PETSC ERROR: SNESLineSearchApply() line 506 in /Users/barrysmith/Src/petsc-dev/src/snes/linesearch/interface/linesearch.c [0]PETSC ERROR: SNESSolve_NRichardson() line 181 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/richardson/snesrichardson.c [0]PETSC ERROR: SNESSolve() line 3570 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESFASUpSmooth_Private() line 447 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESFASCycle_Multiplicative() line 695 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve_FAS() line 774 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve() line 3570 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESFASCoarseCorrection() line 567 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESFASCycle_Multiplicative() line 694 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve_FAS() line 774 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve() line 3570 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESFASCoarseCorrection() line 567 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESFASCycle_Multiplicative() line 694 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve_FAS() line 774 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve() line 3570 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESFASCoarseCorrection() line 567 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESFASCycle_Multiplicative() line 694 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve_FAS() line 774 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve() line 3570 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESFASCoarseCorrection() line 567 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESFASCycle_Multiplicative() line 694 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve_FAS() line 774 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve() line 3570 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESSolve_NGMRES() line 259 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/ngmres/snesngmres.c [0]PETSC ERROR: SNESSolve() line 3570 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: main() line 152 in src/snes/examples/tutorials/ex5.c application called MPI_Abort(MPI_COMM_WORLD, 72) - process 0 [cli_0]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 72) - process 0 /Users/barrysmith/Src/petsc-dev/src/snes/examples/tutorials Possible problem with with ex5_5_ngmres_fas, diffs above ========================================= 2,51c2,46 < 1 SNES Function norm 0.648993 < 2 SNES Function norm 0.518893 < 3 SNES Function norm 0.439196 < 4 SNES Function norm 0.386902 < 5 SNES Function norm 0.350239 < 6 SNES Function norm 0.322173 < 7 SNES Function norm 0.298913 < 8 SNES Function norm 0.279868 < 9 SNES Function norm 0.26322 < 10 SNES Function norm 0.249106 < 11 SNES Function norm 0.236341 < 12 SNES Function norm 0.225242 < 13 SNES Function norm 0.214976 < 14 SNES Function norm 0.205853 < 15 SNES Function norm 0.168755 < 16 SNES Function norm 0.150005 < 17 SNES Function norm 0.146453 < 18 SNES Function norm 0.122221 < 19 SNES Function norm 0.109397 < 20 SNES Function norm 0.107737 < 21 SNES Function norm 0.106311 < 22 SNES Function norm 0.0907226 < 23 SNES Function norm 0.0813505 < 24 SNES Function norm 0.0796973 < 25 SNES Function norm 0.0782834 < 26 SNES Function norm 0.0770559 < 27 SNES Function norm 0.0758003 < 28 SNES Function norm 0.0746744 < 29 SNES Function norm 0.0740446 < 30 SNES Function norm 0.061927 < 31 SNES Function norm 0.0549197 < 32 SNES Function norm 0.054583 < 33 SNES Function norm 0.0519673 < 34 SNES Function norm 0.0498744 < 35 SNES Function norm 0.0491159 < 36 SNES Function norm 0.0482318 < 37 SNES Function norm 0.047538 < 38 SNES Function norm 0.0468002 < 39 SNES Function norm 0.0461691 < 40 SNES Function norm 0.0459809 < 41 SNES Function norm 0.0420109 < 42 SNES Function norm 0.0389469 < 43 SNES Function norm 0.0383224 < 44 SNES Function norm 0.0378171 < 45 SNES Function norm 0.0375595 < 46 SNES Function norm 0.0373267 < 47 SNES Function norm 0.0370906 < 48 SNES Function norm 0.0368703 < 49 SNES Function norm 0.0366439 < 50 SNES Function norm 0.0364308
1 SNES Function norm 1.73715 2 SNES Function norm 44.1655 3 SNES Function norm 2.72388e+10 [0]PETSC ERROR: --------------------- Error Message
[0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Vec entry at local location 8 is not-a-number or infinite at end of function: Parameter number 3! [0]PETSC ERROR:
[0]PETSC ERROR: Petsc Development HG revision: bd03fdcd3cf87c5b29ea28a5f4737ae936d9df04 HG Date: Sat Aug 11 20:22:42 2012 -0500 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR:
[0]PETSC ERROR: ./ex5 on a arch-gnu named Barrys-MacBook-Pro.local by barrysmith Mon Aug 13 21:12:21 2012 [0]PETSC ERROR: Libraries linked from /Users/barrysmith/Src/petsc-dev/arch-gnu/lib [0]PETSC ERROR: Configure run at Sat Aug 11 20:25:02 2012 [0]PETSC ERROR: Configure options --download-blacs --download-fftw --download-hypre --download-libyaml --download-metis --download-ml --download-mpich --download-mumps --download-parmetis --download-ptscotch --download-scalapack --download-superlu --download-superlu_dist --download-triangle --download-yaml --with-ams-dir=/Users/barrysmith/Src/ams-dev --with-java --with-openmp --with-server --with-shared-libraries PETSC_ARCH=arch-gnu [0]PETSC ERROR:
[0]PETSC ERROR: SNESComputeFunction() line 1927 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESLineSearchApply_Basic() line 41 in /Users/barrysmith/Src/petsc-dev/src/snes/linesearch/impls/basic/linesearchbasic.c [0]PETSC ERROR: SNESLineSearchApply() line 506 in /Users/barrysmith/Src/petsc-dev/src/snes/linesearch/interface/linesearch.c [0]PETSC ERROR: SNESSolve_NRichardson() line 181 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/richardson/snesrichardson.c [0]PETSC ERROR: SNESSolve() line 3570 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESFASDownSmooth_Private() line 420 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESFASCycle_Additive() line 638 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve_FAS() line 776 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve() line 3570 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESFASCycle_Additive() line 635 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve_FAS() line 776 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve() line 3570 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESFASCycle_Additive() line 635 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve_FAS() line 776 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve() line 3570 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESFASCycle_Additive() line 635 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve_FAS() line 776 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve() line 3570 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESFASCycle_Additive() line 635 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve_FAS() line 776 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve() line 3570 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESFASCycle_Additive() line 635 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve_FAS() line 776 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/fas/fas.c [0]PETSC ERROR: SNESSolve() line 3570 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: main() line 152 in src/snes/examples/tutorials/ex5.c application called MPI_Abort(MPI_COMM_WORLD, 72) - process 0 [cli_0]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 72) - process 0 /Users/barrysmith/Src/petsc-dev/src/snes/examples/tutorials Possible problem with with ex5_5_fas_additive, diffs above