November 2011 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Generality of VecScatter
by Matthew Knepley 23 Nov '11

23 Nov '11

On Wed, Nov 23, 2011 at 6:49 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Wed, Nov 23, 2011 at 18:43, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: > >> What does this mean: "where one side has at most one edge"? > > > A "local" point corresponds to at most one "global" point. > > Suppose we have scatter between spaces {a,b,c,d} and {x,y,z}. > > An example of a valid scatter is > > Scatter1 = {(a,y),(b,z),(c,y)} > > This is entirely useful, but I don't know if the following is useful: > > Scatter2 = {(b,y),(b,z),(c,y)} > > Is anyone using scatters that look like Scatter2? For what purpose? > Yes, for mesh distribution Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] Generality of VecScatter
by Jed Brown 23 Nov '11

23 Nov '11

On Wed, Nov 23, 2011 at 18:48, Dmitry Karpeev <karpeev(a)gmail.com> wrote: > > I wonder if we can express all useful communication with a more > restrictive interface similar to a local-to-global map (but mapping between > any vectors) where one side (typically called "local", but doesn't need to > actually be a local Vec) has at most one edge. > That would disallow accumulating a MAX "locally". I don't know what you mean. Say we have a scatter from X (sometimes "local") to Y (sometimes "global"). One way to describe my communication graph is as a single-valued function on X (containing either the index into Y or NULL). I'm asking whether the single-valued constraint is too restrictive for some important use case. The primitives I can readily provide are broadcast from Y to X reduce values on X into Y (min, max, sum, replace, etc) gather values on X into arrays for each point in Y (in some ordering, rank ordering if you like) scatter values in arrays for each point in Y to the point in X These latter two involve a data structure that is as big as the number of sharing processes (because that is explicitly being requested), but are still implemented where the owner of that point in Y has no knowledge of which processes it is communicating with. With the capability of hardware RMA, this can actually be more efficient than if the two-sided graph was available (and it's less code and fewer data structures).

1 0

Re: [petsc-dev] Generality of VecScatter
by Jed Brown 23 Nov '11

23 Nov '11

On Wed, Nov 23, 2011 at 18:43, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: > What does this mean: "where one side has at most one edge"? A "local" point corresponds to at most one "global" point. Suppose we have scatter between spaces {a,b,c,d} and {x,y,z}. An example of a valid scatter is Scatter1 = {(a,y),(b,z),(c,y)} This is entirely useful, but I don't know if the following is useful: Scatter2 = {(b,y),(b,z),(c,y)} Is anyone using scatters that look like Scatter2? For what purpose?

1 0

Re: [petsc-dev] Generality of VecScatter
by Dmitry Karpeev 23 Nov '11

23 Nov '11

On Nov 23, 2011, at 18:28, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > I had a useful conversation with the MPICH guys today about the unstructured communication primitives that Mark, Matt, Barry, and I have been discussing. It's looking like we can do a very thin communication library that has the nice semantics I was looking for, namely that the local-to-global is stored only as indices in the local space. To implement the communication primitives, the owner of a global node never needs to know how many or which processes have that point in their local space. There is an MPI-3 feature that we can use to elegantly avoid two-way knowledge for pointwise gather/scatter. (We can always implement the same interface using vanilla MPI-1, but the MPI-2/3 implementation would have less overhead.) > > If I/we write this layer, VecScatter (with arbitrary data types, etc) could be implemented very easily on top of it. I'm in no rush to do this, but I could imagine getting there eventually. The VecScatter interface allows some things that might not be important. For example, any process can specify an edge in the communication graph, even if it does not own the source or the destination vertex. Also, both source and destination vertices can have degree higher than one. We can even put a link between the same points twice (effectively multiplying that contribution by 2 when using ADD_VALUES). I don't know if this ever makes semantic sense, but the interface allows it. > > I wonder if we can express all useful communication with a more restrictive interface similar to a local-to-global map (but mapping between any vectors) where one side (typically called "local", but doesn't need to actually be a local Vec) has at most one edge. That would disallow accumulating a MAX "locally". > > What weird things do people use VecScatter for that might break with this more restrictive model?

1 0

Re: [petsc-dev] Generality of VecScatter
by Dmitry Karpeev 23 Nov '11

23 Nov '11

On Wed, Nov 23, 2011 at 6:28 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > I had a useful conversation with the MPICH guys today about the > unstructured communication primitives that Mark, Matt, Barry, and I have > been discussing. It's looking like we can do a very thin communication > library that has the nice semantics I was looking for, namely that the > local-to-global is stored only as indices in the local space. To implement > the communication primitives, the owner of a global node never needs to > know how many or which processes have that point in their local space. > There is an MPI-3 feature that we can use to elegantly avoid two-way > knowledge for pointwise gather/scatter. (We can always implement the same > interface using vanilla MPI-1, but the MPI-2/3 implementation would have > less overhead.) > > If I/we write this layer, VecScatter (with arbitrary data types, etc) > could be implemented very easily on top of it. I'm in no rush to do this, > but I could imagine getting there eventually. The VecScatter interface > allows some things that might not be important. For example, any process > can specify an edge in the communication graph, even if it does not own the > source or the destination vertex. Also, both source and destination > vertices can have degree higher than one. We can even put a link between > the same points twice (effectively multiplying that contribution by 2 when > using ADD_VALUES). I don't know if this ever makes semantic sense, but the > interface allows it. > > I wonder if we can express all useful communication with a more > restrictive interface similar to a local-to-global map (but mapping between > any vectors) where one side (typically called "local", but doesn't need to > actually be a local Vec) has at most one edge. > What does this mean: "where one side has at most one edge"? > > What weird things do people use VecScatter for that might break with this > more restrictive model? >

1 0

Generality of VecScatter
by Jed Brown 23 Nov '11

23 Nov '11

I had a useful conversation with the MPICH guys today about the unstructured communication primitives that Mark, Matt, Barry, and I have been discussing. It's looking like we can do a very thin communication library that has the nice semantics I was looking for, namely that the local-to-global is stored only as indices in the local space. To implement the communication primitives, the owner of a global node never needs to know how many or which processes have that point in their local space. There is an MPI-3 feature that we can use to elegantly avoid two-way knowledge for pointwise gather/scatter. (We can always implement the same interface using vanilla MPI-1, but the MPI-2/3 implementation would have less overhead.) If I/we write this layer, VecScatter (with arbitrary data types, etc) could be implemented very easily on top of it. I'm in no rush to do this, but I could imagine getting there eventually. The VecScatter interface allows some things that might not be important. For example, any process can specify an edge in the communication graph, even if it does not own the source or the destination vertex. Also, both source and destination vertices can have degree higher than one. We can even put a link between the same points twice (effectively multiplying that contribution by 2 when using ADD_VALUES). I don't know if this ever makes semantic sense, but the interface allows it. I wonder if we can express all useful communication with a more restrictive interface similar to a local-to-global map (but mapping between any vectors) where one side (typically called "local", but doesn't need to actually be a local Vec) has at most one edge. What weird things do people use VecScatter for that might break with this more restrictive model?

1 0

Re: [petsc-dev] uni
by Mark F. Adams 23 Nov '11

23 Nov '11

On Nov 23, 2011, at 6:10 PM, Barry Smith wrote: > > On Nov 23, 2011, at 3:14 PM, Mark F. Adams wrote: > >> I fixed my code error with valgrind and now I see the same thing (appended) as before (eg, valgrind does not seem to help). >> >> I'm suspecting that MatConvert is messed, but I will try not deleting this matrix and see what happens. >> >> >> ==69836== Conditional jump or move depends on uninitialised value(s) >> ==69836== at 0x1E8A5DE: ATL_dgemvT_a1_x1_b0_y1 (in /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib) >> ==69836== >> ==69836== Conditional jump or move depends on uninitialised value(s) >> ==69836== at 0x1EBAC7A: ATL_dger1_a1_x1_yX (in /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib) >> ==69836== >> [0] > > Yes it pisses me off to no end that Apple got suckered into using that ATLAS crap and it is buggy with basic memory errors and Jack doesn't give a shit. > Wow .... So I fixed the problem. The errors happened when calling the repartitioning code w/o MPI. THe code works fine on one processor with the MPI version. So I think there is a bug in PETSc having to do with MatConvert or Partitioning code w/o MPI. I have pushed fix today, but if you really want to debug it ... I'm sure that any no-MPI test using GAMG would trip this. All that needs to change is on line 191 of gamg.c: if( s_avoid_repart || npe==1 ) { change it to if( s_avoid_repart ) { This will run the partitioning code even on one processor. Let me know if I can help. Thanks, Mark > > It you can tell us how to run a code (or push an example and tell use it) that reproduces this we can also debug it to see what the problem is. > > Barry > >> >> On Nov 23, 2011, at 2:01 PM, Barry Smith wrote: >> >>> >>> On Nov 23, 2011, at 10:31 AM, Mark F. Adams wrote: >>> >>>> Yep, need to keep the extra include with uni, but not the lib anymore. >>>> >>>> Now I get this error, any ideas? I need a uni regression test apparently. >>> >>> After you do the "make all test" on PETSc do "make alltests" and it should run all the tests that run on one process. >>> >>> Something is wrong with your reference counting likely here. Does this work with --with-debugging=0? I would run debug version with valgrind first. Then debug version in debugger. >>> >>> Barry >>> >>> >>>> >>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>> [0]PETSC ERROR: Corrupt argument: >>>> see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind! >>>> [0]PETSC ERROR: Inner MPI_Comm does not have expected reference to outer comm! >>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: Petsc Development HG revision: 3dda30872eeec1a2ea5c130f267935c5e0b2534a HG Date: Mon Nov 21 11:19:39 2011 -0800 >>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: ./viscousTensorSolve2d.Darwin.g++-4.gfortran.DEBUG.OPT.PETSC.ex on a arch-maco named madams-macbk-3.local by markadams Wed Nov 23 11:25:30 2011 >>>> [0]PETSC ERROR: Libraries linked from /Users/markadams/Codes/petsc-dev/arch-macosx-gnu-seq/lib >>>> [0]PETSC ERROR: Configure run at Tue Nov 22 12:41:46 2011 >>>> [0]PETSC ERROR: Configure options CXX=/sw/lib/gcc4.4/bin/g++-4 CC=/sw/lib/gcc4.4/bin/gcc-4 FC=/sw/lib/gcc4.4/bin/gfortran --with-x=0 --with-clanguage=c++ --with-debugging=0 --with-mpi=0 PETSC_ARCH=arch-macosx-gnu-seq >>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: Petsc_DelComm() line 448 in /Users/markadams/Codes/petsc-dev/src/sys/objects/pinit.c >>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>>> [0]PETSC ERROR: Corrupt argument: >>>> see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind! >>>> [0]PETSC ERROR: MPI_Comm does not have tag/name counter nor does it have inner MPI_Comm! >>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: Petsc Development HG revision: 3dda30872eeec1a2ea5c130f267935c5e0b2534a HG Date: Mon Nov 21 11:19:39 2011 -0800 >>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>> [0]PETSC ERROR: See docs/index.html for manual pages. >>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: ./viscousTensorSolve2d.Darwin.g++-4.gfortran.DEBUG.OPT.PETSC.ex on a arch-maco named madams-macbk-3.local by markadams Wed Nov 23 11:25:30 2011 >>>> [0]PETSC ERROR: Libraries linked from /Users/markadams/Codes/petsc-dev/arch-macosx-gnu-seq/lib >>>> [0]PETSC ERROR: Configure run at Tue Nov 22 12:41:46 2011 >>>> [0]PETSC ERROR: Configure options CXX=/sw/lib/gcc4.4/bin/g++-4 CC=/sw/lib/gcc4.4/bin/gcc-4 FC=/sw/lib/gcc4.4/bin/gfortran --with-x=0 --with-clanguage=c++ --with-debugging=0 --with-mpi=0 PETSC_ARCH=arch-macosx-gnu-seq >>>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>>> [0]PETSC ERROR: PetscCommDestroy() line 229 in /Users/markadams/Codes/petsc-dev/src/sys/objects/tagm.c >>>> [0]PETSC ERROR: PetscHeaderDestroy_Private() line 110 in /Users/markadams/Codes/petsc-dev/src/sys/objects/inherit.c >>>> [0]PETSC ERROR: MatDestroy() line 1045 in /Users/markadams/Codes/petsc-dev/src/mat/interface/matrix.c >>>> [0]PETSC ERROR: partitionLevel() line 347 in /Users/markadams/Codes/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>>> [0]PETSC ERROR: PCSetUp_GAMG() line 599 in /Users/markadams/Codes/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>>> [0]PETSC ERROR: PCSetUp() line 819 in /Users/markadams/Codes/petsc-dev/src/ksp/pc/interface/precon.c >>>> [0]PETSC ERROR: KSPSetUp() line 261 in /Users/markadams/Codes/petsc-dev/src/ksp/ksp/interface/itfunc.c >>>> [0]PETSC ERROR: solveprivate() line 191 in "unknowndirectory/"PetscLinearSolverI.H >>>> >>>> >>>> >>>> >>>> On Nov 22, 2011, at 3:19 PM, Barry Smith wrote: >>>> >>>>> >>>>> You have som other MPI include file hanging around that is getting intercepted before the MPI Uni one. >>>>> >>>>> Barry >>>>> >>>>> On Nov 22, 2011, at 12:00 PM, Mark F. Adams wrote: >>>>> >>>>>> Scorched earth did not work: >>>>>> >>>>>> "VecCreate(ompi_communicator_t*, _p_Vec**)", referenced from: >>>>>> PetscLinearSolver<LevelData<FArrayBox> >::solveprivate(LevelData<FArrayBox>&, LevelData<FArrayBox> const&) in libamrelliptic2d.Darwin.g++-4.gfortran.DEBUG.OPT.PETSC.a(PetscLinearSolver.o) >>>>>> "_ompi_mpi_comm_self", referenced from: >>>>>> _ompi_mpi_comm_self$non_lazy_ptr in libamrelliptic2d.Darwin.g++-4.gfortran.DEBUG.OPT.PETSC.a(PetscLinearSolver.o) >>>>>> >>>>>> <configure.log.gz> >>>>>> >>>>>> On Nov 22, 2011, at 11:47 AM, Satish Balay wrote: >>>>>> >>>>>>> Something is strange. >>>>>>> >>>>>>> I'll suggeset >>>>>>> >>>>>>> rm -rf externalpackages arch-macosx-gnu-seq >>>>>>> >>>>>>> and then doing a fresh build. >>>>>>> >>>>>>> And then verifying the build with 'make test' >>>>>>> >>>>>>> If 'make test' works - and the application build is not working- I'd >>>>>>> like to see the complete compile log thats producing this error. >>>>>>> >>>>>>> >>>>>>> wrt libmpiuni.a - all the relavent stuff is added to libpetsc.a and a separate >>>>>>> libmpiuni.a is not created. >>>>>>> >>>>>>> Satish >>>>>>> >>>>>>> On Tue, 22 Nov 2011, Mark F. Adams wrote: >>>>>>> >>>>>>>> I'm trying to build a non MPI code and get this error: >>>>>>>> >>>>>>>> Undefined symbols: >>>>>>>> "_ompi_mpi_comm_self", referenced from: >>>>>>>> _ompi_mpi_comm_self$non_lazy_ptr in libamrelliptic2d.Darwin.g++-4.gfortran.DEBUG.OPT.PETSC.a(PetscLinearSolver.o) >>>>>>>> (maybe you meant: _ompi_mpi_comm_self$non_lazy_ptr) >>>>>>>> >>>>>>>> >>>>>>>> In the past I have linked libmpiuni.a but that did not seem to get created. Did libmpiuni.a go away or did my make fail ... >>>>>>>> >>>>>>>> Mark >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>> >>> >>> >> > >

1 0

Re: [petsc-dev] uni
by Barry Smith 23 Nov '11

23 Nov '11

On Nov 23, 2011, at 3:07 PM, Mark F. Adams wrote: The test suite sucks in that in you need to then go into the directories and run the problem examples to see what is going wrong or why the file is the wrong size. Someday when we are not writing code we can fix up the test harness. Barry >> >> After you do the "make all test" on PETSc do "make alltests" and it should run all the tests that run on one process. > > This does not look promising: > > ~/Codes/petsc-dev>make alltests > ---------------- > ##1 <== > ##1 #>3 ==> 1 MPI processes > @ Line 1 in file "output/ex101.out" is shorter than expected! > ---------------- > ##12 <== > ##12 #>3 ==> 1 MPI processes > @ Line 12 in file "output/ex101.out" is shorter than expected! > ---------------- > ##23 <== > ##23 #>3 ==> 1 MPI processes > @ Line 23 in file "output/ex101.out" is shorter than expected! > /Users/markadams/Codes/petsc-dev/src/mat/examples/tests > Possible problem with with ex101_1, diffs above > ========================================= > ---------------- > ##1 <== > ##1 #>3 ==> 1 MPI processes > @ Line 1 in file "output/ex6f_1.out" is shorter than expected! > ---------------- > ##12 <== > ##12 #>3 ==> 1 MPI processes > @ Line 12 in file "output/ex6f_1.out" is shorter than expected! > ---------------- > ##27 <== > ##27 #>3 ==> 1 MPI processes > @ Line 27 in file "output/ex6f_1.out" is shorter than expected! > /Users/markadams/Codes/petsc-dev/src/ksp/ksp/examples/tutorials > Possible problem with with ex6f_1, diffs above > ========================================= > > >> >> Something is wrong with your reference counting likely here. Does this work with --with-debugging=0? > > No > >> I would run debug version with valgrind first. > > I'm getting an error in my code" > > ==69160== Process terminating with default action of signal 11 (SIGSEGV) > ==69160== General Protection Fault > ==69160== at 0x13DE930: misaligned_stack_error_ (in /usr/lib/libSystem.B.dylib) > ==69160== by 0x187214: TraceTimer::sampleMemUsage() (CH_Timer.cpp:1700) > > but CH_Timer.cpp:1700 is simply: > > sampleFile = fopen("memory.samples", "w"); > > So that is puzzling ... > > Any thoughts? I'll start hacking at this. > > Mark > >> Then debug version in debugger. >> >> Barry >> >> >>> >>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> [0]PETSC ERROR: Corrupt argument: >>> see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind! >>> [0]PETSC ERROR: Inner MPI_Comm does not have expected reference to outer comm! >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Development HG revision: 3dda30872eeec1a2ea5c130f267935c5e0b2534a HG Date: Mon Nov 21 11:19:39 2011 -0800 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: ./viscousTensorSolve2d.Darwin.g++-4.gfortran.DEBUG.OPT.PETSC.ex on a arch-maco named madams-macbk-3.local by markadams Wed Nov 23 11:25:30 2011 >>> [0]PETSC ERROR: Libraries linked from /Users/markadams/Codes/petsc-dev/arch-macosx-gnu-seq/lib >>> [0]PETSC ERROR: Configure run at Tue Nov 22 12:41:46 2011 >>> [0]PETSC ERROR: Configure options CXX=/sw/lib/gcc4.4/bin/g++-4 CC=/sw/lib/gcc4.4/bin/gcc-4 FC=/sw/lib/gcc4.4/bin/gfortran --with-x=0 --with-clanguage=c++ --with-debugging=0 --with-mpi=0 PETSC_ARCH=arch-macosx-gnu-seq >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc_DelComm() line 448 in /Users/markadams/Codes/petsc-dev/src/sys/objects/pinit.c >>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> [0]PETSC ERROR: Corrupt argument: >>> see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind! >>> [0]PETSC ERROR: MPI_Comm does not have tag/name counter nor does it have inner MPI_Comm! >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Development HG revision: 3dda30872eeec1a2ea5c130f267935c5e0b2534a HG Date: Mon Nov 21 11:19:39 2011 -0800 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: ./viscousTensorSolve2d.Darwin.g++-4.gfortran.DEBUG.OPT.PETSC.ex on a arch-maco named madams-macbk-3.local by markadams Wed Nov 23 11:25:30 2011 >>> [0]PETSC ERROR: Libraries linked from /Users/markadams/Codes/petsc-dev/arch-macosx-gnu-seq/lib >>> [0]PETSC ERROR: Configure run at Tue Nov 22 12:41:46 2011 >>> [0]PETSC ERROR: Configure options CXX=/sw/lib/gcc4.4/bin/g++-4 CC=/sw/lib/gcc4.4/bin/gcc-4 FC=/sw/lib/gcc4.4/bin/gfortran --with-x=0 --with-clanguage=c++ --with-debugging=0 --with-mpi=0 PETSC_ARCH=arch-macosx-gnu-seq >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: PetscCommDestroy() line 229 in /Users/markadams/Codes/petsc-dev/src/sys/objects/tagm.c >>> [0]PETSC ERROR: PetscHeaderDestroy_Private() line 110 in /Users/markadams/Codes/petsc-dev/src/sys/objects/inherit.c >>> [0]PETSC ERROR: MatDestroy() line 1045 in /Users/markadams/Codes/petsc-dev/src/mat/interface/matrix.c >>> [0]PETSC ERROR: partitionLevel() line 347 in /Users/markadams/Codes/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>> [0]PETSC ERROR: PCSetUp_GAMG() line 599 in /Users/markadams/Codes/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>> [0]PETSC ERROR: PCSetUp() line 819 in /Users/markadams/Codes/petsc-dev/src/ksp/pc/interface/precon.c >>> [0]PETSC ERROR: KSPSetUp() line 261 in /Users/markadams/Codes/petsc-dev/src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: solveprivate() line 191 in "unknowndirectory/"PetscLinearSolverI.H >>> >>> >>> >>> >>> On Nov 22, 2011, at 3:19 PM, Barry Smith wrote: >>> >>>> >>>> You have som other MPI include file hanging around that is getting intercepted before the MPI Uni one. >>>> >>>> Barry >>>> >>>> On Nov 22, 2011, at 12:00 PM, Mark F. Adams wrote: >>>> >>>>> Scorched earth did not work: >>>>> >>>>> "VecCreate(ompi_communicator_t*, _p_Vec**)", referenced from: >>>>> PetscLinearSolver<LevelData<FArrayBox> >::solveprivate(LevelData<FArrayBox>&, LevelData<FArrayBox> const&) in libamrelliptic2d.Darwin.g++-4.gfortran.DEBUG.OPT.PETSC.a(PetscLinearSolver.o) >>>>> "_ompi_mpi_comm_self", referenced from: >>>>> _ompi_mpi_comm_self$non_lazy_ptr in libamrelliptic2d.Darwin.g++-4.gfortran.DEBUG.OPT.PETSC.a(PetscLinearSolver.o) >>>>> >>>>> <configure.log.gz> >>>>> >>>>> On Nov 22, 2011, at 11:47 AM, Satish Balay wrote: >>>>> >>>>>> Something is strange. >>>>>> >>>>>> I'll suggeset >>>>>> >>>>>> rm -rf externalpackages arch-macosx-gnu-seq >>>>>> >>>>>> and then doing a fresh build. >>>>>> >>>>>> And then verifying the build with 'make test' >>>>>> >>>>>> If 'make test' works - and the application build is not working- I'd >>>>>> like to see the complete compile log thats producing this error. >>>>>> >>>>>> >>>>>> wrt libmpiuni.a - all the relavent stuff is added to libpetsc.a and a separate >>>>>> libmpiuni.a is not created. >>>>>> >>>>>> Satish >>>>>> >>>>>> On Tue, 22 Nov 2011, Mark F. Adams wrote: >>>>>> >>>>>>> I'm trying to build a non MPI code and get this error: >>>>>>> >>>>>>> Undefined symbols: >>>>>>> "_ompi_mpi_comm_self", referenced from: >>>>>>> _ompi_mpi_comm_self$non_lazy_ptr in libamrelliptic2d.Darwin.g++-4.gfortran.DEBUG.OPT.PETSC.a(PetscLinearSolver.o) >>>>>>> (maybe you meant: _ompi_mpi_comm_self$non_lazy_ptr) >>>>>>> >>>>>>> >>>>>>> In the past I have linked libmpiuni.a but that did not seem to get created. Did libmpiuni.a go away or did my make fail ... >>>>>>> >>>>>>> Mark >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> >>> >> >> >

1 0

Re: [petsc-dev] uni
by Barry Smith 23 Nov '11

23 Nov '11

On Nov 23, 2011, at 3:14 PM, Mark F. Adams wrote: > I fixed my code error with valgrind and now I see the same thing (appended) as before (eg, valgrind does not seem to help). > > I'm suspecting that MatConvert is messed, but I will try not deleting this matrix and see what happens. > > > ==69836== Conditional jump or move depends on uninitialised value(s) > ==69836== at 0x1E8A5DE: ATL_dgemvT_a1_x1_b0_y1 (in /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib) > ==69836== > ==69836== Conditional jump or move depends on uninitialised value(s) > ==69836== at 0x1EBAC7A: ATL_dger1_a1_x1_yX (in /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib) > ==69836== > [0] Yes it pisses me off to no end that Apple got suckered into using that ATLAS crap and it is buggy with basic memory errors and Jack doesn't give a shit. It you can tell us how to run a code (or push an example and tell use it) that reproduces this we can also debug it to see what the problem is. Barry > > On Nov 23, 2011, at 2:01 PM, Barry Smith wrote: > >> >> On Nov 23, 2011, at 10:31 AM, Mark F. Adams wrote: >> >>> Yep, need to keep the extra include with uni, but not the lib anymore. >>> >>> Now I get this error, any ideas? I need a uni regression test apparently. >> >> After you do the "make all test" on PETSc do "make alltests" and it should run all the tests that run on one process. >> >> Something is wrong with your reference counting likely here. Does this work with --with-debugging=0? I would run debug version with valgrind first. Then debug version in debugger. >> >> Barry >> >> >>> >>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> [0]PETSC ERROR: Corrupt argument: >>> see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind! >>> [0]PETSC ERROR: Inner MPI_Comm does not have expected reference to outer comm! >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Development HG revision: 3dda30872eeec1a2ea5c130f267935c5e0b2534a HG Date: Mon Nov 21 11:19:39 2011 -0800 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: ./viscousTensorSolve2d.Darwin.g++-4.gfortran.DEBUG.OPT.PETSC.ex on a arch-maco named madams-macbk-3.local by markadams Wed Nov 23 11:25:30 2011 >>> [0]PETSC ERROR: Libraries linked from /Users/markadams/Codes/petsc-dev/arch-macosx-gnu-seq/lib >>> [0]PETSC ERROR: Configure run at Tue Nov 22 12:41:46 2011 >>> [0]PETSC ERROR: Configure options CXX=/sw/lib/gcc4.4/bin/g++-4 CC=/sw/lib/gcc4.4/bin/gcc-4 FC=/sw/lib/gcc4.4/bin/gfortran --with-x=0 --with-clanguage=c++ --with-debugging=0 --with-mpi=0 PETSC_ARCH=arch-macosx-gnu-seq >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc_DelComm() line 448 in /Users/markadams/Codes/petsc-dev/src/sys/objects/pinit.c >>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------ >>> [0]PETSC ERROR: Corrupt argument: >>> see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind! >>> [0]PETSC ERROR: MPI_Comm does not have tag/name counter nor does it have inner MPI_Comm! >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: Petsc Development HG revision: 3dda30872eeec1a2ea5c130f267935c5e0b2534a HG Date: Mon Nov 21 11:19:39 2011 -0800 >>> [0]PETSC ERROR: See docs/changes/index.html for recent updates. >>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>> [0]PETSC ERROR: See docs/index.html for manual pages. >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: ./viscousTensorSolve2d.Darwin.g++-4.gfortran.DEBUG.OPT.PETSC.ex on a arch-maco named madams-macbk-3.local by markadams Wed Nov 23 11:25:30 2011 >>> [0]PETSC ERROR: Libraries linked from /Users/markadams/Codes/petsc-dev/arch-macosx-gnu-seq/lib >>> [0]PETSC ERROR: Configure run at Tue Nov 22 12:41:46 2011 >>> [0]PETSC ERROR: Configure options CXX=/sw/lib/gcc4.4/bin/g++-4 CC=/sw/lib/gcc4.4/bin/gcc-4 FC=/sw/lib/gcc4.4/bin/gfortran --with-x=0 --with-clanguage=c++ --with-debugging=0 --with-mpi=0 PETSC_ARCH=arch-macosx-gnu-seq >>> [0]PETSC ERROR: ------------------------------------------------------------------------ >>> [0]PETSC ERROR: PetscCommDestroy() line 229 in /Users/markadams/Codes/petsc-dev/src/sys/objects/tagm.c >>> [0]PETSC ERROR: PetscHeaderDestroy_Private() line 110 in /Users/markadams/Codes/petsc-dev/src/sys/objects/inherit.c >>> [0]PETSC ERROR: MatDestroy() line 1045 in /Users/markadams/Codes/petsc-dev/src/mat/interface/matrix.c >>> [0]PETSC ERROR: partitionLevel() line 347 in /Users/markadams/Codes/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>> [0]PETSC ERROR: PCSetUp_GAMG() line 599 in /Users/markadams/Codes/petsc-dev/src/ksp/pc/impls/gamg/gamg.c >>> [0]PETSC ERROR: PCSetUp() line 819 in /Users/markadams/Codes/petsc-dev/src/ksp/pc/interface/precon.c >>> [0]PETSC ERROR: KSPSetUp() line 261 in /Users/markadams/Codes/petsc-dev/src/ksp/ksp/interface/itfunc.c >>> [0]PETSC ERROR: solveprivate() line 191 in "unknowndirectory/"PetscLinearSolverI.H >>> >>> >>> >>> >>> On Nov 22, 2011, at 3:19 PM, Barry Smith wrote: >>> >>>> >>>> You have som other MPI include file hanging around that is getting intercepted before the MPI Uni one. >>>> >>>> Barry >>>> >>>> On Nov 22, 2011, at 12:00 PM, Mark F. Adams wrote: >>>> >>>>> Scorched earth did not work: >>>>> >>>>> "VecCreate(ompi_communicator_t*, _p_Vec**)", referenced from: >>>>> PetscLinearSolver<LevelData<FArrayBox> >::solveprivate(LevelData<FArrayBox>&, LevelData<FArrayBox> const&) in libamrelliptic2d.Darwin.g++-4.gfortran.DEBUG.OPT.PETSC.a(PetscLinearSolver.o) >>>>> "_ompi_mpi_comm_self", referenced from: >>>>> _ompi_mpi_comm_self$non_lazy_ptr in libamrelliptic2d.Darwin.g++-4.gfortran.DEBUG.OPT.PETSC.a(PetscLinearSolver.o) >>>>> >>>>> <configure.log.gz> >>>>> >>>>> On Nov 22, 2011, at 11:47 AM, Satish Balay wrote: >>>>> >>>>>> Something is strange. >>>>>> >>>>>> I'll suggeset >>>>>> >>>>>> rm -rf externalpackages arch-macosx-gnu-seq >>>>>> >>>>>> and then doing a fresh build. >>>>>> >>>>>> And then verifying the build with 'make test' >>>>>> >>>>>> If 'make test' works - and the application build is not working- I'd >>>>>> like to see the complete compile log thats producing this error. >>>>>> >>>>>> >>>>>> wrt libmpiuni.a - all the relavent stuff is added to libpetsc.a and a separate >>>>>> libmpiuni.a is not created. >>>>>> >>>>>> Satish >>>>>> >>>>>> On Tue, 22 Nov 2011, Mark F. Adams wrote: >>>>>> >>>>>>> I'm trying to build a non MPI code and get this error: >>>>>>> >>>>>>> Undefined symbols: >>>>>>> "_ompi_mpi_comm_self", referenced from: >>>>>>> _ompi_mpi_comm_self$non_lazy_ptr in libamrelliptic2d.Darwin.g++-4.gfortran.DEBUG.OPT.PETSC.a(PetscLinearSolver.o) >>>>>>> (maybe you meant: _ompi_mpi_comm_self$non_lazy_ptr) >>>>>>> >>>>>>> >>>>>>> In the past I have linked libmpiuni.a but that did not seem to get created. Did libmpiuni.a go away or did my make fail ... >>>>>>> >>>>>>> Mark >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> >>> >> >> >

1 0

Re: [petsc-dev] Bug in example code
by Barry Smith 23 Nov '11

23 Nov '11

On Nov 23, 2011, at 1:28 PM, Tim Gallagher wrote: > I couldn't agree more about using the modern versions of Fortran! Oh the fights we've had here about that one... > > I thought I read somewhere that petsc F90 wasn't working correctly/had limited support, but perhaps that was old or incorrect. I think I got that impression here: > > http://www.mcs.anl.gov/petsc/petsc-3.1/docs/manualpages/Sys/UsingFortran.ht… "We recommend either 2 or 3." Only 1 is pure F77 style and there is probably no reason to use it. > > I will recompile it with the F90 turned on instead. You don't really have to change anything to use 2 or 3 , the same libraries, no different configuration. Just other examples Barry > > Thanks, > > Tim > > ----- Original Message ----- > From: "Barry Smith" <bsmith(a)mcs.anl.gov> > To: gtg085x(a)mail.gatech.edu, "For users of the development version of PETSc" <petsc-dev(a)mcs.anl.gov> > Sent: Wednesday, November 23, 2011 2:16:59 PM > Subject: Re: [petsc-dev] Bug in example code > > > Tim, > > You must have array bounds checking turned on for Fortran with the Intel debug version. The lines > > subroutine FormInitialGuess(X,ierr) > implicit none > > #include "ex5f.h" > > ! Input/output variables: > Vec X > PetscErrorCode ierr > > ! Declarations for use with local arrays: > PetscScalar lx_v(0:1) > PetscOffset lx_i > Vec localX > > will break that because we access outside th lx_f(0:1) array. Our Fortran examples that use F77 have to violate the array bounds so must be compiled without that checking. > > The Fortran 90 examples like ex5f90.F use F90 constructs and thus don't violate bounds. If you plan to use Fortran it makes sense to use the F90 version; after all 2011 >> 1977 :-) > > Barry > > > > On Nov 23, 2011, at 12:04 PM, Tim Gallagher wrote: > >> Hi, >> >> I just cloned the petsc-dev branch for the first time and built it with GCC with both release and debug options and the tests during make test worked fine. I then built it with Intel with release options and the tests worked. Then I compiled with debug options and one of the tests generates an error: >> >> Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process >> See http://www.mcs.anl.gov/petsc/documentation/faq.html >> forrtl: severe (408): fort: (2): Subscript #1 of the array LX_V has value 15092 which is greater than the upper bound of 1 >> >> Image PC Routine Line Source >> libirc.so 00007F4245C589AA Unknown Unknown Unknown >> libirc.so 00007F4245C574A6 Unknown Unknown Unknown >> libifcore.so.5 00007F4240A5B6FC Unknown Unknown Unknown >> libifcore.so.5 00007F42409CEDA2 Unknown Unknown Unknown >> libifcore.so.5 00007F42409CF323 Unknown Unknown Unknown >> ex5f 0000000000403F58 forminitialguess_ 294 ex5f.F >> ex5f 0000000000403719 MAIN__ 224 ex5f.F >> ex5f 00000000004028AC Unknown Unknown Unknown >> libc.so.6 00007F42401AB23D Unknown Unknown Unknown >> ex5f 00000000004027A9 Unknown Unknown Unknown >> >> I am using: >> >> Intel(R) Fortran Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 12.1.1.256 Build 20111011 >> Copyright (C) 1985-2011 Intel Corporation. All rights reserved. >> >> and petsc was configured with: >> >> ./configure PETSC_ARCH=linux-intel-debug --prefix=/opt/petsc/intel/2011.sp1.1/debug --with-make-np=12 --with-clanguage=C++ --with-c-support=1--with-fortran-kernels=1 --with-shared-libraries=1--download-scalapack=yes --download-parmetis=yes --download-metis=yes --with-imagemagick=1 --download-sundials=yes --download-mpi4py=yes --download-petsc4py=yes --download-boost=yes --with-hdf5=1 --with-hdf5-dir=$HDF5_DIR --with-imagemagick=0 --with-shared-libraries --with-debugging=no COPTFLAGS='-g -C -traceback -fpe0 -check noarg_temp_created -fp-stack-check -check uninit' FOPTFLAGS='-g -C -traceback -fpe0 -check noarg_temp_created -fp-stack-check -check uninit' >> >> I am going to try looking for the problem, but since this is the first 10 minutes I've looked at the code, it will probably take me a lot longer than somebody else to find the problem. Let me know if there's anything I can do to help track down the problem. >> >> Thanks, >> >> Tim >

1 0