November 2011 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Parmetis / Metis changes
by Sean Farley 20 Nov '11

20 Nov '11

> > It appears that George is still living in the Linux desert and Sean has so > pimped out his Mac it is barely running OS X. > Oh really? My system is so messed up that I have to have special code in BuildSystem? Exhibit A: http://petsc.cs.iit.edu/petsc/BuildSystem/rev/39d608c7ffd5 Let the record show that this only happens on Barry's (barely OS X) machine. > One problem I don't know how to fix. > > metis has the file gk_externs.h with the lines > > #ifndef _GK_ERROR_C_ > /* declared in error.c */ > extern __thread int gk_cur_jbufs; > extern __thread jmp_buf gk_jbufs[]; > extern __thread jmp_buf gk_jbuf; > > #endif > > The default Mac compilers won't compile it; I don't see how this worked > for Sean. I think this is similar to the MPICH bug I reported with clang. __thread is reported funky with the newest XCode, it seems (which is why I couldn't compile MPICH with it). > Anyways no big deal except because of metis's cmake system I don't know > how I can pass the flag _GK_ERROR_C_ done into the metis cmake system so > the compiler doesn't see this thing. Actually, you'd want to pass -D__thread= which is how metis handles it with other compilers. I don't know why llvm is reporting __thread incorrectly. Any ideas on how to pass it from metis.py?????? This is an annoying fact of the way that metis's makefile passes variables to cmake. We could circumvent the top-level makefile entirely, but that would lock us into a permanent battle with doing the same with every metis release. Or I could just patch the top-level makefile to take a "cflags='-blah'" argument and hope that George accepts the patch. I'm leaning towards the patch. I've implemented both but will see how I feel about it tomorrow before I push any changes.

1 0

Re: [petsc-dev] Parmetis / Metis changes
by Barry Smith 19 Nov '11

19 Nov '11

Thanks Sean, It appears that George is still living in the Linux desert and Sean has so pimped out his Mac it is barely running OS X. One problem I don't know how to fix. metis has the file gk_externs.h with the lines #ifndef _GK_ERROR_C_ /* declared in error.c */ extern __thread int gk_cur_jbufs; extern __thread jmp_buf gk_jbufs[]; extern __thread jmp_buf gk_jbuf; #endif The default Mac compilers won't compile it; I don't see how this worked for Sean. Anyways no big deal except because of metis's cmake system I don't know how I can pass the flag _GK_ERROR_C_ done into the metis cmake system so the compiler doesn't see this thing. Any ideas on how to pass it from metis.py?????? Thanks Barry On Nov 18, 2011, at 7:25 PM, Sean Farley wrote: > I've just pushed some changes that update parmetis to newest revision, and this has some consequences on packages that depend on parmetis. > > Parmetis now depends on metis, and both can use 64-bit integers. This required an update to SuperLU_DIST to use this new interface which can now use 64-bit integers everywhere. Also, MUMPS needed to be updated (but is not 64-bit). All of these are available with --download-packagename (BuildSystem is always an adventure!). Patches are available upon request but I hope that the code maintainers accept them (I've sent them to their respective authors). > > CHOLMOD needed updating too, but that isn't available as a --download-cholmod because it will be refactored into --download-suitesparse. Tim has graciously posted my patch on CHOLMOD's website: > > http://www.cise.ufl.edu/research/sparse/cholmod/ > > People that use sieve will need to rebuild as well since Matt uses parmetis directly. Potentially, sieve could work with 64-bit integers now, but I tried to go through sieve ... and then ran screaming in terror. I think I may have tripped on the bones of the last intrepid adventurer who ventured into sieve. It's hairy. At the very least, triangle and tetgen will have to be updated. Matt might also want to look at updating the other mesh partitioners (zoltan, hmetis, chaco, etc.) > > If you use any of these packages, I recommend deleting your PETSC_ARCH and externalpackages directories, and re-running configure. > > I tested each of the situations: > > Matrix partitioning: > > $ cd src/mat/examples/tutorials > $ make ex15 > $ ./ex15 -mat_partitioning_type parmetis > > SuperLU_DIST (w/ metis): > > $ cd src/ksp/ksp/examples/tutorials > $ make ex15 > $ ./ex15 -pc_type lu -pc_factor_mat_solver_package superlu_dist -ksp_view > > SuperLU_DIST (w/ parmetis): > > $ cd src/ksp/ksp/examples/tutorials > $ make ex15 > $ mpiexec -n 2 ./ex15 -pc_type lu -pc_factor_mat_solver_package superlu_dist -ksp_view -mat_superlu_dist_matinput DISTRIBUTED -mat_superlu_dist_colperm PARMETIS > > MUMPS (w/ metis): > > $ cd src/ksp/ksp/examples/tutorials > $ make ex15 > $ /ex15 -pc_type lu -pc_factor_mat_solver_package mumps -ksp_view -mat_mumps_icntl_7 5 > > MUMPS (w/ parmetis): > > $ cd src/ksp/ksp/examples/tutorials > $ make ex15 > $ mpiexec -n 2 ./ex15 -pc_type lu -pc_factor_mat_solver_package mumps -ksp_view -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2 > > I tried to do my best with testing, but I'm sure I missed something. Please let me know if there are any problems.

1 0

Parmetis / Metis changes
by Sean Farley 18 Nov '11

18 Nov '11

I've just pushed some changes that update parmetis to newest revision, and this has some consequences on packages that depend on parmetis. Parmetis now depends on metis, and both can use 64-bit integers. This required an update to SuperLU_DIST to use this new interface which can now use 64-bit integers everywhere. Also, MUMPS needed to be updated (but is not 64-bit). All of these are available with --download-packagename (BuildSystem is always an adventure!). Patches are available upon request but I hope that the code maintainers accept them (I've sent them to their respective authors). CHOLMOD needed updating too, but that isn't available as a --download-cholmod because it will be refactored into --download-suitesparse. Tim has graciously posted my patch on CHOLMOD's website: http://www.cise.ufl.edu/research/sparse/cholmod/ People that use sieve will need to rebuild as well since Matt uses parmetis directly. Potentially, sieve could work with 64-bit integers now, but I tried to go through sieve ... and then ran screaming in terror. I think I may have tripped on the bones of the last intrepid adventurer who ventured into sieve. It's hairy. At the very least, triangle and tetgen will have to be updated. Matt might also want to look at updating the other mesh partitioners (zoltan, hmetis, chaco, etc.) If you use any of these packages, I recommend deleting your PETSC_ARCH and externalpackages directories, and re-running configure. I tested each of the situations: Matrix partitioning: $ cd src/mat/examples/tutorials $ make ex15 $ ./ex15 -mat_partitioning_type parmetis SuperLU_DIST (w/ metis): $ cd src/ksp/ksp/examples/tutorials $ make ex15 $ ./ex15 -pc_type lu -pc_factor_mat_solver_package superlu_dist -ksp_view SuperLU_DIST (w/ parmetis): $ cd src/ksp/ksp/examples/tutorials $ make ex15 $ mpiexec -n 2 ./ex15 -pc_type lu -pc_factor_mat_solver_package superlu_dist -ksp_view -mat_superlu_dist_matinput DISTRIBUTED -mat_superlu_dist_colperm PARMETIS MUMPS (w/ metis): $ cd src/ksp/ksp/examples/tutorials $ make ex15 $ /ex15 -pc_type lu -pc_factor_mat_solver_package mumps -ksp_view -mat_mumps_icntl_7 5 MUMPS (w/ parmetis): $ cd src/ksp/ksp/examples/tutorials $ make ex15 $ mpiexec -n 2 ./ex15 -pc_type lu -pc_factor_mat_solver_package mumps -ksp_view -mat_mumps_icntl_28 2 -mat_mumps_icntl_29 2 I tried to do my best with testing, but I'm sure I missed something. Please let me know if there are any problems.

1 0

Re: [petsc-dev] CUDA 4.1 RC1
by John Fettig 18 Nov '11

18 Nov '11

The good news is it builds. The bad news is that it doesn't function properly. This same code works fine with cuda 4.0, so I'll have to dig deeper to figure out where it is going wrong. I tried running one of the examples and it worked with sacusp, so perhaps something isn't working right in my code. [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Floating point exception! [0]PETSC ERROR: Infinite or not-a-number generated in norm! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: VecNorm() line 167 in src/vec/vec/interface/rvector.c [0]PETSC ERROR: KSPSolve_CG() line 140 in src/ksp/ksp/impls/cg/cg.c [0]PETSC ERROR: KSPSolve() line 429 in src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: --------------------- Error Message ------------------------------------ John On Wed, Nov 16, 2011 at 4:36 PM, Daniel Lowell <redratio1(a)gmail.com> wrote: > Not yet. But I'm considering it. > > > On Wed, Nov 16, 2011 at 3:33 PM, John Fettig <john.fettig(a)gmail.com>wrote: > >> Has anybody tried compiling petsc-dev using CUDA 4.1? I'm just wondering >> if I should bother trying. >> >> Thanks, >> John >> >> > > > -- > Daniel Lowell > >

1 0

Re: [petsc-dev] CUDA 4.1 RC1
by Daniel Lowell 16 Nov '11

16 Nov '11

Not yet. But I'm considering it. On Wed, Nov 16, 2011 at 3:33 PM, John Fettig <john.fettig(a)gmail.com> wrote: > Has anybody tried compiling petsc-dev using CUDA 4.1? I'm just wondering > if I should bother trying. > > Thanks, > John > > -- Daniel Lowell

1 0

CUDA 4.1 RC1
by John Fettig 16 Nov '11

16 Nov '11

Has anybody tried compiling petsc-dev using CUDA 4.1? I'm just wondering if I should bother trying. Thanks, John

1 0

Re: [petsc-dev] What is this for?
by Jed Brown 16 Nov '11

16 Nov '11

On Wed, Nov 16, 2011 at 09:07, Matthew Knepley <knepley(a)gmail.com> wrote: > I hate extra step annotations, which can be screwed up. I think we could > do a rule based on the #ifdef. This could possibly work, but we > would have to distinguish #ifdefs which test features from those that > indicate packages. > Trouble is that it has to be obvious to the user how to get a suitable build environment. How are you going to tell which package #include <blocks.h> comes from? And how should you error if SPAI is not available? I'm all for eliminating redundancy, but I think it is better for users to have (checked) redundancy than magic. > If we only support some Linux distros, things would be easy. > My point was that they are requiring all packages to do this anyway, so we have to support the strict model if we want PETSc to be packaged for those distros. Falling back to something more verbose on systems with broken shared libraries is understandable, static linking needs everything all the time. > I hate annotations. There has to be a better way to do this. How many > times are the bfort annotations screwed up? Those are a problem because they are too slow to always be updated. A while back, I put in a hack to avoid touching bfort output that did not need to be updated. Then Barry made bfort support more local scoping (so that we could, in principle make it an automatic part of the build sytem), but my hack got lost in that change, so all the ftn-auto are regenerated when you rebuild the stubs. We can set up CMake (or a different system) to track proper dependencies for the generated files, at which point those bugs should go away because the generated files will always be updated. (Note that we want to make sure the time stamp doesn't change if the contents of the generated file doesn't change.)

1 0

Re: [petsc-dev] What is this for?
by Matthew Knepley 16 Nov '11

16 Nov '11

On Wed, Nov 16, 2011 at 8:11 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Wed, Nov 16, 2011 at 07:46, Matthew Knepley <knepley(a)gmail.com> wrote: > >> 1) Nope, we have some package structs in our public interface. >> > > So those use > > #ifdef PETSC_HAVE_HDF5 > #include <hdf5.h> /*R INCLUDES=PETSC_HDF5_INCLUDES R*/ > #endif > > which, since it's in a public header, the build system would recognize > means that those include flags need to go in PETSC_INCLUDES. > I hate extra step annotations, which can be screwed up. I think we could do a rule based on the #ifdef. This could possibly work, but we would have to distinguish #ifdefs which test features from those that indicate packages. > With shared libs > > >> 2) Of course they can dig in and pull something out, but the reason we >> have "make getincludedirs" is that this is beyond most users >> > > Yeah, I want something like > > CFLAGS = -DMY_THING `/path/to/petsc-tool cflags > --required-packages=hdf5:parmetis --optional-packages=mumps > --arch=mpich-clang-opt` > > this would check that the required packages are available with this > PETSC_ARCH and set up CFLAGS to include them directly (and error if the > packages are not available). If available, the flags for the optional > packages would also be added to CFLAGS. As always, the PETSC_HAVE_* macros > from petscconf.h would be visible so the user could check whatever they > wanted. > > Note that if the user was not calling these packages directly, they would > not need to write anything for required/optional packages. Maybe there is a > better way to specify this. > This would not be hard to code up. > 3) We have been a package manager since Barry convinced us to download and >> install packages >> > > I'm only distinguishing between intent. Installing a package for someone > else to use directly should be independent from making that package > available through our common interface (which does not need to expose it to > the user). > > >> >> >>> If shared libraries are being used, we _should_ be linking libpetsc.so >>> such that -lpetsc is enough. Overlinking is bad because it makes the ABI >>> more fragile. >>> >> >> Nice if we can make it work. >> > > That is how shared libs are intended to be used. Some of the Linux distros > are getting quite strict about enforcing this proper usage. > If we only support some Linux distros, things would be easy. > How do you distinguish these cases by looking at the source? > > > The non-portable system include is wrapped in an #ifdef, perhaps with an > #else, and usually does not need special paths. The other has the "require > package" annotation that activates a suitable build environment and > disables building the file if the environment is not available. > I hate annotations. There has to be a better way to do this. How many times are the bfort annotations screwed up? Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] What is this for?
by Jed Brown 16 Nov '11

16 Nov '11

On Wed, Nov 16, 2011 at 07:46, Matthew Knepley <knepley(a)gmail.com> wrote: > 1) Nope, we have some package structs in our public interface. > So those use #ifdef PETSC_HAVE_HDF5 #include <hdf5.h> /*R INCLUDES=PETSC_HDF5_INCLUDES R*/ #endif which, since it's in a public header, the build system would recognize means that those include flags need to go in PETSC_INCLUDES. With shared libs > 2) Of course they can dig in and pull something out, but the reason we > have "make getincludedirs" is that this is beyond most users > Yeah, I want something like CFLAGS = -DMY_THING `/path/to/petsc-tool cflags --required-packages=hdf5:parmetis --optional-packages=mumps --arch=mpich-clang-opt` this would check that the required packages are available with this PETSC_ARCH and set up CFLAGS to include them directly (and error if the packages are not available). If available, the flags for the optional packages would also be added to CFLAGS. As always, the PETSC_HAVE_* macros from petscconf.h would be visible so the user could check whatever they wanted. Note that if the user was not calling these packages directly, they would not need to write anything for required/optional packages. Maybe there is a better way to specify this. > 3) We have been a package manager since Barry convinced us to download and > install packages > I'm only distinguishing between intent. Installing a package for someone else to use directly should be independent from making that package available through our common interface (which does not need to expose it to the user). > > >> If shared libraries are being used, we _should_ be linking libpetsc.so >> such that -lpetsc is enough. Overlinking is bad because it makes the ABI >> more fragile. >> > > Nice if we can make it work. > That is how shared libs are intended to be used. Some of the Linux distros are getting quite strict about enforcing this proper usage. How do you distinguish these cases by looking at the source? The non-portable system include is wrapped in an #ifdef, perhaps with an #else, and usually does not need special paths. The other has the "require package" annotation that activates a suitable build environment and disables building the file if the environment is not available.

1 0

Re: [petsc-dev] What is this for?
by Matthew Knepley 16 Nov '11

16 Nov '11

On Wed, Nov 16, 2011 at 7:37 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Wed, Nov 16, 2011 at 07:19, Matthew Knepley <knepley(a)gmail.com> wrote: > >> There is a big difference between >> >> 1) Using only the necessary include directories for a given compile, >> which might work >> >> and >> >> 2) Removing includes we "don't need", which I do not think will work >> >> Starting with 1). This will break things like "make getincludedirs", >> unless you want to maintain one >> thing with the kitchen sink for users, and one tailored thing for PETSc. >> I think this will generate weird >> errors where the full line breaks something, but we never see it because >> we use the custom build. >> I could still support this if we tested the antique version along with >> the custom one. >> > > Since our public headers don't need to include those implementation > headers, the user does not have access to them unless they personally > include them. In that case, they should pull the variable out of the > makefile/cmake/python configuration. We just want to make it completely > consistent how the necessary variable is named. (Note that this is using > PETSc as a package manager, not just a library.) > 1) Nope, we have some package structs in our public interface. 2) Of course they can dig in and pull something out, but the reason we have "make getincludedirs" is that this is beyond most users 3) We have been a package manager since Barry convinced us to download and install packages > If shared libraries are being used, we _should_ be linking libpetsc.so > such that -lpetsc is enough. Overlinking is bad because it makes the ABI > more fragile. > Nice if we can make it work. > >> However with 2), how would we distinguish includes we only want if the >> package is found from those >> that just do not work because of a configuration/build bug? If its >> annonation, as Jed suggests, how is >> that a step up from #ifdef? >> > > So there are two cases > > 2a. Include a system header if it exists. It is expected to provide a > typedef or function that is needed in the public interface (either directly > by the user or in an inline function). Configure would have checked what is > available and we have fallback logic if it is not provided. I don't think > #ifdefs are wrong here. > > 2b. Compile an optional implementation which requires some headers. If the > package is not available, this implementation file is not even compiled. > This is where that annotation could be useful, so that the build > environment can stay in the source file instead of being written into the > makefile or some other independent file. This also reduces the granularity > with which we can specify compiler flags (it's currently directory-level). > How do you distinguish these cases by looking at the source? Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0