November 2011 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] What is this for?
by Jed Brown 16 Nov '11

16 Nov '11

On Wed, Nov 16, 2011 at 07:19, Matthew Knepley <knepley(a)gmail.com> wrote: > There is a big difference between > > 1) Using only the necessary include directories for a given compile, > which might work > > and > > 2) Removing includes we "don't need", which I do not think will work > > Starting with 1). This will break things like "make getincludedirs", > unless you want to maintain one > thing with the kitchen sink for users, and one tailored thing for PETSc. I > think this will generate weird > errors where the full line breaks something, but we never see it because > we use the custom build. > I could still support this if we tested the antique version along with the > custom one. > Since our public headers don't need to include those implementation headers, the user does not have access to them unless they personally include them. In that case, they should pull the variable out of the makefile/cmake/python configuration. We just want to make it completely consistent how the necessary variable is named. (Note that this is using PETSc as a package manager, not just a library.) If shared libraries are being used, we _should_ be linking libpetsc.so such that -lpetsc is enough. Overlinking is bad because it makes the ABI more fragile. > However with 2), how would we distinguish includes we only want if the > package is found from those > that just do not work because of a configuration/build bug? If its > annonation, as Jed suggests, how is > that a step up from #ifdef? > So there are two cases 2a. Include a system header if it exists. It is expected to provide a typedef or function that is needed in the public interface (either directly by the user or in an inline function). Configure would have checked what is available and we have fallback logic if it is not provided. I don't think #ifdefs are wrong here. 2b. Compile an optional implementation which requires some headers. If the package is not available, this implementation file is not even compiled. This is where that annotation could be useful, so that the build environment can stay in the source file instead of being written into the makefile or some other independent file. This also reduces the granularity with which we can specify compiler flags (it's currently directory-level).

1 0

Re: [petsc-dev] What is this for?
by Matthew Knepley 16 Nov '11

16 Nov '11

On Wed, Nov 16, 2011 at 6:09 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Wed, Nov 16, 2011 at 00:47, Sean Farley <sean(a)mcs.anl.gov> wrote: > >> So, this is basically what I think Barry meant ... except the whole /*R >> <flags> R*/ part. > > > I would want something equivalently verbose because it's HUMAN > documentation for how the variables are spelled. Even if the build system > is clairvoyant such that it knows what package each header comes from and > which variables it needs to link with, a human won't know that and likely > won't know where to find out. Spelling out those variables, which they can > query through the makefile include, a CMake include, or Python (preferably > with the same spelling in each case) makes it understandable. > > I don't care about the specifics of the syntax, but I think the variables > should be spelled out. > There is a big difference between 1) Using only the necessary include directories for a given compile, which might work and 2) Removing includes we "don't need", which I do not think will work Starting with 1). This will break things like "make getincludedirs", unless you want to maintain one thing with the kitchen sink for users, and one tailored thing for PETSc. I think this will generate weird errors where the full line breaks something, but we never see it because we use the custom build. I could still support this if we tested the antique version along with the custom one. However with 2), how would we distinguish includes we only want if the package is found from those that just do not work because of a configuration/build bug? If its annonation, as Jed suggests, how is that a step up from #ifdef? Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] What is this for?
by Jed Brown 16 Nov '11

16 Nov '11

On Wed, Nov 16, 2011 at 00:47, Sean Farley <sean(a)mcs.anl.gov> wrote: > So, this is basically what I think Barry meant ... except the whole /*R > <flags> R*/ part. I would want something equivalently verbose because it's HUMAN documentation for how the variables are spelled. Even if the build system is clairvoyant such that it knows what package each header comes from and which variables it needs to link with, a human won't know that and likely won't know where to find out. Spelling out those variables, which they can query through the makefile include, a CMake include, or Python (preferably with the same spelling in each case) makes it understandable. I don't care about the specifics of the syntax, but I think the variables should be spelled out.

1 0

Re: [petsc-dev] What is this for?
by Sean Farley 16 Nov '11

16 Nov '11

> > I still don't like this. I think modifying (or copying) source files is > bad, not least because of interaction with the debugger, compilation > errors, and other builds with different configurations. > >From what Barry and I talked about earlier today, he doesn't mean copying or modify source files, but rather, having a tool that builds a (partial?) AST that would parse / gather / (clang?) this kind of information automatically. I wouldn't mind something like > > #include <parmetis.h> /*R CFLAGS:PETSC_PARMETIS_FLAGS > INCLUDES:PETSC_PARMETIS_INCLUDES LIBS:PETSC_PARMETIS_LIBS R*/ > > that would inform the build system that PETSC_PARMETIS_INCLUDES and flags > are required to build that file and that the libs are required to link. The > "normal" user can read the file and add that to their build system manually. > So, this is basically what I think Barry meant ... except the whole /*R <flags> R*/ part. Modern compilers ("compilers not designed 40 years ago" - Barry Smith) should be able to give access to things like BuildSystem to build this structure automatically without human interaction. I, for one, welcome our new Compiler Overlords.

1 0

Re: [petsc-dev] What is this for?
by Jed Brown 15 Nov '11

15 Nov '11

On Tue, Nov 15, 2011 at 22:50, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > Yes, we'll need to make sure that the public PETSc include files don't > require goofy preprocessing but I think that is possible, it is only the > source code files (and maybe some private include files) that need the > goofy preprocessing. I still don't like this. I think modifying (or copying) source files is bad, not least because of interaction with the debugger, compilation errors, and other builds with different configurations. I wouldn't mind something like #include <parmetis.h> /*R CFLAGS:PETSC_PARMETIS_FLAGS INCLUDES:PETSC_PARMETIS_INCLUDES LIBS:PETSC_PARMETIS_LIBS R*/ that would inform the build system that PETSC_PARMETIS_INCLUDES and flags are required to build that file and that the libs are required to link. The "normal" user can read the file and add that to their build system manually.

1 0

Re: [petsc-dev] What is this for?
by Barry Smith 15 Nov '11

15 Nov '11

On Nov 15, 2011, at 10:44 PM, Jed Brown wrote: > On Tue, Nov 15, 2011 at 22:39, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > Ideally we would not need to list the information separate from the source file. What should happen is the BuildSystem parses each source file, determines what includes are used and then passes to the compiler exactly the correct -I and no extra. Or as alternative the BuildSystem behaves like CPP and actually puts the exact full path for each include into the source code just before passing it to the compiler. This takes the whole bothersome problem of the compiler searching through the -I paths to find the first include file with the same name and including it. In other words BuildSystem takes on more parts of the compile process away from the untrustworthy UNIX tools for compiling and linking. > > I agree with the rest of your mail, but not this part. I think that the build environment should be managed in essentially the same way for library source code and examples. I also think it's important that examples be relocatable to a user's own project and built with simple tools. If the build system is doing goofy preprocessing, then users won't be able to build it easily with their own tools. This concern is appropriate. But the examples will not need "googy preprocessing" they will be regular old C/C++/Fortran code. Note that in general the examples do not need includes for particular external packages or #if PETSC_HAVE_... stuff. If the user wants one of the external package include then they can use PETSC_PARMETIS_INCLUDE or whatever in their makefiles for those includes to be found. So in short, no goofy preprocessing for examples! Barry Yes, we'll need to make sure that the public PETSc include files don't require goofy preprocessing but I think that is possible, it is only the source code files (and maybe some private include files) that need the goofy preprocessing.

1 0

Re: [petsc-dev] What is this for?
by Jed Brown 15 Nov '11

15 Nov '11

On Tue, Nov 15, 2011 at 22:39, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > Ideally we would not need to list the information separate from the > source file. What should happen is the BuildSystem parses each source file, > determines what includes are used and then passes to the compiler exactly > the correct -I and no extra. Or as alternative the BuildSystem behaves > like CPP and actually puts the exact full path for each include into the > source code just before passing it to the compiler. This takes the whole > bothersome problem of the compiler searching through the -I paths to find > the first include file with the same name and including it. In other words > BuildSystem takes on more parts of the compile process away from the > untrustworthy UNIX tools for compiling and linking. I agree with the rest of your mail, but not this part. I think that the build environment should be managed in essentially the same way for library source code and examples. I also think it's important that examples be relocatable to a user's own project and built with simple tools. If the build system is doing goofy preprocessing, then users won't be able to build it easily with their own tools.

1 0

Re: [petsc-dev] What is this for?
by Barry Smith 15 Nov '11

15 Nov '11

On Nov 15, 2011, at 6:52 PM, Jed Brown wrote: > On Tue, Nov 15, 2011 at 18:48, Matthew Knepley <knepley(a)gmail.com> wrote: > Yes, but you are advocating a really shitty infrastructure for doing this. How do you expose these when the user turns out to want > them (like ParMetis for instance)? How do you follow the chain back to find a bug if you have build logic scattered all over? It is not scattered all over. The information about a particular source file should be with that file (currently that is done using Unix conventions of a makefile in the directory of the source file). The information should not be in some centralized location away from the source code. Ideally we would not need to list the information separate from the source file. What should happen is the BuildSystem parses each source file, determines what includes are used and then passes to the compiler exactly the correct -I and no extra. Or as alternative the BuildSystem behaves like CPP and actually puts the exact full path for each include into the source code just before passing it to the compiler. This takes the whole bothersome problem of the compiler searching through the -I paths to find the first include file with the same name and including it. In other words BuildSystem takes on more parts of the compile process away from the untrustworthy UNIX tools for compiling and linking. Related is the absurd fact that we have in source code things like #if defined(PETSC_HAVE_MEMORY_H) #include <memory.h> #endif and then in BuildSystem we need to list a test for memory.h; instead BuildSystem should scarf all the #if defined(PETSC_HAVE_MEMORY_H) from the source code and do the tests removing the need for us to manually list the test for memory.h in BuildSystem. This need to manually list facts elsewhere that are actually imbedded in the code is simply due to crappy tools, not because it is a good idea. Satish was the one who advocated many years ago that the actual configure tests be scarfed out of the real source code and it is still a good idea (though maybe hard to implement). One could argue that ParMetis.py should be in the PETSc source code tree in the directory that manages the PETSc interface to ParMetis and not in the packages directory with all the rest. Same with other package files. We can't explore all these possibilities if whenever someone changes something someone else gets upset that someone is using a shitty infrastructure. Barry > > The makefile is already used to indicate which files require optional packages. > > If the user wants those headers, they can add $PETSC_PARMETIS_INCLUDES to their compile line and $PETSC_PARMETIS_LIBS to their link line. (Yes, I think that EVERY variable brought in by conf/variables should be namespaced.) > > We shouldn't break the standard practices for shared libraries just because you might want to get access to all these optional things without stating that you want access.

1 0

Re: [petsc-dev] What is this for?
by Barry Smith 15 Nov '11

15 Nov '11

On Nov 15, 2011, at 6:14 PM, Jed Brown wrote: > On Tue, Nov 15, 2011 at 18:12, Matthew Knepley <knepley(a)gmail.com> wrote: > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/0910059a9184 > > One small step in a plan to be less promiscuous. ^^^^ possible Note that we used to do this a bit, but somehow long the way we changed to just listing all the package include files together in one macro and exposing all the include directories everywhere. I just put in this change set to show Sean how it used to be done and how it could be done. What is the right model? See my next email. Barry

1 0

Re: [petsc-dev] What is this for?
by Jed Brown 15 Nov '11

15 Nov '11

On Tue, Nov 15, 2011 at 18:48, Matthew Knepley <knepley(a)gmail.com> wrote: > Yes, but you are advocating a really shitty infrastructure for doing this. > How do you expose these when the user turns out to want > them (like ParMetis for instance)? How do you follow the chain back to > find a bug if you have build logic scattered all over? > The makefile is already used to indicate which files require optional packages. If the user wants those headers, they can add $PETSC_PARMETIS_INCLUDES to their compile line and $PETSC_PARMETIS_LIBS to their link line. (Yes, I think that EVERY variable brought in by conf/variables should be namespaced.) We shouldn't break the standard practices for shared libraries just because you might want to get access to all these optional things without stating that you want access.

1 0