March 2012 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] buildsystem: cusp location
by Jed Brown 25 Mar '12

25 Mar '12

On Sun, Mar 25, 2012 at 16:32, Matthew Knepley <knepley(a)gmail.com> wrote: > Isn't the final point here that we are trying to let the user do the > dumbest, easiest thing? Maybe no Linux distribution > does this, but we are trying to cope with bad input from a user not using > a nice distro. > > The price we would pay for cleaning up our internals is demanding that > users think more about the configure input. This > is not always a good tradeoff. > Simple compromise proposal: --with-cusp-dir=/prefix Looks in /prefix/include/ for the headers <cusp/version.h>. If unsuccessful, it tries /prefix/. If you insisted, it could even search in the other order. What I don't like is --with-cusp-dir=/prefix not looking for headers in /prefix/include/.

1 0

Re: [petsc-dev] buildsystem: cusp location
by Satish Balay 25 Mar '12

25 Mar '12

On Sun, 25 Mar 2012, Jed Brown wrote: > On Sun, Mar 25, 2012 at 16:14, Satish Balay <balay(a)mcs.anl.gov> wrote: > > > Note: there is no such package for cusp/thrust to base a default - so > > the upstream default thats currently implemented is fine [we are > > supporting actuall use cases instead of fictious ones that don't > > exist]. If such a package actually existed - then compilers would find > > cusp/thrust automatically. > > > > This one puts it in /usr/include/cusp/*.h, therefore we might expect > --with-cusp-dir=/usr to work (just like every other package; even ignoring > the "automatically looking in /usr"). > > https://aur.archlinux.org/packages/cu/cusp/PKGBUILD > > > I will eat my hat if any mainstream linux distro does not put the headers > in the appropriate include directory. >>> echo "export CUSP_PATH=/usr/include" > $pkgdir/etc/profile.d/${pkgname}.sh <<<< Isn't this contradictary to the prefix rule you mentioned before? Also - do they package cuda aswell? If so where is is installed [wrt includes]? My contention here is 'nvcc' will automatically find cusp - so user never needs --with-cusp-dir= option. Satish

1 0

Re: [petsc-dev] buildsystem: cusp location
by Matthew Knepley 25 Mar '12

25 Mar '12

On Sun, Mar 25, 2012 at 4:26 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Sun, Mar 25, 2012 at 16:14, Satish Balay <balay(a)mcs.anl.gov> wrote: > >> Note: there is no such package for cusp/thrust to base a default - so >> the upstream default thats currently implemented is fine [we are >> supporting actuall use cases instead of fictious ones that don't >> exist]. If such a package actually existed - then compilers would find >> cusp/thrust automatically. >> > > This one puts it in /usr/include/cusp/*.h, therefore we might expect > --with-cusp-dir=/usr to work (just like every other package; even ignoring > the "automatically looking in /usr"). > > https://aur.archlinux.org/packages/cu/cusp/PKGBUILD > > > I will eat my hat if any mainstream linux distro does not put the headers > in the appropriate include directory. > Isn't the final point here that we are trying to let the user do the dumbest, easiest thing? Maybe no Linux distribution does this, but we are trying to cope with bad input from a user not using a nice distro. The price we would pay for cleaning up our internals is demanding that users think more about the configure input. This is not always a good tradeoff. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] buildsystem: cusp location
by Jed Brown 25 Mar '12

25 Mar '12

On Sun, Mar 25, 2012 at 16:14, Satish Balay <balay(a)mcs.anl.gov> wrote: > Note: there is no such package for cusp/thrust to base a default - so > the upstream default thats currently implemented is fine [we are > supporting actuall use cases instead of fictious ones that don't > exist]. If such a package actually existed - then compilers would find > cusp/thrust automatically. > This one puts it in /usr/include/cusp/*.h, therefore we might expect --with-cusp-dir=/usr to work (just like every other package; even ignoring the "automatically looking in /usr"). https://aur.archlinux.org/packages/cu/cusp/PKGBUILD I will eat my hat if any mainstream linux distro does not put the headers in the appropriate include directory.

1 0

Re: [petsc-dev] buildsystem: cusp location
by Satish Balay 25 Mar '12

25 Mar '12

On Sun, 25 Mar 2012, Sean Farley wrote: > > > > They give 2 options . [one /usr/local/cuda/include, the other > > /home/nathan/cuda_libraries/cusp] > > > > I'd like to point out that many (if not the majority of) times we (i.e. > BuildSystem logic) isn't working with what the user installed by hand but > rather by one of the many package managers: apt-get, ports, rpm, arch, etc. > These package managers all dictate a prefix install with few exceptions; > regardless of what the package writer really prefers. Again, we even do the > same with --download-package (placing headers in $PETSC_ARCH/include) For one if package managers install stuff [that are distribution compliant] - users don't need to specify --with-package-dir option at-all. Compilers automatically find includes/libs [ and package.py defaults to checking this mode first.] package-dir option is used primarily for non-[apt,yum] installed stuff - either by user or sysadmin. Note: there is no such package for cusp/thrust to base a default - so the upstream default thats currently implemented is fine [we are supporting actuall use cases instead of fictious ones that don't exist]. If such a package actually existed - then compilers would find cusp/thrust automatically. > For your consistancy [wrt to santity of prefix definition] you say - > > petsc configure should support 1 wirh packaage-dir - and users who > > choose 2 should use --with-package-includ option. I don't agree with > > this logic. > I understand you don't agree. The --with-package-dir=prefix would be able > to clean up and unify a lot of code in BuildSystem and I haven't seen a > good counter-point besides the fact that you don't like it. I agree about the cleanup part. But thats beside the point. I thought the motivation so far was user convinence [and autodetect as many user configs as possible] - hence the extra code in buildsystem. > We don't really insist on --with-package-dir=prefix. So far we've been > > accomodating varations of 'package installs' that we could reasonably > > support. > > And the code in each package.py reflects this schism. > > But based on the your previous statement - we should not support mkl > > organiztion with --package-dir option [as it doesn't comply with > > prefix organizaton], insist users should always --with-package-lib > > option. > > > We're using python. We can overload and use inheritance (or whatever fancy > design pattern you want; i.e. decorator) and change the default behavior if > a package really misbehaves like MKL. > > To recap, what I'm suggesting is the following: > > 1) Change the meaning of --with-package-dir to be a prefixed install I think we support this for packages that actually have a concept of prefix-install. But what you are actually saying is: if a user installs an externalpackage separately [that doesn't comply with a prefix install] - then s/he should reinstall [with some manual hacks] per some prefix guidelines specified by petsc - so that s/he might provide --with-package-dir option to petsc configure. I don't claim the current mode works for all curent user installs - but so far - when it failed - we would recommend --download-package as a better alternative. [it also deals with other types of incompatibilities aswell]. One of the sugestions in this thread was for the user to specify --with-include/with-lib options instead of --with-dir - for such cases. > > 2) Report a meaningful error if a wanted file is not found. In the cusp > example: > > =============================================================================== > Configuring PETSc to compile on your system > > =============================================================================== > TESTING: checkInclude from > config.headers(config/BuildSystem/config/headers.py:86) > > > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > ------------------------------------------------------------------------------- > --with-cusp-dir=/Users/sean/local/include does not seem to contain > cusp/version.h; did you want to specify this with --with-cusp-includedir? > ******************************************************************************* Sure better errors are useful. This is independent of the package-dir=prefix issue. [so you can add this stuff now.] > 3) Override this behavior for certain packages like MKL > > I like this more than the current behavior of --with-package-dir because > the location is well-defined and not "how did this particular developer > layout his package? i have no idea since I installed using apt-get!!" apt-get install should already work. And --with-package-dir is generally for manual installs. Satish

1 0

Re: [petsc-dev] buildsystem: cusp location
by Sean Farley 25 Mar '12

25 Mar '12

> > They give 2 options . [one /usr/local/cuda/include, the other > /home/nathan/cuda_libraries/cusp] > I'd like to point out that many (if not the majority of) times we (i.e. BuildSystem logic) isn't working with what the user installed by hand but rather by one of the many package managers: apt-get, ports, rpm, arch, etc. These package managers all dictate a prefix install with few exceptions; regardless of what the package writer really prefers. Again, we even do the same with --download-package (placing headers in $PETSC_ARCH/include) For your consistancy [wrt to santity of prefix definition] you say - > petsc configure should support 1 wirh packaage-dir - and users who > choose 2 should use --with-package-includ option. I don't agree with > this logic. > I understand you don't agree. The --with-package-dir=prefix would be able to clean up and unify a lot of code in BuildSystem and I haven't seen a good counter-point besides the fact that you don't like it. We don't really insist on --with-package-dir=prefix. So far we've been > accomodating varations of 'package installs' that we could reasonably > support. > And the code in each package.py reflects this schism. But based on the your previous statement - we should not support mkl > organiztion with --package-dir option [as it doesn't comply with > prefix organizaton], insist users should always --with-package-lib > option. We're using python. We can overload and use inheritance (or whatever fancy design pattern you want; i.e. decorator) and change the default behavior if a package really misbehaves like MKL. To recap, what I'm suggesting is the following: 1) Change the meaning of --with-package-dir to be a prefixed install 2) Report a meaningful error if a wanted file is not found. In the cusp example: =============================================================================== Configuring PETSc to compile on your system =============================================================================== TESTING: checkInclude from config.headers(config/BuildSystem/config/headers.py:86) ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- --with-cusp-dir=/Users/sean/local/include does not seem to contain cusp/version.h; did you want to specify this with --with-cusp-includedir? ******************************************************************************* 3) Override this behavior for certain packages like MKL I like this more than the current behavior of --with-package-dir because the location is well-defined and not "how did this particular developer layout his package? i have no idea since I installed using apt-get!!"

1 0

Re: [petsc-dev] buildsystem: cusp location
by Satish Balay 25 Mar '12

25 Mar '12

On Sun, 25 Mar 2012, Jed Brown wrote: > On Sun, Mar 25, 2012 at 13:47, Satish Balay <balay(a)mcs.anl.gov> wrote: > > > The default organization for --with-package-dir should be the default > > organization used by the package - not what petsc configure dictates > > whats suitable for itself. For ex: look at MKL - and the wierd path > > scheems it has developed over the past many versions. We support most > > of them. > > > > The cusp installation instructions _say_ to put it in /some/path/include. > That is reasonable, considering that the package consists of headers. Of > course the user could choose to put it somewhere else, but they can inflict > that hardship on themselves using --includedir=. They give 2 options . [one /usr/local/cuda/include, the other /home/nathan/cuda_libraries/cusp] For your consistancy [wrt to santity of prefix definition] you say - petsc configure should support 1 wirh packaage-dir - and users who choose 2 should use --with-package-includ option. I don't agree with this logic. > I don't see how the fact that cusp does not provide an install target > implies that --with-cusp-dir=$prefix has some meaning different from > "prefix" with other packages. We don't really insist on --with-package-dir=prefix. So far we've been accomodating varations of 'package installs' that we could reasonably support. > I don't care that MKL has a funny layout, there is still a reasonably > well-defined concept of what the prefix is. But based on the your previous statement - we should not support mkl organiztion with --package-dir option [as it doesn't comply with prefix organizaton], insist users should always --with-package-lib option. Satish

1 0

Re: [petsc-dev] buildsystem: cusp location
by Jed Brown 25 Mar '12

25 Mar '12

On Sun, Mar 25, 2012 at 13:51, Satish Balay <balay(a)mcs.anl.gov> wrote: > So far its been the 'default organiation of the package' determeined > by the package owner. Yes - ideally everyone complies with some > packaging standard [including us] - but it hasn't been there. > So, hypothetically, suppose cusp had a library too, but still no installer. Then the directions would say to put the headers in /usr/local/cuda/include and the library in /usr/local/cuda/lib, right? Would that change the definition of a prefix? What if it was a manual page? I don't see how lack of an automatic install changes the meaning of a prefix. Any reasonable person would be outraged if a header-only package had an install target and installed to $prefix/*.h instead of $prefix/include/*.h.

1 0

Re: [petsc-dev] buildsystem: cusp location
by Jed Brown 25 Mar '12

25 Mar '12

On Sun, Mar 25, 2012 at 13:47, Satish Balay <balay(a)mcs.anl.gov> wrote: > The default organization for --with-package-dir should be the default > organization used by the package - not what petsc configure dictates > whats suitable for itself. For ex: look at MKL - and the wierd path > scheems it has developed over the past many versions. We support most > of them. > The cusp installation instructions _say_ to put it in /some/path/include. That is reasonable, considering that the package consists of headers. Of course the user could choose to put it somewhere else, but they can inflict that hardship on themselves using --includedir=. I don't see how the fact that cusp does not provide an install target implies that --with-cusp-dir=$prefix has some meaning different from "prefix" with other packages. I don't care that MKL has a funny layout, there is still a reasonably well-defined concept of what the prefix is.

1 0

Re: [petsc-dev] buildsystem: cusp location
by Satish Balay 25 Mar '12

25 Mar '12

On Sun, 25 Mar 2012, Jed Brown wrote: > 2012/3/25 Satish Balay <balay(a)mcs.anl.gov> > > > The key here is for the user to be aware that the layout is not normal. > > > > Well, what is "normal"? The way the package recommends and the way all > other packages are organized, or the way an unknowing user might lay things > out? So far its been the 'default organiation of the package' determeined by the package owner. Yes - ideally everyone complies with some packaging standard [including us] - but it hasn't been there. For ex: MLK on windows [or any other package on windows - don't comply with the linux organization]... > > I agree - There is a bit of autodetection with --with-package-dir > > option. But if one knows include,lib details and want to specify these > > details --with-package-include, --with-package-lib are appropriate > > options. > > > > And with default layout - one should expect both modes to work. > > > > > > > > > What other package ever uses --prefix=.../include? Every one I know of > > > would expect you to say --prefix=/opt/local/ and it would place headers > > in > > > /opt/local/include. > > > > Well this package is trying to organize itself within 'cuda include' > > package orgainzation. I think this is smilar to the way 'perl' or > > 'python' modules organize themselves within their respective trees. > > > > With Python, I say --prefix=/opt/local and it puts things in > /opt/local/lib/python3.2/site-packages. There is a PYTHONPATH environment > variable that points to the inner directory, but I have always thought > --with-foo-dir= should be thought of as matching a --prefix option. Yes - but some packages don't have installers like cusp thrust. Satish >

1 0