September 2005 - petsc-dev - mailman3.cels.anl.gov

make_lineal.py
by S V N Vishwanathan 28 Sep '05

28 Sep '05

Hi! If you check out the latest Numerix repository, under externalpackages/petsc you will find make_lineal.py which essentially is your make.py with minor tweaks to build our python wrappers into $PETSC_DIR/lib/$PETSC_ARCH/pypetsc We would ideally like to run this file from within our root directory i.e. use $LINEAL_DIR but the hardcoded dependencies built into the PETSc configure prevent us from doing this. We can submit patches to PETSc configure if give us specific pointers on how to go around doing this. Alternatively, if you make changes, we are willing to be perfect gunieapigs :) vishy ----- Allen's Axiom: When all else fails, read the instructions.

1 0

Re: BuildSystem question
by S V N Vishwanathan 26 Sep '05

26 Sep '05

Hi! >> Ideally, I would like to use your BuildSystem to hack up a quick build >> for Numerix. Then as we add and integrate more tools in, I would work on >> making it fancier. Matt> Cool, we can do that. Step one in that direction would be for me to adapt make.py in $PETSC_DIR to build our shared libraries. I have made some effort in this direction (see attachment). But when I run it I get the following error: [dhcp-215-103]:~/Repositories/numerix % export PYTHONPATH=~/Repositories/numerix/externalpackages/petsc/python/BuildSystem:~/Repositories/numerix/externalpackages/petsc/python/PETSc:$PYTHONPATH [dhcp-215-103]:~/Repositories/numerix % python make.py Tue 09:09 Traceback (most recent call last): File "make.py", line 82, in ? Make().run() File "/Users/vishy/Repositories/numerix/externalpackages/petsc/python/BuildSystem/maker.py", line 213, in run self.executeSection(self.configure, self.builder) File "/Users/vishy/Repositories/numerix/externalpackages/petsc/python/BuildSystem/maker.py", line 207, in executeSection return section(*args) File "make.py", line 46, in configure framework = maker.BasicMake.configure(self, builder) File "/Users/vishy/Repositories/numerix/externalpackages/petsc/python/BuildSystem/maker.py", line 291, in configure framework = Make.configure(self, builder) File "/Users/vishy/Repositories/numerix/externalpackages/petsc/python/BuildSystem/maker.py", line 149, in configure if not self.setupConfigure(self.framework): File "make.py", line 36, in setupConfigure maker.BasicMake.setupConfigure(self, framework) File "/Users/vishy/Repositories/numerix/externalpackages/petsc/python/BuildSystem/maker.py", line 283, in setupConfigure Make.setupConfigure(self, framework) File "/Users/vishy/Repositories/numerix/externalpackages/petsc/python/BuildSystem/maker.py", line 136, in setupConfigure self.configureObj.setupPackageDependencies(framework) File "/Users/vishy/Repositories/numerix/externalpackages/petsc/python/PETSc/petsc.py", line 88, in setupPackageDependencies framework.addPackageDependency(petscConf, confPath) File "/Users/vishy/Repositories/numerix/externalpackages/petsc/python/BuildSystem/config/framework.py", line 297, in addPackageDependency self.childGraph.replaceVertex(self.require(child.__module__, None), child) File "/Users/vishy/Repositories/numerix/externalpackages/petsc/python/BuildSystem/config/framework.py", line 255, in require config = self.getChild(moduleName, keywordArgs) File "/Users/vishy/Repositories/numerix/externalpackages/petsc/python/BuildSystem/config/framework.py", line 235, in getChild config.setupDependencies(self) File "/Users/vishy/Repositories/numerix/externalpackages/petsc/python/PETSc/Configure.py", line 40, in setupDependencies for utility in os.listdir(os.path.join('python', 'PETSc', d)): OSError: [Errno 2] No such file or directory: 'python/PETSc/utilities' You can also find the latest version checked into SVN. vishy ----- Parting is such sweet sorrow. -William Shakespeare

1 0

Re: BuildSystem question
by Matthew Knepley 26 Sep '05

26 Sep '05

S V N Vishwanathan <vishy(a)mail.rsise.anu.edu.au> writes: > Ideally, I would like to use your BuildSystem to hack up a quick build > for Numerix. Then as we add and integrate more tools in, I would work on > making it fancier. Cool, we can do that. > Here is the plan I came up with (after sleeping over it for a night): > > 1) I assume that the PETSc Framework object can be made modular i.e. can > be invoked as long as the PETSC_DIR is set and points to a sane > directory. Yes. I currently have nothing that does this, so you sending bug reports would be fine for now, and I will fix what is wrong. > 2) I will create a new Configure object for Numerix and add it to the > PETSc framework using require(). My configure script will also set > the PETSC_DIR env variable to a sane directory. Okay. > 3) All options that are passed to my build are first passed to the PETSc > build. I also pass it > --with-external-packages-dir=$LINEAL_DIR/externalpackages Actually, the first thing that happens is all options are stored in the RDict, so all you have to do is instantiate the Petsc part with the same RDict. This is what happens with the ParMetis build for instance. > 4) Any extra flags which are not handled by PETSc (or are shared between > packages) will be handled and passed to appropriate builds by the > Numerix configure. See above answer. > Does this make sense? > > How do we handle the case when say PETSc libs are installed by using a > package manager like rpm or apt? If they follow the PETSc build process (which they should), the RDict will be right there waiting for you. Otherwise, we will have to smack them around for a while. There is code to look for the libraries/includes, but it also wants the RDict right now. > I guess my problem at a high level is as follows: > > What is the best approach to take for a package which depends on PETSc > to use > a) The BuildSystem module > b) The PETSc framework object itself > > I am sorry if some of my questions are incoherent. I am still trying to > figure out how to structure out build and I realize it is not an easy > task. Hmmm, I do not quite understand the question. The framework object does configuration, and other parts of BuildSystem do build. I admit that the build stuff is not as good or as mature, but I think it is harder to do. Matt -- "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness

1 0

Re: BuildSystem question
by S V N Vishwanathan 26 Sep '05

26 Sep '05

Hi! >> I am unclear about where external packages required by PETSc should >> go. Many packages (like PETSc, TAO, OOQP) might want to use some common >> external package (like MPICH). But it seems like a wasted effort if we >> replicate the PETSc build scripts which already install these packages >> into $PETSC_DIR/externalpackages. Matt> PETSc does have --with-external-packages-dir, which defaults to Matt> $PETSC_DIR/externalpackages, but you can initialize it to whatever Matt> you want. This I realized after I had a careful look at the configure options yesterday night. >> When I try to grab the framework object from RDict.db and use a >> require method on it, it seems to bomb because many of the scripts in >> $PETSC_DIR/python/PETSc seem to make an implicit assumption that the >> configure scripts are run from $PETSC_DIR (a bad assumption I think, >> esp given that you have the PETSC_DIR variable available to you). Matt> We would be willing to work on this with you. That requirement has Matt> been discussed. The point of view from the other side is that Matt> users (very) often get confused and we would like to do as much as Matt> possible to make sure they are doing the correct thing (like Matt> installing from PETSC_DIR). Also, we never assume that PETSC_DIR Matt> is defined during configure. That said, I'm sure these outright Matt> dependencies can be fixed. Yes that would be very good. I am sure there are other pieces of software (TAO and SLEPc come to mind) which might benefit from just making their build depend on the PETSc configure. As for the PETSC_DIR problem, I think your configure.py script already does enough sanity checking that you can safely set a variable and use it. I am sure it is more portable than the current approach. Matt> I tried to document the configure (Python) part as much as I Matt> could. I know the make part is not documented, but it is an Matt> artifact of an earlier age when make systems were considered the Matt> equivalent of hammers, which are rarely documented. Feel free to Matt> mail questions. Good. I will probably mail you and bug you a bit more :) >> So my question is simple, if you were in my position i.e. designing the >> build scripts for LINEAL, how would approach the problem? Matt> I would decide up front whether I wanted to use: Matt> a) make For heavens sake NO! Matt> b) autotools I know they are quite powerful but never got my head around to actually using them. They are still a bit of a black box to me and I would keep away from them. Matt> c) something totally new I am seriously considering using your BuildSystem. I like it since it is Python and modular. My only concern is whether it might be a bit of an overkill to begin with. But going forward it might be more extensible and more comprehensive. I am also considering Scons. The main advantage of Scons is that it is well documented and hence the learning curve is not too steep. Ideally, I would like to use your BuildSystem to hack up a quick build for Numerix. Then as we add and integrate more tools in, I would work on making it fancier. Matt> From this basic design decision flows almost every other Matt> decision. Also, it gives you an idea of what kind of investment is Matt> necessary. I will try and indicate why I choose c). The kind of Matt> work involved in making a new system is different and appeals to The idea of working with a system which has good support appeals to me. For me having good support is more important than having good documentation :) While I would not mind trying out new things I am generally loathe to develop new things unless there is a clear and pressing need for me to do so. Matt> me. If you use a), generally you spend lots of time writing shell Matt> and working out bizzare tricks with stamp files to get flow I hate Make too. I tried out only simple things with it but already things got very very messy. Matt> work things out, but you can fix many things and perhaps introduce Matt> a more powerful paradigm. However, all projects are different and Matt> all options shuold be considered every time. I find that many of the aims I outlined in my last mail are very similar to what the PETSc build already does. I am willing to live with a build completely done in Python. Here is the plan I came up with (after sleeping over it for a night): 1) I assume that the PETSc Framework object can be made modular i.e. can be invoked as long as the PETSC_DIR is set and points to a sane directory. 2) I will create a new Configure object for Numerix and add it to the PETSc framework using require(). My configure script will also set the PETSC_DIR env variable to a sane directory. 3) All options that are passed to my build are first passed to the PETSc build. I also pass it --with-external-packages-dir=$LINEAL_DIR/externalpackages 4) Any extra flags which are not handled by PETSc (or are shared between packages) will be handled and passed to appropriate builds by the Numerix configure. Does this make sense? How do we handle the case when say PETSc libs are installed by using a package manager like rpm or apt? I guess my problem at a high level is as follows: What is the best approach to take for a package which depends on PETSc to use a) The BuildSystem module b) The PETSc framework object itself I am sorry if some of my questions are incoherent. I am still trying to figure out how to structure out build and I realize it is not an easy task. vishy ----- We're mortal -- which is to say, we're ignorant, stupid, and sinful -- but those are only handicaps. Our pride is that nevertheless, now and then, we do our best. A few times we succeed. What more dare we ask for? -- Ensign Flandry

1 0

Re: BuildSystem question
by Matthew Knepley 25 Sep '05

25 Sep '05

S V N Vishwanathan <vishy(a)mail.rsise.anu.edu.au> writes: > Hi! > > I am tying myself up in knots in trying to figure out your BuildSystem > and how to adapt it to LINEAL. PETSc is one primary component of LINEAL > but going forward there will be others (viz. TAO, OOQP). > > I want that my users should be exposed to one and only one config + make > step if possible. A typical user interaction should be as follows: > > 1) If PETSc exists under $LINEAL_DIR/externalpackages and is configured > and built with the right switches then we just build LINEAL. Good. > 2) If the user wishes to use their own custom PETSc then they just > specify --with-petsc-dir= and I should be able to pick it up from > there. Also good. > 3) If neither 1 nor 2 is true then I should offer to download, configure > and build PETSc (much akin to what the current PETSc build scripts > currently do). Exactly. > I am unclear about where external packages required by PETSc should > go. Many packages (like PETSc, TAO, OOQP) might want to use some common > external package (like MPICH). But it seems like a wasted effort if we > replicate the PETSc build scripts which already install these packages > into $PETSC_DIR/externalpackages. PETSc does have --with-external-packages-dir, which defaults to $PETSC_DIR/externalpackages, but you can initialize it to whatever you want. > When I try to grab the framework object from RDict.db and use a require > method on it, it seems to bomb because many of the scripts in > $PETSC_DIR/python/PETSc seem to make an implicit assumption that the > configure scripts are run from $PETSC_DIR (a bad assumption I think, esp > given that you have the PETSC_DIR variable available to you). We would be willing to work on this with you. That requirement has been discussed. The point of view from the other side is that users (very) often get confused and we would like to do as much as possible to make sure they are doing the correct thing (like installing from PETSC_DIR). Also, we never assume that PETSC_DIR is defined during configure. That said, I'm sure these outright dependencies can be fixed. > The lack of documentation adds to my woes. I don't want to be a cry > baby, but I did spend considerable time and effort trying to figure > things out and still don't have a good feel for how different scripts > interact. I tried to document the configure (Python) part as much as I could. I know the make part is not documented, but it is an artifact of an earlier age when make systems were considered the equivalent of hammers, which are rarely documented. Feel free to mail questions. > So my question is simple, if you were in my position i.e. designing the > build scripts for LINEAL, how would approach the problem? I would decide up front whether I wanted to use: a) make b) autotools c) something totally new >From this basic design decision flows almost every other decision. Also, it gives you an idea of what kind of investment is necessary. I will try and indicate why I choose c). The kind of work involved in making a new system is different and appeals to me. If you use a), generally you spend lots of time writing shell and working out bizzare tricks with stamp files to get flow control. If you choose b), you spend lots of time understanding someone else's system, and for autotools that means even more, even more baroque shell. With c), you have to write much more and work things out, but you can fix many things and perhaps introduce a more powerful paradigm. However, all projects are different and all options shuold be considered every time. Matt -- "Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness

1 0

BuildSystem question
by S V N Vishwanathan 25 Sep '05

25 Sep '05

Hi! I am tying myself up in knots in trying to figure out your BuildSystem and how to adapt it to LINEAL. PETSc is one primary component of LINEAL but going forward there will be others (viz. TAO, OOQP). I want that my users should be exposed to one and only one config + make step if possible. A typical user interaction should be as follows: 1) If PETSc exists under $LINEAL_DIR/externalpackages and is configured and built with the right switches then we just build LINEAL. 2) If the user wishes to use their own custom PETSc then they just specify --with-petsc-dir= and I should be able to pick it up from there. 3) If neither 1 nor 2 is true then I should offer to download, configure and build PETSc (much akin to what the current PETSc build scripts currently do). I am unclear about where external packages required by PETSc should go. Many packages (like PETSc, TAO, OOQP) might want to use some common external package (like MPICH). But it seems like a wasted effort if we replicate the PETSc build scripts which already install these packages into $PETSC_DIR/externalpackages. When I try to grab the framework object from RDict.db and use a require method on it, it seems to bomb because many of the scripts in $PETSC_DIR/python/PETSc seem to make an implicit assumption that the configure scripts are run from $PETSC_DIR (a bad assumption I think, esp given that you have the PETSC_DIR variable available to you). The lack of documentation adds to my woes. I don't want to be a cry baby, but I did spend considerable time and effort trying to figure things out and still don't have a good feel for how different scripts interact. So my question is simple, if you were in my position i.e. designing the build scripts for LINEAL, how would approach the problem? vishy ----- Where you stand depends on where you sit. -- Rufus Miles, HEW

1 0

Re: PETSc for Python
by Simon Burton 16 Sep '05

16 Sep '05

On Thu, 15 Sep 2005 20:14:02 -0300 Lisandro Dalcin <dalcinl(a)gmail.com> wrote: > > > Simon, some time ago I wrote to Matt Knepley to propose him the > development of a Python interface to PETSc. He agreed, but I have not > received any answer yet, I suppose he is busy. > > My bindings are rather mature, I am actually runing nontrivial FEM > problems in parallel with them. > > I understand you are a "Python guru" in your team. So I really want to > know if you are interested in something like that. It would be really > nice to have a 'official' interface for accesing PETSc from Python, > this will help a lot in comparing differente implementations (Matt > uses SIDL, I am using SWIG, Matt told me you use pyrex, perhaps other > guys can try boost libraries). Hi Lisandro! Happy to hear from you again :) (I'm a keen tango dancer, and i know you live in the land of tango). Not sure what to say here; it seems we have three competing python wrappers. I started work on pypetsc because we also needed a *complete* wrapper to PETSc. The pypetsc API is similar to the C API, and to Matt's sidl wrapper (and petsc4py). So as you say, hopefully we can all converge to the same interface then it won't matter which wrapper is used. Our wrapper is also complete in that we can access the whole C API, although it's a little rough to use at the moment. It has other advantages: -- autogenerated from the (.h) headers, so can track the source -- generates pyrex code: array operations (for-loops) can be written in pyrex (==easy) and then compiled to c (==fast code). We plan to release the tool itself (pyxelator) soon, as it can be used to wrap other c libraries. There is also the whole python numeric/numarray community to consider: can we expose a matrix API that looks anything like what they use ? And: we are working with academic institutions who are migrating from Matlab to python/PETSc, so they have their preferences too. A much more profound issue is how to employ python to express parallel programming (with all these different matrix types). That really excites me, but i guess it's beside the point right now. I'm cc'ing the petsc-dev list, hope thats OK. ciao! Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com

1 0

Re: PETSc for Python
by S V N Vishwanathan 15 Sep '05

15 Sep '05

Hi! Let me make a bit of our aims clear. >> My bindings are rather mature, I am actually runing nontrivial FEM >> problems in parallel with them. For us wrapping PETSc in Python is just the first step. We basically want to write a thin layer on top of it such that it is "natural" to use PETSc as the backend while doing linear algebra computation in Python very similar in spirit to numarray. Till now we have not paid much attention to the parallel aspects but going forward we will. There is an entire consortium of labs who are interested in developing this platform (thin layer). Our ultimate goal is to have a stable mature platform on which we can develop higher level machine learning algorithms. More information about our efforts can be found at http://sml.nicta.com.au/crest We intend to release the numerical platform under the free BSD license so that it can be adapted by a wider community. >> I understand you are a "Python guru" in your team. So I really want to >> know if you are interested in something like that. It would be really >> nice to have a 'official' interface for accesing PETSc from Python, >> this will help a lot in comparing differente implementations (Matt >> uses SIDL, I am using SWIG, Matt told me you use pyrex, perhaps other >> guys can try boost libraries). Simon> Not sure what to say here; it seems we have three competing Simon> python wrappers. I started work on pypetsc because we also needed Simon> a *complete* wrapper to PETSc. The pypetsc API is similar to the Simon> C API, and to Matt's sidl wrapper (and petsc4py). So as you say, Simon> hopefully we can all converge to the same interface then it won't Simon> matter which wrapper is used. But I do agree that having three different python bindings is not the optimal strategy. Maybe we should think about merging the best aspects of all three bindings into one gold standard. Matt: Is there a way we could do a phone conf or email thread discussion and converge on a gold standard? We intend to do an alpha release of our platform by the end of this month. So I would prefer this to happen sooner than later. I have another related but slightly tangential question. We would also like to wrap TAO and other related optimization packages. My understanding is that our method does not allow for this to happen seamlessly since TAO relies very heavily on C++ and our wrapping scripts have problems with that. Maybe when we converge on *the* standard we will also take this into account. Please let me know your thoughts. vishy

1 0

Build system
by S V N Vishwanathan 14 Sep '05

14 Sep '05

Hi! We are looking to piggy bank our build on top of the PETSc build system. I see that many packages (viz. TAO) have managed to do that. I am looking for some documentation. In general, I am a bit confused about what is the best way to achieve this integration. Ideally, we would like to distribute a patched version of the PETSc build system which when given the flags --with-pypetsc=1 and --download-pypetsc=1 does the needful. BTW we are inching closer towards an alpha release (extremely unstable but usable) by the end of this month. vishy

1 0

Re: patch for seqdense
by Simon Burton 06 Sep '05

06 Sep '05

On Fri, 2 Sep 2005 15:41:50 -0500 (CDT) Hong Zhang <hzhang(a)mcs.anl.gov> wrote: > > Simon, > > I tested MatMatMult* using a new file > petsc-dev/src/mat/examples/tests/ex104.c. > > Why you did not implement > MatMatMultSymbolic_SeqDense_SeqDense() > and > MatMatMultTransposeSymbolic_SeqDense_SeqDense()? > > Did you use inplace mat product in your python test? We ended up wrapping BLASgemm_ and using it directly. This call provides a kind of MatMatMultAdd (except the result gets stored in the add matrix) which we need. But obviously this will not work for the other matrix types. bye, Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com

1 0