September 2007 - petsc-dev - mailman3.cels.anl.gov

Re: Release
by Lisandro Dalcin 29 Sep '07

29 Sep '07

Barry, please take a look at this thread in Python-Dev mailing list, just the first and second mail. http://mail.python.org/pipermail/python-dev/2003-July/036898.html Python 2.X does exactly the same as PETSc for defining derived object structures. #define PyObject_HEAD \ _PyObject_HEAD_EXTRA \ Py_ssize_t ob_refcnt; \ struct _typeobject *ob_type; typedef struct _object { PyObject_HEAD } PyObject; typedef struct { PyObject_HEAD double ob_fval; } PyFloatObject; But For Python 3.X, they will be using... typedef struct _object { Py_ssize_t ob_refcnt; struct _typeobject *ob_type; } PyObject; #define PyObject_HEAD PyObject ob_base; typedef struct { PyObject_HEAD double ob_fval; } PyFloatObject; This is the way I'm trying to redefine PetscObject for PETSc. On 9/29/07, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > > Well, I followed the approach in Python 3.0 for > > defining the base object header for derived objects. > > Lisandro, > > Please send a reference to where this is discussed and an explanation > why this type of caste is allowed with strict aliasing? (the web page > http://www.cellperformance.com/mike_acton/2006/06/understanding_strict_alia… > does not discuss this case nor does it give any indication that this > "trick" is legal with strict alaising). > > Thanks > > Barry > > > > On Wed, 26 Sep 2007, Lisandro Dalcin wrote: > > > On 9/25/07, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > Please explain in detail. > > > > Well, this article explains strict aliasing in some detail (just in > > case you does not know the details) > > > > http://www.cellperformance.com/mike_acton/2006/06/understanding_strict_alia… > > > > In the particular case of PETSc, if I understand correctly, the > > designed object's struct inheritance based in the current PETSCHEADER > > approach will not make it possible to enable strict aliasing rules. > > For the compiler, PetscObject and Vec are different types of objects, > > so if you do > > > > PetscObject o = (PetscObject)v; > > > > you are breacking strict aliasing rules, using o->comm and v->comm to > > access the vector communicator could possibly fail with aggresive > > compiler optimization. > > > > How to solve this?? Well, I followed the approach in Python 3.0 for > > defining the base object header for derived objects. In the case of > > PETSc, this amounts to the following: > > > > #define PETSCHEADERBASE \ > > PetscCookie cookie; > > MPI_Comm comm; > > char* prefix; > > char* type_name; > > /* etc ....*/ > > > > struct _p_PetscObject { > > PETSCHEADERBASE > > }; > > > > #define PETSCHEADER(ObjectOps) \ > > struct _p_PetscObject header; > > ObjectOps *ops; > > > > And now the Vec struct is declared as before: > > > > struct _p_Vec { > > PETSCHEADER(struct _VecOps); > > /* etc ... */ > > }; > > > > Now, there is no way to access things like cookie/comm/prefix fro a > > Vec, for doing it you need to cast to PetscObject. This removes the > > possibility of accesing 'comm' or 'prefix' though pointers of > > different types. As a side effect, you need to always cast to > > PetscObject to get comm/prefix/type_name (by far the more commonly > > accessed members when implementing Vec/Mat/KSP etc...) > > > > I'm started this work at home, I do not have the code here at my > > office right now, but I could remove almost all the warnings related > > to strict aliasing in 'src/sys' and 'src > > > > > > I have also some problems when calling PetscObjectQueryFunction, > > > > Yes this function pointer casting stuff is a royal pain and ugly. > > > > Perhaps I can find better way of doing this using a macro. > > > > > > > > Might be easier to just rewrite everything in C++. :-) > > > > > > > Well, having error handling based on C++ exceptions is really > > tempting. But C++ polymorphism would make it easier to derive specific > > implementations of objects, and using smart pointer would help for the > > automatic reference counting of owned objects, and a lot of more good > > things. BUT... the PETSc way of doing polimorphism is a bit more > > powerful than the C++ virtual table. That is, each object instance > > have its own table of functions pointers, and you can freely replace > > it in a object by object base (and done in some MatConvert() calls). > > Of course, a C++ implementation could still use tables of function > > pointer, and dispatching code based on that table. > > > > All this would require a lot, lot of redesing and work. :) > > > > BTW, Could you please teach me some easy way of replacing in PETSc > > source code the following: > > > > 'something>comm' by '((PetscObject)something)->comm' > > > > I never found the time to learn regular expresions and 'sed' :-(. > > > > > > > > > > > > Barry > > > > > > > In case > > > > you want me to continue this work, I will need a bit of help in a few > > > > places related to low-level calls related to sockets and sys calls. > > > > > > > > > > > > > > > > > > > > > > -- Lisandro Dalcï¿½n --------------- Centro Internacional de Mï¿½todos Computacionales en Ingenierï¿½a (CIMEC) Instituto de Desarrollo Tecnolï¿½gico para la Industria Quï¿½mica (INTEC) Consejo Nacional de Investigaciones Cientï¿½ficas y Tï¿½cnicas (CONICET) PTLC - Gï¿½emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594

1 0

Re: Release
by Stephan Kramer 29 Sep '07

29 Sep '07

Lisandro Dalcin wrote: > BTW, Could you please teach me some easy way of replacing in PETSc > source code the following: > > 'something>comm' by '((PetscObject)something)->comm' > > I never found the time to learn regular expresions and 'sed' :-(. > > Try something like the following: sed -e 's/\(\w*\)\(\s*->\s*comm\)/((PetscObject)\1)\2/g' here \w* should match any number of alfanumerics + underscores \s* matches any number of blanks in the second part of s/part1/part2/, the \1 substitutes the bit in part1 between the first pair of \( .. \) and \2 substitutes the second pair. Always be careful using this type of automated replacements as unexpected behaviour is always to be expected! You might want to xxdiff the results for instance. Cheers Stephan

1 0

Re: Release
by Barry Smith 28 Sep '07

28 Sep '07

> Well, I followed the approach in Python 3.0 for > defining the base object header for derived objects. Lisandro, Please send a reference to where this is discussed and an explanation why this type of caste is allowed with strict aliasing? (the web page http://www.cellperformance.com/mike_acton/2006/06/understanding_strict_alia… does not discuss this case nor does it give any indication that this "trick" is legal with strict alaising). Thanks Barry On Wed, 26 Sep 2007, Lisandro Dalcin wrote: > On 9/25/07, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Please explain in detail. > > Well, this article explains strict aliasing in some detail (just in > case you does not know the details) > > http://www.cellperformance.com/mike_acton/2006/06/understanding_strict_alia… > > In the particular case of PETSc, if I understand correctly, the > designed object's struct inheritance based in the current PETSCHEADER > approach will not make it possible to enable strict aliasing rules. > For the compiler, PetscObject and Vec are different types of objects, > so if you do > > PetscObject o = (PetscObject)v; > > you are breacking strict aliasing rules, using o->comm and v->comm to > access the vector communicator could possibly fail with aggresive > compiler optimization. > > How to solve this?? Well, I followed the approach in Python 3.0 for > defining the base object header for derived objects. In the case of > PETSc, this amounts to the following: > > #define PETSCHEADERBASE \ > PetscCookie cookie; > MPI_Comm comm; > char* prefix; > char* type_name; > /* etc ....*/ > > struct _p_PetscObject { > PETSCHEADERBASE > }; > > #define PETSCHEADER(ObjectOps) \ > struct _p_PetscObject header; > ObjectOps *ops; > > And now the Vec struct is declared as before: > > struct _p_Vec { > PETSCHEADER(struct _VecOps); > /* etc ... */ > }; > > Now, there is no way to access things like cookie/comm/prefix fro a > Vec, for doing it you need to cast to PetscObject. This removes the > possibility of accesing 'comm' or 'prefix' though pointers of > different types. As a side effect, you need to always cast to > PetscObject to get comm/prefix/type_name (by far the more commonly > accessed members when implementing Vec/Mat/KSP etc...) > > I'm started this work at home, I do not have the code here at my > office right now, but I could remove almost all the warnings related > to strict aliasing in 'src/sys' and 'src > > > > I have also some problems when calling PetscObjectQueryFunction, > > > Yes this function pointer casting stuff is a royal pain and ugly. > > Perhaps I can find better way of doing this using a macro. > > > > > Might be easier to just rewrite everything in C++. :-) > > > > Well, having error handling based on C++ exceptions is really > tempting. But C++ polymorphism would make it easier to derive specific > implementations of objects, and using smart pointer would help for the > automatic reference counting of owned objects, and a lot of more good > things. BUT... the PETSc way of doing polimorphism is a bit more > powerful than the C++ virtual table. That is, each object instance > have its own table of functions pointers, and you can freely replace > it in a object by object base (and done in some MatConvert() calls). > Of course, a C++ implementation could still use tables of function > pointer, and dispatching code based on that table. > > All this would require a lot, lot of redesing and work. :) > > BTW, Could you please teach me some easy way of replacing in PETSc > source code the following: > > 'something>comm' by '((PetscObject)something)->comm' > > I never found the time to learn regular expresions and 'sed' :-(. > > > > > > > Barry > > > > > In case > > > you want me to continue this work, I will need a bit of help in a few > > > places related to low-level calls related to sockets and sys calls. > > > > > > > > > > > > >

1 0

Re: Contents of library archives
by Matthew Knepley 26 Sep '07

26 Sep '07

If you look in any makefile, the LIBBASE gives you the library in which the code is put. The contrib library is code that we accept from outside. Matt On 9/26/07, Keita Teranishi <keita(a)cray.com> wrote: > > > > > Hi PETSc team, > > > > I would like to know the contents of each library archive (.a file) of PETSc > library. > > It looks some of them are self-explanatory such as libpetscmat.a, but I am > wondering other archives such as > > libpetsccontrib.a. Could you explain what each of them contains? > > > > Thanks, > > > > =================== > > Keita Teranishi, Ph.D. > > Math Software Team > > Cray Inc. > > =================== -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Contents of library archives
by Keita Teranishi 26 Sep '07

26 Sep '07

Hi PETSc team, I would like to know the contents of each library archive (.a file) of PETSc library. It looks some of them are self-explanatory such as libpetscmat.a, but I am wondering other archives such as libpetsccontrib.a. Could you explain what each of them contains? Thanks, =================== Keita Teranishi, Ph.D. Math Software Team Cray Inc. ===================

1 0

Re: particle tracking on a DA
by Boyce Griffith 26 Sep '07

26 Sep '07

Hi, Rich -- I essentially do Lagrangian particle tracking in my AMR immersed boundary code using PETSc, although if you are looking for a quick-and-dirty solution, I'm pretty sure you don't want to do things the way that I do. Also, I apologize in advance for the rambling response. The basic idea is very simple: ============================== Store the coordinates of the Lagrangian points in a ghosted Vec, and use an AO object to keep track of the mapping from the current distribution of Lagrangian points to some arbitrary initial ordering of points. The devil's in the details: =========================== During the computation, each point is assigned to the processor that owns the Cartesian grid subdomain in which that point is presently located. The ghosted Vec is constructed to provide the coordinates of all Lagrangian points that lie within the ghost cell region of each Cartesian grid subdomain. I think of each processor as "owning" only those points that lie within the interior of the Cartesian grid subdomain. In particular, processors should only modify data associated with Lagrangian points that they "own". Updated values associated with the ghost points must be provided via VecGhostUpdateBegin / VecGhostUpdateEnd. The drawback of this approach is that you have to re-distribute the Lagrangian point data at regular intervals, as determined by the width of your Lagrangian point ghost cell region and a CFL-type restriction. E.g., if your CFL number is 2 and your ghost cell region is two cells wide, you must redistribute points every timestep. If your CFL number is 0.1 and your ghost cell region is one cell wide, your must redistribute data at least once every 10 timesteps. The advantage to doing things this way is that the code required to update the positions of the Lagrangian points is basically a single-processor code. All you have to do is provide ghost cell values for the Cartesian grid velocity field. This ghost cell width must be sufficiently large to allow you to interpolate the velocities to the positions of the Lagrangian points that are "owned" by the processor. E.g., if you are using bilinear interpolation and node-centered velocities, you probably can get away with a ghost cell width of 1 for the velocity field, depending on the CFL number. When it is time to redistribute the Lagrangian data, it is easy to construct a VecScatter that will perform the data redistribution using AO. The trick is that you have to know which points belong to which processor before you can build the VecScatter that redistributes everything, and I'm not sure if it is straightforward to do this using only PETSc data structures. I do this by storing in each Cartesian grid cell a list of the Lagrangian points that are located in that cell. It should be fairly easy to manage this data efficiently using an appropriate STL container in C++ (e.g., map). Note that these lists must be available both for the interior grid cells as well as the grid cells in the ghost cell region of the subdomain. Summarizing the algorithm: ========================== During each timestep: - On each processor, update the Lagrangian coordinates of all points that are presently assigned to that processor. (Note that the physical positions of these points may have drifted outside of the extents of the subdomain, but that is OK as long as your Cartesian grid velocity ghost cell width is sufficiently large.) When it is time to redistribute points: - Communicate the ghosted Lagrangian coordinate information, - Using the Lagrangian coordinate information, update the lists of Lagrangian point indices in each Cartesian grid cell in the interior of each Cartesian subdomain, - Communicate these lists to the ghost cell regions of all of the Cartesian subdomains, and - Construct a new AO, build the appropriate VecScatter, and redistribute your Lagrangian coordinate data. Good luck, -- Boyce Richard Katz wrote: > Hi All, > > Is anyone aware of PETSc particle tracking code? I'm looking for > something that will do Lagrangian particle tracking in parallel for 2D > fluid flow on a regular cartesian mesh. > > If this doesn't exist, I may try to create it---quick and dirty style. > My thinking was to a master-slaves parallelism: keep a complete > directory of the particle locations on node 0 then send the entire > directory to each node after each timestep to do an update. Each node > updates any points that fall into it's subdomain, flags those particle > entries, then returns the entire directory to node 0. Node 0 merges the > updates into the master directory. > > This isn't scalable but it is simpler to program. Does anyone have any > suggestions about strategy, implementation, PETSc tools that would > facilitate this, etc? > > Cheers > Rich > > Richard Foa Katz NSF IRFP Postdoc > http://www.damtp.cam.ac.uk/user/rfk22 > >

1 0

Re: particle tracking on a DA
by Mehdi Bostandoost 26 Sep '07

26 Sep '07

Hi Richard I am working on Lagrangian particles as well(using DA objects). the way that I am implementing it,is using DA for domain decompositioning. but the problem will appears later on. problems: 1-How do you want to store your particles(using link-list or arrays) 2-how are you going to take care of particle communications? right now,by using DA object you can get rid of domain decomposition. I did not use the master/slave approach.for my case,I am sending the particles using the point to point communication. if a particle goes out of a sub-domain,it should be send to its neighbor. if you want to use this approach,you don't know who are the neighbors using DA object. so I wrote simple code to find out who are the neighboring sub-domains(I attached the code). I am still working on it, but it would be great if there was partcile object (using link list) in petsc as well. Mehdi Richard Katz <rfk22(a)cam.ac.uk> wrote: Hi All, Is anyone aware of PETSc particle tracking code? I'm looking for something that will do Lagrangian particle tracking in parallel for 2D fluid flow on a regular cartesian mesh. If this doesn't exist, I may try to create it---quick and dirty style. My thinking was to a master-slaves parallelism: keep a complete directory of the particle locations on node 0 then send the entire directory to each node after each timestep to do an update. Each node updates any points that fall into it's subdomain, flags those particle entries, then returns the entire directory to node 0. Node 0 merges the updates into the master directory. This isn't scalable but it is simpler to program. Does anyone have any suggestions about strategy, implementation, PETSc tools that would facilitate this, etc? Cheers Rich Richard Foa Katz NSF IRFP Postdoc http://www.damtp.cam.ac.uk/user/rfk22 --------------------------------- Catch up on fall's hot new shows on Yahoo! TV. Watch previews, get listings, and more! #include "include/finclude/petscdef.h" c********************************************************************* subroutine findneighbours use sharedvars implicit none integer :: ierr integer :: i xsdomains(rank+1)=xs call MPI_ALLGATHER(xs,1,MPI_INTEGER, & xsdomains ,1,MPI_INTEGER, & MPI_COMM_WORLD,ierr) CHKERRQ(ierr) call MPI_ALLGATHER(xe,1,MPI_INTEGER, & xedomains ,1,MPI_INTEGER, & MPI_COMM_WORLD,ierr) CHKERRQ(ierr) call MPI_ALLGATHER(ys,1,MPI_INTEGER, & ysdomains ,1,MPI_INTEGER, & MPI_COMM_WORLD,ierr) CHKERRQ(ierr) call MPI_ALLGATHER(ye,1,MPI_INTEGER, & yedomains ,1,MPI_INTEGER, & MPI_COMM_WORLD,ierr) CHKERRQ(ierr) call MPI_ALLGATHER(zs,1,MPI_INTEGER, & zsdomains ,1,MPI_INTEGER, & MPI_COMM_WORLD,ierr) CHKERRQ(ierr) call MPI_ALLGATHER(ze,1,MPI_INTEGER, & zedomains ,1,MPI_INTEGER, & MPI_COMM_WORLD,ierr) CHKERRQ(ierr) write(logfile,*)'xs',xsdomains write(logfile,*)'xe',xedomains write(logfile,*)'ys',ysdomains write(logfile,*)'ye',yedomains write(logfile,*)'zs',zsdomains write(logfile,*)'ze',zedomains xleftdomainrank=-10 yleftdomainrank=-10 zleftdomainrank=-10 xrightdomainrank=-10 yrightdomainrank=-10 zrightdomainrank=-10 do i=1,domainsize if(xs==1)then xleftdomainrank=-10 endif if(ys==1)then yleftdomainrank=-10 endif if(zs==1)then zleftdomainrank=-10 endif if(xe==nx)then xrightdomainrank=-10 endif if(ye==ny)then yrightdomainrank=-10 endif if(ze==nz)then zrightdomainrank=-10 endif enddo do i=1,domainsize if((xs-1)==xedomains(i) .and. & ys==ysdomains(i) .and. ye==yedomains(i) .and. & zs==zsdomains(i) .and. ze==zedomains(i) & )then xleftdomainrank=i-1 endif if((ys-1)==yedomains(i) .and. & xs==xsdomains(i) .and. xe==xedomains(i) .and. & zs==zsdomains(i) .and. ze==zedomains(i) & )then yleftdomainrank=i-1 endif if((zs-1)==zedomains(i) .and. & xs==xsdomains(i) .and. xe==xedomains(i) .and. & ys==ysdomains(i) .and. ye==yedomains(i) & )then zleftdomainrank=i-1 endif enddo do i=1,domainsize if((xe+1)==xsdomains(i) .and. & ys==ysdomains(i) .and. ye==yedomains(i) .and. & zs==zsdomains(i) .and. ze==zedomains(i) & )then xrightdomainrank=i-1 endif if((ye+1)==ysdomains(i) .and. & xs==xsdomains(i) .and. xe==xedomains(i) .and. & zs==zsdomains(i) .and. ze==zedomains(i) & )then yrightdomainrank=i-1 endif if((ze+1)==zsdomains(i) .and. & xs==xsdomains(i) .and. xe==xedomains(i) .and. & ys==ysdomains(i) .and. ye==yedomains(i) & )then zrightdomainrank=i-1 endif enddo return end c*********************************************************************

1 0

Re: particle tracking on a DA
by Berend van Wachem 26 Sep '07

26 Sep '07

Hi, Yes I use PETSc to do particle tracking with DA's. But it is not very simple and I get bad scaling. I have done geometric decomposition of the problem and are keeping "ghost cells". The cells are the size of the lookup for collisions. Questions you should address are, 1. What kind of particle-particle interactions? 2. How many particles? 3. Is there a fluid phase as well? I think the communication with the scheme you propose would be major if the particle numbers go over 500.000 Berend. > Hi All, > > Is anyone aware of PETSc particle tracking code? I'm looking for > something that will do Lagrangian particle tracking in parallel for > 2D fluid flow on a regular cartesian mesh. > > If this doesn't exist, I may try to create it---quick and dirty > style. My thinking was to a master-slaves parallelism: keep a > complete directory of the particle locations on node 0 then send the > entire directory to each node after each timestep to do an update. > Each node updates any points that fall into it's subdomain, flags > those particle entries, then returns the entire directory to node 0. > Node 0 merges the updates into the master directory. > > This isn't scalable but it is simpler to program. Does anyone have > any suggestions about strategy, implementation, PETSc tools that > would facilitate this, etc? > > Cheers > Rich > > Richard Foa Katz NSF IRFP Postdoc > http://www.damtp.cam.ac.uk/user/rfk22 > > >

1 0

Re: Release
by Lisandro Dalcin 26 Sep '07

26 Sep '07

On 9/25/07, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > Please explain in detail. Well, this article explains strict aliasing in some detail (just in case you does not know the details) http://www.cellperformance.com/mike_acton/2006/06/understanding_strict_alia… In the particular case of PETSc, if I understand correctly, the designed object's struct inheritance based in the current PETSCHEADER approach will not make it possible to enable strict aliasing rules. For the compiler, PetscObject and Vec are different types of objects, so if you do PetscObject o = (PetscObject)v; you are breacking strict aliasing rules, using o->comm and v->comm to access the vector communicator could possibly fail with aggresive compiler optimization. How to solve this?? Well, I followed the approach in Python 3.0 for defining the base object header for derived objects. In the case of PETSc, this amounts to the following: #define PETSCHEADERBASE \ PetscCookie cookie; MPI_Comm comm; char* prefix; char* type_name; /* etc ....*/ struct _p_PetscObject { PETSCHEADERBASE }; #define PETSCHEADER(ObjectOps) \ struct _p_PetscObject header; ObjectOps *ops; And now the Vec struct is declared as before: struct _p_Vec { PETSCHEADER(struct _VecOps); /* etc ... */ }; Now, there is no way to access things like cookie/comm/prefix fro a Vec, for doing it you need to cast to PetscObject. This removes the possibility of accesing 'comm' or 'prefix' though pointers of different types. As a side effect, you need to always cast to PetscObject to get comm/prefix/type_name (by far the more commonly accessed members when implementing Vec/Mat/KSP etc...) I'm started this work at home, I do not have the code here at my office right now, but I could remove almost all the warnings related to strict aliasing in 'src/sys' and 'src > > I have also some problems when calling PetscObjectQueryFunction, > > Yes this function pointer casting stuff is a royal pain and ugly. Perhaps I can find better way of doing this using a macro. > > Might be easier to just rewrite everything in C++. :-) > Well, having error handling based on C++ exceptions is really tempting. But C++ polymorphism would make it easier to derive specific implementations of objects, and using smart pointer would help for the automatic reference counting of owned objects, and a lot of more good things. BUT... the PETSc way of doing polimorphism is a bit more powerful than the C++ virtual table. That is, each object instance have its own table of functions pointers, and you can freely replace it in a object by object base (and done in some MatConvert() calls). Of course, a C++ implementation could still use tables of function pointer, and dispatching code based on that table. All this would require a lot, lot of redesing and work. :) BTW, Could you please teach me some easy way of replacing in PETSc source code the following: 'something>comm' by '((PetscObject)something)->comm' I never found the time to learn regular expresions and 'sed' :-(. > > Barry > > > In case > > you want me to continue this work, I will need a bit of help in a few > > places related to low-level calls related to sockets and sys calls. > > > > > > -- Lisandro Dalcï¿½n --------------- Centro Internacional de Mï¿½todos Computacionales en Ingenierï¿½a (CIMEC) Instituto de Desarrollo Tecnolï¿½gico para la Industria Quï¿½mica (INTEC) Consejo Nacional de Investigaciones Cientï¿½ficas y Tï¿½cnicas (CONICET) PTLC - Gï¿½emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594

1 0

particle tracking on a DA
by Richard Katz 26 Sep '07

26 Sep '07

Hi All, Is anyone aware of PETSc particle tracking code? I'm looking for something that will do Lagrangian particle tracking in parallel for 2D fluid flow on a regular cartesian mesh. If this doesn't exist, I may try to create it---quick and dirty style. My thinking was to a master-slaves parallelism: keep a complete directory of the particle locations on node 0 then send the entire directory to each node after each timestep to do an update. Each node updates any points that fall into it's subdomain, flags those particle entries, then returns the entire directory to node 0. Node 0 merges the updates into the master directory. This isn't scalable but it is simpler to program. Does anyone have any suggestions about strategy, implementation, PETSc tools that would facilitate this, etc? Cheers Rich Richard Foa Katz NSF IRFP Postdoc http://www.damtp.cam.ac.uk/user/rfk22

1 0