November 2009 - petsc-dev - mailman3.cels.anl.gov

Re: -malign-double
by Barry Smith 16 Nov '09

16 Nov '09

On Nov 16, 2009, at 2:07 PM, Satish Balay wrote: > The whole reason for having PetscMalloc2() PetscMalloc3() etc is to > reduce the number of calls to malloc() - thus hoping for a performance > boost. > > For one - this usage was triggered by old SunOS boxes [where malloc > took a long time]. We don't know if this is still true for any of the > current OSes. > > With the increased complexity of managing alignment here - is it still > worth keeping these merged mallocs? [do we still get any payoff from > this complexity? - instead of relying directly on malloc() for the > necessary alignment?] I would argue yes. Not only is there the issue of improved performance of the malloc but it also is a nice way to have clarity in the code as to what allocated arrays are associated together. Barry > > I'm not sure.. > > Satish > > On Mon, 16 Nov 2009, Jed Brown wrote: > >> Barry Smith wrote: >>> >>> Agreed this would be a good option to have. The question is how >>> to do >>> it without having a morass of nasty nested if-defs. Note that >>> portably >>> getting alignment out of malloc() alone is already ugly and not as >>> simple a code as I would like. >>> >>> To simplify things we could always require 16 byte alignment >>> everywhere? But is that a desirable? >> >> I doubt it's harmful, I don't think I understand where the nested >> ifdefs >> come in. The required alignment only needs to be stated in one >> place, I >> have something like the following in one of my projects. >> >> /* current */ >> #define >> PetscMalloc3(m1,t1,r1,m2,t2,r2,m3,t3,r3) \ >> (PetscMalloc((m1)*sizeof(t1)+(m2)*sizeof(t2)+ >> (m3)*sizeof(t3),r1) \ >> || (*(r2) = (t2*) >> (*(r1)+m1), \ >> *(r3) = (t3*)(*(r2)+m2),0)) >> >> /* aligned */ >> #define >> PetscMalloc3(m1,t1,r1,m2,t2,r2,m3,t3,r3) \ >> (PetscMalloc((m1)*sizeof(t1)+(m2)*sizeof(t2)+ >> (m3)*sizeof(t3)+2*(PETSC_MEMALIGN-1),r1) \ >> || (*(r2) = >> (t2*)PETSC_ALIGN(*(r1)+m1), \ >> *(r3) = (t3*)PETSC_ALIGN(*(r2)+m2),0)) >> >> #define PETSC_ALIGN(p) PetscNextAligned((uintptr_t) >> (p),PETSC_MEMALIGN-1) >> >> static inline void *PetscNextAligned(uintptr_t base,uintptr_t mask) >> {return (void*)((base + mask) & ~mask);} >> >> >> Note that this is a no-op if PETSC_MEMALIGN=1 and thus compiles to >> exactly what we have now. >> >> Jed >> >> >

1 0

Re: -malign-double
by Satish Balay 16 Nov '09

16 Nov '09

The whole reason for having PetscMalloc2() PetscMalloc3() etc is to reduce the number of calls to malloc() - thus hoping for a performance boost. For one - this usage was triggered by old SunOS boxes [where malloc took a long time]. We don't know if this is still true for any of the current OSes. With the increased complexity of managing alignment here - is it still worth keeping these merged mallocs? [do we still get any payoff from this complexity? - instead of relying directly on malloc() for the necessary alignment?] I'm not sure.. Satish On Mon, 16 Nov 2009, Jed Brown wrote: > Barry Smith wrote: > > > > Agreed this would be a good option to have. The question is how to do > > it without having a morass of nasty nested if-defs. Note that portably > > getting alignment out of malloc() alone is already ugly and not as > > simple a code as I would like. > > > > To simplify things we could always require 16 byte alignment > > everywhere? But is that a desirable? > > I doubt it's harmful, I don't think I understand where the nested ifdefs > come in. The required alignment only needs to be stated in one place, I > have something like the following in one of my projects. > > /* current */ > #define PetscMalloc3(m1,t1,r1,m2,t2,r2,m3,t3,r3) \ > (PetscMalloc((m1)*sizeof(t1)+(m2)*sizeof(t2)+(m3)*sizeof(t3),r1) \ > || (*(r2) = (t2*)(*(r1)+m1), \ > *(r3) = (t3*)(*(r2)+m2),0)) > > /* aligned */ > #define PetscMalloc3(m1,t1,r1,m2,t2,r2,m3,t3,r3) \ > (PetscMalloc((m1)*sizeof(t1)+(m2)*sizeof(t2)+(m3)*sizeof(t3)+2*(PETSC_MEMALIGN-1),r1) \ > || (*(r2) = (t2*)PETSC_ALIGN(*(r1)+m1), \ > *(r3) = (t3*)PETSC_ALIGN(*(r2)+m2),0)) > > #define PETSC_ALIGN(p) PetscNextAligned((uintptr_t)(p),PETSC_MEMALIGN-1) > > static inline void *PetscNextAligned(uintptr_t base,uintptr_t mask) > {return (void*)((base + mask) & ~mask);} > > > Note that this is a no-op if PETSC_MEMALIGN=1 and thus compiles to > exactly what we have now. > > Jed > >

1 0

Re: -malign-double
by Jed Brown 16 Nov '09

16 Nov '09

Barry Smith wrote: > > Jed, > > Seems ok to me. Feel free to update PETSc. (But I think PETSc style > would detect that your PETSC_ALIGN() macro would be called > PetscMemAlign() or something, we only use caps for macros that are fixed > values, not for macros that like functions.) Indeed (though there are a few exceptions). Although stdint.h (which contains uintptr_t) is required by C99, I suppose it's likely that it's broken somewhere. On Win64, unsigned long is not sufficient so we need unsigned long long (also required by C99, but not in C++). This is messy enough that we may as well work around it by truncating the pointer into an unsigned, computing the shift, and adding the shift through a (char*) cast. Please speak up if uintptr_t is actually reliable. (I just require a C99 compiler and don't tolerate non-compliance, but PETSc can't do that). Jed

1 0

Re: Should PetscObjectName be collective?
by Lisandro Dalcin 16 Nov '09

16 Nov '09

On Mon, Nov 16, 2009 at 4:04 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Ok, we could just put the "counter" into the MPI attribute and then just > have uniqueness of the names within each communicator. This would solve > Jed's problem cleanly. > I agree. +1 on this. > > We don't have any way of requiring the user to use Unique names anyways. > With great power comes great responsibility... -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594

1 0

Re: [petsc-maint #37691] problem with petsc4py
by Lisandro Dalcin 16 Nov '09

16 Nov '09

On Mon, Nov 16, 2009 at 4:16 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > On Nov 16, 2009, at 9:37 AM, Lisandro Dalcin wrote: > >> On Sat, Nov 14, 2009 at 12:36 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >>> >>> petsc-dev should be setup to always use the appropriate petsc4py so this >>> shit works automatically. >>> >> >> However, please note that I cannot guarantee that petsc4py-dev will be >> in sync with latest changes in petsc-dev. Sometimes, a delay of a week >> or so should be expected. > > This we understand. > >>> $ hg clone http://petsc.cs.iit.edu/petsc4py/petsc4py-dev >> > > So the only thing holding us up is having Matt add support for using hg to > download in BuildSystem. There was support for bk but hg was never added. > > > Matt, can you add this? > IIRC, there were issues about that a URL like "http://petsc.cs.iit.edu/petsc4py/petsc4py-dev" provides no clue about the repo being a Mercurial one. However, I want to mention a "convention" it is being used elsewere (see for example: http://pip.openplans.org/#requirements-files) Basically, you have to prefix the URL like this: "hg+http://petsc.cs.iit.edu/petsc4py/petsc4py-dev".. then the hg+ svn+ git+ bzr+ prefix provides the required clue. -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594

1 0

Re: -malign-double
by Barry Smith 16 Nov '09

16 Nov '09

Jed, Seems ok to me. Feel free to update PETSc. (But I think PETSc style would detect that your PETSC_ALIGN() macro would be called PetscMemAlign() or something, we only use caps for macros that are fixed values, not for macros that like functions.) Thanks for thinking about this, Barry On Nov 16, 2009, at 4:11 AM, Jed Brown wrote: > Barry Smith wrote: >> >> Agreed this would be a good option to have. The question is how >> to do >> it without having a morass of nasty nested if-defs. Note that >> portably >> getting alignment out of malloc() alone is already ugly and not as >> simple a code as I would like. >> >> To simplify things we could always require 16 byte alignment >> everywhere? But is that a desirable? > > I doubt it's harmful, I don't think I understand where the nested > ifdefs > come in. The required alignment only needs to be stated in one > place, I > have something like the following in one of my projects. > > /* current */ > #define > PetscMalloc3(m1,t1,r1,m2,t2,r2,m3,t3,r3) \ > (PetscMalloc((m1)*sizeof(t1)+(m2)*sizeof(t2)+ > (m3)*sizeof(t3),r1) \ > || (*(r2) = (t2*) > (*(r1)+m1), \ > *(r3) = (t3*)(*(r2)+m2),0)) > > /* aligned */ > #define > PetscMalloc3(m1,t1,r1,m2,t2,r2,m3,t3,r3) \ > (PetscMalloc((m1)*sizeof(t1)+(m2)*sizeof(t2)+ > (m3)*sizeof(t3)+2*(PETSC_MEMALIGN-1),r1) \ > || (*(r2) = > (t2*)PETSC_ALIGN(*(r1)+m1), \ > *(r3) = (t3*)PETSC_ALIGN(*(r2)+m2),0)) > > #define PETSC_ALIGN(p) PetscNextAligned((uintptr_t) > (p),PETSC_MEMALIGN-1) > > static inline void *PetscNextAligned(uintptr_t base,uintptr_t mask) > {return (void*)((base + mask) & ~mask);} > > > Note that this is a no-op if PETSC_MEMALIGN=1 and thus compiles to > exactly what we have now. > > Jed >

1 0

Re: Should PetscObjectName be collective?
by Barry Smith 16 Nov '09

16 Nov '09

Ok, we could just put the "counter" into the MPI attribute and then just have uniqueness of the names within each communicator. This would solve Jed's problem cleanly. Barry We don't have any way of requiring the user to use Unique names anyways. On Nov 16, 2009, at 11:17 AM, Jed Brown wrote: > Barry Smith wrote: > >> I do not know a way to generate unique MPI-comm-id's. Any thoughts? >> Some feature of MPI I don't know? > > This seems very hard. One could (the first time PETSc sees a > communicator) hash the hostname+pid of every process on the > communicator, start a counter on this hash, and set the communicator > name to be the hash concatenated with this counter. I think this is > sufficient, but it's kinda nasty. > > Jed >

1 0

Re: Should PetscObjectName be collective?
by Jed Brown 16 Nov '09

16 Nov '09

Barry Smith wrote: > I do not know a way to generate unique MPI-comm-id's. Any thoughts? > Some feature of MPI I don't know? This seems very hard. One could (the first time PETSc sees a communicator) hash the hostname+pid of every process on the communicator, start a counter on this hash, and set the communicator name to be the hash concatenated with this counter. I think this is sufficient, but it's kinda nasty. Jed

1 0

Re: Should PetscObjectName be collective?
by Lisandro Dalcin 16 Nov '09

16 Nov '09

On Mon, Nov 16, 2009 at 2:03 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > I do not know a way to generate unique MPI-comm-id's. Any thoughts? Some > feature of MPI I don't know? > I think the only way is to use a (truly) global counter, but you know, implementing NXTVAL is a bit hard and will require MPI-2 features. (http://code.google.com/p/mpi4py/source/browse/trunk/demo/nxtval/) And now that I think better, a truly global counter is perhaps not so easy... How do you handle Spawn&Merge? MPI-2 made COMM_WORLD far less special that it was in MPI-1. -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594

1 0

Re: Should PetscObjectName be collective?
by Barry Smith 16 Nov '09

16 Nov '09

Jed points out a flaw in the simple minded naming scheme we have. Lisandro points out the lack of generality of Jed's fix. To insure unique names and consistent names across ranks for a parallel object we could use the following scheme: classname_MPI-comm-id_counter where MPI-comm-id is obtained somehow from the communicator (and then set as an attribute) and counter is a another attribute of that communicator that is cranked up each time it is obtained. For a given communicator after setting the MPI-comm-id no communication is needed for each new name it hands out (and the debug version of PETSc would do a communication to check that everyone's counter is the same as a sanity check). I do not know a way to generate unique MPI-comm-id's. Any thoughts? Some feature of MPI I don't know? Barry On Nov 16, 2009, at 9:48 AM, Lisandro Dalcin wrote: > On Mon, Nov 16, 2009 at 10:49 AM, Jed Brown <jed(a)59a2.org> wrote: >> Since it is not collective, the names can get out of sync, even with >> "reasonable" usage. Consider a case where rank 0 views a serial Vec >> (say with the Matlab viewer), and then a parallel Vec (e.g. to >> HDF5). The result would either be deadlock (with collective IO) or a >> file where Vec_0 contains all but rank 0's part, and Vec_1 contains >> only >> rank 0's part (and an HDF5 error once another vector is viewed >> because >> the Vec_1 dataspace cannot be created if it already exists). >> >> Is it reasonable to make PetscObjectName (and necessary dependents) >> collective where it performs this? >> >> MPI_Allreduce(MPI_IN_PLACE,&counter,1,MPI_INT,MPI_MAX,obj->comm); >> > > And what about VecView() of serial objects on two different processes? > What about parallel Vec's on subcommunicators of COMM_WORLD? > > > -- > Lisandro Dalcín > --------------- > Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) > Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) > Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) > PTLC - Güemes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594

1 0