November 2009 - petsc-dev - mailman3.cels.anl.gov

Re: Should PetscObjectName be collective?
by Jed Brown 16 Nov '09

16 Nov '09

Lisandro Dalcin wrote: > And what about VecView() of serial objects on two different processes? > What about parallel Vec's on subcommunicators of COMM_WORLD? Of course there is nothing special about COMM_WORLD, and there is no way to get consistent naming of serial objects. This would only ensure that all processes in a communicator have the same name for parallel objects and that no process on that communicator uses the name for a different object. There is no guarantee that this name is never used for an object on a different communicator. Jed

1 0

Re: Should PetscObjectName be collective?
by Lisandro Dalcin 16 Nov '09

16 Nov '09

On Mon, Nov 16, 2009 at 10:49 AM, Jed Brown <jed(a)59a2.org> wrote: > Since it is not collective, the names can get out of sync, even with > "reasonable" usage. Consider a case where rank 0 views a serial Vec > (say with the Matlab viewer), and then a parallel Vec (e.g. to > HDF5). The result would either be deadlock (with collective IO) or a > file where Vec_0 contains all but rank 0's part, and Vec_1 contains only > rank 0's part (and an HDF5 error once another vector is viewed because > the Vec_1 dataspace cannot be created if it already exists). > > Is it reasonable to make PetscObjectName (and necessary dependents) > collective where it performs this? > > MPI_Allreduce(MPI_IN_PLACE,&counter,1,MPI_INT,MPI_MAX,obj->comm); > And what about VecView() of serial objects on two different processes? What about parallel Vec's on subcommunicators of COMM_WORLD? -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594

1 0

Should PetscObjectName be collective?
by Jed Brown 16 Nov '09

16 Nov '09

Since it is not collective, the names can get out of sync, even with "reasonable" usage. Consider a case where rank 0 views a serial Vec (say with the Matlab viewer), and then a parallel Vec (e.g. to HDF5). The result would either be deadlock (with collective IO) or a file where Vec_0 contains all but rank 0's part, and Vec_1 contains only rank 0's part (and an HDF5 error once another vector is viewed because the Vec_1 dataspace cannot be created if it already exists). Is it reasonable to make PetscObjectName (and necessary dependents) collective where it performs this? MPI_Allreduce(MPI_IN_PLACE,&counter,1,MPI_INT,MPI_MAX,obj->comm); Jed

1 0

Re: -malign-double
by Jed Brown 16 Nov '09

16 Nov '09

Barry Smith wrote: > > Agreed this would be a good option to have. The question is how to do > it without having a morass of nasty nested if-defs. Note that portably > getting alignment out of malloc() alone is already ugly and not as > simple a code as I would like. > > To simplify things we could always require 16 byte alignment > everywhere? But is that a desirable? I doubt it's harmful, I don't think I understand where the nested ifdefs come in. The required alignment only needs to be stated in one place, I have something like the following in one of my projects. /* current */ #define PetscMalloc3(m1,t1,r1,m2,t2,r2,m3,t3,r3) \ (PetscMalloc((m1)*sizeof(t1)+(m2)*sizeof(t2)+(m3)*sizeof(t3),r1) \ || (*(r2) = (t2*)(*(r1)+m1), \ *(r3) = (t3*)(*(r2)+m2),0)) /* aligned */ #define PetscMalloc3(m1,t1,r1,m2,t2,r2,m3,t3,r3) \ (PetscMalloc((m1)*sizeof(t1)+(m2)*sizeof(t2)+(m3)*sizeof(t3)+2*(PETSC_MEMALIGN-1),r1) \ || (*(r2) = (t2*)PETSC_ALIGN(*(r1)+m1), \ *(r3) = (t3*)PETSC_ALIGN(*(r2)+m2),0)) #define PETSC_ALIGN(p) PetscNextAligned((uintptr_t)(p),PETSC_MEMALIGN-1) static inline void *PetscNextAligned(uintptr_t base,uintptr_t mask) {return (void*)((base + mask) & ~mask);} Note that this is a no-op if PETSC_MEMALIGN=1 and thus compiles to exactly what we have now. Jed

1 0

Re: -malign-double
by Barry Smith 15 Nov '09

15 Nov '09

Agreed this would be a good option to have. The question is how to do it without having a morass of nasty nested if-defs. Note that portably getting alignment out of malloc() alone is already ugly and not as simple a code as I would like. To simplify things we could always require 16 byte alignment everywhere? But is that a desirable? Barry On Nov 14, 2009, at 12:43 PM, Jed Brown wrote: > Barry Smith wrote: >> >> On Nov 14, 2009, at 11:28 AM, Jed Brown wrote: >> >>> Barry Smith wrote: >>>> >>>> >>>> Good point, I have removed it. >>>> >>>> I put it in because I wanted an easy way to test that PETSc double >>>> arrays are always 8 byte aligned (and the unaligned struct values >>>> were >>>> giving me lots of false alarms). >>> >>> A related issue is PetscMallocN which (in optimized mode) gives >>> unaligned arrays even if malloc always returns aligned memory. >>> Consider >>> >>> PetscMalloc2(3,PetscInt,&ai,3,PetscScalar,&a); >> >> This call is illegal. You are required to pass the longest aligned >> arrays first leading the shortest. Hence The PetscScalar should be >> before the PetscInt. >> >> The debug mode should check that this requirement is satisfied; it >> currently does not check anything. > > If nothing ever checks and the docs don't specify, it's valid. > >> In the real world when any of these beasts can be 32 bit or 64 bit >> one cannot always put them in the right order; but for now they >> should >> be ordered, PetscScalar, pointer, PetscInt > > What advantage does this have over just aligning all the pointers > (even > to 16-byte to enable movapd and prevent many loads across a cacheline > split, see http://x264dev.multimedia.cx/?p=8)? The arithmetic and few > wasted bytes is trivial compared to the cost of the allocation, so I > don't see a reason not to. > > Jed >

1 0

Re: -malign-double
by Satish Balay 15 Nov '09

15 Nov '09

On Sat, 14 Nov 2009, Lisandro Dalcin wrote: > On Fri, Nov 13, 2009 at 11:09 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > The IBM BlueGene doesn't like 4 byte aligned doubles and sends a bus error > > after a certain number are generated, Thus I was motivated to make PETSc a > > bit clearer on that. > > > > But then the compiler in the BlueGene does not -malign-double by > default? That would be weird and annoying... Malloc() returns 16 byte aligned types. I'm not sure what happens in structs.. [I'm guessing datatypes do get aligned properly] The issue came up with us malloc()ing a big array and spliting it up between (double* array) and (double array). This [manual] split didn't happen on a '8'byte boundary. Satish

1 0

Re: -malign-double
by Barry Smith 14 Nov '09

14 Nov '09

On Nov 14, 2009, at 4:01 PM, Lisandro Dalcin wrote: > On Fri, Nov 13, 2009 at 11:09 PM, Barry Smith <bsmith(a)mcs.anl.gov> > wrote: >> >> >> Good point, I have removed it. >> >> I put it in because I wanted an easy way to test that PETSc >> double arrays >> are always 8 byte aligned (and the unaligned struct values were >> giving me >> lots of false alarms). >> >> I guess we can test for double align on x86_64 systems. >> >> Barry >> >> The IBM BlueGene doesn't like 4 byte aligned doubles and sends a >> bus error >> after a certain number are generated, Thus I was motivated to make >> PETSc a >> bit clearer on that. >> > > But then the compiler in the BlueGene does not -malign-double by > default? That would be weird and annoying... BlueGene has two compilers, IBM's XLc and gcc. I do not know if either of them are rigged to 8 byte align by default. Barry > >> >> On Nov 12, 2009, at 9:29 AM, Lisandro Dalcin wrote: >> >>> On Wed, Nov 11, 2009 at 11:02 PM, Barry Smith <bsmith(a)mcs.anl.gov> >>> wrote: >>>> >>>> Satish, >>>> >>>> gcc supports the option -malign-double on some systems. Please >>>> add it >>>> to >>>> the flags that configure checks for c and c++ and have it added >>>> if it >>>> exists. >>>> >>> >>> Definitely -1. Why are you asking for this? On x86_64 this is the >>> default. In i386, this flag is likely going to break ABI's. >>> >>> >>> -- >>> Lisandro Dalcín >>> --------------- >>> Centro Internacional de Métodos Computacionales en Ingeniería >>> (CIMEC) >>> Instituto de Desarrollo Tecnológico para la Industria Química >>> (INTEC) >>> Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) >>> PTLC - Güemes 3450, (3000) Santa Fe, Argentina >>> Tel/Fax: +54-(0)342-451.1594 >> >> > > > > -- > Lisandro Dalcín > --------------- > Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) > Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) > Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) > PTLC - Güemes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594

1 0

Re: -malign-double
by Lisandro Dalcin 14 Nov '09

14 Nov '09

On Sat, Nov 14, 2009 at 3:43 PM, Jed Brown <jed(a)59a2.org> wrote: > > What advantage does this have over just aligning all the pointers (even > to 16-byte to enable movapd and prevent many loads across a cacheline > split, see http://x264dev.multimedia.cx/?p=8)? The arithmetic and few > wasted bytes is trivial compared to the cost of the allocation, so I > don't see a reason not to. > I think Jed has a VERY good point here... -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594

1 0

Re: -malign-double
by Lisandro Dalcin 14 Nov '09

14 Nov '09

On Fri, Nov 13, 2009 at 11:09 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > Good point, I have removed it. > > I put it in because I wanted an easy way to test that PETSc double arrays > are always 8 byte aligned (and the unaligned struct values were giving me > lots of false alarms). > > I guess we can test for double align on x86_64 systems. > > Barry > > The IBM BlueGene doesn't like 4 byte aligned doubles and sends a bus error > after a certain number are generated, Thus I was motivated to make PETSc a > bit clearer on that. > But then the compiler in the BlueGene does not -malign-double by default? That would be weird and annoying... > > On Nov 12, 2009, at 9:29 AM, Lisandro Dalcin wrote: > >> On Wed, Nov 11, 2009 at 11:02 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >>> >>> Satish, >>> >>> gcc supports the option -malign-double on some systems. Please add it >>> to >>> the flags that configure checks for c and c++ and have it added if it >>> exists. >>> >> >> Definitely -1. Why are you asking for this? On x86_64 this is the >> default. In i386, this flag is likely going to break ABI's. >> >> >> -- >> Lisandro Dalcín >> --------------- >> Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) >> Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) >> Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) >> PTLC - Güemes 3450, (3000) Santa Fe, Argentina >> Tel/Fax: +54-(0)342-451.1594 > > -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594

1 0

Re: -malign-double
by Jed Brown 14 Nov '09

14 Nov '09

Barry Smith wrote: > > On Nov 14, 2009, at 11:28 AM, Jed Brown wrote: > >> Barry Smith wrote: >>> >>> >>> Good point, I have removed it. >>> >>> I put it in because I wanted an easy way to test that PETSc double >>> arrays are always 8 byte aligned (and the unaligned struct values were >>> giving me lots of false alarms). >> >> A related issue is PetscMallocN which (in optimized mode) gives >> unaligned arrays even if malloc always returns aligned memory. Consider >> >> PetscMalloc2(3,PetscInt,&ai,3,PetscScalar,&a); > > This call is illegal. You are required to pass the longest aligned > arrays first leading the shortest. Hence The PetscScalar should be > before the PetscInt. > > The debug mode should check that this requirement is satisfied; it > currently does not check anything. If nothing ever checks and the docs don't specify, it's valid. > In the real world when any of these beasts can be 32 bit or 64 bit > one cannot always put them in the right order; but for now they should > be ordered, PetscScalar, pointer, PetscInt What advantage does this have over just aligning all the pointers (even to 16-byte to enable movapd and prevent many loads across a cacheline split, see http://x264dev.multimedia.cx/?p=8)? The arithmetic and few wasted bytes is trivial compared to the cost of the allocation, so I don't see a reason not to. Jed

1 0