September 2008 - petsc-dev - mailman3.cels.anl.gov

Re: PETSc on GPUs
by Harald Pfeiffer 25 Sep '08

25 Sep '08

Hi, I ran across another cool way to speed up Petsc on GPUs: GPUs have way faster single precision FLOPS counts than double precision. For iterative solvers, the update-computation can easily be single precision. But that's where all the work is. For example, one could run Newton-Raphson such that: - the non-linear residual is computed in double precision. - the inversion of the Jacobian is done in single precision. One gets the single precision FLOPS count while keeping double-precision accuracy in the end result. Harald For iterative solvers, run the update-computation in *single* precision. On 09/18/2008 02:09 PM, Barry Smith wrote: > > In implementing Vec in the GPU a VecScatter would take the entries > directly from GPU copy of the > vector and put them appropriately into the GPU copy of the other vector, > thus it would be completely > transparent from the MatMult_MPIAIJ() code. In other words, if you have > to modify the MatMult_MPIAIJ > IN ANYWAY for the GPU then there is something wrong. > > Barry > > Note the VecScatter would need to marshal the values up from the GPU > into the regular memory, > do the appropriate message passing to the resulting processes and then > marshal the values back > down to the destination GPU. Since the current VecScatter handles all > the message passing I think > the GPU VecScatter() could be written to 1) move the needed GPU values > up to regular memory > (inside work MPI Vecs) then 2) use a regular VecScatter on those work > vectors then 3) move the results > down from the regular memory to the GPU. > > In other words PETSc has been designed since 1991 to be able to reuse > almost all the infrastructure > and only need new low level kernels written to run on GPUs. :-) > > Fun stuff, sure beats working :-) > > > > On Sep 18, 2008, at 3:18 PM, Richard Tran Mills wrote: > >> Ahmed, >> >> If you take a look in >> >> src/mat/impls/aij/mpi/ >> >> at mpiaij.h and mpiaij.c, you can see that an MPIAIJ matrix is stored >> locally as two "sequential" matrics, A and B. A contains the >> "diagonal" portion of the matrix, and B contains the "off-diagonal" >> portion. See page 59 of the PETSc User manual for a quick >> illustration of how this partitioning works. Because vectors are >> partitioned row-wise among processors in the same manner as the >> matrix, each process already has the vector entries required to form >> the portion of the matrix-vector product ("mat-vec") involving the >> diagonal portion 'A' can be done with data that reside locally. The >> off-diagonal portion 'B', however, requires gathering vector entries >> that reside off-processor. If you look in mpiaij.c at the >> MatMult_MPIAIJ, you will see the following lines: >> >> ierr = >> VecScatterBegin(a->Mvctx,xx,a->lvec,INSERT_VALUES,SCATTER_FORWARD);CHKERRQ(ierr); >> >> ierr = (*a->A->ops->mult)(a->A,xx,yy);CHKERRQ(ierr); >> ierr = >> VecScatterEnd(a->Mvctx,xx,a->lvec,INSERT_VALUES,SCATTER_FORWARD);CHKERRQ(ierr); >> >> ierr = (*a->B->ops->multadd)(a->B,a->lvec,yy,yy);CHKERRQ(ierr); >> >> The mat-vec involving a->A can be completed without any VecScatter, >> but the one involving a->B needs the VecScatter to complete before it >> can be done. (Note that the VecScatterBegin() occurs before the >> mat-vec for a->A so that there might be some overlap of communication >> and computation). >> >> Hopefully this helps elucidate what Matt meant, and didn't just >> confuse you. >> >> --Richard >> >> Ahmed El Zein wrote: >>> On Thu, 2008-09-18 at 10:20 -0500, Matthew Knepley wrote: >>>>> A question that I had regarding the PETSc code when I was thinking >>>> about >>>>> this was: >>>>> You have the SeqAIJ matrix type and the the MPIAIJ type built around >>>> it >>>>> (or that is what I understand from the code). So basically you >>>> implement >>>>> the SeqAIJ type for the GPU and you get the MPI type for free? >>>> Yes, that is true. However, note that in the MPI step, you will need a >>>> gather >>>> operation to get the second matrix multiply to work. >>> Matt, >>> Could you explain a bit more? >>> Ahmed >> >> -- >> Richard Tran Mills, Ph.D. | E-mail: rmills(a)climate.ornl.gov >> Computational Scientist | Phone: (865) 241-3198 >> Computational Earth Sciences Group | Fax: (865) 574-0405 >> Oak Ridge National Laboratory | http://climate.ornl.gov/~rmills >> -- Harald P. Pfeiffer harald(a)tapir.caltech.edu Theoretical Astrophysics Phone (626) 395-8413 Caltech 130-33 Fax (626) 796-5675 Pasadena, CA 91125, USA

1 0

RE: PetscTruth bool?
by Chetan Jhurani 24 Sep '08

24 Sep '08

> -----Original Message----- > From: owner-petsc-dev(a)mcs.anl.gov On Behalf Of Boyce Griffith > Sent: Wednesday, September 24, 2008 12:24 PM > To: petsc-dev(a)mcs.anl.gov > Subject: Re: PetscTruth bool? > > Hi, Barry et al. -- > > I may be mistaken, but I believe that sizeof(bool) is almost always one > bit in C++. This results in various problems, including the need to > have a specialized implementation of std::vector<bool> in the > STL. If I am not mistaken, the reason std::vector is specialized for bool is to represent each boolean entry by 1 bit instead of 1 byte. The vector<bool> specialization is not needed because sizeof(bool) happens to be 1. For example, vector<char> is not specialized even if sizeof(char) is 1. vector<bool> was made different only to save 7 bits per entry by using only 1 bit to check for the boolean. Sutter mentions that "sizeof(bool) is implementation-defined, but it must be at least 1." From http://www.gotw.ca/gotw/050.htm . > See, e.g., > > http://www.informit.com/guides/content.aspx?g=cplusplus&seqNum=98 BTW, the code written in that article does not compile on the compilers of that era (MSVC 2003). I think gcc would have behaved the same way since it is usually stricter than the MS compiler. I am not sure what compiler the author used to claim that the code compiles and crashes, but the crash could be because the vector was empty and he called the front() method. Chetan

1 0

Re: PetscTruth bool?
by Boyce Griffith 24 Sep '08

24 Sep '08

Hi, Barry et al. -- I may be mistaken, but I believe that sizeof(bool) is almost always one bit in C++. This results in various problems, including the need to have a specialized implementation of std::vector<bool> in the STL. See, e.g., http://www.informit.com/guides/content.aspx?g=cplusplus&seqNum=98 -- Boyce Barry Smith wrote: > > Interesting, they don't mind totally screwing up memory alignments in > order to save some > space on a bool. (I mean, come on, who makes an array of a million bools > anyways?) > > From this, if we did switch to bool it would totally screw up our > Fortran interface (Fortran does > use integer for logical). > > So forget what I suggested, keep putting the ugly (PetscTruth) casts > in argument lists > all over PETSc because the C++ compiler stubbornly refuses to auto-cast > a bool to an enum > and we'll leave it the way it is now. > > Barry > > On Sep 24, 2008, at 11:27 AM, Lisandro Dalcin wrote: > >> Barry, see this >> >> $ cat testbool.c >> #include <stdio.h> >> #include <stdbool.h> >> int main(){ >> printf("sizeof(bool): %d\n", sizeof(bool)); >> printf("sizeof(int): %d\n", sizeof(int)); >> } >> $ gcc testbool.c >> $ a.out >> sizeof(bool): 1 >> sizeof(int): 4 >> >> And now this: >> >> $ cat testbool.cpp >> #include <iostream> >> int main(){ >> std::cout << "sizeof(bool): " << sizeof(bool) << std::endl; >> std::cout << "sizeof(int): " << sizeof(int) << std::endl; >> } >> $ g++ testbool.cpp >> $ a.out >> sizeof(bool): 1 >> sizeof(int): 4 >> >> Unless I'm doing something wrong, it seems that in my system (32bits >> box, FedoraCore 6) 'bool' do not have the size of an 'int' in both C >> and C++. >> >> >> On Wed, Sep 24, 2008 at 12:59 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >>> >>> On Sep 24, 2008, at 9:30 AM, Matthew Knepley wrote: >>> >>>> On Wed, Sep 24, 2008 at 8:51 AM, Barry Smith <bsmith(a)mcs.anl.gov> >>>> wrote: >>>>> >>>>> Since C++ has bool and C99 has bool, should PetscTruth be changed >>>>> to be >>>>> defined to be bool? Instead of int. >>>> >>>> I see the motivation, but we would have to recode any MPI usages of >>>> PetscTruth >>>> since we could not guarantee the size of bool. >>> >>> Matt, >>> >>> YOU, of all people, being pedantic :-) >>> >>> Show me a single system where a bool is not an int size. >>> >>> Actually, some Fortran compilers use a -1 as true (not a 1!), this is >>> already >>> painful and could be tricky. >>> >>> >>> Barry >>> >>> >>>> >>>> >>>> Matt >>>> >>>>> Barry >>>>> >>>>> Should we go further and simply remove PetscTruth and use bool >>>>> directly >>>>> in >>>>> its place? (logical in Fortran). >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which >>>> their experiments lead. >>>> -- Norbert Wiener >>>> >>> >>> >> >> >> >> -- >> Lisandro Dalcï¿½n >> --------------- >> Centro Internacional de Mï¿½todos Computacionales en Ingenierï¿½a (CIMEC) >> Instituto de Desarrollo Tecnolï¿½gico para la Industria Quï¿½mica (INTEC) >> Consejo Nacional de Investigaciones Cientï¿½ficas y Tï¿½cnicas (CONICET) >> PTLC - Gï¿½emes 3450, (3000) Santa Fe, Argentina >> Tel/Fax: +54-(0)342-451.1594 >> >

1 0

Re: PetscTruth bool?
by Barry Smith 24 Sep '08

24 Sep '08

Interesting, they don't mind totally screwing up memory alignments in order to save some space on a bool. (I mean, come on, who makes an array of a million bools anyways?) From this, if we did switch to bool it would totally screw up our Fortran interface (Fortran does use integer for logical). So forget what I suggested, keep putting the ugly (PetscTruth) casts in argument lists all over PETSc because the C++ compiler stubbornly refuses to auto- cast a bool to an enum and we'll leave it the way it is now. Barry On Sep 24, 2008, at 11:27 AM, Lisandro Dalcin wrote: > Barry, see this > > $ cat testbool.c > #include <stdio.h> > #include <stdbool.h> > int main(){ > printf("sizeof(bool): %d\n", sizeof(bool)); > printf("sizeof(int): %d\n", sizeof(int)); > } > $ gcc testbool.c > $ a.out > sizeof(bool): 1 > sizeof(int): 4 > > And now this: > > $ cat testbool.cpp > #include <iostream> > int main(){ > std::cout << "sizeof(bool): " << sizeof(bool) << std::endl; > std::cout << "sizeof(int): " << sizeof(int) << std::endl; > } > $ g++ testbool.cpp > $ a.out > sizeof(bool): 1 > sizeof(int): 4 > > Unless I'm doing something wrong, it seems that in my system (32bits > box, FedoraCore 6) 'bool' do not have the size of an 'int' in both C > and C++. > > > On Wed, Sep 24, 2008 at 12:59 PM, Barry Smith <bsmith(a)mcs.anl.gov> > wrote: >> >> On Sep 24, 2008, at 9:30 AM, Matthew Knepley wrote: >> >>> On Wed, Sep 24, 2008 at 8:51 AM, Barry Smith <bsmith(a)mcs.anl.gov> >>> wrote: >>>> >>>> Since C++ has bool and C99 has bool, should PetscTruth be changed >>>> to be >>>> defined to be bool? Instead of int. >>> >>> I see the motivation, but we would have to recode any MPI usages of >>> PetscTruth >>> since we could not guarantee the size of bool. >> >> Matt, >> >> YOU, of all people, being pedantic :-) >> >> Show me a single system where a bool is not an int size. >> >> Actually, some Fortran compilers use a -1 as true (not a 1!), this >> is >> already >> painful and could be tricky. >> >> >> Barry >> >> >>> >>> >>> Matt >>> >>>> Barry >>>> >>>> Should we go further and simply remove PetscTruth and use bool >>>> directly >>>> in >>>> its place? (logical in Fortran). >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which >>> their experiments lead. >>> -- Norbert Wiener >>> >> >> > > > > -- > Lisandro Dalcï¿½n > --------------- > Centro Internacional de Mï¿½todos Computacionales en Ingenierï¿½a (CIMEC) > Instituto de Desarrollo Tecnolï¿½gico para la Industria Quï¿½mica (INTEC) > Consejo Nacional de Investigaciones Cientï¿½ficas y Tï¿½cnicas (CONICET) > PTLC - Gï¿½emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 >

1 0

Re: PetscTruth bool?
by Lisandro Dalcin 24 Sep '08

24 Sep '08

Barry, see this $ cat testbool.c #include <stdio.h> #include <stdbool.h> int main(){ printf("sizeof(bool): %d\n", sizeof(bool)); printf("sizeof(int): %d\n", sizeof(int)); } $ gcc testbool.c $ a.out sizeof(bool): 1 sizeof(int): 4 And now this: $ cat testbool.cpp #include <iostream> int main(){ std::cout << "sizeof(bool): " << sizeof(bool) << std::endl; std::cout << "sizeof(int): " << sizeof(int) << std::endl; } $ g++ testbool.cpp $ a.out sizeof(bool): 1 sizeof(int): 4 Unless I'm doing something wrong, it seems that in my system (32bits box, FedoraCore 6) 'bool' do not have the size of an 'int' in both C and C++. On Wed, Sep 24, 2008 at 12:59 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > On Sep 24, 2008, at 9:30 AM, Matthew Knepley wrote: > >> On Wed, Sep 24, 2008 at 8:51 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >>> >>> Since C++ has bool and C99 has bool, should PetscTruth be changed to be >>> defined to be bool? Instead of int. >> >> I see the motivation, but we would have to recode any MPI usages of >> PetscTruth >> since we could not guarantee the size of bool. > > Matt, > > YOU, of all people, being pedantic :-) > > Show me a single system where a bool is not an int size. > > Actually, some Fortran compilers use a -1 as true (not a 1!), this is > already > painful and could be tricky. > > > Barry > > >> >> >> Matt >> >>> Barry >>> >>> Should we go further and simply remove PetscTruth and use bool directly >>> in >>> its place? (logical in Fortran). >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which >> their experiments lead. >> -- Norbert Wiener >> > > -- Lisandro Dalcï¿½n --------------- Centro Internacional de Mï¿½todos Computacionales en Ingenierï¿½a (CIMEC) Instituto de Desarrollo Tecnolï¿½gico para la Industria Quï¿½mica (INTEC) Consejo Nacional de Investigaciones Cientï¿½ficas y Tï¿½cnicas (CONICET) PTLC - Gï¿½emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594

1 0

Re: PetscTruth bool?
by Barry Smith 24 Sep '08

24 Sep '08

On Sep 24, 2008, at 9:30 AM, Matthew Knepley wrote: > On Wed, Sep 24, 2008 at 8:51 AM, Barry Smith <bsmith(a)mcs.anl.gov> > wrote: >> >> Since C++ has bool and C99 has bool, should PetscTruth be changed >> to be >> defined to be bool? Instead of int. > > I see the motivation, but we would have to recode any MPI usages of > PetscTruth > since we could not guarantee the size of bool. Matt, YOU, of all people, being pedantic :-) Show me a single system where a bool is not an int size. Actually, some Fortran compilers use a -1 as true (not a 1!), this is already painful and could be tricky. Barry > > > Matt > >> Barry >> >> Should we go further and simply remove PetscTruth and use bool >> directly in >> its place? (logical in Fortran). > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener >

1 0

Re: PetscTruth bool?
by Barry Smith 24 Sep '08

24 Sep '08

On Sep 24, 2008, at 9:27 AM, Lisandro Dalcin wrote: > On Wed, Sep 24, 2008 at 10:51 AM, Barry Smith <bsmith(a)mcs.anl.gov> > wrote: >> >> Since C++ has bool and C99 has bool, should PetscTruth be changed >> to be >> defined to be bool? Instead of int. > > You mean instead of an enumeration, C++ is a bit more strict than C in > this point. At this point I'm +0 on this, but perhaps you could > elaborate a bit more the rationale for the change? All the ugly (PetscTruth) cast that are in the code and needed by C ++. If we can use a language feature 100% and it does not impose any constraints that we would not have otherwise then a good general policy would be to use it. Barry > > > >> >> Should we go further and simply remove PetscTruth and use bool >> directly in >> its place? (logical in Fortran). >> > > Well, not sure again. In the same spirit, we could introduce > 'PetscIndex' for integers that works as indices in vectors and > matrices, and plain 'int' for other things, like an iteration counter. > > > In short, such changes are more or less good, of course users will cry > porting their codes, and we have to update a lot of code. But anyway > PETSc developers never hesitate to do that if really needed. The > question is if such changes would really pay in the long term. > > > > -- > Lisandro Dalcï¿½n > --------------- > Centro Internacional de Mï¿½todos Computacionales en Ingenierï¿½a (CIMEC) > Instituto de Desarrollo Tecnolï¿½gico para la Industria Quï¿½mica (INTEC) > Consejo Nacional de Investigaciones Cientï¿½ficas y Tï¿½cnicas (CONICET) > PTLC - Gï¿½emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 >

1 0

Re: PetscTruth bool?
by Matthew Knepley 24 Sep '08

24 Sep '08

On Wed, Sep 24, 2008 at 8:51 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Since C++ has bool and C99 has bool, should PetscTruth be changed to be > defined to be bool? Instead of int. I see the motivation, but we would have to recode any MPI usages of PetscTruth since we could not guarantee the size of bool. Matt > Barry > > Should we go further and simply remove PetscTruth and use bool directly in > its place? (logical in Fortran). -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: PetscTruth bool?
by Lisandro Dalcin 24 Sep '08

24 Sep '08

On Wed, Sep 24, 2008 at 10:51 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Since C++ has bool and C99 has bool, should PetscTruth be changed to be > defined to be bool? Instead of int. You mean instead of an enumeration, C++ is a bit more strict than C in this point. At this point I'm +0 on this, but perhaps you could elaborate a bit more the rationale for the change? > > Should we go further and simply remove PetscTruth and use bool directly in > its place? (logical in Fortran). > Well, not sure again. In the same spirit, we could introduce 'PetscIndex' for integers that works as indices in vectors and matrices, and plain 'int' for other things, like an iteration counter. In short, such changes are more or less good, of course users will cry porting their codes, and we have to update a lot of code. But anyway PETSc developers never hesitate to do that if really needed. The question is if such changes would really pay in the long term. -- Lisandro Dalcï¿½n --------------- Centro Internacional de Mï¿½todos Computacionales en Ingenierï¿½a (CIMEC) Instituto de Desarrollo Tecnolï¿½gico para la Industria Quï¿½mica (INTEC) Consejo Nacional de Investigaciones Cientï¿½ficas y Tï¿½cnicas (CONICET) PTLC - Gï¿½emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594

1 0

PetscTruth bool?
by Barry Smith 24 Sep '08

24 Sep '08

Since C++ has bool and C99 has bool, should PetscTruth be changed to be defined to be bool? Instead of int. Barry Should we go further and simply remove PetscTruth and use bool directly in its place? (logical in Fortran).

1 0