February 2016 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Not possible to do a VecPlaceArray for veccusp
by Karl Rupp 25 Feb '16

25 Feb '16

Hi Jose, > We are trying to do some GPU developments on the SLEPc side, and we would need a way of placing the array of a VECCUSP vector, providing the GPU address. Specifically, what we want to do is have a large Vec on GPU and slice it in several smaller Vecs. > > For the GetArray/RestoreArray we have all possibilities: > - VecGetArray: gets the pointer to the buffer stored in CPU memory > - VecCUSPGetArray*: returns a CUSPARRAY object that contains some info, including the buffer allocated in GPU memory > - VecCUSPGetCUDAArray*: returns a raw pointer of the GPU buffer > > The problem comes with PlaceArray equivalents. Using VecPlaceArray we can provide a new pointer to CPU memory. We wanted to implement the equivalent thing for GPU, but we found difficulties due to Thrust. If we wanted to provide a VecCUSPPlaceCUDAArray the problem is that Thrust does not allow wrapping an exisiting GPU buffer with a CUSPARRAY object (when creating a CUSPARRAY it always allocates new memory). On the other hand, VecCUSPPlaceArray is possible to implement, but the problem is that one should provide a CUSPARRAY obtained from a VecCUSPGetArray* without modification (it is not possible to do pointer arithmetic with a CUSPARRAY). As far as I can see from browsing the documentation and the web there is indeed no such option. Ouch. > Any thoughts? I'll soon expose the CUDA backend of ViennaCL in PETSc, which will offer such functionality. Is this an option? It will take 2-3 weeks, though. Best regards, Karli

1 0

Not possible to do a VecPlaceArray for veccusp
by Jose E. Roman 25 Feb '16

25 Feb '16

We are trying to do some GPU developments on the SLEPc side, and we would need a way of placing the array of a VECCUSP vector, providing the GPU address. Specifically, what we want to do is have a large Vec on GPU and slice it in several smaller Vecs. For the GetArray/RestoreArray we have all possibilities: - VecGetArray: gets the pointer to the buffer stored in CPU memory - VecCUSPGetArray*: returns a CUSPARRAY object that contains some info, including the buffer allocated in GPU memory - VecCUSPGetCUDAArray*: returns a raw pointer of the GPU buffer The problem comes with PlaceArray equivalents. Using VecPlaceArray we can provide a new pointer to CPU memory. We wanted to implement the equivalent thing for GPU, but we found difficulties due to Thrust. If we wanted to provide a VecCUSPPlaceCUDAArray the problem is that Thrust does not allow wrapping an exisiting GPU buffer with a CUSPARRAY object (when creating a CUSPARRAY it always allocates new memory). On the other hand, VecCUSPPlaceArray is possible to implement, but the problem is that one should provide a CUSPARRAY obtained from a VecCUSPGetArray* without modification (it is not possible to do pointer arithmetic with a CUSPARRAY). Any thoughts?

1 0

Re: [petsc-dev] MatNest in petsc4py with empty submatrices
by Chris Eldred 23 Feb '16

23 Feb '16

Perfect! Thank you! -Chris On Mon, Feb 22, 2016 at 7:16 PM, Lisandro Dalcin <dalcinl(a)gmail.com> wrote: > On 22 February 2016 at 19:18, Chris Eldred <chris.eldred(a)gmail.com> wrote: >> Hey Lisandro, >> >> null = PETSc.Mat() >> >> works as well. >> > > https://bitbucket.org/petsc/petsc4py/commits/00412243acf967acc69229b2e81ff3… > > > -- > Lisandro Dalcin > ============ > Research Scientist > Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) > Extreme Computing Research Center (ECRC) > King Abdullah University of Science and Technology (KAUST) > http://ecrc.kaust.edu.sa/ > > 4700 King Abdullah University of Science and Technology > al-Khawarizmi Bldg (Bldg 1), Office # 4332 > Thuwal 23955-6900, Kingdom of Saudi Arabia > http://www.kaust.edu.sa > > Office Phone: +966 12 808-0459 -- Chris Eldred Postdoctoral Fellow, LAGA, University of Paris 13 PhD, Atmospheric Science, Colorado State University, 2015 DOE Computational Science Graduate Fellow (Alumni) B.S. Applied Computational Physics, Carnegie Mellon University, 2009 chris.eldred(a)gmail.com

1 0

Re: [petsc-dev] MatNest in petsc4py with empty submatrices
by Lisandro Dalcin 22 Feb '16

22 Feb '16

On 22 February 2016 at 19:18, Chris Eldred <chris.eldred(a)gmail.com> wrote: > Hey Lisandro, > > null = PETSc.Mat() > > works as well. > https://bitbucket.org/petsc/petsc4py/commits/00412243acf967acc69229b2e81ff3… -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459

1 0

Re: [petsc-dev] MatNest in petsc4py with empty submatrices
by Chris Eldred 22 Feb '16

22 Feb '16

Hey Lisandro, null = PETSc.Mat() works as well. -Chris On Sun, Feb 21, 2016 at 8:55 PM, Lisandro Dalcin <dalcinl(a)gmail.com> wrote: > Could you try the following? > > null = PETSc.Mat() > nest_list = [[null,null],[null,null]] > > Anyway, I like Lawrence's patch. I'll push a fix to maint > > > On 18 February 2016 at 21:27, Chris Eldred <chris.eldred(a)gmail.com> wrote: >> Hey Lawrence, >> >> I will give that a try, thanks! >> >> -Chris >> >> On Wed, Feb 17, 2016 at 3:53 PM, Lawrence Mitchell >> <lawrence.mitchell(a)imperial.ac.uk> wrote: >>> On 17/02/16 14:47, Chris Eldred wrote: >>>> Hey All, >>>> >>>> I was wondering if anyone had used MatNest with petsc4py. I am having >>>> issues trying to create a MatNest where some of the submatrices are >>>> empty. In PETSc, I can just pass NULL in the array that I give to >>>> MatCreateNest and it works fine. However, the following code >>> >>> We use it, but haven't used it with empty slots. >>> >>>> nest_list = [[None,None],[None,None]] >>>> nest_list[0][0] = some matrix >>>> nest_list[0][1] = some matrix >>>> nest_list[1][0] = some matrix >>>> A = PETSc.Mat().createNest(nest_list,comm=PETSc.COMM_WORLD) >>>> >>>> fails with: >>>> File "Mat.pyx", line 445, in petsc4py.PETSc.Mat.createNest >>>> (src/petsc4py.PETSc.c:103202) >>>> TypeError: Cannot convert NoneType to petsc4py.PETSc.Mat >>>> >>>> Is there an equivalent to NULL in petsc4py (there doesn't appear to be one)? >>>> >>>> If all of the entries in nest_list are filled, everything works fine. >>>> >>>> Any help would be appreciated! >>> >>> Maybe a patch like this to petsc4py? >>> >>> diff --git a/src/PETSc/Mat.pyx b/src/PETSc/Mat.pyx >>> index f281ba0..490c58e 100644 >>> --- a/src/PETSc/Mat.pyx >>> +++ b/src/PETSc/Mat.pyx >>> @@ -460,7 +460,10 @@ cdef class Mat(Object): >>> tmp1 = oarray_p(empty_p(nr*nc), NULL, <void**>&cmats) >>> for i from 0 <= i < mr: >>> for j from 0 <= j < mc: >>> - cmats[i*mc+j] = (<Mat?>mats[i][j]).mat >>> + if mats[i][j] is None: >>> + cmats[i*mc+j] = NULL >>> + else: >>> + cmats[i*mc+j] = (<Mat?>mats[i][j]).mat >>> if isrows is not None: >>> tmp2 = oarray_p(empty_p(nr), NULL, <void**>&cisrows) >>> for i from 0 <= i < mr: cisrows[i] = (<IS?>isrows[i]).iset >>> >>> >>> Untested. >>> >>> If you want to pass null isrows and iscols as well you'll need to do a >>> similar thing there. >>> >>> Cheers, >>> >>> Lawrence >>> >> >> >> >> -- >> Chris Eldred >> Postdoctoral Fellow, LAGA, University of Paris 13 >> PhD, Atmospheric Science, Colorado State University, 2015 >> DOE Computational Science Graduate Fellow (Alumni) >> B.S. Applied Computational Physics, Carnegie Mellon University, 2009 >> chris.eldred(a)gmail.com > > > > -- > Lisandro Dalcin > ============ > Research Scientist > Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) > Extreme Computing Research Center (ECRC) > King Abdullah University of Science and Technology (KAUST) > http://ecrc.kaust.edu.sa/ > > 4700 King Abdullah University of Science and Technology > al-Khawarizmi Bldg (Bldg 1), Office # 4332 > Thuwal 23955-6900, Kingdom of Saudi Arabia > http://www.kaust.edu.sa > > Office Phone: +966 12 808-0459 -- Chris Eldred Postdoctoral Fellow, LAGA, University of Paris 13 PhD, Atmospheric Science, Colorado State University, 2015 DOE Computational Science Graduate Fellow (Alumni) B.S. Applied Computational Physics, Carnegie Mellon University, 2009 chris.eldred(a)gmail.com

1 0

Re: [petsc-dev] MatNest in petsc4py with empty submatrices
by Lisandro Dalcin 21 Feb '16

21 Feb '16

Could you try the following? null = PETSc.Mat() nest_list = [[null,null],[null,null]] Anyway, I like Lawrence's patch. I'll push a fix to maint On 18 February 2016 at 21:27, Chris Eldred <chris.eldred(a)gmail.com> wrote: > Hey Lawrence, > > I will give that a try, thanks! > > -Chris > > On Wed, Feb 17, 2016 at 3:53 PM, Lawrence Mitchell > <lawrence.mitchell(a)imperial.ac.uk> wrote: >> On 17/02/16 14:47, Chris Eldred wrote: >>> Hey All, >>> >>> I was wondering if anyone had used MatNest with petsc4py. I am having >>> issues trying to create a MatNest where some of the submatrices are >>> empty. In PETSc, I can just pass NULL in the array that I give to >>> MatCreateNest and it works fine. However, the following code >> >> We use it, but haven't used it with empty slots. >> >>> nest_list = [[None,None],[None,None]] >>> nest_list[0][0] = some matrix >>> nest_list[0][1] = some matrix >>> nest_list[1][0] = some matrix >>> A = PETSc.Mat().createNest(nest_list,comm=PETSc.COMM_WORLD) >>> >>> fails with: >>> File "Mat.pyx", line 445, in petsc4py.PETSc.Mat.createNest >>> (src/petsc4py.PETSc.c:103202) >>> TypeError: Cannot convert NoneType to petsc4py.PETSc.Mat >>> >>> Is there an equivalent to NULL in petsc4py (there doesn't appear to be one)? >>> >>> If all of the entries in nest_list are filled, everything works fine. >>> >>> Any help would be appreciated! >> >> Maybe a patch like this to petsc4py? >> >> diff --git a/src/PETSc/Mat.pyx b/src/PETSc/Mat.pyx >> index f281ba0..490c58e 100644 >> --- a/src/PETSc/Mat.pyx >> +++ b/src/PETSc/Mat.pyx >> @@ -460,7 +460,10 @@ cdef class Mat(Object): >> tmp1 = oarray_p(empty_p(nr*nc), NULL, <void**>&cmats) >> for i from 0 <= i < mr: >> for j from 0 <= j < mc: >> - cmats[i*mc+j] = (<Mat?>mats[i][j]).mat >> + if mats[i][j] is None: >> + cmats[i*mc+j] = NULL >> + else: >> + cmats[i*mc+j] = (<Mat?>mats[i][j]).mat >> if isrows is not None: >> tmp2 = oarray_p(empty_p(nr), NULL, <void**>&cisrows) >> for i from 0 <= i < mr: cisrows[i] = (<IS?>isrows[i]).iset >> >> >> Untested. >> >> If you want to pass null isrows and iscols as well you'll need to do a >> similar thing there. >> >> Cheers, >> >> Lawrence >> > > > > -- > Chris Eldred > Postdoctoral Fellow, LAGA, University of Paris 13 > PhD, Atmospheric Science, Colorado State University, 2015 > DOE Computational Science Graduate Fellow (Alumni) > B.S. Applied Computational Physics, Carnegie Mellon University, 2009 > chris.eldred(a)gmail.com -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459

1 0

Re: [petsc-dev] MatNest in petsc4py with empty submatrices
by Chris Eldred 18 Feb '16

18 Feb '16

Hey Lawrence, I will give that a try, thanks! -Chris On Wed, Feb 17, 2016 at 3:53 PM, Lawrence Mitchell <lawrence.mitchell(a)imperial.ac.uk> wrote: > On 17/02/16 14:47, Chris Eldred wrote: >> Hey All, >> >> I was wondering if anyone had used MatNest with petsc4py. I am having >> issues trying to create a MatNest where some of the submatrices are >> empty. In PETSc, I can just pass NULL in the array that I give to >> MatCreateNest and it works fine. However, the following code > > We use it, but haven't used it with empty slots. > >> nest_list = [[None,None],[None,None]] >> nest_list[0][0] = some matrix >> nest_list[0][1] = some matrix >> nest_list[1][0] = some matrix >> A = PETSc.Mat().createNest(nest_list,comm=PETSc.COMM_WORLD) >> >> fails with: >> File "Mat.pyx", line 445, in petsc4py.PETSc.Mat.createNest >> (src/petsc4py.PETSc.c:103202) >> TypeError: Cannot convert NoneType to petsc4py.PETSc.Mat >> >> Is there an equivalent to NULL in petsc4py (there doesn't appear to be one)? >> >> If all of the entries in nest_list are filled, everything works fine. >> >> Any help would be appreciated! > > Maybe a patch like this to petsc4py? > > diff --git a/src/PETSc/Mat.pyx b/src/PETSc/Mat.pyx > index f281ba0..490c58e 100644 > --- a/src/PETSc/Mat.pyx > +++ b/src/PETSc/Mat.pyx > @@ -460,7 +460,10 @@ cdef class Mat(Object): > tmp1 = oarray_p(empty_p(nr*nc), NULL, <void**>&cmats) > for i from 0 <= i < mr: > for j from 0 <= j < mc: > - cmats[i*mc+j] = (<Mat?>mats[i][j]).mat > + if mats[i][j] is None: > + cmats[i*mc+j] = NULL > + else: > + cmats[i*mc+j] = (<Mat?>mats[i][j]).mat > if isrows is not None: > tmp2 = oarray_p(empty_p(nr), NULL, <void**>&cisrows) > for i from 0 <= i < mr: cisrows[i] = (<IS?>isrows[i]).iset > > > Untested. > > If you want to pass null isrows and iscols as well you'll need to do a > similar thing there. > > Cheers, > > Lawrence > -- Chris Eldred Postdoctoral Fellow, LAGA, University of Paris 13 PhD, Atmospheric Science, Colorado State University, 2015 DOE Computational Science Graduate Fellow (Alumni) B.S. Applied Computational Physics, Carnegie Mellon University, 2009 chris.eldred(a)gmail.com

1 0

Re: [petsc-dev] instead of generating a bad fortran stub why don't we have sowing barf on functions with character strings
by Barry Smith 18 Feb '16

18 Feb '16

> On Feb 17, 2016, at 11:54 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > Barry Smith <bsmith(a)mcs.anl.gov> writes: >> It is absurd that we get into this situation with a bad fortran stub due to strings. Sowing should just barf on them not generate a bad stub and then waste everyone's time! > > I'd say we'd be better off making sowing handle character strings correctly. Don't let perfection stop us from at least doing a stop gap measure that saves us time and frustration.

1 0

Re: [petsc-dev] instead of generating a bad fortran stub why don't we have sowing barf on functions with character strings
by Jed Brown 17 Feb '16

17 Feb '16

Barry Smith <bsmith(a)mcs.anl.gov> writes: > It is absurd that we get into this situation with a bad fortran stub due to strings. Sowing should just barf on them not generate a bad stub and then waste everyone's time! I'd say we'd be better off making sowing handle character strings correctly.

1 0

instead of generating a bad fortran stub why don't we have sowing barf on functions with character strings
by Barry Smith 17 Feb '16

17 Feb '16

Everyone, It is absurd that we get into this situation with a bad fortran stub due to strings. Sowing should just barf on them not generate a bad stub and then waste everyone's time! Barry

1 0