October 2014 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Do you have any suggestion in order to reduce my memory footprint?
by Jed Brown 19 Oct '14

19 Oct '14

"Alletto, John M" <john.m.alletto(a)lmco.com> writes: > All, > > I have a 64 bit linux machine with 256 GByte of memory and 30 processors. > I am solving Laplace equations and the current solver I am using is taking every bit of memory I have available. What is the current problem size that's filling your memory? How regular is the domain and coefficients? Barry's suggestion won't necessarily reduce memory usage, but it does tend to be fast and easy. How much memory is used by the solver versus other parts of your application? Is the current configuration realistic? It's not worth minimizing solver memory usage if it's going to be a small part of the total application, for example. If you have simple geometry, smooth coefficients, and a standard discretization, a matrix-free full multigrid can solve the problem using about 50 bytes of memory per DOF at a cost of around 5 times as much as a residual evaluation. But many applications don't benefit from such a refined solver.

1 0

Re: [petsc-dev] Do you have any suggestion in order to reduce my memory footprint?
by Barry Smith 19 Oct '14

19 Oct '14

What do you get for -ksp_view ? You should very likely be using the -pc_type gamg if it is an unstructured grid problem or you are using your own mesh data structure or -pc_type mg if you are using the DMDACreate1/2/3() Barry > On Oct 19, 2014, at 3:58 PM, Alletto, John M <john.m.alletto(a)lmco.com> wrote: > > All, > > I have a 64 bit linux machine with 256 GByte of memory and 30 processors. > I am solving Laplace equations and the current solver I am using is taking every bit of memory I have available. > > Do you have any suggestions (such as changing precision or changing solvers or something else) in order to reduce my memory footprint? > > Many Thanks > John

1 0

Do you have any suggestion in order to reduce my memory footprint?
by Alletto, John M 19 Oct '14

19 Oct '14

All, I have a 64 bit linux machine with 256 GByte of memory and 30 processors. I am solving Laplace equations and the current solver I am using is taking every bit of memory I have available. Do you have any suggestions (such as changing precision or changing solvers or something else) in order to reduce my memory footprint? Many Thanks John

1 0

Re: [petsc-dev] Backend independent VecGetArray for GPUs
by Ashwin Srinath 19 Oct '14

19 Oct '14

Thanks Karl! Ashwin On Sun, Oct 19, 2014 at 9:45 AM, Karl Rupp <rupp(a)iue.tuwien.ac.at> wrote: > Hi Ashwin, > > I'll add two functions returning the bare CUDA and OpenCL handles. > > Best regards, > Karli > > > On 10/19/2014 03:42 PM, Ashwin Srinath wrote: > >> Hi everyone >> >> Just wondering what the consensus is. I'm happy to submit a PR if >> someone can tell me what goes where! >> >> Thanks >> Ashwin >> >> On 18 Oct 2014 01:46, "Karl Rupp" <rupp(a)iue.tuwien.ac.at >> <mailto:[email protected]>> wrote: >> >> Hi, >> >> >> > Why would we want this? The packages themselves >> (CUDA/ViennaCL) only >> >> expose >> memory using these specific types. What use is it to wrap >> these up in >> a void * if you >> just have to caste back down to use them. Isn't it better to >> maintain >> type-specific, and >> type safe, interfaces for this stuff? >> >> The biggest problem I faced was that there was no way to >> access device >> memory using petsc4py - since there is no equivalent for >> VecCUSPGetArray. So returning a raw pointer may not be very >> helpful >> for C++/CUSP users (they already have a nice way to access >> device >> memory) but it would definitely make things a lot easier for >> Python users. >> >> >> To me it sounds like something that should be dealt with in the >> library >> that does the python bindings, not in PETSc itself. (...) >> >> >> Unfortunately, this is not so easy: If the Python wrapper has to >> take care of such a conversion, then it needs to use the *exactly >> same build environment* as PETSc. The reason is that the CUSP and >> ViennaCL types are C++ beasts, not having a defined ABI, so one can >> run into all sorts of hard-to-debug problems when finally linking >> libpetsc.so with the Python wrapper. If, however, PETSc provides >> these low-level memory buffers, the Python wrapper can attach to a >> well-defined ABI. >> >> Best regards, >> Karli >> >> >

1 0

Re: [petsc-dev] Backend independent VecGetArray for GPUs
by Karl Rupp 19 Oct '14

19 Oct '14

Hi Ashwin, I'll add two functions returning the bare CUDA and OpenCL handles. Best regards, Karli On 10/19/2014 03:42 PM, Ashwin Srinath wrote: > Hi everyone > > Just wondering what the consensus is. I'm happy to submit a PR if > someone can tell me what goes where! > > Thanks > Ashwin > > On 18 Oct 2014 01:46, "Karl Rupp" <rupp(a)iue.tuwien.ac.at > <mailto:[email protected]>> wrote: > > Hi, > > >> > Why would we want this? The packages themselves > (CUDA/ViennaCL) only > > expose > memory using these specific types. What use is it to wrap > these up in > a void * if you > just have to caste back down to use them. Isn't it better to > maintain > type-specific, and > type safe, interfaces for this stuff? > > The biggest problem I faced was that there was no way to > access device > memory using petsc4py - since there is no equivalent for > VecCUSPGetArray. So returning a raw pointer may not be very > helpful > for C++/CUSP users (they already have a nice way to access > device > memory) but it would definitely make things a lot easier for > Python users. > > > To me it sounds like something that should be dealt with in the > library > that does the python bindings, not in PETSc itself. (...) > > > Unfortunately, this is not so easy: If the Python wrapper has to > take care of such a conversion, then it needs to use the *exactly > same build environment* as PETSc. The reason is that the CUSP and > ViennaCL types are C++ beasts, not having a defined ABI, so one can > run into all sorts of hard-to-debug problems when finally linking > libpetsc.so with the Python wrapper. If, however, PETSc provides > these low-level memory buffers, the Python wrapper can attach to a > well-defined ABI. > > Best regards, > Karli >

1 0

Re: [petsc-dev] Backend independent VecGetArray for GPUs
by Karl Rupp 18 Oct '14

18 Oct '14

Hi, >> > Why would we want this? The packages themselves (CUDA/ViennaCL) only >> expose >> memory using these specific types. What use is it to wrap these up in >> a void * if you >> just have to caste back down to use them. Isn't it better to maintain >> type-specific, and >> type safe, interfaces for this stuff? >> >> The biggest problem I faced was that there was no way to access device >> memory using petsc4py - since there is no equivalent for >> VecCUSPGetArray. So returning a raw pointer may not be very helpful >> for C++/CUSP users (they already have a nice way to access device >> memory) but it would definitely make things a lot easier for Python users. > > To me it sounds like something that should be dealt with in the library > that does the python bindings, not in PETSc itself. (...) Unfortunately, this is not so easy: If the Python wrapper has to take care of such a conversion, then it needs to use the *exactly same build environment* as PETSc. The reason is that the CUSP and ViennaCL types are C++ beasts, not having a defined ABI, so one can run into all sorts of hard-to-debug problems when finally linking libpetsc.so with the Python wrapper. If, however, PETSc provides these low-level memory buffers, the Python wrapper can attach to a well-defined ABI. Best regards, Karli

1 0

Re: [petsc-dev] Backend independent VecGetArray for GPUs
by Barry Smith 17 Oct '14

17 Oct '14

> On Oct 17, 2014, at 4:13 PM, Dominic Meiser <dmeiser(a)txcorp.com> wrote: > > On 10/17/2014 02:42 PM, Ashwin Srinath wrote: >> > Why would we want this? The packages themselves (CUDA/ViennaCL) only expose >> memory using these specific types. What use is it to wrap these up in a void * if you >> just have to caste back down to use them. Isn't it better to maintain type-specific, and >> type safe, interfaces for this stuff? >> >> The biggest problem I faced was that there was no way to access device memory using petsc4py - since there is no equivalent for VecCUSPGetArray. So returning a raw pointer may not be very helpful for C++/CUSP users (they already have a nice way to access device memory) but it would definitely make things a lot easier for Python users. > > To me it sounds like something that should be dealt with in the library that does the python bindings, Yup. That library is petsc4py :-) We want to try to have as much as possible the same API in petsc4py and PETSc so the question is if we provide the API all the way in PETSc or just petsc4py. For consistency across libraries it might belong in PETSc, plus it might actually be useful directly in C/C++ so it may belong there. Barry > not in PETSc itself. For example, it would be easy to write a wrapper functions like the following: > > void *getDevicePtr(Vec v) > { > void *ptr; > > /* Use VecCUSPGetArray* or VecViennaCLGetArray* to obtain device ptr */ > > return ptr; > } > > void releaseDevicePtr(Vec v, void *ptr) > { > /* Vec type specific code to release the pointer */ > } > > What do we gain by having these wrappers in PETSc rather than in a separate library? > > Cheers, > Dominic > >> >> Thanks! >> >> On Fri, Oct 17, 2014 at 12:09 PM, Dominic Meiser <dmeiser(a)txcorp.com> wrote: >> On 10/17/2014 09:45 AM, Lisandro Dalcin wrote: >> On 17 October 2014 17:54, Dominic Meiser <dmeiser(a)txcorp.com> wrote: >> Hi Ashwin, >> >> Are you suggesting that `VecGetGPUArray*` be added to the `Vec` interface? >> That might be problematic because these methods only makes sense for GPU >> vectors. The interface of `Vec` would then be tied to the PETSc >> configuration. >> >> These methods could be always available simply generate an error for >> non-GPU vector types. >> That's a possible approach. Barry, Jed, what's the preferred way of doing this? Make base class interface wider or subtyping via compose and query? >> >> An alternative might be to compose the various `VecCUSPGetArray` and related >> methods with `Vec` objects. You could then query these methods and use them >> if available (and handle absence of the methods if needed, e.g. throw an >> exception). This would be easy to add. >> >> Yes, however note that what we really need is a backend-independent >> way to get the RAW pointer to the GPU buffers. Right now we have calls >> to get the CUSP or ViennaCL array, to handle them you need C++ and >> depend on CUDA/OpenCL at compile-time. >> >> >> In what sense should it be backend-independent? The method name should be independent of the backend? Or the presence of the method should be backend independent? >> >> I don't see what your gaining by having the methods in Vec. You still need to handle the error that occurs if you're dealing with a non-GPU vector and you still have to know the vector type in order to interpret the return value (device pointer vs OpenCL buffer). >> >> Dominic >> >> >> -- >> Dominic Meiser >> Tech-X Corporation >> 5621 Arapahoe Avenue >> Boulder, CO 80303 >> USA >> Telephone: 303-996-2036 >> Fax: 303-448-7756 >> www.txcorp.com >> >> > > > -- > Dominic Meiser > Tech-X Corporation > 5621 Arapahoe Avenue > Boulder, CO 80303 > USA > Telephone: 303-996-2036 > Fax: 303-448-7756 > > www.txcorp.com

1 0

Re: [petsc-dev] Backend independent VecGetArray for GPUs
by Dominic Meiser 17 Oct '14

17 Oct '14

On 10/17/2014 02:42 PM, Ashwin Srinath wrote: > > Why would we want this? The packages themselves (CUDA/ViennaCL) only > expose > memory using these specific types. What use is it to wrap these up in > a void * if you > just have to caste back down to use them. Isn't it better to maintain > type-specific, and > type safe, interfaces for this stuff? > > The biggest problem I faced was that there was no way to access device > memory using petsc4py - since there is no equivalent for > VecCUSPGetArray. So returning a raw pointer may not be very helpful > for C++/CUSP users (they already have a nice way to access device > memory) but it would definitely make things a lot easier for Python users. To me it sounds like something that should be dealt with in the library that does the python bindings, not in PETSc itself. For example, it would be easy to write a wrapper functions like the following: void *getDevicePtr(Vec v) { void *ptr; /* Use VecCUSPGetArray* or VecViennaCLGetArray* to obtain device ptr */ return ptr; } void releaseDevicePtr(Vec v, void *ptr) { /* Vec type specific code to release the pointer */ } What do we gain by having these wrappers in PETSc rather than in a separate library? Cheers, Dominic > > Thanks! > > On Fri, Oct 17, 2014 at 12:09 PM, Dominic Meiser <dmeiser(a)txcorp.com > <mailto:[email protected]>> wrote: > > On 10/17/2014 09:45 AM, Lisandro Dalcin wrote: > > On 17 October 2014 17:54, Dominic Meiser <dmeiser(a)txcorp.com > <mailto:[email protected]>> wrote: > > Hi Ashwin, > > Are you suggesting that `VecGetGPUArray*` be added to the > `Vec` interface? > That might be problematic because these methods only makes > sense for GPU > vectors. The interface of `Vec` would then be tied to the > PETSc > configuration. > > These methods could be always available simply generate an > error for > non-GPU vector types. > > That's a possible approach. Barry, Jed, what's the preferred way > of doing this? Make base class interface wider or subtyping via > compose and query? > > > An alternative might be to compose the various > `VecCUSPGetArray` and related > methods with `Vec` objects. You could then query these > methods and use them > if available (and handle absence of the methods if needed, > e.g. throw an > exception). This would be easy to add. > > Yes, however note that what we really need is a > backend-independent > way to get the RAW pointer to the GPU buffers. Right now we > have calls > to get the CUSP or ViennaCL array, to handle them you need C++ and > depend on CUDA/OpenCL at compile-time. > > > In what sense should it be backend-independent? The method name > should be independent of the backend? Or the presence of the > method should be backend independent? > > I don't see what your gaining by having the methods in Vec. You > still need to handle the error that occurs if you're dealing with > a non-GPU vector and you still have to know the vector type in > order to interpret the return value (device pointer vs OpenCL buffer). > > Dominic > > > -- > Dominic Meiser > Tech-X Corporation > 5621 Arapahoe Avenue > Boulder, CO 80303 > USA > Telephone: 303-996-2036 <tel:303-996-2036> > Fax: 303-448-7756 <tel:303-448-7756> > www.txcorp.com <http://www.txcorp.com> > > -- Dominic Meiser Tech-X Corporation 5621 Arapahoe Avenue Boulder, CO 80303 USA Telephone: 303-996-2036 Fax: 303-448-7756 www.txcorp.com

1 0

Re: [petsc-dev] Backend independent VecGetArray for GPUs
by Ashwin Srinath 17 Oct '14

17 Oct '14

Thanks, Domomic To be honest, I'm not sure where it would be appropriate. I'm just hoping to see the functionality *somewhere* :) On 17 Oct 2014 16:02, "Dominic Meiser" <dmeiser(a)txcorp.com> wrote: > On 10/17/2014 02:42 PM, Ashwin Srinath wrote: > > > Why would we want this? The packages themselves (CUDA/ViennaCL) only > expose > memory using these specific types. What use is it to wrap these up in a > void * if you > just have to caste back down to use them. Isn't it better to maintain > type-specific, and > type safe, interfaces for this stuff? > > The biggest problem I faced was that there was no way to access device > memory using petsc4py - since there is no equivalent for VecCUSPGetArray. > So returning a raw pointer may not be very helpful for C++/CUSP users (they > already have a nice way to access device memory) but it would definitely > make things a lot easier for Python users. > > > To me it sounds like something that should be dealt with in the library > that does the python bindings, not in PETSc itself. For example, it would > be easy to write a wrapper functions like the following: > > void *getDevicePtr(Vec v) > { > void *ptr; > > /* Use VecCUSPGetArray* or VecViennaCLGetArray* to obtain device ptr */ > > return ptr; > } > > void releaseDevicePtr(Vec v, void *ptr) > { > /* Vec type specific code to release the pointer */ > } > > What do we gain by having these wrappers in PETSc rather than in a > separate library? > > Cheers, > Dominic > > > Thanks! > > On Fri, Oct 17, 2014 at 12:09 PM, Dominic Meiser <dmeiser(a)txcorp.com> > wrote: > >> On 10/17/2014 09:45 AM, Lisandro Dalcin wrote: >> >>> On 17 October 2014 17:54, Dominic Meiser <dmeiser(a)txcorp.com> wrote: >>> >>>> Hi Ashwin, >>>> >>>> Are you suggesting that `VecGetGPUArray*` be added to the `Vec` >>>> interface? >>>> That might be problematic because these methods only makes sense for GPU >>>> vectors. The interface of `Vec` would then be tied to the PETSc >>>> configuration. >>>> >>>> These methods could be always available simply generate an error for >>> non-GPU vector types. >>> >> That's a possible approach. Barry, Jed, what's the preferred way of >> doing this? Make base class interface wider or subtyping via compose and >> query? >> >>> >>> An alternative might be to compose the various `VecCUSPGetArray` and >>>> related >>>> methods with `Vec` objects. You could then query these methods and use >>>> them >>>> if available (and handle absence of the methods if needed, e.g. throw an >>>> exception). This would be easy to add. >>>> >>>> Yes, however note that what we really need is a backend-independent >>> way to get the RAW pointer to the GPU buffers. Right now we have calls >>> to get the CUSP or ViennaCL array, to handle them you need C++ and >>> depend on CUDA/OpenCL at compile-time. >>> >>> >>> In what sense should it be backend-independent? The method name should >> be independent of the backend? Or the presence of the method should be >> backend independent? >> >> I don't see what your gaining by having the methods in Vec. You still >> need to handle the error that occurs if you're dealing with a non-GPU >> vector and you still have to know the vector type in order to interpret the >> return value (device pointer vs OpenCL buffer). >> >> Dominic >> >> >> -- >> Dominic Meiser >> Tech-X Corporation >> 5621 Arapahoe Avenue >> Boulder, CO 80303 >> USA >> Telephone: 303-996-2036 >> Fax: 303-448-7756 >> www.txcorp.com >> >> > > > -- > Dominic Meiser > Tech-X Corporation > 5621 Arapahoe Avenue > Boulder, CO 80303 > USA > Telephone: 303-996-2036 > Fax: 303-448-7756www.txcorp.com > >

1 0

Re: [petsc-dev] Backend independent VecGetArray for GPUs
by Ashwin Srinath 17 Oct '14

17 Oct '14

> Why would we want this? The packages themselves (CUDA/ViennaCL) only expose memory using these specific types. What use is it to wrap these up in a void * if you just have to caste back down to use them. Isn't it better to maintain type-specific, and type safe, interfaces for this stuff? The biggest problem I faced was that there was no way to access device memory using petsc4py - since there is no equivalent for VecCUSPGetArray. So returning a raw pointer may not be very helpful for C++/CUSP users (they already have a nice way to access device memory) but it would definitely make things a lot easier for Python users. Thanks! On Fri, Oct 17, 2014 at 12:09 PM, Dominic Meiser <dmeiser(a)txcorp.com> wrote: > On 10/17/2014 09:45 AM, Lisandro Dalcin wrote: > >> On 17 October 2014 17:54, Dominic Meiser <dmeiser(a)txcorp.com> wrote: >> >>> Hi Ashwin, >>> >>> Are you suggesting that `VecGetGPUArray*` be added to the `Vec` >>> interface? >>> That might be problematic because these methods only makes sense for GPU >>> vectors. The interface of `Vec` would then be tied to the PETSc >>> configuration. >>> >>> These methods could be always available simply generate an error for >> non-GPU vector types. >> > That's a possible approach. Barry, Jed, what's the preferred way of doing > this? Make base class interface wider or subtyping via compose and query? > >> >> An alternative might be to compose the various `VecCUSPGetArray` and >>> related >>> methods with `Vec` objects. You could then query these methods and use >>> them >>> if available (and handle absence of the methods if needed, e.g. throw an >>> exception). This would be easy to add. >>> >>> Yes, however note that what we really need is a backend-independent >> way to get the RAW pointer to the GPU buffers. Right now we have calls >> to get the CUSP or ViennaCL array, to handle them you need C++ and >> depend on CUDA/OpenCL at compile-time. >> >> >> In what sense should it be backend-independent? The method name should > be independent of the backend? Or the presence of the method should be > backend independent? > > I don't see what your gaining by having the methods in Vec. You still need > to handle the error that occurs if you're dealing with a non-GPU vector and > you still have to know the vector type in order to interpret the return > value (device pointer vs OpenCL buffer). > > Dominic > > > -- > Dominic Meiser > Tech-X Corporation > 5621 Arapahoe Avenue > Boulder, CO 80303 > USA > Telephone: 303-996-2036 > Fax: 303-448-7756 > www.txcorp.com > >

1 0