July 2013 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Improving and stabilizing GPU support
by Karl Rupp 19 Jul '13

19 Jul '13

Hi Dave, > Your list sounds great to me. Glad that you and Paul are working on this together. > > My main interests are in better preconditioner support and better multi-GPU/MPI > scalability. This is follow-up work then. There are a couple of 'simple' preconditioners (polynomial preconditioning, maybe some point-block Jacobi) which can also be useful as smoothers and which we can add in the near future. We should just get the 'infrastructure' work done first so that we don't have to unnecessarily adjust too much code later on. > Is there any progress on Steve Dalton's work on the cusp algebraic multigrid > preconditioner with PETSc? I believe Jed said in a previous email that Steve > was going to be working on adding MPI support for that as well as other > enhancements. Yes, Steve is working on this right here at our division. Jed can give a more detailed answer on this. > Will there be any improvements for GPU preconditioners in ViennaCL 1.5.0? > When do you expect ViennaCL 1.5.0 to be available in PETSc? Jed gave me a good hint with respect to D-ILU0, which I'll also add to PETSc. As with other GPU-accelerations using ILU, it will require a proper matrix ordering to give good performance. I'm somewhat tempted to port the SA-AMG implementation in CUSP to OpenCL as well, but this certainly won't be in 1.5.0. > I'm also interested in trying the PETSc ViennaCL support on the Xeon Phi. > Do you have a schedule for when that might be ready for friendly testing? With OpenCL you can already test this now. Just install the Intel OpenCL SDK on your Xeon Phi machine, configure with --download-viennacl, --with-opencl-include=..., --with-opencl-lib=..., and pass the -viennacl_device_accelerator flag in addition to -vec_type viennacl -mat_type aijviennacl when executing. Unfortunately the application memory bandwidth we get on the Xeon Phi is too limited to be useful for off-loaded execution as it is the case with OpenCL: Even the folks at Intel couldn't obtain more than ~95 GB/sec even when filling up the whole MIC with just two vectors for benchmarking a simple copy operation. Thus, I don't think our efforts are currently well spent on trying a fully native execution of PETSc on the MIC, because the trend is going more towards a tighter CPU/accelerator integration on the same die rather than piggy-backing via PCI-Express. Anyway, I'll let you know if there are any updates on this front. Best regards, Karli

1 0

Re: [petsc-dev] Improving and stabilizing GPU support
by Nystrom, William D 19 Jul '13

19 Jul '13

Hi Karli, Your list sounds great to me. Glad that you and Paul are working on this together. My main interests are in better preconditioner support and better multi-GPU/MPI scalability. Is there any progress on Steve Dalton's work on the cusp algebraic multigrid preconditioner with PETSc? I believe Jed said in a previous email that Steve was going to be working on adding MPI support for that as well as other enhancements. Will there be any improvements for GPU preconditioners in ViennaCL 1.5.0? When do you expect ViennaCL 1.5.0 to be available in PETSc? I'm also interested in trying the PETSc ViennaCL support on the Xeon Phi. Do you have a schedule for when that might be ready for friendly testing? Thanks, Dave -- Dave Nystrom LANL HPC-5 Phone: 505-667-7913 Email: wdn(a)lanl.gov Smail: Mail Stop B272 Group HPC-5 Los Alamos National Laboratory Los Alamos, NM 87545 ________________________________________ From: petsc-dev-bounces(a)mcs.anl.gov [petsc-dev-bounces(a)mcs.anl.gov] on behalf of Karl Rupp [rupp(a)mcs.anl.gov] Sent: Friday, July 19, 2013 1:12 PM To: For users of the development version of PETSc Subject: [petsc-dev] Improving and stabilizing GPU support Hi guys, now as the Paul's pull request for largely removing the txpetscgpu dependency is merged to next, I will proceed with further improving our GPU support. My ideas and TODO-list are as follows: * Reduce CUSP dependency: The current elementary operations are mainly realized via CUSP. With better support via CUSPARSE and CUBLAS, I'd add a separate 'native' CUDA backend so that we can provide a full set of vector and sparse matrix operations out of the default NVIDIA toolchain. We will still keep CUSP for its preconditioners, yet we no longer depend on it. * Integrate last bits of txpetscgpu package. I assume Paul will provide a helping hand here. * Better ViennaCL bindings: The OpenCL version of VecMDot() will experience a boost with the ViennaCL 1.5.0 release, the CUDA version was fixed a couple of month back. Also, VecCopySome() will get improved in order to provide better MPI performance (similar to what Paul applied for CUSPARSE) * Documentation: Add a chapter on GPUs to the manual, particularly on what to expect and what not to expect. Update documentation on webpage regarding installation. * Integration of FEM quadrature from SNES ex52. The CUDA part requiring code generation is not very elegant, while the OpenCL approach is better suited for a library integration thanks to JIT. However, this requires user code to be provided as a string (again not very elegant) or loaded from file (more reasonable). How much FEM functionality do we want to provide via PETSc? Please don't hesitate to post other GPU wishes. Now it's the best time for doing so :-) Best regards, Karli

1 0

Re: [petsc-dev] Improving and stabilizing GPU support
by Paul Mullowney 19 Jul '13

19 Jul '13

>> * Reduce CUSP dependency: The current elementary operations are mainly realized via CUSP. With better support via CUSPARSE and CUBLAS, I'd add a separate 'native' CUDA backend so that we can provide a full set of vector and sparse matrix operations out of the default NVIDIA toolchain. We will still keep CUSP for its preconditioners, yet we no longer depend on it. Agreed. In the past, I've suggested a -vec_type cuda (not cusp). All the CUSP operations can be done with Thrust algorithms. Since Thrust comes default with CUDA, one can have only a CUDA dependency. >> * Integrate last bits of txpetscgpu package. I assume Paul will provide a helping hand here. Of course. This will go much faster as much of the hard work is done. Do people want support for different matrix formats in the CUSP classes : i.e. diagonal, ellpack, hybrid? I think the CUSP preconditioners can be derived from matrices stored in non-csr format (although they're likely just doing a convert under the hood). >> * Better ViennaCL bindings: The OpenCL version of VecMDot() will experience a boost with the ViennaCL 1.5.0 release, the CUDA version was fixed a couple of month back. Also, VecCopySome() will get improved in order to provide better MPI performance (similar to what Paul applied for CUSPARSE) >> >> * Documentation: Add a chapter on GPUs to the manual, particularly on what to expect and what not to expect. Update documentation on webpage regarding installation. I will help with the manual. >> * Integration of FEM quadrature from SNES ex52. The CUDA part requiring code generation is not very elegant, while the OpenCL approach is better suited for a library integration thanks to JIT. However, this requires user code to be provided as a string (again not very elegant) or loaded from file (more reasonable). How much FEM functionality do we want to provide via PETSc? Multi-GPU is a highly pressing need, IMO. Need to figure out how to make Block Jacobi and ASM run efficiently. -Paul >> Please don't hesitate to post other GPU wishes. Now it's the best time for doing so :-) >> >> Best regards, >> Karli

1 0

Re: [petsc-dev] Improving and stabilizing GPU support
by Barry Smith 19 Jul '13

19 Jul '13

All sounds good. On Jul 19, 2013, at 2:12 PM, Karl Rupp <rupp(a)mcs.anl.gov> wrote: > Hi guys, > > now as the Paul's pull request for largely removing the txpetscgpu dependency is merged to next, I will proceed with further improving our GPU support. My ideas and TODO-list are as follows: > > * Reduce CUSP dependency: The current elementary operations are mainly realized via CUSP. With better support via CUSPARSE and CUBLAS, I'd add a separate 'native' CUDA backend so that we can provide a full set of vector and sparse matrix operations out of the default NVIDIA toolchain. We will still keep CUSP for its preconditioners, yet we no longer depend on it. > > * Integrate last bits of txpetscgpu package. I assume Paul will provide a helping hand here. > > * Better ViennaCL bindings: The OpenCL version of VecMDot() will experience a boost with the ViennaCL 1.5.0 release, the CUDA version was fixed a couple of month back. Also, VecCopySome() will get improved in order to provide better MPI performance (similar to what Paul applied for CUSPARSE) > > * Documentation: Add a chapter on GPUs to the manual, particularly on what to expect and what not to expect. Update documentation on webpage regarding installation. > > * Integration of FEM quadrature from SNES ex52. The CUDA part requiring code generation is not very elegant, while the OpenCL approach is better suited for a library integration thanks to JIT. However, this requires user code to be provided as a string (again not very elegant) or loaded from file (more reasonable). How much FEM functionality do we want to provide via PETSc? > > Please don't hesitate to post other GPU wishes. Now it's the best time for doing so :-) > > Best regards, > Karli

1 0

Re: [petsc-dev] [petsc-maint] CUDA Version Compatibility
by Barry Smith 19 Jul '13

19 Jul '13

On Jul 19, 2013, at 2:11 PM, Satish Balay <balay(a)mcs.anl.gov> wrote: > On Fri, 19 Jul 2013, Barry Smith wrote: > >> >> On Jul 19, 2013, at 10:47 AM, Matthew Knepley <petsc-maint(a)mcs.anl.gov> wrote: >> >>> On Fri, Jul 19, 2013 at 10:38 AM, Jason Sarich <sarich(a)mcs.anl.gov> wrote: >>> Shiva, >>> >>> This is from my bad merge. PETSc used to allow cuda without cusp/thrust, this is apparently no longer an option. I'll fix this. >>> >>> We do. You configure with --with-cuda-only >> >> In the long run this is a terrible way to indicate this and needs to be changed. It is inconsistent with how we do everything else by simply adding additional supports using --with-xxx. We should have --with-cuda for basic cuda support and --with-cusp for cusp support (which may automatically add the --with-cuda). > > For one --with-cuda-only=1 is the default [and all its doing the following] > >>>>> > if not self.framework.argDB['with-cuda-only']: > if not self.cusp.found or not self.thrust.found: > raise RuntimeError('PETSc CUDA support requires the CUSP and Thrust packages\nRerun configure using --with-cusp-dir and --with-thrust-dir') > self.checkThrustVersion() > self.checkCUSPVersion() > <<<<< > > i.e currently none of these checks [including the version check for cusp/thrust] is being done. > > So with-cuda-only option can be removed? Seems like cuda-only works even without cuda? That's not right :-( > > Also we have checkThrustVersion() and checkCUSPVersion() in cuda.py. > Perhaps they belong in thrust.py and cusp.py? [but for some reason > they were moved int cuda.py?] Sure sounds like it. > > And thrust.include is used in checkCUDAVersion(). Is this really > needed? Seems totally wrong. We use to think of cuda, thrust and cusp as always going together. Since we no longer think that it sounds like some refactoring is in order. Barry > > > Satish

1 0

Improving and stabilizing GPU support
by Karl Rupp 19 Jul '13

19 Jul '13

Hi guys, now as the Paul's pull request for largely removing the txpetscgpu dependency is merged to next, I will proceed with further improving our GPU support. My ideas and TODO-list are as follows: * Reduce CUSP dependency: The current elementary operations are mainly realized via CUSP. With better support via CUSPARSE and CUBLAS, I'd add a separate 'native' CUDA backend so that we can provide a full set of vector and sparse matrix operations out of the default NVIDIA toolchain. We will still keep CUSP for its preconditioners, yet we no longer depend on it. * Integrate last bits of txpetscgpu package. I assume Paul will provide a helping hand here. * Better ViennaCL bindings: The OpenCL version of VecMDot() will experience a boost with the ViennaCL 1.5.0 release, the CUDA version was fixed a couple of month back. Also, VecCopySome() will get improved in order to provide better MPI performance (similar to what Paul applied for CUSPARSE) * Documentation: Add a chapter on GPUs to the manual, particularly on what to expect and what not to expect. Update documentation on webpage regarding installation. * Integration of FEM quadrature from SNES ex52. The CUDA part requiring code generation is not very elegant, while the OpenCL approach is better suited for a library integration thanks to JIT. However, this requires user code to be provided as a string (again not very elegant) or loaded from file (more reasonable). How much FEM functionality do we want to provide via PETSc? Please don't hesitate to post other GPU wishes. Now it's the best time for doing so :-) Best regards, Karli

1 0

Re: [petsc-dev] [petsc-maint] CUDA Version Compatibility
by Satish Balay 19 Jul '13

19 Jul '13

On Fri, 19 Jul 2013, Barry Smith wrote: > > On Jul 19, 2013, at 10:47 AM, Matthew Knepley <petsc-maint(a)mcs.anl.gov> wrote: > > > On Fri, Jul 19, 2013 at 10:38 AM, Jason Sarich <sarich(a)mcs.anl.gov> wrote: > > Shiva, > > > > This is from my bad merge. PETSc used to allow cuda without cusp/thrust, this is apparently no longer an option. I'll fix this. > > > > We do. You configure with --with-cuda-only > > In the long run this is a terrible way to indicate this and needs to be changed. It is inconsistent with how we do everything else by simply adding additional supports using --with-xxx. We should have --with-cuda for basic cuda support and --with-cusp for cusp support (which may automatically add the --with-cuda). For one --with-cuda-only=1 is the default [and all its doing the following] >>>> if not self.framework.argDB['with-cuda-only']: if not self.cusp.found or not self.thrust.found: raise RuntimeError('PETSc CUDA support requires the CUSP and Thrust packages\nRerun configure using --with-cusp-dir and --with-thrust-dir') self.checkThrustVersion() self.checkCUSPVersion() <<<<< i.e currently none of these checks [including the version check for cusp/thrust] is being done. So with-cuda-only option can be removed? Also we have checkThrustVersion() and checkCUSPVersion() in cuda.py. Perhaps they belong in thrust.py and cusp.py? [but for some reason they were moved int cuda.py?] And thrust.include is used in checkCUDAVersion(). Is this really needed? Satish

1 0

Re: [petsc-dev] [petsc-maint] CUDA Version Compatibility
by Barry Smith 19 Jul '13

19 Jul '13

On Jul 19, 2013, at 10:47 AM, Matthew Knepley <petsc-maint(a)mcs.anl.gov> wrote: > On Fri, Jul 19, 2013 at 10:38 AM, Jason Sarich <sarich(a)mcs.anl.gov> wrote: > Shiva, > > This is from my bad merge. PETSc used to allow cuda without cusp/thrust, this is apparently no longer an option. I'll fix this. > > We do. You configure with --with-cuda-only In the long run this is a terrible way to indicate this and needs to be changed. It is inconsistent with how we do everything else by simply adding additional supports using --with-xxx. We should have --with-cuda for basic cuda support and --with-cusp for cusp support (which may automatically add the --with-cuda). Barry > > Matt > > Jason > > > > > On Thu, Jul 18, 2013 at 4:06 PM, Chekuri Choudary <cchoudary(a)rnet-tech.com> wrote: > Apparently, I was using a branch that deviated from petsc-dev before this > got fixed. I am testing the master and will let you know the result. Thank > you very much. > > > > -----Original Message----- > From: Satish Balay [mailto:[email protected]] > Sent: Thursday, July 18, 2013 4:56 PM > To: petsc-maint > Cc: sarich(a)mcs.anl.gov; Chekuri Choudary > Subject: Re: [petsc-maint] CUDA Version Compatibility > > On Thu, 18 Jul 2013, Satish Balay wrote: > > > https://bitbucket.org/petsc/petsc/src/3123b27f26841f5e48c5aadd3e04f879 > > f1d21372/config/PETSc/packages/cuda.py?at=sarich/rnet-all#cl-52 > > > > 3ae844c2 (Daniel Lowell 2011-10-26 09:17:02 -0500 29)# > self.ThrustVersionStr = str(int(self.ThrustVersion)/100000) + '.' + > str(int(self.ThrustVersion)/100%1000) + '.' + str(int(self.ThrustVersio > > 3ae844c2 (Daniel Lowell 2011-10-26 09:17:02 -0500 30)# > self.CUSPVersionStr = str(int(self.CUSPVersion)/100000) + '.' + > str(int(self.CUSPVersion)/100%1000) + '.' + str(int(self.CUSPVersion)%100) > > ops - I meant to refer to the following lines that got commneted - causing > configure to give this error. > > 3ae844c2 (Daniel Lowell 2011-10-26 09:17:02 -0500 51) #### > self.cusp = framework.require('config.packages.cusp', self) > 3ae844c2 (Daniel Lowell 2011-10-26 09:17:02 -0500 52) #### > self.thrust = framework.require('config.packages.thrust', self) > > > satish > > > > > > Jason might have to take a look on the rbase? > > > > Satish > > > > On Thu, 18 Jul 2013, Matthew Knepley wrote: > > > > > On Thu, Jul 18, 2013 at 3:24 PM, Chekuri Choudary > > > <cchoudary(a)rnet-tech.com>wrote: > > > > > > > > > > > It appears that there is an issue with checkCUDAVersion. I have > > > > downloaded the most recent versions of both CUSP and THRUST and > > > > tested with both CUDA > > > > 4.0 and CUDA 5.0. The configure.log files are attached. > > > > > > > > > > Send $PETSC_DIR/config/PETSc/packages/cuda.py. > > > > > > Matt > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: Satish Balay [mailto:[email protected]] > > > > Sent: Thursday, July 18, 2013 2:35 PM > > > > To: Chekuri Choudary > > > > Cc: petsc-maint(a)mcs.anl.gov > > > > Subject: Re: [petsc-maint] CUDA Version Compatibility > > > > > > > > CUDA5.0 should work [and perhaps all cuda 4 versions aswell] > > > > > > > > > > > > >>> > > > > Configure Options: --configModules=PETSc.Configure > > > > --optionsModule=PETSc.compilerOptions --with-x=1 --CFLAGS="-O3 -g > > > > -ggdb -fopenmp" --FFLAGS="-O3 -g" --with-batch=0 > > > > --known-mpi-shared-libraries=no > > > > --with-cuda=1 --CUDAC=nvcc --with-cuda-dir=/usr/local/cuda-5.0/ > > > > PETSC_ARCH=PFLOTRAN1 <<<< > > > > > > > > >> > > > > 'Configure' object has no attribute 'thrust' File > > > > "./config/configure.py", line 293, in petsc_configure > > > > framework.configure(out = sys.stdout) << > > > > > > > > Perhaps there should be a better error check somewere. You need > > > > --with-thrust=1 [which is packaged in cuda] - and also > > > > --with-cusp=1 [perhaps with appropriate --with-cusp-dir option - > > > > or installed at '/usr/local/cuda-5.0/include/cusp'] > > > > > > > > Check config/examples/arch-cuda-double.py > > > > > > > > Satish > > > > > > > > On Thu, 18 Jul 2013, Chekuri Choudary wrote: > > > > > > > > > Hi, > > > > > > > > > > Is it necessary to use CUDA 4.1 or 4.2 to use the GPU > > > > > capabilities in > > > > PETSc. > > > > > I am having problems compiling PETSc with CUDA. I have both > > > > > CUDA4.0 and > > > > > CUDA5.0 installed. Please find the configure.log files attached. > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener

1 0

Re: [petsc-dev] Multiple PetscBags in a single file
by Barry Smith 18 Jul '13

18 Jul '13

Glenn, This is certainly suppose to work. Please report any bugs Barry On Jul 18, 2013, at 3:46 PM, "Hammond, Glenn" <glenn.hammond(a)pnnl.gov> wrote: > For the purposes of checkpoint/restart, can one write multiple PetscBags (mingled with Vecs) to a single binary file and read them in the same order? We want to split up the writing/reading of checkpoint files by process model and thus, create a separate PetscBag for each process model. E.g. > > Order as written in file: > > Flow Bag > Flow Solution Vec > Flow State Variable 1 Vec > Flow State Variable 2 Vec > ... > Flow State Variable N Vec > Transport Bag > Transport Solution Vec > Transport State Variable 1 Vec > Transport State Variable 2 Vec > ... > Transport State Variable N Vec > > > Thanks, > > Glenn

1 0

Multiple PetscBags in a single file
by Hammond, Glenn 18 Jul '13

18 Jul '13

For the purposes of checkpoint/restart, can one write multiple PetscBags (mingled with Vecs) to a single binary file and read them in the same order? We want to split up the writing/reading of checkpoint files by process model and thus, create a separate PetscBag for each process model. E.g. Order as written in file: Flow Bag Flow Solution Vec Flow State Variable 1 Vec Flow State Variable 2 Vec ... Flow State Variable N Vec Transport Bag Transport Solution Vec Transport State Variable 1 Vec Transport State Variable 2 Vec ... Transport State Variable N Vec Thanks, Glenn

1 0