September 2008 - petsc-dev - mailman3.cels.anl.gov

Re: PETSc on GPUs
by Ahmed El Zein 18 Sep '08

18 Sep '08

On Thu, 2008-09-18 at 10:20 -0500, Matthew Knepley wrote: > > A question that I had regarding the PETSc code when I was thinking > about > > this was: > > You have the SeqAIJ matrix type and the the MPIAIJ type built around > it > > (or that is what I understand from the code). So basically you > implement > > the SeqAIJ type for the GPU and you get the MPI type for free? > > Yes, that is true. However, note that in the MPI step, you will need a > gather > operation to get the second matrix multiply to work. Matt, Could you explain a bit more? Ahmed

1 0

Re: PETSc on GPUs
by Matthew Knepley 18 Sep '08

18 Sep '08

On Wed, Sep 17, 2008 at 10:51 PM, Ahmed El Zein <ahmed(a)azein.com> wrote: > On Wed, 2008-09-17 at 21:33 -0500, Barry Smith wrote: >> Ahmed, >> >> This is very cool. >> On Sep 17, 2008, at 8:05 PM, Ahmed El Zein wrote: >> >> > Harald, >> > I am working on implementing SpMV on a NVIDIA GPU with CUDA for SML >> > applications (just about finished actually). >> > >> > As the SML application I am planing to base my work on uses PETSc, I >> > have written some functions that will convert AIJ matrices and Vectors >> > to SP and copy them to the GPU, multiply them and copy them back to >> > the >> > host. I would be happy to share them with you if you want. >> >> I think to really make the GPU truly a large step forward in >> performance, the Vec's need to >> be kept on the GPU and only transported back to the main CPU when >> absolutely needed. For example, consider a KSP solver like CG, it would >> run on the main CPU but each Vec is actually just a handle for the >> true vector >> entries that are in the GPU memory, a call to VecAXPY(), for example, >> would >> pass the scalar and the two Vec handles down to the GPU where the actual >> axpy is performed. With this paradigm the only values passed back to the >> main CPU are scalars. This is why I think this work has to be done only >> on the latests GPU systems with lots of memory. >> > You are right! That is what I do for the SML application. I convert and > copy the matrix to the GPU and then iteratively send a new vector to the > GPU for multiplication. In fact unless the matrix will be reused at > least 4 times, there would be no performance gain! > > The 8800 GTX has 768 MB of memory but you can have multiple GPUs running > on your machine and split your data amongst them to effectively get more > memory. > > What I was thinking of was to add GPU pointers to a PETSC Mat or Vec > object with an optional shadow parameter. If shadow is enabled the host > will keep a copy of what is on the GPU in main memory. That way if there > are changes to the original matrix: > 1. It might be less expensive to make the modification on the host. > 2. It might be possible to update the matrix on the GPU by sending > diffs, as the copying of data back and forth is the most expensive > operation. > > It also will allow the use of that matrix on both the host and the GPU > to offer maximum flexibility. Matrix assembly would be done on the host > anyway. > > (I am not sure if the above points are loads of rubbish or not. But I > think that there are many options to be considered in a PETSc GPU > implementation.) > > > A question that I had regarding the PETSc code when I was thinking about > this was: > You have the SeqAIJ matrix type and the the MPIAIJ type built around it > (or that is what I understand from the code). So basically you implement > the SeqAIJ type for the GPU and you get the MPI type for free? Yes, that is true. However, note that in the MPI step, you will need a gather operation to get the second matrix multiply to work. Matt > Ahmed > >> Barry >> >> > >> > >> > While this would be outside the scope of my MPhil I would be very >> > interested in helping to add GPU support for PETSC. I have not yet had >> > any experience with CTM for programming ATI GPUs but I believe there >> > would not be a huge difference. >> > >> > I have access to a GeForce 8800GTX GPU (single precession only) at the >> > ANU. I have been talking with my supervisor about getting a GTX280 or >> > GTX 260 (supports double precision) but I don't know if we will be >> > getting one. >> > >> > Anyway I would like to help. So if anyone would like to start thinking >> > how this would be best implemented, I am available. :) >> > >> > Ahmed >> > >> > On Wed, 2008-09-17 at 08:24 -0700, Harald Pfeiffer wrote: >> >> Hi, >> >> >> >> do you know whether there are any efforts to run PETSc on GPUs >> >> (graphical processing units)? >> >> >> >> Thanks, >> >> Harald >> >> >> >> >> >> >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: PETSc on GPUs
by Ahmed El Zein 17 Sep '08

17 Sep '08

On Wed, 2008-09-17 at 21:33 -0500, Barry Smith wrote: > Ahmed, > > This is very cool. > On Sep 17, 2008, at 8:05 PM, Ahmed El Zein wrote: > > > Harald, > > I am working on implementing SpMV on a NVIDIA GPU with CUDA for SML > > applications (just about finished actually). > > > > As the SML application I am planing to base my work on uses PETSc, I > > have written some functions that will convert AIJ matrices and Vectors > > to SP and copy them to the GPU, multiply them and copy them back to > > the > > host. I would be happy to share them with you if you want. > > I think to really make the GPU truly a large step forward in > performance, the Vec's need to > be kept on the GPU and only transported back to the main CPU when > absolutely needed. For example, consider a KSP solver like CG, it would > run on the main CPU but each Vec is actually just a handle for the > true vector > entries that are in the GPU memory, a call to VecAXPY(), for example, > would > pass the scalar and the two Vec handles down to the GPU where the actual > axpy is performed. With this paradigm the only values passed back to the > main CPU are scalars. This is why I think this work has to be done only > on the latests GPU systems with lots of memory. > You are right! That is what I do for the SML application. I convert and copy the matrix to the GPU and then iteratively send a new vector to the GPU for multiplication. In fact unless the matrix will be reused at least 4 times, there would be no performance gain! The 8800 GTX has 768 MB of memory but you can have multiple GPUs running on your machine and split your data amongst them to effectively get more memory. What I was thinking of was to add GPU pointers to a PETSC Mat or Vec object with an optional shadow parameter. If shadow is enabled the host will keep a copy of what is on the GPU in main memory. That way if there are changes to the original matrix: 1. It might be less expensive to make the modification on the host. 2. It might be possible to update the matrix on the GPU by sending diffs, as the copying of data back and forth is the most expensive operation. It also will allow the use of that matrix on both the host and the GPU to offer maximum flexibility. Matrix assembly would be done on the host anyway. (I am not sure if the above points are loads of rubbish or not. But I think that there are many options to be considered in a PETSc GPU implementation.) A question that I had regarding the PETSc code when I was thinking about this was: You have the SeqAIJ matrix type and the the MPIAIJ type built around it (or that is what I understand from the code). So basically you implement the SeqAIJ type for the GPU and you get the MPI type for free? Ahmed > Barry > > > > > > > While this would be outside the scope of my MPhil I would be very > > interested in helping to add GPU support for PETSC. I have not yet had > > any experience with CTM for programming ATI GPUs but I believe there > > would not be a huge difference. > > > > I have access to a GeForce 8800GTX GPU (single precession only) at the > > ANU. I have been talking with my supervisor about getting a GTX280 or > > GTX 260 (supports double precision) but I don't know if we will be > > getting one. > > > > Anyway I would like to help. So if anyone would like to start thinking > > how this would be best implemented, I am available. :) > > > > Ahmed > > > > On Wed, 2008-09-17 at 08:24 -0700, Harald Pfeiffer wrote: > >> Hi, > >> > >> do you know whether there are any efforts to run PETSc on GPUs > >> (graphical processing units)? > >> > >> Thanks, > >> Harald > >> > >> > >> > >

1 0

Re: PETSc on GPUs
by Barry Smith 17 Sep '08

17 Sep '08

Ahmed, This is very cool. On Sep 17, 2008, at 8:05 PM, Ahmed El Zein wrote: > Harald, > I am working on implementing SpMV on a NVIDIA GPU with CUDA for SML > applications (just about finished actually). > > As the SML application I am planing to base my work on uses PETSc, I > have written some functions that will convert AIJ matrices and Vectors > to SP and copy them to the GPU, multiply them and copy them back to > the > host. I would be happy to share them with you if you want. I think to really make the GPU truly a large step forward in performance, the Vec's need to be kept on the GPU and only transported back to the main CPU when absolutely needed. For example, consider a KSP solver like CG, it would run on the main CPU but each Vec is actually just a handle for the true vector entries that are in the GPU memory, a call to VecAXPY(), for example, would pass the scalar and the two Vec handles down to the GPU where the actual axpy is performed. With this paradigm the only values passed back to the main CPU are scalars. This is why I think this work has to be done only on the latests GPU systems with lots of memory. Barry > > > While this would be outside the scope of my MPhil I would be very > interested in helping to add GPU support for PETSC. I have not yet had > any experience with CTM for programming ATI GPUs but I believe there > would not be a huge difference. > > I have access to a GeForce 8800GTX GPU (single precession only) at the > ANU. I have been talking with my supervisor about getting a GTX280 or > GTX 260 (supports double precision) but I don't know if we will be > getting one. > > Anyway I would like to help. So if anyone would like to start thinking > how this would be best implemented, I am available. :) > > Ahmed > > On Wed, 2008-09-17 at 08:24 -0700, Harald Pfeiffer wrote: >> Hi, >> >> do you know whether there are any efforts to run PETSc on GPUs >> (graphical processing units)? >> >> Thanks, >> Harald >> >> >> >

1 0

Re: PETSc on GPUs
by Ahmed El Zein 17 Sep '08

17 Sep '08

Harald, I am working on implementing SpMV on a NVIDIA GPU with CUDA for SML applications (just about finished actually). As the SML application I am planing to base my work on uses PETSc, I have written some functions that will convert AIJ matrices and Vectors to SP and copy them to the GPU, multiply them and copy them back to the host. I would be happy to share them with you if you want. While this would be outside the scope of my MPhil I would be very interested in helping to add GPU support for PETSC. I have not yet had any experience with CTM for programming ATI GPUs but I believe there would not be a huge difference. I have access to a GeForce 8800GTX GPU (single precession only) at the ANU. I have been talking with my supervisor about getting a GTX280 or GTX 260 (supports double precision) but I don't know if we will be getting one. Anyway I would like to help. So if anyone would like to start thinking how this would be best implemented, I am available. :) Ahmed On Wed, 2008-09-17 at 08:24 -0700, Harald Pfeiffer wrote: > Hi, > > do you know whether there are any efforts to run PETSc on GPUs > (graphical processing units)? > > Thanks, > Harald > > >

1 0

Re: PETSc on GPUs
by Barry Smith 17 Sep '08

17 Sep '08

Harald, No money, no time at our end. With the absolute latest offerings: double precision and 1+ gigabytes memory this could well be a viable way to significantly higher performance. We plan to start to play around with this in the next few months, outside help would be great. Barry On Sep 17, 2008, at 10:24 AM, Harald Pfeiffer wrote: > Hi, > > do you know whether there are any efforts to run PETSc on GPUs > (graphical processing units)? > > Thanks, > Harald > > > > -- > Harald P. Pfeiffer harald(a)tapir.caltech.edu > Theoretical Astrophysics Phone (626) 395-8413 > Caltech 130-33 Fax (626) 796-5675 > Pasadena, CA 91125, USA >

1 0

Re: About SNES solver of TS objects
by Hong Zhang 17 Sep '08

17 Sep '08

On Wed, 17 Sep 2008, Lisandro Dalcin wrote: > BTW, perhaps you should do the same in TSGetKSP(). What do you think? Yes. It is pushed to petsc-dev. Hong > > 2008/9/17 Hong Zhang <hzhang(a)mcs.anl.gov>: >> >> TSGetSNES() must be called after TSSetType(), >> otherwise the nonlinear context 'snes' is >> being created. >> Attached please find updated ex7.c, which >> illustrates how to set uer-defined SNESMonitor(). >> >> This example is pushed to petsc-dev. >> I also added an error flag in petsc-dev to prevent >> calling of TSGetSNES() before TSSetType(). >> Thanks for reporting the problem! >> >> Best, >> >> Hong >> >> On Fri, 12 Sep 2008, tsjb00 wrote: >> >>> >>> Many thanks for your reply! The code I tried was very similar to the code in: >>> ts/examples/tutorials/ex7.c >>> except that I changed the r.h.s. function. >>> >>> The code related to TS: >>> ierr = TSCreate(PETSC_COMM_WORLD,&ts);CHKERRQ(ierr); >>> ierr = TSSetProblemType(ts,TS_NONLINEAR);CHKERRQ(ierr); >>> ierr = TSSetRHSFunction(ts,FormFunction,&appctx);CHKERRQ(ierr); >>> >>> ierr = DAGetColoring(appctx.da,IS_COLORING_GLOBAL,&iscoloring);CHKERRQ(ierr); >>> ierr = MatFDColoringCreate(J,iscoloring,&matfdcoloring);CHKERRQ(ierr); >>> ierr = ISColoringDestroy(iscoloring);CHKERRQ(ierr); >>> ierr = MatFDColoringSetFunction(matfdcoloring,(PetscErrorCode (*)(void))FormFunction,&appctx);CHKERRQ(ierr); >>> ierr = MatFDColoringSetFromOptions(matfdcoloring);CHKERRQ(ierr); >>> ierr = TSSetRHSJacobian(ts,J,J,TSDefaultComputeJacobianColor,matfdcoloring);CHKERRQ(ierr); >>> >>> dt = appctx.dtmin; >>> ierr = TSSetInitialTimeStep(ts,0.0,dt);CHKERRQ(ierr); >>> ierr = TSSetType(ts,TS_BEULER);CHKERRQ(ierr); >>> >>> Then I tried to add the snes monitor: >>> PetscViewerASCIIOpen(PETSC_COMM_WORLD,"snes.log",&rviewer); >>> ierr = TSGetSNES(ts,&ts_snes); >>> ierr = SNESMonitorSet(ts_snes,SNESMonitorDefault,&rviewer,PETSC_NULL); >>> I got the snes.log with long lines which started with countless blank/space and ended at the very end with the SNES iteration info. As a result, the file took a lot of memory space even with one line of outputs. >>> >>> Please let me know if I did something wrong. >>> >>> Have a nice weekend! >>> >>> >>> _________________________________________________________________ >>> �°��ֻ�MSN��󣡲μ����ȡ��ɫ��Ʒ�� >>> http://mobile.msn.com.cn/ >>> >>> > > > > -- > Lisandro Dalc��n > --------------- > Centro Internacional de M��todos Computacionales en Ingenier��a (CIMEC) > Instituto de Desarrollo Tecnol��gico para la Industria Qu��mica (INTEC) > Consejo Nacional de Investigaciones Cient��ficas y T��cnicas (CONICET) > PTLC - G��emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > >

1 0

PETSc on GPUs
by Harald Pfeiffer 17 Sep '08

17 Sep '08

Hi, do you know whether there are any efforts to run PETSc on GPUs (graphical processing units)? Thanks, Harald -- Harald P. Pfeiffer harald(a)tapir.caltech.edu Theoretical Astrophysics Phone (626) 395-8413 Caltech 130-33 Fax (626) 796-5675 Pasadena, CA 91125, USA

1 0

Re: Parallel direct solvers with PFLOTRAN
by Barry Smith 17 Sep '08

17 Sep '08

Given that constructing the LU factorization will be time- consuming you may want to also try lagging the computation of the LU, for example, to have the LU done once for an entire Newton solve you can use -flow_ksp_type ibcgs (or bcgs) -flow_snes_lag_preconditioner 100 (some large number here) To keep the LU factors the same for several timesteps you can call SNESSetLagPreconditioner(snes,-1,ierr) Then you need to call with SNESSetLagPreconditioner(snes,largenumber,ierr) to trigger a recompute and then before the next SNESSolve call with -1 to stop the recomputation. Barry I realize I need to add a SNESSetLagPreconditioner(snes,-2,ierr) to trigger a single recomputation of the LU. I will do this as soon as I can. On Sep 16, 2008, at 11:00 PM, Richard Tran Mills wrote: > Peter and Glenn, > > You had asked about using parallel direct solvers with PFLOTRAN. If > you install a package such as mumps or superlu_dist with the > corresponding PETSc interfaces, you can easily try using a direct > solver using command line options such as the following: > > -flow_mat_type mpiaij -flow_ksp_type preonly -flow_pc_type lu - > flow_pc_factor_mat_solver_package mumps > > You'll need to pull and update to include the push I made tonight > that allows you to specify the matrix type on the command line. > (Things default to BAIJ, which isn't supported for the direct > solvers.) > > The above looks pretty messy but we can worry about making a cleaner > way to do this if this turns out to be something that you might want > to do on a routine basis. > > I have tested PFLOTRAN with both MUMPS and SuperLU_dist on a Linux > workstation using the very simple 'TAO/100_10_10' problem in the > examples directory. This works fine. I was having some problems > getting things to work properly on Jaguar (no surprise) and wasn't > quite able to get things resolved before the machine went down for > most of today. I will try again tomorrow. > > In your own experiments, I would start with trying out MUMPS, as > anecdotal evidence suggests that it is fairly robust. > > --Richard >

1 0

Re: About SNES solver of TS objects
by Lisandro Dalcin 16 Sep '08

16 Sep '08

BTW, perhaps you should do the same in TSGetKSP(). What do you think? 2008/9/17 Hong Zhang <hzhang(a)mcs.anl.gov>: > > TSGetSNES() must be called after TSSetType(), > otherwise the nonlinear context 'snes' is > being created. > Attached please find updated ex7.c, which > illustrates how to set uer-defined SNESMonitor(). > > This example is pushed to petsc-dev. > I also added an error flag in petsc-dev to prevent > calling of TSGetSNES() before TSSetType(). > Thanks for reporting the problem! > > Best, > > Hong > > On Fri, 12 Sep 2008, tsjb00 wrote: > >> >> Many thanks for your reply! The code I tried was very similar to the code in: >> ts/examples/tutorials/ex7.c >> except that I changed the r.h.s. function. >> >> The code related to TS: >> ierr = TSCreate(PETSC_COMM_WORLD,&ts);CHKERRQ(ierr); >> ierr = TSSetProblemType(ts,TS_NONLINEAR);CHKERRQ(ierr); >> ierr = TSSetRHSFunction(ts,FormFunction,&appctx);CHKERRQ(ierr); >> >> ierr = DAGetColoring(appctx.da,IS_COLORING_GLOBAL,&iscoloring);CHKERRQ(ierr); >> ierr = MatFDColoringCreate(J,iscoloring,&matfdcoloring);CHKERRQ(ierr); >> ierr = ISColoringDestroy(iscoloring);CHKERRQ(ierr); >> ierr = MatFDColoringSetFunction(matfdcoloring,(PetscErrorCode (*)(void))FormFunction,&appctx);CHKERRQ(ierr); >> ierr = MatFDColoringSetFromOptions(matfdcoloring);CHKERRQ(ierr); >> ierr = TSSetRHSJacobian(ts,J,J,TSDefaultComputeJacobianColor,matfdcoloring);CHKERRQ(ierr); >> >> dt = appctx.dtmin; >> ierr = TSSetInitialTimeStep(ts,0.0,dt);CHKERRQ(ierr); >> ierr = TSSetType(ts,TS_BEULER);CHKERRQ(ierr); >> >> Then I tried to add the snes monitor: >> PetscViewerASCIIOpen(PETSC_COMM_WORLD,"snes.log",&rviewer); >> ierr = TSGetSNES(ts,&ts_snes); >> ierr = SNESMonitorSet(ts_snes,SNESMonitorDefault,&rviewer,PETSC_NULL); >> I got the snes.log with long lines which started with countless blank/space and ended at the very end with the SNES iteration info. As a result, the file took a lot of memory space even with one line of outputs. >> >> Please let me know if I did something wrong. >> >> Have a nice weekend! >> >> >> _________________________________________________________________ >> �°��ֻ�MSN��󣡲μ����ȡ��ɫ��Ʒ�� >> http://mobile.msn.com.cn/ >> >> -- Lisandro Dalc��n --------------- Centro Internacional de M��todos Computacionales en Ingenier��a (CIMEC) Instituto de Desarrollo Tecnol��gico para la Industria Qu��mica (INTEC) Consejo Nacional de Investigaciones Cient��ficas y T��cnicas (CONICET) PTLC - G��emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594

1 0