February 2012 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] DMGetMatrix --> DMGetMatrices?
by Jed Brown 10 Feb '12

10 Feb '12

On Fri, Feb 10, 2012 at 16:06, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: > I'm not sure I follow. If a matrix has been created, its type has been > set and it has been preallocated (e.g., MATMPIAIJ in > ksp/ksp/examples/tutorials/ex2), it is not marked as assembled until after > MatAssemblyEnd. Does it still make sense to duplicate it? In my opinion, > it does. > MatDuplicateOption specifies what to copy, but all cases are supposed to provide a correct nonzero pattern.

1 0

Re: [petsc-dev] PETSc GPU capabilities
by Jed Brown 10 Feb '12

10 Feb '12

On Fri, Feb 10, 2012 at 16:04, Paul Mullowney <paulm(a)txcorp.com> wrote: > (2) Triangular Solve used in ILU preconditioners; i.e. MatSolve. The > performance of this ... is what it is :| Is this a custom sparse triangular solve or is it from CUSPARSE? If different, how does the performance compare with the CUSPARSE implementation?

1 0

Re: [petsc-dev] PETSc GPU capabilities
by Paul Mullowney 10 Feb '12

10 Feb '12

One other point, my code supports (1) CUDA 4.1, Thrust 1.5, CUSP 0.3 or (2) CUDA 4.0, Thurst 1.4, CUSP 0.2 While it may work with mercurial versions of Thrust and CUSP, I make no guarantees. -Paul > Hi All, > > I've been developing GPU capabilities for PETSc. The development has > focused mostly on > (1) An efficient multi-GPU SpMV, i.e. MatMult. This is working well. > (2) Triangular Solve used in ILU preconditioners; i.e. MatSolve. The > performance of this ... is what it is :| > This code is in beta mode. Keep that in mind, if you decide to use it. > It supports single and double precision, real numbers only! Complex > will be supported at some point in the future, but not any time soon. > > To build with these capabilities, add the following to your configure > line. > --download-txpetscgpu=yes > > The capabilities of the SpMV code are accessed with the following 2 > command line flags > -cusp_storage_format csr (other options are coo (coordinate), ell > (ellpack), dia (diagonal). hyb (hybrid) is not yet supported) > -use_cusparse (this is a boolean and at the moment is only supported > with csr format matrices. In the future, cusparse will work with ell, > coo, and hyb formats). > > Regarding the number of GPUs to run on: > Imagine a system with P nodes, N cores per node, and M GPUs per node. > Then, to use only the GPUs, I would run with M ranks per node over P > nodes. As an example, I have a system with 2 nodes. Each node has 8 > cores, and 4 GPUs attached to each node (P=2, N=8, M=4). In a PBS > queue script, one would use 2 nodes at 4 processors per node. Each mpi > rank (CPU processor) will be attached to a GPU. > > You do not need to explicitly manage the GPUs, apart from > understanding what type of system you are running on. To learn how > many devices are available per node, use the command line flag: > -cuda_show_devices > > -Paul

1 0

Re: [petsc-dev] DMGetMatrix --> DMGetMatrices?
by Dmitry Karpeev 10 Feb '12

10 Feb '12

On Fri, Feb 10, 2012 at 4:05 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Fri, Feb 10, 2012 at 15:59, Matthew Knepley <knepley(a)gmail.com> wrote: > >> This makes no sense to me unless the two matrices being returned have >> different sparsity patterns, and we have no >> good way for specifying that, or even a good rationale for doing it. >> > > Preconditioning with a STAR stencil is often pretty good. > > Dmitry is suggesting specifying it by having two "slots" in which we can > request a matrix. > > DMSetMatrixPreallocateOnly(), DMSetMatrixType(), and perhaps a couple > others would need two arguments so the two matrices can be addressed > separately. > > It's not clear to me that two is always the right number of ways to > distinguish matrices. > Well, two is the number that's used in KSP and SNES. With the current API SNES actually uses the same matrix for the Jacobian and preconditioner, if the matrix is obtained from a DM, which is limiting. It is natural to have a "geometric" backend (e.g., libMesh or MOOSE) interact with the algebraic objects (PETSc's SNES, KSP) through the DM as much as possible, since the DM (its impl, rather) carries all of the necessary mesh and discretization information in one place. Yet, currently this interaction is limited as compared to the "bare" SNES/KSP interface, in part because of the one matrix limitation. Dmitry.

1 0

Re: [petsc-dev] DMGetMatrix --> DMGetMatrices?
by Dmitry Karpeev 10 Feb '12

10 Feb '12

On Fri, Feb 10, 2012 at 4:02 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Fri, Feb 10, 2012 at 15:59, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: > >> As a separate matter, however, should preallocated, but not assembled >> matrices be able to be duplicated or copied? >> > > It doesn't even make semantic sense for Copy. > Fine. > > Duplicate only makes sense in that the sizes and types get set, but > obviously the sparsity can't be set, so it doesn't have the (stated) > semantics of Duplicate. > I'm not sure I follow. If a matrix has been created, its type has been set and it has been preallocated (e.g., MATMPIAIJ in ksp/ksp/examples/tutorials/ex2), it is not marked as assembled until after MatAssemblyEnd. Does it still make sense to duplicate it? In my opinion, it does. Dmitry. > > As currently implemented, MatDuplicate() and MatCopy() will refuse to >> operate on a matrix that is !assembled. >> > >

1 0

Re: [petsc-dev] DMGetMatrix --> DMGetMatrices?
by Jed Brown 10 Feb '12

10 Feb '12

On Fri, Feb 10, 2012 at 15:59, Matthew Knepley <knepley(a)gmail.com> wrote: > This makes no sense to me unless the two matrices being returned have > different sparsity patterns, and we have no > good way for specifying that, or even a good rationale for doing it. > Preconditioning with a STAR stencil is often pretty good. Dmitry is suggesting specifying it by having two "slots" in which we can request a matrix. DMSetMatrixPreallocateOnly(), DMSetMatrixType(), and perhaps a couple others would need two arguments so the two matrices can be addressed separately. It's not clear to me that two is always the right number of ways to distinguish matrices.

1 0

PETSc GPU capabilities
by Paul Mullowney 10 Feb '12

10 Feb '12

Hi All, I've been developing GPU capabilities for PETSc. The development has focused mostly on (1) An efficient multi-GPU SpMV, i.e. MatMult. This is working well. (2) Triangular Solve used in ILU preconditioners; i.e. MatSolve. The performance of this ... is what it is :| This code is in beta mode. Keep that in mind, if you decide to use it. It supports single and double precision, real numbers only! Complex will be supported at some point in the future, but not any time soon. To build with these capabilities, add the following to your configure line. --download-txpetscgpu=yes The capabilities of the SpMV code are accessed with the following 2 command line flags -cusp_storage_format csr (other options are coo (coordinate), ell (ellpack), dia (diagonal). hyb (hybrid) is not yet supported) -use_cusparse (this is a boolean and at the moment is only supported with csr format matrices. In the future, cusparse will work with ell, coo, and hyb formats). Regarding the number of GPUs to run on: Imagine a system with P nodes, N cores per node, and M GPUs per node. Then, to use only the GPUs, I would run with M ranks per node over P nodes. As an example, I have a system with 2 nodes. Each node has 8 cores, and 4 GPUs attached to each node (P=2, N=8, M=4). In a PBS queue script, one would use 2 nodes at 4 processors per node. Each mpi rank (CPU processor) will be attached to a GPU. You do not need to explicitly manage the GPUs, apart from understanding what type of system you are running on. To learn how many devices are available per node, use the command line flag: -cuda_show_devices -Paul

1 0

Re: [petsc-dev] DMGetMatrix --> DMGetMatrices?
by Dmitry Karpeev 10 Feb '12

10 Feb '12

On Fri, Feb 10, 2012 at 4:00 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Fri, Feb 10, 2012 at 15:58, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: > >> It seems to me that the sparsity pattern (and even the type of the >> matrices) is up to the particular DM type, >> so it is, in principle, opaque. DMDA can control it by manipulating >> stencil width or widths, while FEM-oriented DM >> types (e.g., libMesh) could control it via the element type, etc. The >> generic DM interface doesn't have a way to specify >> the sparsity pattern. As it should be, in my opinion. >> > > Yes, it's fine for DM to have a way to allocate a sparser preconditioning > matrix. The most common case in practice is MFFD for the true Jacobian and > more compact stencil for the preconditioner. It just makes the interface > and implementation a bit ugly/confusing to support both. It would be nice > to have a more elegant way to state it. > No uglier than having SNES/KSP take in two matrices, in my view. Dmitry.

1 0

Re: [petsc-dev] DMGetMatrix --> DMGetMatrices?
by Jed Brown 10 Feb '12

10 Feb '12

On Fri, Feb 10, 2012 at 15:59, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: > As a separate matter, however, should preallocated, but not assembled > matrices be able to be duplicated or copied? > It doesn't even make semantic sense for Copy. Duplicate only makes sense in that the sizes and types get set, but obviously the sparsity can't be set, so it doesn't have the (stated) semantics of Duplicate. As currently implemented, MatDuplicate() and MatCopy() will refuse to > operate on a matrix that is !assembled. >

1 0

Re: [petsc-dev] DMGetMatrix --> DMGetMatrices?
by Dmitry Karpeev 10 Feb '12

10 Feb '12

On Fri, Feb 10, 2012 at 3:59 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > On Fri, Feb 10, 2012 at 3:58 PM, Dmitry Karpeev <karpeev(a)mcs.anl.gov>wrote: > >> >> >> On Fri, Feb 10, 2012 at 3:53 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: >> >>> On Fri, Feb 10, 2012 at 15:50, Dmitry Karpeev <karpeev(a)mcs.anl.gov>wrote: >>> >>>> Why not simply allow DM to return two matrices as SNES and KSP would >>>> want anyway? >>> >>> >>> How do you specify the reduced stencil to use for the preconditioning >>> matrix? Maybe it wouldn't add too many more if statements to the current >>> code which is already polluted with them. >>> >> >> It seems to me that the sparsity pattern (and even the type of the >> matrices) is up to the particular DM type, >> so it is, in principle, opaque. DMDA can control it by manipulating >> stencil width or widths, while FEM-oriented DM >> types (e.g., libMesh) could control it via the element type, etc. The >> generic DM interface doesn't have a way to specify >> the sparsity pattern. As it should be, in my opinion. >> > > This makes no sense to me unless the two matrices being returned have > different sparsity patterns, and we have no > good way for specifying that, or even a good rationale for doing it. > The rationale is that the preconditioning matrix can have a different sparsity pattern. L2 projection leads to a mass matrix, and the lumped mass matrix is a good preconditioner. It is not DM's job to specify the sparsity pattern -- it is up to a particular implementation. Dmitry. > > Matt > > >> Dmitry. >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener >

1 0