December 2011 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Question About Petsc ILU
by Dave Nystrom 29 Dec '11

29 Dec '11

Jed Brown writes: > On Thu, Dec 29, 2011 at 21:42, Dave Nystrom <Dave.Nystrom(a)tachyonlogic.com> wrote: > > > I just browsed through the output of configure -help and don't see > > suitesparse. I do see stuff for cholmod but not a download option for > > cholmod. > > You have to install it yourself. Tim Davis' installation plan is too much > work to maintain separately for each of his packages separately. PETSc > should have a --download-suitesparse, but we haven't written it yet, so you > currently have to install it yourself. Linux distros have it in their > package managers anyway. OK. Thanks. I did not realize that was what you meant. I'll check for it in my distro and/or download and install it. Thanks for the tip.

1 0

Re: [petsc-dev] Question About Petsc ILU
by Jed Brown 29 Dec '11

29 Dec '11

On Thu, Dec 29, 2011 at 21:42, Dave Nystrom <Dave.Nystrom(a)tachyonlogic.com>wrote: > I just browsed through the output of configure -help and don't see > suitesparse. I do see stuff for cholmod but not a download option for > cholmod. > You have to install it yourself. Tim Davis' installation plan is too much work to maintain separately for each of his packages separately. PETSc should have a --download-suitesparse, but we haven't written it yet, so you currently have to install it yourself. Linux distros have it in their package managers anyway.

1 0

Re: [petsc-dev] Question About Petsc ILU
by Dave Nystrom 29 Dec '11

29 Dec '11

Jed Brown writes: > On Thu, Dec 29, 2011 at 19:30, Dave Nystrom > <Dave.Nystrom(a)tachyonlogic.com>wrote: > > > Generally I use CG and LU from petsc. Cholesky runs slower than LU and I > > was > > told in a previous email that it was because of the extra data movement for > > Cholesky versus LU. I have also tried minres instead of CG and tend to get > > reduced iteration counts. But I think minres may have an extra dot product > > and so is a little more expensive than cg. But minres is usually cheaper > > over all because of the reduction in iterations. > > > > If you are concerned about solve time for serial runs, I would either use > Cholesky from MUMPS or CHOLMOD (install suitesparse, then -pc_type cholesky > -pc_factor_mat_solver_type cholmod). I just browsed through the output of configure -help and don't see suitesparse. I do see stuff for cholmod but not a download option for cholmod.

1 0

Re: [petsc-dev] Question About Petsc ILU
by Dave Nystrom 29 Dec '11

29 Dec '11

Jed Brown writes: > On Thu, Dec 29, 2011 at 19:30, Dave Nystrom > <Dave.Nystrom(a)tachyonlogic.com>wrote: > > > Generally I use CG and LU from petsc. Cholesky runs slower than LU and I > > was > > told in a previous email that it was because of the extra data movement for > > Cholesky versus LU. I have also tried minres instead of CG and tend to get > > reduced iteration counts. But I think minres may have an extra dot product > > and so is a little more expensive than cg. But minres is usually cheaper > > over all because of the reduction in iterations. > > > > If you are concerned about solve time for serial runs, I would either use > Cholesky from MUMPS or CHOLMOD (install suitesparse, then -pc_type cholesky > -pc_factor_mat_solver_type cholmod). Thanks. I am very interested in improving the direct solution time for my Hall matrix problem for use in smaller runs. My impression is that sparse direct solves don't really scale to large problems but maybe it will be good enough for this 2d code. I will try suitesparse and cholmod. I have been having problems with mumps but will try it again and at least document the problem on petsc-maint. So far, I have tried with success the following sparse direct solvers from petsc: petsc, superlu, spooles, umfpack. The best performer has been umfpack and it gives a speedup in solving my Hall matrix of between 2x and 3x when using sequential MKL for blas. But I would definitely like to get a result with mumps and cholmod. Thanks for the suggestion.

1 0

Re: [petsc-dev] Seeking Advice on Petsc Preconditioners to Try
by Mark F. Adams 29 Dec '11

29 Dec '11

On Dec 29, 2011, at 8:24 PM, Dave Nystrom wrote: > Mark F. Adams writes: >> On Dec 29, 2011, at 12:51 PM, Dave Nystrom wrote: >> >>> Jed Brown writes: >>>> On Thu, Dec 29, 2011 at 11:15, Dave Nystrom <dnystrom1(a)comcast.net> wrote: >>>> >>>>> I have tried out -ksp_type cg -pc_type gamg -pc_gamg_type sa on my problem >>>>> and am encouraged enough with the results that I would like to try taking >>>>> the >>>>> next step with using gamg. Could you provide some advice on how to do >>>>> that? >>>>> I'm not sure how to provide the null space info you say is important. >>>>> >>>> >>>> Are you solving a scalar or vector problem? >>> >>> Both although the most challenging problem right now is one for a Hall matrix >>> system that is block banded with 2x2 blocks. Right now, I am just inputting >>> the matrix into a petsc seqaij matrix. >> >> Thats fine, actually require. Just call this: >> >> ierr = MatSetBlockSize(Amat,3); CHKERRQ(ierr); > > Shouldn't it be > > ierr = MatSetBlockSize(Amat,2); CHKERRQ(ierr); > > since I have 2x2 blocks. yes > >>> I guess I need to put it into a petsc >>> block matrix but have not yet attempted to do that. I'm not sure how much >>> work is involved - I need to read the section in the users manual on that. >>> >>> Anyway, I have 5 scalar systems and on vector system with 2x2 blocks. I am >>> interested in applying gamg to all of them to see how well it works. >>> >> >> >

1 0

Re: [petsc-dev] Question About Petsc ILU
by Jed Brown 29 Dec '11

29 Dec '11

On Thu, Dec 29, 2011 at 19:30, Dave Nystrom <Dave.Nystrom(a)tachyonlogic.com>wrote: > Generally I use CG and LU from petsc. Cholesky runs slower than LU and I > was > told in a previous email that it was because of the extra data movement for > Cholesky versus LU. I have also tried minres instead of CG and tend to get > reduced iteration counts. But I think minres may have an extra dot product > and so is a little more expensive than cg. But minres is usually cheaper > over all because of the reduction in iterations. > If you are concerned about solve time for serial runs, I would either use Cholesky from MUMPS or CHOLMOD (install suitesparse, then -pc_type cholesky -pc_factor_mat_solver_type cholmod).

1 0

Re: [petsc-dev] Question About Petsc ILU
by Dave Nystrom 29 Dec '11

29 Dec '11

Mark F. Adams writes: > > On Dec 29, 2011, at 12:45 PM, Dave Nystrom wrote: > > > Jed Brown writes: > >> On Thu, Dec 29, 2011 at 11:01, Dave Nystrom <dnystrom1(a)comcast.net> wrote: > >> > >>> I have recently added the capability to have a separate preconditioning > >>> matrix in the petsc interface for the code I am working with. I have two > >>> types of preconditioning matrices that I have questions about. One is > >>> tridiagonal and the other is 7 diagonals. In both cases, the the diagonals > >>> are all lexically adjacent. Or phrased differently, the tridiagonal matrix > >>> has a bandwidth of 3 and the 7 diagonal matrix has a bandwidth of 7 and so > >>> they are compact or dense band systems. > >>> > >>> I was wondering what petsc ilu will do for preconditioning matrices like > >>> these. Will it produce an exact lu factorization or a nearly exact > >>> factorization? > >> > >> Yes > >> > >>> I'm interested in the answer to this question because I am thinking I > >>> might be able to run this preconditioner on the gpu using the txpetscgpu > >>> package. > >> > >> Likely pointless because this solve is probably not a big part of run-time. > >> The bigger concern is the convergence rate that you lose by using this > >> approximation. Matt and I mentioned this the last time you brought it up, > >> but I recommend getting familiar with the literature. Then, get the math > >> straight before looking for ways to transform into problems that you can > >> run fast on a GPU or whatever. If you just optimize kernels, you're likely > >> to optimize something that takes a small part of run-time and isn't really > >> helping you anyway. > > > > So the application code is one that I am not the primary author on and it is > > a 2d resistive MHD with extensions code for magnetized fusion problems. > > Magnetized fusion problems tend to be highly anisotropic with qualitatively > > different physics parallel and perpendicular to the magnetic field. The code > > has 3 approaches to solving the linear systems. One is a suite of native cg > > solvers that use a cholesky solver for the inner band of the matrix as the > > preconditioner. As noted, this preconditioner ignores the coupling to the > > second coordinate. However, this preconditioner produces the lowest > > iteration count of the various preconditioner approaches that I have tried. > > > > This sounds like it is and SPD matrix. I would think GAMG would be OK. > If you can set -pc_gamg_verbose -ksp_monitor' and send me the output I can > see of anything jumps out. Sure. I'll try this as soon as I can. > > The second solver approach is use of the agmg package which works well on all > > of the linear solves except for the Hall matrix. For the Hall matrix, agmg > > fails to produce a solution. > > > > Are you using CG and Cholesky for the Hall matrix? If so it is probably a > div(alpha(x)grad(u)). Generally I use CG and LU from petsc. Cholesky runs slower than LU and I was told in a previous email that it was because of the extra data movement for Cholesky versus LU. I have also tried minres instead of CG and tend to get reduced iteration counts. But I think minres may have an extra dot product and so is a little more expensive than cg. But minres is usually cheaper over all because of the reduction in iterations. > Mark > > > The third approach is an interface to petsc that I have added in the last few > > months. So far, I have been able to get really good results with petsc on > > all of the linear systems except for the Hall matrix. In general, the > > fastest solution on these other linear systems has been petsc using jacobi > > preconditioning running on the gpu. However, I think I may be able to do > > even better on these other systems using petsc with the separate > > preconditioning matrix and using ilu plus txpetscgpu. > > > > But, having invested the time to interface to petsc, I am now trying to > > explore the vast set of options available to me to see what works best for my > > problems from the petsc toolbox. But solving the Hall matrix, especially for > > larger mesh sizes, still remains problematic. The lead author of this code > > is investigating the Hall matrix formulation to see if there are better ways > > to either formulate it or solve it and I am exploring what petsc has to offer > > for solving it. The matrix is very ill conditioned. The formulation is > > complicated enough that perhaps there is just a bug somewhere. But these > > sort of questions are being investigated by the main author of the code. > > > > Anyway, I have so far gotten good performance gains using the gpu with petsc > > but am totally open to and am trying to explore the whole petsc toolbox. > > >

1 0

Re: [petsc-dev] Seeking Advice on Petsc Preconditioners to Try
by Dave Nystrom 29 Dec '11

29 Dec '11

Mark F. Adams writes: > On Dec 29, 2011, at 12:51 PM, Dave Nystrom wrote: > > > Jed Brown writes: > >> On Thu, Dec 29, 2011 at 11:15, Dave Nystrom <dnystrom1(a)comcast.net> wrote: > >> > >>> I have tried out -ksp_type cg -pc_type gamg -pc_gamg_type sa on my problem > >>> and am encouraged enough with the results that I would like to try taking > >>> the > >>> next step with using gamg. Could you provide some advice on how to do > >>> that? > >>> I'm not sure how to provide the null space info you say is important. > >>> > >> > >> Are you solving a scalar or vector problem? > > > > Both although the most challenging problem right now is one for a Hall matrix > > system that is block banded with 2x2 blocks. Right now, I am just inputting > > the matrix into a petsc seqaij matrix. > > Thats fine, actually require. Just call this: > > ierr = MatSetBlockSize(Amat,3); CHKERRQ(ierr); Shouldn't it be ierr = MatSetBlockSize(Amat,2); CHKERRQ(ierr); since I have 2x2 blocks. > > I guess I need to put it into a petsc > > block matrix but have not yet attempted to do that. I'm not sure how much > > work is involved - I need to read the section in the users manual on that. > > > > Anyway, I have 5 scalar systems and on vector system with 2x2 blocks. I am > > interested in applying gamg to all of them to see how well it works. > > > >

1 0

Re: [petsc-dev] Question About Petsc ILU
by Mark F. Adams 29 Dec '11

29 Dec '11

On Dec 29, 2011, at 12:45 PM, Dave Nystrom wrote: > Jed Brown writes: >> On Thu, Dec 29, 2011 at 11:01, Dave Nystrom <dnystrom1(a)comcast.net> wrote: >> >>> I have recently added the capability to have a separate preconditioning >>> matrix in the petsc interface for the code I am working with. I have two >>> types of preconditioning matrices that I have questions about. One is >>> tridiagonal and the other is 7 diagonals. In both cases, the the diagonals >>> are all lexically adjacent. Or phrased differently, the tridiagonal matrix >>> has a bandwidth of 3 and the 7 diagonal matrix has a bandwidth of 7 and so >>> they are compact or dense band systems. >>> >>> I was wondering what petsc ilu will do for preconditioning matrices like >>> these. Will it produce an exact lu factorization or a nearly exact >>> factorization? >> >> Yes >> >>> I'm interested in the answer to this question because I am thinking I >>> might be able to run this preconditioner on the gpu using the txpetscgpu >>> package. >> >> Likely pointless because this solve is probably not a big part of run-time. >> The bigger concern is the convergence rate that you lose by using this >> approximation. Matt and I mentioned this the last time you brought it up, >> but I recommend getting familiar with the literature. Then, get the math >> straight before looking for ways to transform into problems that you can >> run fast on a GPU or whatever. If you just optimize kernels, you're likely >> to optimize something that takes a small part of run-time and isn't really >> helping you anyway. > > So the application code is one that I am not the primary author on and it is > a 2d resistive MHD with extensions code for magnetized fusion problems. > Magnetized fusion problems tend to be highly anisotropic with qualitatively > different physics parallel and perpendicular to the magnetic field. The code > has 3 approaches to solving the linear systems. One is a suite of native cg > solvers that use a cholesky solver for the inner band of the matrix as the > preconditioner. As noted, this preconditioner ignores the coupling to the > second coordinate. However, this preconditioner produces the lowest > iteration count of the various preconditioner approaches that I have tried. > This sounds like it is and SPD matrix. I would think GAMG would be OK. If you can set -pc_gamg_verbose -ksp_monitor' and send me the output I can see of anything jumps out. > The second solver approach is use of the agmg package which works well on all > of the linear solves except for the Hall matrix. For the Hall matrix, agmg > fails to produce a solution. > Are you using CG and Cholesky for the Hall matrix? If so it is probably a div(alpha(x)grad(u)). Mark > The third approach is an interface to petsc that I have added in the last few > months. So far, I have been able to get really good results with petsc on > all of the linear systems except for the Hall matrix. In general, the > fastest solution on these other linear systems has been petsc using jacobi > preconditioning running on the gpu. However, I think I may be able to do > even better on these other systems using petsc with the separate > preconditioning matrix and using ilu plus txpetscgpu. > > But, having invested the time to interface to petsc, I am now trying to > explore the vast set of options available to me to see what works best for my > problems from the petsc toolbox. But solving the Hall matrix, especially for > larger mesh sizes, still remains problematic. The lead author of this code > is investigating the Hall matrix formulation to see if there are better ways > to either formulate it or solve it and I am exploring what petsc has to offer > for solving it. The matrix is very ill conditioned. The formulation is > complicated enough that perhaps there is just a bug somewhere. But these > sort of questions are being investigated by the main author of the code. > > Anyway, I have so far gotten good performance gains using the gpu with petsc > but am totally open to and am trying to explore the whole petsc toolbox. >

1 0

Re: [petsc-dev] Seeking Advice on Petsc Preconditioners to Try
by Mark F. Adams 29 Dec '11

29 Dec '11

On Dec 29, 2011, at 12:51 PM, Dave Nystrom wrote: > Jed Brown writes: >> On Thu, Dec 29, 2011 at 11:15, Dave Nystrom <dnystrom1(a)comcast.net> wrote: >> >>> I have tried out -ksp_type cg -pc_type gamg -pc_gamg_type sa on my problem >>> and am encouraged enough with the results that I would like to try taking >>> the >>> next step with using gamg. Could you provide some advice on how to do >>> that? >>> I'm not sure how to provide the null space info you say is important. >>> >> >> Are you solving a scalar or vector problem? > > Both although the most challenging problem right now is one for a Hall matrix > system that is block banded with 2x2 blocks. Right now, I am just inputting > the matrix into a petsc seqaij matrix. Thats fine, actually require. Just call this: ierr = MatSetBlockSize(Amat,3); CHKERRQ(ierr); > I guess I need to put it into a petsc > block matrix but have not yet attempted to do that. I'm not sure how much > work is involved - I need to read the section in the users manual on that. > > Anyway, I have 5 scalar systems and on vector system with 2x2 blocks. I am > interested in applying gamg to all of them to see how well it works. >

1 0