Some general questions
Hello, I have a few questions on how to improve performance of my program. I'm solving Poisson's equation on a (large) 3D FD grid with Dirichlet boundary conditions and multiple right hand sides. I set up the matrix and everything's working fine so far, but I'm sure the solving process could go faster. I know multigrid is generally the best preconditioner in such a case and algebraic multigrid currently works best. So generally speaking: Should I make the effort of symmetrizising the system matrix? I know how to do it, but it would probably take some time. CG does currently work, but is not competitive against other methods, so I guess the matrix might not be "symmetric enough"? For the various multigrid preconditioners: I always read that the problem should be solved exactly on the coarsest grid, but wouldn't an iterative solver do the same job if its provided accuracy is high enough, since the coarse discretization and the subsequent interpolation process introduce errors themselves? I submit my program to a batch system, but PETSc was compiled on the login node with different hardware. Is this affecting performance? What parts of the configuration process should I perform on a compute node then? Thanks.
On 12 May 2017 at 07:50, Matt Baker <[email protected]> wrote:
Hello,
I have a few questions on how to improve performance of my program. I'm solving Poisson's equation on a (large) 3D FD grid with Dirichlet boundary conditions and multiple right hand sides. I set up the matrix and everything's working fine so far, but I'm sure the solving process could go faster. I know multigrid is generally the best preconditioner in such a case and algebraic multigrid currently works best.
If you use a DMDA for your FD problem, consider using PCMG with Galerkin. It will set up a geometric multigrid hierarchy. Depending on the specifics of your Poisson problem (constant coefficient versus highly hetegoneous), geometric MG is likely superior (faster time to solution) than AMG.
So generally speaking:
Should I make the effort of symmetrizising the system matrix? I know how to do it, but it would probably take some time. CG does currently work, but is not competitive against other methods, so I guess the matrix might not be "symmetric enough"?
In its basic form, CG is only guaranteed to converge with with an SPD operator. If you want to use CG, definitely do the work and make the operator symmetric.
For the various multigrid preconditioners: I always read that the problem should be solved exactly on the coarsest grid, but wouldn't an iterative solver do the same job if its provided accuracy is high enough, since the coarse discretization and the subsequent interpolation process introduce errors themselves?
Yes iterative can work well. If your Poisson problem has a constant coefficient, rtol 1.0e-1 is likely a sufficient tolerance to use for an the coarse grid solve (e.g. overall convergence of solve won't be affected). If the Poisson problem has a highly variable coefficient (jumps of O(1e3) or more), or it has very large gradients say 1e3 variation over a few cells, then you will have to perform a more accurate iterative coarse level solve (say rtol 1e-4 to 1e-6). Note that the numbers for rtol I quote are purely empirical.
I submit my program to a batch system, but PETSc was compiled on the login node with different hardware. Is this affecting performance? What parts of the configuration process should I perform on a compute node then?
If the login and compute nodes are fundamentally different, you should configure petsc with the option --with-batch and following the instructions. Thanks, Dave
Thanks.
On Fri, May 12, 2017 at 2:25 AM, Dave May <[email protected]> wrote:
On 12 May 2017 at 07:50, Matt Baker <[email protected]> wrote:
Hello,
I have a few questions on how to improve performance of my program. I'm solving Poisson's equation on a (large) 3D FD grid with Dirichlet boundary conditions and multiple right hand sides. I set up the matrix and everything's working fine so far, but I'm sure the solving process could go faster. I know multigrid is generally the best preconditioner in such a case and algebraic multigrid currently works best.
If you use a DMDA for your FD problem, consider using PCMG with Galerkin. It will set up a geometric multigrid hierarchy. Depending on the specifics of your Poisson problem (constant coefficient versus highly hetegoneous), geometric MG is likely superior (faster time to solution) than AMG.
So generally speaking:
Should I make the effort of symmetrizising the system matrix? I know how to do it, but it would probably take some time. CG does currently work, but is not competitive against other methods, so I guess the matrix might not be "symmetric enough"?
In its basic form, CG is only guaranteed to converge with with an SPD operator. If you want to use CG, definitely do the work and make the operator symmetric.
For Poisson, CG is never, ever ever, ever ever faster than Full Multigrid (FMG). Don't use it. All the people publishing that are idiots :) but of course try it out for yourself with -pc_mg_type full It should converge to discretization error in 1 iterate if the smoother is strong enough (you might need to use 2 iterates on the downsmooth).
For the various multigrid preconditioners: I always read that the problem should be solved exactly on the coarsest grid, but wouldn't an iterative solver do the same job if its provided accuracy is high enough, since the coarse discretization and the subsequent interpolation process introduce errors themselves?
Yes iterative can work well. If your Poisson problem has a constant coefficient, rtol 1.0e-1 is likely a sufficient tolerance to use for an the coarse grid solve (e.g. overall convergence of solve won't be affected). If the Poisson problem has a highly variable coefficient (jumps of O(1e3) or more), or it has very large gradients say 1e3 variation over a few cells, then you will have to perform a more accurate iterative coarse level solve (say rtol 1e-4 to 1e-6). Note that the numbers for rtol I quote are purely empirical.
Always compare to direct. I don't think anything beats direct on problems the size of your coarse problem. If iterative is winning, likely your coarse problem is too big. However, again you can try it yourself easily with options. Matt
I submit my program to a batch system, but PETSc was compiled on the login node with different hardware. Is this affecting performance? What parts of the configuration process should I perform on a compute node then?
If the login and compute nodes are fundamentally different, you should configure petsc with the option --with-batch and following the instructions.
Thanks, Dave
Thanks.
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
Hello, just a quick question: The CG method is generally derived for spd matrices. However, the PETSc man page states Notes: The PCG method requires both the matrix and preconditioner to be symmetric positive (or negative) (semi) definite Only left preconditioning is supported. Does this mean CG works for semi-definite problems as well? Is that guaranteed then? Thanks.
On Thu, May 18, 2017 at 8:38 AM, Matt Baker <[email protected]> wrote:
Hello,
just a quick question:
The CG method is generally derived for spd matrices. However, the PETSc man page states
Notes: The PCG method requires both the matrix and preconditioner to be symmetric positive (or negative) (semi) definite Only left preconditioning is supported.
Does this mean CG works for semi-definite problems as well? Is that guaranteed then?
I believe that CG converges to A^+ b if b is in the range space of A, but I would have to look it up. Its probably in Hestenes, "Optimization theory: the finite dimensional case" 1975 Thanks, Matt
Thanks.
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
participants (3)
-
Dave May -
Matt Baker -
Matthew Knepley