Parallel matrix assembly questions
Hello, I need to build a dense matrix in parallel and looking at examples in ksp/examples/tutorials it seems there are two approaches. In ex3.c MPI_Comm_rank and MPI_Comm_size are used to compute the index range for the current processor and ex5.c uses MatGetOwnershipRange to figure out the index range. 1- Is there any advantage using either of these methods or they will behave the same when it comes to performance and portability ? Based on comments in ex5.c, PETSc partitions the matrix by contiguous chunks of rows. 2- Is there a way of changing this scheme using PETSc routines ? 3- If not, can I partition my matrix in a different way, say 2D block cycling method, and still be able to use PETSc to solve it ? Since my matrix is dense I am going to start with direct solvers and I am concerned whether the partitioning scheme will affect the solver's performance. thanks in advance, Hamid
On Jan 12, 2011, at 4:53 PM, Hamid M. wrote:
Hello,
I need to build a dense matrix in parallel and looking at examples in ksp/examples/tutorials it seems there are two approaches. In ex3.c MPI_Comm_rank and MPI_Comm_size are used to compute the index range for the current processor and ex5.c uses MatGetOwnershipRange to figure out the index range.
1- Is there any advantage using either of these methods or they will behave the same when it comes to performance and portability ?
Use MatGetOwnershipRange because it works with any Mat layout.
Based on comments in ex5.c, PETSc partitions the matrix by contiguous chunks of rows. 2- Is there a way of changing this scheme using PETSc routines ?
No
3- If not, can I partition my matrix in a different way, say 2D block cycling method, and still be able to use PETSc to solve it ?
No
Since my matrix is dense I am going to start with direct solvers and I am concerned whether the partitioning scheme will affect the solver's performance.
It will affect performance, unless you hope/plan to use iterative solvers there is no good reason to use PETSc for dense matrices with direct solvers. That is another world with a different world of software. Barry
thanks in advance, Hamid
It will affect performance, unless you hope/plan to use iterative solvers there is no good reason to use PETSc for dense matrices with direct solvers. That is another world with a different world of software.
Thanks for the helpful response. Just from theoretical point of view, are iterative solvers suitable for a dense matrix or it doesn't really matter ? I know direct solvers would perform faster for problems with multiple RHS values and a fixed LHS matrix, but how what are the main factors one needs to consider when dealing with dense matrices and various solvers. Hamid
On Jan 12, 2011, at 5:31 PM, Hamid M. wrote:
It will affect performance, unless you hope/plan to use iterative solvers there is no good reason to use PETSc for dense matrices with direct solvers. That is another world with a different world of software.
Thanks for the helpful response.
Just from theoretical point of view, are iterative solvers suitable for a dense matrix
It depends on the conditioning of the matrix. For some matrices (for example from some boundary element methods) the system can be solved with a couple dozen GMES iterations which is much much faster than using a direct solver. For other problems GMRES would be slower than a direct solver. Barry
or it doesn't really matter ? I know direct solvers would perform faster for problems with multiple RHS values and a fixed LHS matrix, but how what are the main factors one needs to consider when dealing with dense matrices and various solvers.
Hamid
participants (2)
-
Barry Smith -
Hamid M.