February 2012 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Understanding Some Parallel Results with PETSc
by Nystrom, William D 23 Feb '12

23 Feb '12

Hi Mark, Thanks for the suggestions. Sounds like you would say that there is something wrong with the performance of the cpu only calculation. Is that a fair conclusion? I have been looking at smaller problem sizes since the original run. Limiting the iteration count also seems like a good way to look at the performance of larger problem sizes. Thanks again for your suggestions. Dave ________________________________ From: petsc-dev-bounces(a)mcs.anl.gov [petsc-dev-bounces(a)mcs.anl.gov] on behalf of Mark F. Adams [mark.adams(a)columbia.edu] Sent: Thursday, February 23, 2012 2:20 PM To: For users of the development version of PETSc Subject: Re: [petsc-dev] Understanding Some Parallel Results with PETSc The difference in performance for VecAXPY and VecAYPX is dramatic (~35x) and these are dead simple methods that are almost identical and are not parallel so they may be a good place to start looking. You might look a simpler example like src/vec/vec/example/tutorial/ex1.c. You could add a loop around the calls to VecAXPY and VecAYPX to get some meaningful timings. Also, you might limit the number of iterations to say 100, so it does not take 10 hours to run these tests. You could also try scaling the problem up (or down) to see when these problems kick in (eg, when you go off a node ...). Mark On Feb 23, 2012, at 2:49 PM, Nystrom, William D wrote: Hi Matt, Attached are the log files for the two runs. Thanks, Dave ________________________________ From: petsc-dev-bounces(a)mcs.anl.gov<mailto:[email protected]> [petsc-dev-bounces(a)mcs.anl.gov] on behalf of Matthew Knepley [knepley(a)gmail.com] Sent: Thursday, February 23, 2012 11:17 AM To: For users of the development version of PETSc Subject: Re: [petsc-dev] Understanding Some Parallel Results with PETSc On Thu, Feb 23, 2012 at 11:06 AM, Nystrom, William D <wdn(a)lanl.gov<mailto:[email protected]>> wrote: I recently ran a couple of test runs with petsc-dev that I do not understand. I'm running on a test bed machine that has 4 nodes with two Tesla 2090 gpus per node. Each node is dual socket and populated with Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz processors. These are 8 core processors and so each node has 16 cores. On the gpu, I'm running with Paul's latest version of the txpetscgpu package. I'm running the src/ksp/ksp/examples/tutorials/ex2.c petsc example with m=n=10000. My objective was to compare the performance running on 4 nodes using all 8 gpus to that of running on the same 4 nodes with all 64 cores. This problem uses about a third of the memory available on the gpus. I was using cg with jacobi preconditioning on both the gpu run and the cpu run. What is puzzling to me is that the cpu case ran 44x times slower than the gpu case and the big difference was in the time spend in functions like VecTDot, VecNorm and VecAXPY. Below is a table that summarizes the performance of the main functions that were using time in the two runs. Times are in seconds. | GPU | CPU | Ratio ------------------------------------------------------------------------- MatMult | 450.64 | 5484.7 | 12.17 ------------------------------------------------------------------------- VecTDot | 285.35 | 16688.0 | 58.48 ------------------------------------------------------------------------- VecNorm | 19.03 | 9058.8 | 476.03 ------------------------------------------------------------------------- VecAXPY | 106.88 | 5636.3 | 52.73 ------------------------------------------------------------------------- VecAYPX | 53.69 | 85.1 | 1.58 ------------------------------------------------------------------------- KSPSolve | 811.95 | 35930.0 | 44.25 ------------------------------------------------------------------------- The ratio of MatMult for CPU versus GPU is what I typically see when I am comparing a CPU run on a single core versus a run on a single GPU. Since both runs are communicating across node via mpi, I'm puzzled about why the CPU case is so much slower than the GPU case especially since there is communication for the MatMult as well. Both runs compute the same final error norm using the exact same number of iterations. Do these results make sense to people who understand the performance issues of parallel sparse linear solvers much better than I? Or do these results look abnormal. I had wondered if part of the performance issue was related to my running 8 times as many mpi processes for the CPU case. However, I ran a smaller problem with m=n=1000 and using 8 mpi processes and 2 cores per node and I see the same extreme differences in the times spent in VecTDot, VecNorm and VecAXPY. Here are the command lines I used for the two runs: CPU: mpirun -np 64 -mca btl self,sm,openib ex2 -m 10000 -n 10000 -ksp_type cg -ksp_max_it 100000 -pc_type jacobi -log_summary -options_left GPU: mpirun -np 8 -npernode 2 -mca btl self,sm,openib ex2 -m 10000 -n 10000 -ksp_type cg -ksp_max_it 100000 -pc_type jacobi -log_summary -options_left -mat_type aijcusp -vec_type cusp -cusp_storage_format dia 1) Always send -log_summary with performance questions 2) Comparing two things will not make any sense beyond "one ran faster" without a model for execution time 3) In order to make sense of my model, I need flop rates for those events Matt Thanks, Dave -- Dave Nystrom LANL HPC-5 Phone: 505-667-7913<tel:505-667-7913> Email: wdn(a)lanl.gov<mailto:[email protected]> Smail: Mail Stop B272 Group HPC-5 Los Alamos National Laboratory Los Alamos, NM 87545 -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener <ex2_10000_10000_cg_jacobi_mpi_64.log><ex2_10000_10000_cg_jacobi_cusp_dia_mpi_8.log>

1 0

Re: [petsc-dev] Understanding Some Parallel Results with PETSc
by Mark F. Adams 23 Feb '12

23 Feb '12

The difference in performance for VecAXPY and VecAYPX is dramatic (~35x) and these are dead simple methods that are almost identical and are not parallel so they may be a good place to start looking. You might look a simpler example like src/vec/vec/example/tutorial/ex1.c. You could add a loop around the calls to VecAXPY and VecAYPX to get some meaningful timings. Also, you might limit the number of iterations to say 100, so it does not take 10 hours to run these tests. You could also try scaling the problem up (or down) to see when these problems kick in (eg, when you go off a node ...). Mark On Feb 23, 2012, at 2:49 PM, Nystrom, William D wrote: > Hi Matt, > > Attached are the log files for the two runs. > > Thanks, > > Dave > > From: petsc-dev-bounces(a)mcs.anl.gov [petsc-dev-bounces(a)mcs.anl.gov] on behalf of Matthew Knepley [knepley(a)gmail.com] > Sent: Thursday, February 23, 2012 11:17 AM > To: For users of the development version of PETSc > Subject: Re: [petsc-dev] Understanding Some Parallel Results with PETSc > > On Thu, Feb 23, 2012 at 11:06 AM, Nystrom, William D <wdn(a)lanl.gov> wrote: > I recently ran a couple of test runs with petsc-dev that I do not understand. I'm running on a test bed > machine that has 4 nodes with two Tesla 2090 gpus per node. Each node is dual socket and populated > with Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz processors. These are 8 core processors and so each > node has 16 cores. On the gpu, I'm running with Paul's latest version of the txpetscgpu package. I'm > running the src/ksp/ksp/examples/tutorials/ex2.c petsc example with m=n=10000. My objective was > to compare the performance running on 4 nodes using all 8 gpus to that of running on the same 4 nodes > with all 64 cores. This problem uses about a third of the memory available on the gpus. I was using cg > with jacobi preconditioning on both the gpu run and the cpu run. What is puzzling to me is that the cpu > case ran 44x times slower than the gpu case and the big difference was in the time spend in functions > like VecTDot, VecNorm and VecAXPY. > > Below is a table that summarizes the performance of the main functions that were using time in the > two runs. Times are in seconds. > > | GPU | CPU | Ratio > ------------------------------------------------------------------------- > MatMult | 450.64 | 5484.7 | 12.17 > ------------------------------------------------------------------------- > VecTDot | 285.35 | 16688.0 | 58.48 > ------------------------------------------------------------------------- > VecNorm | 19.03 | 9058.8 | 476.03 > ------------------------------------------------------------------------- > VecAXPY | 106.88 | 5636.3 | 52.73 > ------------------------------------------------------------------------- > VecAYPX | 53.69 | 85.1 | 1.58 > ------------------------------------------------------------------------- > KSPSolve | 811.95 | 35930.0 | 44.25 > ------------------------------------------------------------------------- > > The ratio of MatMult for CPU versus GPU is what I typically see when I am comparing a CPU run on > a single core versus a run on a single GPU. Since both runs are communicating across node via mpi, > I'm puzzled about why the CPU case is so much slower than the GPU case especially since there is > communication for the MatMult as well. Both runs compute the same final error norm using the exact > same number of iterations. Do these results make sense to people who understand the performance > issues of parallel sparse linear solvers much better than I? Or do these results look abnormal. I had > wondered if part of the performance issue was related to my running 8 times as many mpi processes > for the CPU case. However, I ran a smaller problem with m=n=1000 and using 8 mpi processes and > 2 cores per node and I see the same extreme differences in the times spent in VecTDot, VecNorm > and VecAXPY. > > Here are the command lines I used for the two runs: > > CPU: > > mpirun -np 64 -mca btl self,sm,openib ex2 -m 10000 -n 10000 -ksp_type cg -ksp_max_it 100000 -pc_type jacobi -log_summary -options_left > > GPU: > > mpirun -np 8 -npernode 2 -mca btl self,sm,openib ex2 -m 10000 -n 10000 -ksp_type cg -ksp_max_it 100000 -pc_type jacobi -log_summary -options_left -mat_type aijcusp -vec_type cusp -cusp_storage_format dia > > 1) Always send -log_summary with performance questions > > 2) Comparing two things will not make any sense beyond "one ran faster" without a model for execution time > > 3) In order to make sense of my model, I need flop rates for those events > > Matt > > Thanks, > > Dave > > -- > Dave Nystrom > LANL HPC-5 > Phone: 505-667-7913 > Email: wdn(a)lanl.gov > Smail: Mail Stop B272 > Group HPC-5 > Los Alamos National Laboratory > Los Alamos, NM 87545 > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > <ex2_10000_10000_cg_jacobi_mpi_64.log><ex2_10000_10000_cg_jacobi_cusp_dia_mpi_8.log>

1 0

Re: [petsc-dev] non-symmetric permutation in PCFieldSplit
by Barry Smith 23 Feb '12

23 Feb '12

On Feb 23, 2012, at 3:08 PM, Jed Brown wrote: > On Thu, Feb 23, 2012 at 13:05, Jungho Lee <julee(a)mcs.anl.gov> wrote: > [C D;A B][x;y] = [v;w] (to be able to explore different > preconditioning strategies), which currently doesn't seem to be > supported. At first glance it seems like this can be easily solved by > adding extra IS (is) and PetscInt* (fields) objects in > _PC_FieldSplitLink, read in integer arrays for rows and columns (the > symmetric case where the rows and columns have the same indices is > already handled by -pc_fieldsplit_%d_fields), fill in the extra IS > object, and allow the user to use these two different IS's in the > MatGetSubMatrix calls in PCSetUp_FieldSplit. Is it really as simple as > that, or am I missing something here? > > I suggest leaving the original field ordering. Add a new function PCFieldSplitSetPermutation(PC,const PetscInt *rowperm,const PetscInt *colperm). Then your desired permutation is > > -pc_fieldsplit_permutation_row 1,0 > -pc_fieldsplit_permutation_col 0,1 I don't understand this. Will this only affect the -pc_fieldsplit_%d_fields and PCFieldSplitSetFields() commands and not PCFieldSplitIS()? What happens if one wants non symmetric access when using IS to provide the parts, won't you still need to provide 2 one for rows and one for columns? Barry > > When we refactor this, we have to get rid of the goofy restriction that Schur only works directly on 2x2 block systems, and we should really get rid of the distinction between Schur and relaxation. I think we should be able to select on a block-entry basis whether to use the entry that appears in a relaxation-based method (just that entry of the original matrix) or whether to use the block that appears in a factorization method (most blocks involve a solve or a Schur complement).

1 0

Re: [petsc-dev] non-symmetric permutation in PCFieldSplit
by Jed Brown 23 Feb '12

23 Feb '12

On Thu, Feb 23, 2012 at 13:05, Jungho Lee <julee(a)mcs.anl.gov> wrote: > [C D;A B][x;y] = [v;w] (to be able to explore different > preconditioning strategies), which currently doesn't seem to be > supported. At first glance it seems like this can be easily solved by > adding extra IS (is) and PetscInt* (fields) objects in > _PC_FieldSplitLink, read in integer arrays for rows and columns (the > symmetric case where the rows and columns have the same indices is > already handled by -pc_fieldsplit_%d_fields), fill in the extra IS > object, and allow the user to use these two different IS's in the > MatGetSubMatrix calls in PCSetUp_FieldSplit. Is it really as simple as > that, or am I missing something here? > I suggest leaving the original field ordering. Add a new function PCFieldSplitSetPermutation(PC,const PetscInt *rowperm,const PetscInt *colperm). Then your desired permutation is -pc_fieldsplit_permutation_row 1,0 -pc_fieldsplit_permutation_col 0,1 When we refactor this, we have to get rid of the goofy restriction that Schur only works directly on 2x2 block systems, and we should really get rid of the distinction between Schur and relaxation. I think we should be able to select on a block-entry basis whether to use the entry that appears in a relaxation-based method (just that entry of the original matrix) or whether to use the block that appears in a factorization method (most blocks involve a solve or a Schur complement).

1 0

Re: [petsc-dev] non-symmetric permutation in PCFieldSplit
by Barry Smith 23 Feb '12

23 Feb '12

Jungho will be adding this more generality and looking to simplify by removing the nfields, fields case from the code. For now we plan to keep the nfields, fields information because it is useful in the PCView() and provides much more information than the IS case. Eventually I guess we can augment the ISView() to provide this information and then totally remove the nfields, fields from the PC_FieldSplit object. Barry On Feb 23, 2012, at 1:14 PM, Matthew Knepley wrote: > On Thu, Feb 23, 2012 at 1:05 PM, Jungho Lee <julee(a)mcs.anl.gov> wrote: > I'd like to solve a system with a block structure with PCFieldSplit: > > [A B;C D][x;y] = [v;w] > > (I wrote it this way to simplify notation but x and y are in fact > interlaced, i.e., I'm solving a permuted version of the system above, > where the variables are laid out as x_1 y_1 x_2 y_2 ...., so the > system is fieldsplit-friendly) > > Applying PCFieldSplit to the equivalent system > > [D C;B A][y;x] = [w;v] > > can be done with command line options (-pc_fieldsplit_0_fields 1 > -pc_fieldsplit_1_fields 0), but I'm also interested in solving > > [C D;A B][x;y] = [v;w] (to be able to explore different > preconditioning strategies), which currently doesn't seem to be > supported. At first glance it seems like this can be easily solved by > adding extra IS (is) and PetscInt* (fields) objects in > _PC_FieldSplitLink, read in integer arrays for rows and columns (the > symmetric case where the rows and columns have the same indices is > already handled by -pc_fieldsplit_%d_fields), fill in the extra IS > object, and allow the user to use these two different IS's in the > MatGetSubMatrix calls in PCSetUp_FieldSplit. Is it really as simple as > that, or am I missing something here? > > No, I think that should definitely be available. However, I also think FS needs a > complete rewrite. There is way to much crud in there specific to co-located > discretizations on structured grids which should be moved to the DA. FS should > only takes ISes, which would make the code a LOT simpler. Then the DA has all > the nice routines for easily creating ISes for it. I am putting in the code for > DMComplex now, and it seems clear the interface for all DMs should be the same. > > Matt > > Thanks, > Jungho > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener

1 0

Re: [petsc-dev] Understanding Some Parallel Results with PETSc
by Nystrom, William D 23 Feb '12

23 Feb '12

Hi Matt, Attached are the log files for the two runs. Thanks, Dave ________________________________ From: petsc-dev-bounces(a)mcs.anl.gov [petsc-dev-bounces(a)mcs.anl.gov] on behalf of Matthew Knepley [knepley(a)gmail.com] Sent: Thursday, February 23, 2012 11:17 AM To: For users of the development version of PETSc Subject: Re: [petsc-dev] Understanding Some Parallel Results with PETSc On Thu, Feb 23, 2012 at 11:06 AM, Nystrom, William D <wdn(a)lanl.gov<mailto:[email protected]>> wrote: I recently ran a couple of test runs with petsc-dev that I do not understand. I'm running on a test bed machine that has 4 nodes with two Tesla 2090 gpus per node. Each node is dual socket and populated with Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz processors. These are 8 core processors and so each node has 16 cores. On the gpu, I'm running with Paul's latest version of the txpetscgpu package. I'm running the src/ksp/ksp/examples/tutorials/ex2.c petsc example with m=n=10000. My objective was to compare the performance running on 4 nodes using all 8 gpus to that of running on the same 4 nodes with all 64 cores. This problem uses about a third of the memory available on the gpus. I was using cg with jacobi preconditioning on both the gpu run and the cpu run. What is puzzling to me is that the cpu case ran 44x times slower than the gpu case and the big difference was in the time spend in functions like VecTDot, VecNorm and VecAXPY. Below is a table that summarizes the performance of the main functions that were using time in the two runs. Times are in seconds. | GPU | CPU | Ratio ------------------------------------------------------------------------- MatMult | 450.64 | 5484.7 | 12.17 ------------------------------------------------------------------------- VecTDot | 285.35 | 16688.0 | 58.48 ------------------------------------------------------------------------- VecNorm | 19.03 | 9058.8 | 476.03 ------------------------------------------------------------------------- VecAXPY | 106.88 | 5636.3 | 52.73 ------------------------------------------------------------------------- VecAYPX | 53.69 | 85.1 | 1.58 ------------------------------------------------------------------------- KSPSolve | 811.95 | 35930.0 | 44.25 ------------------------------------------------------------------------- The ratio of MatMult for CPU versus GPU is what I typically see when I am comparing a CPU run on a single core versus a run on a single GPU. Since both runs are communicating across node via mpi, I'm puzzled about why the CPU case is so much slower than the GPU case especially since there is communication for the MatMult as well. Both runs compute the same final error norm using the exact same number of iterations. Do these results make sense to people who understand the performance issues of parallel sparse linear solvers much better than I? Or do these results look abnormal. I had wondered if part of the performance issue was related to my running 8 times as many mpi processes for the CPU case. However, I ran a smaller problem with m=n=1000 and using 8 mpi processes and 2 cores per node and I see the same extreme differences in the times spent in VecTDot, VecNorm and VecAXPY. Here are the command lines I used for the two runs: CPU: mpirun -np 64 -mca btl self,sm,openib ex2 -m 10000 -n 10000 -ksp_type cg -ksp_max_it 100000 -pc_type jacobi -log_summary -options_left GPU: mpirun -np 8 -npernode 2 -mca btl self,sm,openib ex2 -m 10000 -n 10000 -ksp_type cg -ksp_max_it 100000 -pc_type jacobi -log_summary -options_left -mat_type aijcusp -vec_type cusp -cusp_storage_format dia 1) Always send -log_summary with performance questions 2) Comparing two things will not make any sense beyond "one ran faster" without a model for execution time 3) In order to make sense of my model, I need flop rates for those events Matt Thanks, Dave -- Dave Nystrom LANL HPC-5 Phone: 505-667-7913<tel:505-667-7913> Email: wdn(a)lanl.gov<mailto:[email protected]> Smail: Mail Stop B272 Group HPC-5 Los Alamos National Laboratory Los Alamos, NM 87545 -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] Understanding Some Parallel Results with PETSc
by Nystrom, William D 23 Feb '12

23 Feb '12

Yes, I ran with double precision. Thanks, Dave ________________________________ From: petsc-dev-bounces(a)mcs.anl.gov [petsc-dev-bounces(a)mcs.anl.gov] on behalf of recrusader [recrusader(a)gmail.com] Sent: Thursday, February 23, 2012 11:34 AM To: For users of the development version of PETSc Subject: Re: [petsc-dev] Understanding Some Parallel Results with PETSc Dear Dave, Did you run the codes with double precision? Thanks, Yujie On Thu, Feb 23, 2012 at 11:06 AM, Nystrom, William D <wdn(a)lanl.gov<mailto:[email protected]>> wrote: I recently ran a couple of test runs with petsc-dev that I do not understand. I'm running on a test bed machine that has 4 nodes with two Tesla 2090 gpus per node. Each node is dual socket and populated with Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz processors. These are 8 core processors and so each node has 16 cores. On the gpu, I'm running with Paul's latest version of the txpetscgpu package. I'm running the src/ksp/ksp/examples/tutorials/ex2.c petsc example with m=n=10000. My objective was to compare the performance running on 4 nodes using all 8 gpus to that of running on the same 4 nodes with all 64 cores. This problem uses about a third of the memory available on the gpus. I was using cg with jacobi preconditioning on both the gpu run and the cpu run. What is puzzling to me is that the cpu case ran 44x times slower than the gpu case and the big difference was in the time spend in functions like VecTDot, VecNorm and VecAXPY. Below is a table that summarizes the performance of the main functions that were using time in the two runs. Times are in seconds. | GPU | CPU | Ratio ------------------------------------------------------------------------- MatMult | 450.64 | 5484.7 | 12.17 ------------------------------------------------------------------------- VecTDot | 285.35 | 16688.0 | 58.48 ------------------------------------------------------------------------- VecNorm | 19.03 | 9058.8 | 476.03 ------------------------------------------------------------------------- VecAXPY | 106.88 | 5636.3 | 52.73 ------------------------------------------------------------------------- VecAYPX | 53.69 | 85.1 | 1.58 ------------------------------------------------------------------------- KSPSolve | 811.95 | 35930.0 | 44.25 ------------------------------------------------------------------------- The ratio of MatMult for CPU versus GPU is what I typically see when I am comparing a CPU run on a single core versus a run on a single GPU. Since both runs are communicating across node via mpi, I'm puzzled about why the CPU case is so much slower than the GPU case especially since there is communication for the MatMult as well. Both runs compute the same final error norm using the exact same number of iterations. Do these results make sense to people who understand the performance issues of parallel sparse linear solvers much better than I? Or do these results look abnormal. I had wondered if part of the performance issue was related to my running 8 times as many mpi processes for the CPU case. However, I ran a smaller problem with m=n=1000 and using 8 mpi processes and 2 cores per node and I see the same extreme differences in the times spent in VecTDot, VecNorm and VecAXPY. Here are the command lines I used for the two runs: CPU: mpirun -np 64 -mca btl self,sm,openib ex2 -m 10000 -n 10000 -ksp_type cg -ksp_max_it 100000 -pc_type jacobi -log_summary -options_left GPU: mpirun -np 8 -npernode 2 -mca btl self,sm,openib ex2 -m 10000 -n 10000 -ksp_type cg -ksp_max_it 100000 -pc_type jacobi -log_summary -options_left -mat_type aijcusp -vec_type cusp -cusp_storage_format dia Thanks, Dave -- Dave Nystrom LANL HPC-5 Phone: 505-667-7913<tel:505-667-7913> Email: wdn(a)lanl.gov<mailto:[email protected]> Smail: Mail Stop B272 Group HPC-5 Los Alamos National Laboratory Los Alamos, NM 87545

1 0

(no subject)
by Lisandro Dalcin 23 Feb '12

23 Feb '12

MatSetOption() seems to require previous preallocation (I mean, it has the MatCheckPreallocated() call). Isn't it too much? Do we really need a previous preallocation in order to set an option? I agree that many options are tightly related to nonzero pattern and memory allocation, but many options can be safely set before preallocation. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

1 0

Re: [petsc-dev] non-symmetric permutation in PCFieldSplit
by Matthew Knepley 23 Feb '12

23 Feb '12

On Thu, Feb 23, 2012 at 1:05 PM, Jungho Lee <julee(a)mcs.anl.gov> wrote: > I'd like to solve a system with a block structure with PCFieldSplit: > > [A B;C D][x;y] = [v;w] > > (I wrote it this way to simplify notation but x and y are in fact > interlaced, i.e., I'm solving a permuted version of the system above, > where the variables are laid out as x_1 y_1 x_2 y_2 ...., so the > system is fieldsplit-friendly) > > Applying PCFieldSplit to the equivalent system > > [D C;B A][y;x] = [w;v] > > can be done with command line options (-pc_fieldsplit_0_fields 1 > -pc_fieldsplit_1_fields 0), but I'm also interested in solving > > [C D;A B][x;y] = [v;w] (to be able to explore different > preconditioning strategies), which currently doesn't seem to be > supported. At first glance it seems like this can be easily solved by > adding extra IS (is) and PetscInt* (fields) objects in > _PC_FieldSplitLink, read in integer arrays for rows and columns (the > symmetric case where the rows and columns have the same indices is > already handled by -pc_fieldsplit_%d_fields), fill in the extra IS > object, and allow the user to use these two different IS's in the > MatGetSubMatrix calls in PCSetUp_FieldSplit. Is it really as simple as > that, or am I missing something here? > No, I think that should definitely be available. However, I also think FS needs a complete rewrite. There is way to much crud in there specific to co-located discretizations on structured grids which should be moved to the DA. FS should only takes ISes, which would make the code a LOT simpler. Then the DA has all the nice routines for easily creating ISes for it. I am putting in the code for DMComplex now, and it seems clear the interface for all DMs should be the same. Matt > Thanks, > Jungho -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

non-symmetric permutation in PCFieldSplit
by Jungho Lee 23 Feb '12

23 Feb '12

I'd like to solve a system with a block structure with PCFieldSplit: [A B;C D][x;y] = [v;w] (I wrote it this way to simplify notation but x and y are in fact interlaced, i.e., I'm solving a permuted version of the system above, where the variables are laid out as x_1 y_1 x_2 y_2 ...., so the system is fieldsplit-friendly) Applying PCFieldSplit to the equivalent system [D C;B A][y;x] = [w;v] can be done with command line options (-pc_fieldsplit_0_fields 1 -pc_fieldsplit_1_fields 0), but I'm also interested in solving [C D;A B][x;y] = [v;w] (to be able to explore different preconditioning strategies), which currently doesn't seem to be supported. At first glance it seems like this can be easily solved by adding extra IS (is) and PetscInt* (fields) objects in _PC_FieldSplitLink, read in integer arrays for rows and columns (the symmetric case where the rows and columns have the same indices is already handled by -pc_fieldsplit_%d_fields), fill in the extra IS object, and allow the user to use these two different IS's in the MatGetSubMatrix calls in PCSetUp_FieldSplit. Is it really as simple as that, or am I missing something here? Thanks, Jungho

1 0