Re: [petsc-users] PCGASMSetLocalSubdomains
On May 15, 2012, at 5:42 PM, Dmitry Karpeev wrote:
On Tue, May 15, 2012 at 10:30 AM, Hui Zhang <[email protected]> wrote: thanks for reply! On May 15, 2012, at 5:19 PM, Dmitry Karpeev wrote:
On Tue, May 15, 2012 at 9:55 AM, Hui Zhang <[email protected]> wrote: Dmitry,
thanks for remind. I have a new question about PCASM / PCGASM:
can I get the restricted extension operators, which maps an overlapping subdomain solution to the global domain?
I'm not exactly sure what you mean.
Are you talking about embedding the subdomain vectors back into the original vector?
Yes, exactly.
If so, there is substantial difference in how this is handled in ASM and GASM: ASM has a bunch of sequential vectors that can be scattered back into the global vector,
Yes. Is there a method to get the scatter?
In the ASM case it's a bunch of scatters -- one for each subdomain. Currently there is no method to retrieve them.
this hint is very helpful. Thanks!
What requires this functionality?
I am writing some modified ASM method. In construction of energy minimization coarse basis, I need to solve individual subdomain problems and not to sum them, just to extend them separately. I wonder whether you guys have ever done this coarse basis. Thanks, Hui
In the ASM case you can construct the scatters yourself easily enough, since you have all of the requisite information -- the array of subdomain ISs and the global vector x. The only piece of data you might not have is the set of outer subdomains that have been obtained by applying overlap increase to the original inner (nonoverlapping) subdomains.
because the subdomains are always local to at most one processor.
In the GASM case this is rather different, since the subdomains can live on arbitrary subcommunicators and there is only one scatter, which is applied to the direct sum of all the subdomain vectors on the original communicator. I'm not sure how useful that last scatter would be for you, since the details of the structure of the direct sum vector are internal to GASM.
I would prefer to have the scatter for individual subdomain before direct sum. But if I can get the scatter PCGASM has, maybe it is still useful. Please tell me how to get it? There are no individual subdomain scatters, but, as in the case of ASM, you can construct them easily enough, except that those would have to operate on subcommunicators. In GASM we pack them into a single scatter on the original communicator. Currently there is no method to expose this scatter. Why do you need this functionality?
Dmitry. Thanks!
Dmitry.
Thanks!
On May 15, 2012, at 3:29 PM, Dmitry Karpeev wrote:
There are some additional minor fixes that mostly have to do with outputting the subdomain information with -pc_gasm_view_subdomains (in PCView()) and with -pc_gasm_print_subdomains (during PCSetUp()). You might want to pull those latest patches, but it won't interfere with your work if you don't use subdomain output.
Thanks. Dmitry.
On Tue, May 15, 2012 at 7:14 AM, Hui Zhang <[email protected]> wrote: Dmitry,
thanks for reply. I re-download the codes and tried it again and now it works correctly!
Everything seems ok.
Thanks, Hui
On May 15, 2012, at 2:01 PM, Dmitry Karpeev wrote:
Hui, I'm trying to reproduce this problem, unsuccessfully, so far. One thing that looks odd is that the output below claims the PC is of type "asm", even though you are running with -dd_type gasm. Could you verify that's the correct output?
Here's the output I get with ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type asm -dd_ksp_view
PC Object:(dd_) 1 MPI processes type: asm Additive Schwarz: total subdomain blocks = 2, user-defined overlap Additive Schwarz: restriction/interpolation type - RESTRICT Local solve is same for all blocks, in the following KSP and PC objects: KSP Object: (dd_sub_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test <snip>
and with ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type gasm -dd_ksp_view
PC Object:(dd_) 1 MPI processes type: gasm Generalized additive Schwarz: Restriction/interpolation type: RESTRICT user-defined overlap total number of subdomains = 2 number of local subdomains = 2 max number of local subdomains = 2 [0:1] number of locally-supported subdomains = 2 Subdomain solver info is as follows: <snip>
What convergence are you seeing with the two PC types? It should be the same with 1 and 2 procs for both PCASM and PCGASM.
Thanks. Dmitry.
On Tue, May 15, 2012 at 4:03 AM, Hui Zhang <[email protected]> wrote: Dmitry,
I got the newest petsc-dev and I run the test by
mpirun -np 1 ./gasm_test -dd_type gasm -n 64 -dd_ksp_view
which gives the following output
PC Object:(dd_) 1 MPI processes type: asm Additive Schwarz: total subdomain blocks = 1, amount of overlap = 1 ^^^ note the above number, it should be 2
While PCASM has no such problem.
Thanks, Hui
Hui,
The convergence issue should be resolved in the latest petsc-dev. I'm attaching a slightly modified gasm_test.c (reflecting some upcoming API changes) that should verify that.
Let me know if it works for you. Thanks. Dmitry. On Fri, May 11, 2012 at 12:31 PM, Hui Zhang <[email protected]> wrote: Hi Dmitry,
thanks for useful hints. Good day!
Hui
On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling your modification subroutine), but not the subdomains or the subdomain solvers. If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures. As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
Dmitry.
On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <[email protected]> wrote: I just have a question about reuse of PCASM or PCGASM. Suppose I have seted up the PCASM and related KSP and I solved one time. Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with different user context for the modifying routine.
What can I do for this task? Currently, I destroy the KSP and re-construct it. I guess even for PCASM I can re-use it because the partition of subdomains remain the same.
Thanks!
On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
Hui, There've been several changes to PCGASM ahead of the new release. Let me go back and see if it affected the convergence problem. Dmitry.
On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <[email protected]> wrote: Hi Dmitry,
is there any news about PCGASM?
thanks, Hui
On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
Okay, thanks. I'll take a look.
Dmitry.
On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <[email protected]> wrote: For reference, my results are attached.
asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes
thank you, Hui
On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <[email protected]> wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]> wrote: I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not. Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor? I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference. That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
<gasm_test.c>
On Tue, May 15, 2012 at 11:05 AM, Hui Zhang <[email protected]>wrote:
On May 15, 2012, at 5:42 PM, Dmitry Karpeev wrote:
On Tue, May 15, 2012 at 10:30 AM, Hui Zhang <[email protected]>wrote:
thanks for reply! On May 15, 2012, at 5:19 PM, Dmitry Karpeev wrote:
On Tue, May 15, 2012 at 9:55 AM, Hui Zhang <[email protected]>wrote:
Dmitry,
thanks for remind. I have a new question about PCASM / PCGASM:
can I get the restricted extension operators, which maps an overlapping subdomain solution to the global domain?
I'm not exactly sure what you mean.
Are you talking about embedding the subdomain vectors back into the original vector?
Yes, exactly.
If so, there is substantial difference in how this is handled in ASM and GASM: ASM has a bunch of sequential vectors that can be scattered back into the global vector,
Yes. Is there a method to get the scatter?
In the ASM case it's a bunch of scatters -- one for each subdomain. Currently there is no method to retrieve them.
this hint is very helpful. Thanks!
What requires this functionality?
I am writing some modified ASM method. In construction of energy minimization coarse basis, I need to solve individual subdomain problems and not to sum them, just to extend them separately. I wonder whether you guys have ever done this coarse basis.
Is there a reference where the basis is described? Dmtiry.
Thanks, Hui
In the ASM case you can construct the scatters yourself easily enough, since you have all of the requisite information -- the array of subdomain ISs and the global vector x. The only piece of data you might not have is the set of outer subdomains that have been obtained by applying overlap increase to the original inner (nonoverlapping) subdomains.
because the subdomains are always local to at most one processor.
In the GASM case this is rather different, since the subdomains can live on arbitrary subcommunicators and there is only one scatter, which is applied to the direct sum of all the subdomain vectors on the original communicator. I'm not sure how useful that last scatter would be for you, since the details of the structure of the direct sum vector are internal to GASM.
I would prefer to have the scatter for individual subdomain before direct sum. But if I can get the scatter PCGASM has, maybe it is still useful. Please tell me how to get it?
There are no individual subdomain scatters, but, as in the case of ASM, you can construct them easily enough, except that those would have to operate on subcommunicators. In GASM we pack them into a single scatter on the original communicator. Currently there is no method to expose this scatter. Why do you need this functionality?
Dmitry.
Thanks!
Dmitry.
Thanks!
On May 15, 2012, at 3:29 PM, Dmitry Karpeev wrote:
There are some additional minor fixes that mostly have to do with outputting the subdomain information with -pc_gasm_view_subdomains (in PCView()) and with -pc_gasm_print_subdomains (during PCSetUp()). You might want to pull those latest patches, but it won't interfere with your work if you don't use subdomain output.
Thanks. Dmitry.
On Tue, May 15, 2012 at 7:14 AM, Hui Zhang <[email protected]>wrote:
Dmitry,
thanks for reply. I re-download the codes and tried it again and now it works correctly!
Everything seems ok.
Thanks, Hui
On May 15, 2012, at 2:01 PM, Dmitry Karpeev wrote:
Hui, I'm trying to reproduce this problem, unsuccessfully, so far. One thing that looks odd is that the output below claims the PC is of type "asm", even though you are running with -dd_type gasm. Could you verify that's the correct output?
Here's the output I get with ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type asm -dd_ksp_view
PC Object:(dd_) 1 MPI processes type: asm Additive Schwarz: total subdomain blocks = 2, user-defined overlap Additive Schwarz: restriction/interpolation type - RESTRICT Local solve is same for all blocks, in the following KSP and PC objects: KSP Object: (dd_sub_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test <snip>
and with ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type gasm -dd_ksp_view
PC Object:(dd_) 1 MPI processes type: gasm Generalized additive Schwarz: Restriction/interpolation type: RESTRICT user-defined overlap total number of subdomains = 2 number of local subdomains = 2 max number of local subdomains = 2 [0:1] number of locally-supported subdomains = 2 Subdomain solver info is as follows: <snip>
What convergence are you seeing with the two PC types? It should be the same with 1 and 2 procs for both PCASM and PCGASM.
Thanks. Dmitry.
On Tue, May 15, 2012 at 4:03 AM, Hui Zhang <[email protected]>wrote:
Dmitry,
I got the newest petsc-dev and I run the test by
mpirun -np 1 ./gasm_test -dd_type gasm -n 64 -dd_ksp_view
which gives the following output
PC Object:(dd_) 1 MPI processes type: asm Additive Schwarz: total subdomain blocks = 1, amount of overlap = 1 ^^^ note the above number, it should be 2
While PCASM has no such problem.
Thanks, Hui
Hui,
The convergence issue should be resolved in the latest petsc-dev. I'm attaching a slightly modified gasm_test.c (reflecting some upcoming API changes) that should verify that.
Let me know if it works for you. Thanks. Dmitry. On Fri, May 11, 2012 at 12:31 PM, Hui Zhang < [email protected]> wrote:
Hi Dmitry,
thanks for useful hints. Good day!
Hui
On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling your modification subroutine), but not the subdomains or the subdomain solvers. If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures. As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
Dmitry.
On Fri, May 11, 2012 at 11:52 AM, Hui Zhang < [email protected]> wrote:
I just have a question about reuse of PCASM or PCGASM. Suppose I have seted up the PCASM and related KSP and I solved one time. Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with different user context for the modifying routine.
What can I do for this task? Currently, I destroy the KSP and re-construct it. I guess even for PCASM I can re-use it because the partition of subdomains remain the same.
Thanks!
On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
Hui, There've been several changes to PCGASM ahead of the new release. Let me go back and see if it affected the convergence problem. Dmitry.
On Thu, May 10, 2012 at 4:16 AM, Hui Zhang < [email protected]> wrote:
Hi Dmitry,
is there any news about PCGASM?
thanks, Hui
On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
Okay, thanks. I'll take a look.
Dmitry.
On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang < [email protected]> wrote:
For reference, my results are attached.
asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes
thank you, Hui
On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang < [email protected]> wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang < [email protected]> wrote:
I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not.
Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference.
That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <<[email protected]> [email protected]> wrote:
About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
<gasm_test.c>
A question about PCASMGetSubKSP(), it says that we must call KSPSetUp() before calling PCASMGetSubKSP(). But what ksp should be SetUp, the ksp using PCASM or what? Thanks!
On Tue, May 15, 2012 at 10:30 AM, Hui Zhang <[email protected]> wrote: thanks for reply! On May 15, 2012, at 5:19 PM, Dmitry Karpeev wrote:
On Tue, May 15, 2012 at 9:55 AM, Hui Zhang <[email protected]> wrote: Dmitry,
thanks for remind. I have a new question about PCASM / PCGASM:
can I get the restricted extension operators, which maps an overlapping subdomain solution to the global domain?
I'm not exactly sure what you mean.
Are you talking about embedding the subdomain vectors back into the original vector?
Yes, exactly.
If so, there is substantial difference in how this is handled in ASM and GASM: ASM has a bunch of sequential vectors that can be scattered back into the global vector,
Yes. Is there a method to get the scatter?
In the ASM case it's a bunch of scatters -- one for each subdomain. Currently there is no method to retrieve them.
this hint is very helpful. Thanks!
What requires this functionality?
I am writing some modified ASM method. In construction of energy minimization coarse basis, I need to solve individual subdomain problems and not to sum them, just to extend them separately. I wonder whether you guys have ever done this coarse basis. Is there a reference where the basis is described? Dmtiry.
Thanks, Hui
In the ASM case you can construct the scatters yourself easily enough, since you have all of the requisite information -- the array of subdomain ISs and the global vector x. The only piece of data you might not have is the set of outer subdomains that have been obtained by applying overlap increase to the original inner (nonoverlapping) subdomains.
because the subdomains are always local to at most one processor.
In the GASM case this is rather different, since the subdomains can live on arbitrary subcommunicators and there is only one scatter, which is applied to the direct sum of all the subdomain vectors on the original communicator. I'm not sure how useful that last scatter would be for you, since the details of the structure of the direct sum vector are internal to GASM.
I would prefer to have the scatter for individual subdomain before direct sum. But if I can get the scatter PCGASM has, maybe it is still useful. Please tell me how to get it? There are no individual subdomain scatters, but, as in the case of ASM, you can construct them easily enough, except that those would have to operate on subcommunicators. In GASM we pack them into a single scatter on the original communicator. Currently there is no method to expose this scatter. Why do you need this functionality?
Dmitry. Thanks!
Dmitry.
Thanks!
On May 15, 2012, at 3:29 PM, Dmitry Karpeev wrote:
There are some additional minor fixes that mostly have to do with outputting the subdomain information with -pc_gasm_view_subdomains (in PCView()) and with -pc_gasm_print_subdomains (during PCSetUp()). You might want to pull those latest patches, but it won't interfere with your work if you don't use subdomain output.
Thanks. Dmitry.
On Tue, May 15, 2012 at 7:14 AM, Hui Zhang <[email protected]> wrote: Dmitry,
thanks for reply. I re-download the codes and tried it again and now it works correctly!
Everything seems ok.
Thanks, Hui
On May 15, 2012, at 2:01 PM, Dmitry Karpeev wrote:
Hui, I'm trying to reproduce this problem, unsuccessfully, so far. One thing that looks odd is that the output below claims the PC is of type "asm", even though you are running with -dd_type gasm. Could you verify that's the correct output?
Here's the output I get with ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type asm -dd_ksp_view
PC Object:(dd_) 1 MPI processes type: asm Additive Schwarz: total subdomain blocks = 2, user-defined overlap Additive Schwarz: restriction/interpolation type - RESTRICT Local solve is same for all blocks, in the following KSP and PC objects: KSP Object: (dd_sub_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test <snip>
and with ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type gasm -dd_ksp_view
PC Object:(dd_) 1 MPI processes type: gasm Generalized additive Schwarz: Restriction/interpolation type: RESTRICT user-defined overlap total number of subdomains = 2 number of local subdomains = 2 max number of local subdomains = 2 [0:1] number of locally-supported subdomains = 2 Subdomain solver info is as follows: <snip>
What convergence are you seeing with the two PC types? It should be the same with 1 and 2 procs for both PCASM and PCGASM.
Thanks. Dmitry.
On Tue, May 15, 2012 at 4:03 AM, Hui Zhang <[email protected]> wrote: Dmitry,
I got the newest petsc-dev and I run the test by
mpirun -np 1 ./gasm_test -dd_type gasm -n 64 -dd_ksp_view
which gives the following output
PC Object:(dd_) 1 MPI processes type: asm Additive Schwarz: total subdomain blocks = 1, amount of overlap = 1 ^^^ note the above number, it should be 2
While PCASM has no such problem.
Thanks, Hui
Hui,
The convergence issue should be resolved in the latest petsc-dev. I'm attaching a slightly modified gasm_test.c (reflecting some upcoming API changes) that should verify that.
Let me know if it works for you. Thanks. Dmitry. On Fri, May 11, 2012 at 12:31 PM, Hui Zhang <[email protected]> wrote: Hi Dmitry,
thanks for useful hints. Good day!
Hui
On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling your modification subroutine), but not the subdomains or the subdomain solvers. If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures. As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
Dmitry.
On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <[email protected]> wrote: I just have a question about reuse of PCASM or PCGASM. Suppose I have seted up the PCASM and related KSP and I solved one time. Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with different user context for the modifying routine.
What can I do for this task? Currently, I destroy the KSP and re-construct it. I guess even for PCASM I can re-use it because the partition of subdomains remain the same.
Thanks!
On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
Hui, There've been several changes to PCGASM ahead of the new release. Let me go back and see if it affected the convergence problem. Dmitry.
On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <[email protected]> wrote: Hi Dmitry,
is there any news about PCGASM?
thanks, Hui
On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
Okay, thanks. I'll take a look.
Dmitry.
On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <[email protected]> wrote: For reference, my results are attached.
asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes
thank you, Hui
On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <[email protected]> wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]> wrote: I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not. Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor? I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference. That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
<gasm_test.c>
It's ok. I found the example program. On May 16, 2012, at 10:46 AM, Hui Zhang wrote:
A question about PCASMGetSubKSP(), it says that we must call KSPSetUp() before calling PCASMGetSubKSP(). But what ksp should be SetUp, the ksp using PCASM or what?
Thanks!
On Tue, May 15, 2012 at 10:30 AM, Hui Zhang <[email protected]> wrote: thanks for reply! On May 15, 2012, at 5:19 PM, Dmitry Karpeev wrote:
On Tue, May 15, 2012 at 9:55 AM, Hui Zhang <[email protected]> wrote: Dmitry,
thanks for remind. I have a new question about PCASM / PCGASM:
can I get the restricted extension operators, which maps an overlapping subdomain solution to the global domain?
I'm not exactly sure what you mean.
Are you talking about embedding the subdomain vectors back into the original vector?
Yes, exactly.
If so, there is substantial difference in how this is handled in ASM and GASM: ASM has a bunch of sequential vectors that can be scattered back into the global vector,
Yes. Is there a method to get the scatter?
In the ASM case it's a bunch of scatters -- one for each subdomain. Currently there is no method to retrieve them.
this hint is very helpful. Thanks!
What requires this functionality?
I am writing some modified ASM method. In construction of energy minimization coarse basis, I need to solve individual subdomain problems and not to sum them, just to extend them separately. I wonder whether you guys have ever done this coarse basis. Is there a reference where the basis is described? Dmtiry.
Thanks, Hui
In the ASM case you can construct the scatters yourself easily enough, since you have all of the requisite information -- the array of subdomain ISs and the global vector x. The only piece of data you might not have is the set of outer subdomains that have been obtained by applying overlap increase to the original inner (nonoverlapping) subdomains.
because the subdomains are always local to at most one processor.
In the GASM case this is rather different, since the subdomains can live on arbitrary subcommunicators and there is only one scatter, which is applied to the direct sum of all the subdomain vectors on the original communicator. I'm not sure how useful that last scatter would be for you, since the details of the structure of the direct sum vector are internal to GASM.
I would prefer to have the scatter for individual subdomain before direct sum. But if I can get the scatter PCGASM has, maybe it is still useful. Please tell me how to get it? There are no individual subdomain scatters, but, as in the case of ASM, you can construct them easily enough, except that those would have to operate on subcommunicators. In GASM we pack them into a single scatter on the original communicator. Currently there is no method to expose this scatter. Why do you need this functionality?
Dmitry. Thanks!
Dmitry.
Thanks!
On May 15, 2012, at 3:29 PM, Dmitry Karpeev wrote:
There are some additional minor fixes that mostly have to do with outputting the subdomain information with -pc_gasm_view_subdomains (in PCView()) and with -pc_gasm_print_subdomains (during PCSetUp()). You might want to pull those latest patches, but it won't interfere with your work if you don't use subdomain output.
Thanks. Dmitry.
On Tue, May 15, 2012 at 7:14 AM, Hui Zhang <[email protected]> wrote: Dmitry,
thanks for reply. I re-download the codes and tried it again and now it works correctly!
Everything seems ok.
Thanks, Hui
On May 15, 2012, at 2:01 PM, Dmitry Karpeev wrote:
Hui, I'm trying to reproduce this problem, unsuccessfully, so far. One thing that looks odd is that the output below claims the PC is of type "asm", even though you are running with -dd_type gasm. Could you verify that's the correct output?
Here's the output I get with ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type asm -dd_ksp_view
PC Object:(dd_) 1 MPI processes type: asm Additive Schwarz: total subdomain blocks = 2, user-defined overlap Additive Schwarz: restriction/interpolation type - RESTRICT Local solve is same for all blocks, in the following KSP and PC objects: KSP Object: (dd_sub_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test <snip>
and with ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type gasm -dd_ksp_view
PC Object:(dd_) 1 MPI processes type: gasm Generalized additive Schwarz: Restriction/interpolation type: RESTRICT user-defined overlap total number of subdomains = 2 number of local subdomains = 2 max number of local subdomains = 2 [0:1] number of locally-supported subdomains = 2 Subdomain solver info is as follows: <snip>
What convergence are you seeing with the two PC types? It should be the same with 1 and 2 procs for both PCASM and PCGASM.
Thanks. Dmitry.
On Tue, May 15, 2012 at 4:03 AM, Hui Zhang <[email protected]> wrote: Dmitry,
I got the newest petsc-dev and I run the test by
mpirun -np 1 ./gasm_test -dd_type gasm -n 64 -dd_ksp_view
which gives the following output
PC Object:(dd_) 1 MPI processes type: asm Additive Schwarz: total subdomain blocks = 1, amount of overlap = 1 ^^^ note the above number, it should be 2
While PCASM has no such problem.
Thanks, Hui
Hui,
The convergence issue should be resolved in the latest petsc-dev. I'm attaching a slightly modified gasm_test.c (reflecting some upcoming API changes) that should verify that.
Let me know if it works for you. Thanks. Dmitry. On Fri, May 11, 2012 at 12:31 PM, Hui Zhang <[email protected]> wrote: Hi Dmitry,
thanks for useful hints. Good day!
Hui
On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling your modification subroutine), but not the subdomains or the subdomain solvers. If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures. As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
Dmitry.
On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <[email protected]> wrote: I just have a question about reuse of PCASM or PCGASM. Suppose I have seted up the PCASM and related KSP and I solved one time. Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with different user context for the modifying routine.
What can I do for this task? Currently, I destroy the KSP and re-construct it. I guess even for PCASM I can re-use it because the partition of subdomains remain the same.
Thanks!
On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
Hui, There've been several changes to PCGASM ahead of the new release. Let me go back and see if it affected the convergence problem. Dmitry.
On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <[email protected]> wrote: Hi Dmitry,
is there any news about PCGASM?
thanks, Hui
On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
Okay, thanks. I'll take a look.
Dmitry.
On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <[email protected]> wrote: For reference, my results are attached.
asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes
thank you, Hui
On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <[email protected]> wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]> wrote: I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not. Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor? I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference. That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
<gasm_test.c>
On Tue, May 15, 2012 at 10:05 AM, Hui Zhang <[email protected]>wrote:
I am writing some modified ASM method. In construction of energy minimization coarse basis, I need to solve individual subdomain problems and not to sum them, just to extend them separately. I wonder whether you guys have ever done this coarse basis.
How are these subdomain problems defined? They are very often different (in terms of overlap or boundary conditions) than are used in the smoother. You might want to look at PCEXOTIC ( http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/PC/PCEXOTIC.html) which is a basic implementation of a two-level DD method of this type.
On May 15, 2012, at 6:12 PM, Jed Brown wrote:
On Tue, May 15, 2012 at 10:05 AM, Hui Zhang <[email protected]> wrote: I am writing some modified ASM method. In construction of energy minimization coarse basis, I need to solve individual subdomain problems and not to sum them, just to extend them separately. I wonder whether you guys have ever done this coarse basis.
How are these subdomain problems defined? They are very often different (in terms of overlap or boundary conditions) than are used in the smoother
With overlap and modified boundary conditions.
You might want to look at PCEXOTIC (http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/PC/PCEXOTIC.html) which is a basic implementation of a two-level DD method of this type.
Yes, this is interesting :) Thanks!
If PCASMSetSortIndices is used to set false, i.e. DO NOT sort indices, is that true rows of submatrices are in the same order as the IS we passed to PCASMSetLocalSubdomains? Thanks!
On Sat, May 19, 2012 at 4:10 AM, Hui Zhang <[email protected]>wrote:
If PCASMSetSortIndices is used to set false, i.e. DO NOT sort indices, is that true rows of submatrices are in the same order as the IS we passed to PCASMSetLocalSubdomains?
Yes. Dmitry.
Thanks!
Hi, I have got a new question. I'm now constructing a PCComposite from PCASM and another PCKSP (coarse problem). And construction of PCKSP needs to use the subksp's of PCASM. So I need to PCSetUp on the PCASM, right? But the KSP using PCComposite would setup PCASM again. How can I avoid twice setup of PCASM ? Thanks, Hui
PCASM真心不懂。sorry At 2012-05-22 18:23:26,"Hui Zhang" <[email protected]> wrote:
Hi,
I have got a new question. I'm now constructing a PCComposite from PCASM and another PCKSP (coarse problem). And construction of PCKSP needs to use the subksp's of PCASM. So I need to PCSetUp on the PCASM, right? But the KSP using PCComposite would setup PCASM again. How can I avoid twice setup of PCASM ?
Thanks, Hui
It sounds like you need to write a new PC implementation, possibly reusing pieces of PCASM. Then you have control of what is being set up and when. Otherwise you are essentially hacking PCASM and using its internal subsolvers, which are only available after PCSetUp(). PCCOMPOSITE doesn't look like the right framework for this. Dmitry. On Tue, May 22, 2012 at 5:23 AM, Hui Zhang <[email protected]>wrote:
Hi,
I have got a new question. I'm now constructing a PCComposite from PCASM and another PCKSP (coarse problem). And construction of PCKSP needs to use the subksp's of PCASM. So I need to PCSetUp on the PCASM, right? But the KSP using PCComposite would setup PCASM again. How can I avoid twice setup of PCASM ?
Thanks, Hui
On May 22, 2012, at 8:14 PM, Dmitry Karpeev wrote:
It sounds like you need to write a new PC implementation, possibly reusing pieces of PCASM.
Thanks! But then it will be too much work, I think, and what I tried to avoid.
Then you have control of what is being set up and when. Otherwise you are essentially hacking PCASM and using its internal subsolvers, which are only available after PCSetUp().
Yes. Maybe I should follow your advice. Thanks again!
PCCOMPOSITE doesn't look like the right framework for this.
Dmitry.
On Tue, May 22, 2012 at 5:23 AM, Hui Zhang <[email protected]> wrote: Hi,
I have got a new question. I'm now constructing a PCComposite from PCASM and another PCKSP (coarse problem). And construction of PCKSP needs to use the subksp's of PCASM. So I need to PCSetUp on the PCASM, right? But the KSP using PCComposite would setup PCASM again. How can I avoid twice setup of PCASM ?
Thanks, Hui
participants (4)
-
Dmitry Karpeev -
Hui Zhang -
Jed Brown -
w_ang_temp