About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'? The source code of PCGASMCreateSubdomains2D() seemingly does so. Thanks, Hui
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm. There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter. Dmitry. On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]>wrote:
About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
Hi Dmitry, thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx); I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something? thanks, Hui On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same. I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()? thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
try it On Feb 15, 2012, at 10:07 AM, Hui Zhang wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
On Feb 15, 2012, at 5:18 PM, Barry Smith wrote:
try it
Yes, I'm trying. Just one more question: why there is no *Get* LocalToGlobalMapping so that when a Mat is with some LocalToGlobalMapping we can temporarily change with another mapping and reset back to the original mapping.
On Feb 15, 2012, at 10:07 AM, Hui Zhang wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
On Feb 15, 2012, at 10:26 AM, Hui Zhang wrote:
On Feb 15, 2012, at 5:18 PM, Barry Smith wrote:
try it
Yes, I'm trying. Just one more question: why there is no *Get* LocalToGlobalMapping so that when a Mat is with some LocalToGlobalMapping we can temporarily change with another mapping and reset back to the original mapping.
There is. Perhaps it is only in petsc-dev Barry
On Feb 15, 2012, at 10:07 AM, Hui Zhang wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
On Feb 15, 2012, at 10:26 AM, Hui Zhang wrote:
On Feb 15, 2012, at 5:18 PM, Barry Smith wrote:
try it
Yes, I'm trying. Just one more question: why there is no *Get* LocalToGlobalMapping so that when a Mat is with some LocalToGlobalMapping we can temporarily change with another mapping and reset back to the original mapping.
There is. Perhaps it is only in petsc-dev
Barry
I found it. Thank you very much! By GetLocalToGlobalMapping, I also find that the submat's from PCGASM has not been SetLocalToGlobalMapping() yet when ModifySubMatrices was called by PCGASM. Everything is ok and clear now. Thanks again!
On Feb 15, 2012, at 10:07 AM, Hui Zhang wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday. Dmitry On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM. For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem. On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]>wrote:
I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving? Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang < <[email protected]> [email protected]> wrote:
About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]> wrote: I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not. I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <[email protected]>wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]>wrote:
I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not.
Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference.
That would be helpful. Thanks. Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang < <[email protected]> [email protected]> wrote:
About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
Hi, attached is an example modified from ksp/ex11.c. I found even I do not modify matrices there are problems. Just make and run with mpirun -np 1 gasm_test -ksp_monitor -n 64 and mpirun -np 2 gasm_test -ksp_monitor -n 64 You will see how the ASM gives the same result in the two runs. Then, do a replacement for ASM->GASM, make and run again like before. I saw different results from two runs. thank you, Hui On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <[email protected]> wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]> wrote: I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not. Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor? I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference. That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
For reference, my results are attached. asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes thank you, Hui On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <[email protected]> wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]> wrote: I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not. Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor? I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference. That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
Okay, thanks. I'll take a look. Dmitry. On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <[email protected]>wrote:
For reference, my results are attached.
asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes
thank you, Hui
On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <[email protected]>wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]>wrote:
I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not.
Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference.
That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <<[email protected]> [email protected]> wrote:
About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
Hi Dmitry, is there any news about PCGASM? thanks, Hui On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
Okay, thanks. I'll take a look.
Dmitry.
On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <[email protected]> wrote: For reference, my results are attached.
asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes
thank you, Hui
On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <[email protected]> wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]> wrote: I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not. Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor? I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference. That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
Hui, There've been several changes to PCGASM ahead of the new release. Let me go back and see if it affected the convergence problem. Dmitry. On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <[email protected]>wrote:
Hi Dmitry,
is there any news about PCGASM?
thanks, Hui
On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
Okay, thanks. I'll take a look.
Dmitry.
On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <[email protected]>wrote:
For reference, my results are attached.
asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes
thank you, Hui
On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <[email protected]>wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]>wrote:
I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not.
Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference.
That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <<[email protected]> [email protected]> wrote:
About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
I just have a question about reuse of PCASM or PCGASM. Suppose I have seted up the PCASM and related KSP and I solved one time. Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with different user context for the modifying routine. What can I do for this task? Currently, I destroy the KSP and re-construct it. I guess even for PCASM I can re-use it because the partition of subdomains remain the same. Thanks! On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
Hui, There've been several changes to PCGASM ahead of the new release. Let me go back and see if it affected the convergence problem. Dmitry.
On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <[email protected]> wrote: Hi Dmitry,
is there any news about PCGASM?
thanks, Hui
On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
Okay, thanks. I'll take a look.
Dmitry.
On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <[email protected]> wrote: For reference, my results are attached.
asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes
thank you, Hui
On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <[email protected]> wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]> wrote: I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not. Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor? I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference. That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling your modification subroutine), but not the subdomains or the subdomain solvers. If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures. As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback. Dmitry. On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <[email protected]>wrote:
I just have a question about reuse of PCASM or PCGASM. Suppose I have seted up the PCASM and related KSP and I solved one time. Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with different user context for the modifying routine.
What can I do for this task? Currently, I destroy the KSP and re-construct it. I guess even for PCASM I can re-use it because the partition of subdomains remain the same.
Thanks!
On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
Hui, There've been several changes to PCGASM ahead of the new release. Let me go back and see if it affected the convergence problem. Dmitry.
On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <[email protected]>wrote:
Hi Dmitry,
is there any news about PCGASM?
thanks, Hui
On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
Okay, thanks. I'll take a look.
Dmitry.
On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <[email protected]>wrote:
For reference, my results are attached.
asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes
thank you, Hui
On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <[email protected]>wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]>wrote:
I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not.
Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference.
That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <<[email protected]> [email protected]> wrote:
About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
Hi Dmitry, thanks for useful hints. Good day! Hui On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling your modification subroutine), but not the subdomains or the subdomain solvers. If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures. As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
Dmitry.
On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <[email protected]> wrote: I just have a question about reuse of PCASM or PCGASM. Suppose I have seted up the PCASM and related KSP and I solved one time. Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with different user context for the modifying routine.
What can I do for this task? Currently, I destroy the KSP and re-construct it. I guess even for PCASM I can re-use it because the partition of subdomains remain the same.
Thanks!
On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
Hui, There've been several changes to PCGASM ahead of the new release. Let me go back and see if it affected the convergence problem. Dmitry.
On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <[email protected]> wrote: Hi Dmitry,
is there any news about PCGASM?
thanks, Hui
On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
Okay, thanks. I'll take a look.
Dmitry.
On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <[email protected]> wrote: For reference, my results are attached.
asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes
thank you, Hui
On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <[email protected]> wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]> wrote: I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not. Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor? I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference. That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
Hui, The convergence issue should be resolved in the latest petsc-dev. I'm attaching a slightly modified gasm_test.c (reflecting some upcoming API changes) that should verify that. Let me know if it works for you. Thanks. Dmitry. On Fri, May 11, 2012 at 12:31 PM, Hui Zhang <[email protected]>wrote:
Hi Dmitry,
thanks for useful hints. Good day!
Hui
On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling your modification subroutine), but not the subdomains or the subdomain solvers. If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures. As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
Dmitry.
On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <[email protected]>wrote:
I just have a question about reuse of PCASM or PCGASM. Suppose I have seted up the PCASM and related KSP and I solved one time. Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with different user context for the modifying routine.
What can I do for this task? Currently, I destroy the KSP and re-construct it. I guess even for PCASM I can re-use it because the partition of subdomains remain the same.
Thanks!
On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
Hui, There've been several changes to PCGASM ahead of the new release. Let me go back and see if it affected the convergence problem. Dmitry.
On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <[email protected]>wrote:
Hi Dmitry,
is there any news about PCGASM?
thanks, Hui
On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
Okay, thanks. I'll take a look.
Dmitry.
On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <[email protected]>wrote:
For reference, my results are attached.
asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes
thank you, Hui
On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <[email protected]
wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]
wrote:
I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not.
Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference.
That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <<[email protected]> [email protected]> wrote:
About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
Dmitry, I got the newest petsc-dev and I run the test by mpirun -np 1 ./gasm_test -dd_type gasm -n 64 -dd_ksp_view which gives the following output PC Object:(dd_) 1 MPI processes type: asm Additive Schwarz: total subdomain blocks = 1, amount of overlap = 1 ^^^ note the above number, it should be 2 While PCASM has no such problem. Thanks, Hui
Hui,
The convergence issue should be resolved in the latest petsc-dev. I'm attaching a slightly modified gasm_test.c (reflecting some upcoming API changes) that should verify that.
Let me know if it works for you. Thanks. Dmitry. On Fri, May 11, 2012 at 12:31 PM, Hui Zhang <[email protected]> wrote: Hi Dmitry,
thanks for useful hints. Good day!
Hui
On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling your modification subroutine), but not the subdomains or the subdomain solvers. If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures. As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
Dmitry.
On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <[email protected]> wrote: I just have a question about reuse of PCASM or PCGASM. Suppose I have seted up the PCASM and related KSP and I solved one time. Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with different user context for the modifying routine.
What can I do for this task? Currently, I destroy the KSP and re-construct it. I guess even for PCASM I can re-use it because the partition of subdomains remain the same.
Thanks!
On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
Hui, There've been several changes to PCGASM ahead of the new release. Let me go back and see if it affected the convergence problem. Dmitry.
On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <[email protected]> wrote: Hi Dmitry,
is there any news about PCGASM?
thanks, Hui
On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
Okay, thanks. I'll take a look.
Dmitry.
On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <[email protected]> wrote: For reference, my results are attached.
asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes
thank you, Hui
On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <[email protected]> wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]> wrote: I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not. Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor? I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference. That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
<gasm_test.c>
Hui, I'm trying to reproduce this problem, unsuccessfully, so far. One thing that looks odd is that the output below claims the PC is of type "asm", even though you are running with -dd_type gasm. Could you verify that's the correct output? Here's the output I get with ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type asm -dd_ksp_view PC Object:(dd_) 1 MPI processes type: asm Additive Schwarz: total subdomain blocks = 2, user-defined overlap Additive Schwarz: restriction/interpolation type - RESTRICT Local solve is same for all blocks, in the following KSP and PC objects: KSP Object: (dd_sub_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning using NONE norm type for convergence test <snip> and with ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec -np 1 ./gasm_test -n 64 -dd_type gasm -dd_ksp_view PC Object:(dd_) 1 MPI processes type: gasm Generalized additive Schwarz: Restriction/interpolation type: RESTRICT user-defined overlap total number of subdomains = 2 number of local subdomains = 2 max number of local subdomains = 2 [0:1] number of locally-supported subdomains = 2 Subdomain solver info is as follows: <snip> What convergence are you seeing with the two PC types? It should be the same with 1 and 2 procs for both PCASM and PCGASM. Thanks. Dmitry. On Tue, May 15, 2012 at 4:03 AM, Hui Zhang <[email protected]>wrote:
Dmitry,
I got the newest petsc-dev and I run the test by
mpirun -np 1 ./gasm_test -dd_type gasm -n 64 -dd_ksp_view
which gives the following output
PC Object:(dd_) 1 MPI processes type: asm Additive Schwarz: total subdomain blocks = 1, amount of overlap = 1 ^^^ note the above number, it should be 2
While PCASM has no such problem.
Thanks, Hui
Hui,
The convergence issue should be resolved in the latest petsc-dev. I'm attaching a slightly modified gasm_test.c (reflecting some upcoming API changes) that should verify that.
Let me know if it works for you. Thanks. Dmitry. On Fri, May 11, 2012 at 12:31 PM, Hui Zhang <[email protected]>wrote:
Hi Dmitry,
thanks for useful hints. Good day!
Hui
On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling your modification subroutine), but not the subdomains or the subdomain solvers. If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures. As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
Dmitry.
On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <[email protected]>wrote:
I just have a question about reuse of PCASM or PCGASM. Suppose I have seted up the PCASM and related KSP and I solved one time. Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with different user context for the modifying routine.
What can I do for this task? Currently, I destroy the KSP and re-construct it. I guess even for PCASM I can re-use it because the partition of subdomains remain the same.
Thanks!
On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
Hui, There've been several changes to PCGASM ahead of the new release. Let me go back and see if it affected the convergence problem. Dmitry.
On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <[email protected]>wrote:
Hi Dmitry,
is there any news about PCGASM?
thanks, Hui
On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
Okay, thanks. I'll take a look.
Dmitry.
On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <[email protected]
wrote:
For reference, my results are attached.
asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes
thank you, Hui
On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang < [email protected]> wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang < [email protected]> wrote:
I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not.
Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference.
That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <<[email protected]> [email protected]> wrote:
About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
<gasm_test.c>
I have the following MatView output: row 0: (0, -0.0351719 - 0.638814 i) (1, -0.517586 - 0.617823 i) (4, -0.517586 - 0.617823 i) (1, -1.03517) (5, -0.508793) Note that there are two values for the column 1, is this normal? The above matrix is obtained from PCASMGetSubKSP(pc,subksp) and KSPGetOperators(subksp[0], ...). Thanks! On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling your modification subroutine), but not the subdomains or the subdomain solvers. If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures. As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
Dmitry.
On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <[email protected]> wrote: I just have a question about reuse of PCASM or PCGASM. Suppose I have seted up the PCASM and related KSP and I solved one time. Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with different user context for the modifying routine.
What can I do for this task? Currently, I destroy the KSP and re-construct it. I guess even for PCASM I can re-use it because the partition of subdomains remain the same.
Thanks!
On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
Hui, There've been several changes to PCGASM ahead of the new release. Let me go back and see if it affected the convergence problem. Dmitry.
On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <[email protected]> wrote: Hi Dmitry,
is there any news about PCGASM?
thanks, Hui
On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
Okay, thanks. I'll take a look.
Dmitry.
On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <[email protected]> wrote: For reference, my results are attached.
asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes
thank you, Hui
On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <[email protected]> wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]> wrote: I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not. Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor? I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference. That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
On Sun, May 20, 2012 at 5:06 AM, Hui Zhang <[email protected]>wrote:
I have the following MatView output:
row 0: (0, -0.0351719 - 0.638814 i) (1, -0.517586 - 0.617823 i) (4, -0.517586 - 0.617823 i) (1, -1.03517) (5, -0.508793)
Note that there are two values for the column 1, is this normal?
The above matrix is obtained from PCASMGetSubKSP(pc,subksp) and KSPGetOperators(subksp[0], ...).
This is odd. Can you explain a bit more what leads up to this, so we can try and reproduce the problem? Thanks. Dmitry.
Thanks!
On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling your modification subroutine), but not the subdomains or the subdomain solvers. If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures. As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
Dmitry.
On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <[email protected]>wrote:
I just have a question about reuse of PCASM or PCGASM. Suppose I have seted up the PCASM and related KSP and I solved one time. Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with different user context for the modifying routine.
What can I do for this task? Currently, I destroy the KSP and re-construct it. I guess even for PCASM I can re-use it because the partition of subdomains remain the same.
Thanks!
On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
Hui, There've been several changes to PCGASM ahead of the new release. Let me go back and see if it affected the convergence problem. Dmitry.
On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <[email protected]>wrote:
Hi Dmitry,
is there any news about PCGASM?
thanks, Hui
On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
Okay, thanks. I'll take a look.
Dmitry.
On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <[email protected]>wrote:
For reference, my results are attached.
asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes
thank you, Hui
On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <[email protected]
wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]
wrote:
I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not.
Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference.
That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <<[email protected]> [email protected]> wrote:
About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
On May 20, 2012, at 4:36 PM, Dmitry Karpeev wrote:
On Sun, May 20, 2012 at 5:06 AM, Hui Zhang <[email protected]> wrote: I have the following MatView output:
row 0: (0, -0.0351719 - 0.638814 i) (1, -0.517586 - 0.617823 i) (4, -0.517586 - 0.617823 i) (1, -1.03517) (5, -0.508793)
Note that there are two values for the column 1, is this normal?
The above matrix is obtained from PCASMGetSubKSP(pc,subksp) and KSPGetOperators(subksp[0], ...).
This is odd. Can you explain a bit more what leads up to this, so we can try and reproduce the problem?
Yes, I will try to make a simple example program.
Thanks. Dmitry.
Thanks!
On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling your modification subroutine), but not the subdomains or the subdomain solvers. If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures. As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
Dmitry.
On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <[email protected]> wrote: I just have a question about reuse of PCASM or PCGASM. Suppose I have seted up the PCASM and related KSP and I solved one time. Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with different user context for the modifying routine.
What can I do for this task? Currently, I destroy the KSP and re-construct it. I guess even for PCASM I can re-use it because the partition of subdomains remain the same.
Thanks!
On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
Hui, There've been several changes to PCGASM ahead of the new release. Let me go back and see if it affected the convergence problem. Dmitry.
On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <[email protected]> wrote: Hi Dmitry,
is there any news about PCGASM?
thanks, Hui
On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
Okay, thanks. I'll take a look.
Dmitry.
On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <[email protected]> wrote: For reference, my results are attached.
asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes
thank you, Hui
On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <[email protected]> wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]> wrote: I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not. Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor? I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference. That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
Dmitry, finally, I am able to make a simple program and reproduce the error. Since I had moved to petsc-dev (before I used petsc-3.2), the error is a little different. Download the program attached and run the run.sh. You will see the output from PCASM. There is a submatrix (submat[1] on rank 0) which has a row like row 1: (0, -1) (1, 0 + 0.4 i) (4, -1) (2, -1) Note that the column 2 is placed in the last and the column numbers are not in the normal order. Following that, we can also see the error message like [0]PETSC ERROR: New nonzero at (1,2) caused a malloc! You can also try with GASM, use mpirun -np 2 ./gasm_test -dd_type gasm -n 4 -px 1 -py 2 -sx 2 -sy 1 which partitions the domain to 1 * 2 and mapped to the two processors, and for each processor we further partition its local domain to 2 * 1 subdomains. Then, you will see that the row 1 of submat[1] on rank 1 becomes row 1: (0, -1) (1, 0 + 0.4 i) (3, -1) (6, -1) which is totally wrong because from the IS for this overlapping subdomain and the big matrix A, row 1 should have non-zeros like the output from PCASM. I guess the problem is due to that we set PCASMSetSortIndices to FALSE and something goes wrong in asm.c and gasm.c. Thanks! Hui
On Sun, May 20, 2012 at 5:06 AM, Hui Zhang <[email protected]> wrote: I have the following MatView output:
row 0: (0, -0.0351719 - 0.638814 i) (1, -0.517586 - 0.617823 i) (4, -0.517586 - 0.617823 i) (1, -1.03517) (5, -0.508793)
Note that there are two values for the column 1, is this normal?
The above matrix is obtained from PCASMGetSubKSP(pc,subksp) and KSPGetOperators(subksp[0], ...).
This is odd. Can you explain a bit more what leads up to this, so we can try and reproduce the problem? Thanks. Dmitry.
Thanks!
On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling your modification subroutine), but not the subdomains or the subdomain solvers. If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures. As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
Dmitry.
On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <[email protected]> wrote: I just have a question about reuse of PCASM or PCGASM. Suppose I have seted up the PCASM and related KSP and I solved one time. Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with different user context for the modifying routine.
What can I do for this task? Currently, I destroy the KSP and re-construct it. I guess even for PCASM I can re-use it because the partition of subdomains remain the same.
Thanks!
On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
Hui, There've been several changes to PCGASM ahead of the new release. Let me go back and see if it affected the convergence problem. Dmitry.
On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <[email protected]> wrote: Hi Dmitry,
is there any news about PCGASM?
thanks, Hui
On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
Okay, thanks. I'll take a look.
Dmitry.
On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <[email protected]> wrote: For reference, my results are attached.
asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes
thank you, Hui
On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <[email protected]> wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]> wrote: I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not. Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor? I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference. That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
Sorry, I uses PCASMSetSortIndices for GASM. Now, I correct it and the output becomes the same as PCASM. See attached. On May 21, 2012, at 8:59 PM, Hui Zhang wrote:
Dmitry,
finally, I am able to make a simple program and reproduce the error. Since I had moved to petsc-dev (before I used petsc-3.2), the error is a little different.
Download the program attached and run the run.sh. You will see the output from PCASM. There is a submatrix (submat[1] on rank 0) which has a row like
row 1: (0, -1) (1, 0 + 0.4 i) (4, -1) (2, -1)
Note that the column 2 is placed in the last and the column numbers are not in the normal order. Following that, we can also see the error message like
[0]PETSC ERROR: New nonzero at (1,2) caused a malloc!
You can also try with GASM, use
mpirun -np 2 ./gasm_test -dd_type gasm -n 4 -px 1 -py 2 -sx 2 -sy 1
which partitions the domain to 1 * 2 and mapped to the two processors, and for each processor we further partition its local domain to 2 * 1 subdomains.
Then, you will see that the row 1 of submat[1] on rank 1 becomes
row 1: (0, -1) (1, 0 + 0.4 i) (3, -1) (6, -1)
which is totally wrong because from the IS for this overlapping subdomain and the big matrix A, row 1 should have non-zeros like the output from PCASM.
I guess the problem is due to that we set PCASMSetSortIndices to FALSE and something goes wrong in asm.c and gasm.c.
Thanks! Hui
<GASM_test.zip>
On Sun, May 20, 2012 at 5:06 AM, Hui Zhang <[email protected]> wrote: I have the following MatView output:
row 0: (0, -0.0351719 - 0.638814 i) (1, -0.517586 - 0.617823 i) (4, -0.517586 - 0.617823 i) (1, -1.03517) (5, -0.508793)
Note that there are two values for the column 1, is this normal?
The above matrix is obtained from PCASMGetSubKSP(pc,subksp) and KSPGetOperators(subksp[0], ...).
This is odd. Can you explain a bit more what leads up to this, so we can try and reproduce the problem? Thanks. Dmitry.
Thanks!
On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling your modification subroutine), but not the subdomains or the subdomain solvers. If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures. As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
Dmitry.
On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <[email protected]> wrote: I just have a question about reuse of PCASM or PCGASM. Suppose I have seted up the PCASM and related KSP and I solved one time. Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with different user context for the modifying routine.
What can I do for this task? Currently, I destroy the KSP and re-construct it. I guess even for PCASM I can re-use it because the partition of subdomains remain the same.
Thanks!
On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
Hui, There've been several changes to PCGASM ahead of the new release. Let me go back and see if it affected the convergence problem. Dmitry.
On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <[email protected]> wrote: Hi Dmitry,
is there any news about PCGASM?
thanks, Hui
On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
Okay, thanks. I'll take a look.
Dmitry.
On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <[email protected]> wrote: For reference, my results are attached.
asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes
thank you, Hui
On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <[email protected]> wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]> wrote: I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not. Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor? I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference. That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
Okay, thanks. I'll look into it. Dmitry. On Mon, May 21, 2012 at 2:11 PM, Hui Zhang <[email protected]>wrote:
Sorry, I uses PCASMSetSortIndices for GASM. Now, I correct it and the output becomes the same as PCASM. See attached.
On May 21, 2012, at 8:59 PM, Hui Zhang wrote:
Dmitry,
finally, I am able to make a simple program and reproduce the error. Since I had moved to petsc-dev (before I used petsc-3.2), the error is a little different.
Download the program attached and run the run.sh. You will see the output from PCASM. There is a submatrix (submat[1] on rank 0) which has a row like
row 1: (0, -1) (1, 0 + 0.4 i) (4, -1) (2, -1)
Note that the column 2 is placed in the last and the column numbers are not in the normal order. Following that, we can also see the error message like
[0]PETSC ERROR: New nonzero at (1,2) caused a malloc!
You can also try with GASM, use
mpirun -np 2 ./gasm_test -dd_type gasm -n 4 -px 1 -py 2 -sx 2 -sy 1
which partitions the domain to 1 * 2 and mapped to the two processors, and for each processor we further partition its local domain to 2 * 1 subdomains.
Then, you will see that the row 1 of submat[1] on rank 1 becomes
row 1: (0, -1) (1, 0 + 0.4 i) (3, -1) (6, -1)
which is totally wrong because from the IS for this overlapping subdomain and the big matrix A, row 1 should have non-zeros like the output from PCASM.
I guess the problem is due to that we set PCASMSetSortIndices to FALSE and something goes wrong in asm.c and gasm.c.
Thanks! Hui
<GASM_test.zip>
On Sun, May 20, 2012 at 5:06 AM, Hui Zhang <[email protected]>wrote:
I have the following MatView output:
row 0: (0, -0.0351719 - 0.638814 i) (1, -0.517586 - 0.617823 i) (4, -0.517586 - 0.617823 i) (1, -1.03517) (5, -0.508793)
Note that there are two values for the column 1, is this normal?
The above matrix is obtained from PCASMGetSubKSP(pc,subksp) and KSPGetOperators(subksp[0], ...).
This is odd. Can you explain a bit more what leads up to this, so we can try and reproduce the problem? Thanks. Dmitry.
Thanks!
On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling your modification subroutine), but not the subdomains or the subdomain solvers. If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures. As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
Dmitry.
On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <[email protected]>wrote:
I just have a question about reuse of PCASM or PCGASM. Suppose I have seted up the PCASM and related KSP and I solved one time. Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with different user context for the modifying routine.
What can I do for this task? Currently, I destroy the KSP and re-construct it. I guess even for PCASM I can re-use it because the partition of subdomains remain the same.
Thanks!
On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
Hui, There've been several changes to PCGASM ahead of the new release. Let me go back and see if it affected the convergence problem. Dmitry.
On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <[email protected]>wrote:
Hi Dmitry,
is there any news about PCGASM?
thanks, Hui
On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
Okay, thanks. I'll take a look.
Dmitry.
On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <[email protected]
wrote:
For reference, my results are attached.
asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes
thank you, Hui
On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang < [email protected]> wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang < [email protected]> wrote:
I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not.
Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference.
That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <<[email protected]> [email protected]> wrote:
About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
Hi, It might be unclear. The error here is that the submat[1] does have a (1,2) entry, but MatSetValues at (1,2) gives an error message. GASM and ASM give the same error. Thanks! On May 21, 2012, at 9:11 PM, Hui Zhang wrote:
Sorry, I uses PCASMSetSortIndices for GASM. Now, I correct it and the output becomes the same as PCASM. See attached.
<GASM_test.zip>
On May 21, 2012, at 8:59 PM, Hui Zhang wrote:
Dmitry,
finally, I am able to make a simple program and reproduce the error. Since I had moved to petsc-dev (before I used petsc-3.2), the error is a little different.
Download the program attached and run the run.sh. You will see the output from PCASM. There is a submatrix (submat[1] on rank 0) which has a row like
row 1: (0, -1) (1, 0 + 0.4 i) (4, -1) (2, -1)
Note that the column 2 is placed in the last and the column numbers are not in the normal order. Following that, we can also see the error message like
[0]PETSC ERROR: New nonzero at (1,2) caused a malloc!
You can also try with GASM, use
mpirun -np 2 ./gasm_test -dd_type gasm -n 4 -px 1 -py 2 -sx 2 -sy 1
which partitions the domain to 1 * 2 and mapped to the two processors, and for each processor we further partition its local domain to 2 * 1 subdomains.
Then, you will see that the row 1 of submat[1] on rank 1 becomes
row 1: (0, -1) (1, 0 + 0.4 i) (3, -1) (6, -1)
which is totally wrong because from the IS for this overlapping subdomain and the big matrix A, row 1 should have non-zeros like the output from PCASM.
I guess the problem is due to that we set PCASMSetSortIndices to FALSE and something goes wrong in asm.c and gasm.c.
Thanks! Hui
<GASM_test.zip>
On Sun, May 20, 2012 at 5:06 AM, Hui Zhang <[email protected]> wrote: I have the following MatView output:
row 0: (0, -0.0351719 - 0.638814 i) (1, -0.517586 - 0.617823 i) (4, -0.517586 - 0.617823 i) (1, -1.03517) (5, -0.508793)
Note that there are two values for the column 1, is this normal?
The above matrix is obtained from PCASMGetSubKSP(pc,subksp) and KSPGetOperators(subksp[0], ...).
This is odd. Can you explain a bit more what leads up to this, so we can try and reproduce the problem? Thanks. Dmitry.
Thanks!
On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling your modification subroutine), but not the subdomains or the subdomain solvers. If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures. As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
Dmitry.
On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <[email protected]> wrote: I just have a question about reuse of PCASM or PCGASM. Suppose I have seted up the PCASM and related KSP and I solved one time. Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with different user context for the modifying routine.
What can I do for this task? Currently, I destroy the KSP and re-construct it. I guess even for PCASM I can re-use it because the partition of subdomains remain the same.
Thanks!
On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
Hui, There've been several changes to PCGASM ahead of the new release. Let me go back and see if it affected the convergence problem. Dmitry.
On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <[email protected]> wrote: Hi Dmitry,
is there any news about PCGASM?
thanks, Hui
On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
Okay, thanks. I'll take a look.
Dmitry.
On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <[email protected]> wrote: For reference, my results are attached.
asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes
thank you, Hui
On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <[email protected]> wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]> wrote: I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not. Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor? I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference. That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
Hui, There appears to be a bug in MatGetSubMatrices() which inserts submatrix column indices in the same order as they appeared in the original matrix. If a column IS has an inversion -- IS[2] = 8 and IS[6] = 4, as in your case -- then the submatrix column index 2 will appear after the submatrix column index 4, since that's how they were ordered in the original matrix (4 < 8, and the column indices were properly sorted). This violates the ordering of column indices, which prevents binary search from finding column index 2. That results in an apparent new nonzero insertion, which is in turn caught, since by default no new nonzeros are allowed. I'm looking into this, and I'm not sure that there is an easy workaround for you at this time: if you sort the IS indices, it will violate the implicit local-to-global submatrix mapping you are using in your ModifySubMatrices(). Thanks. Dmitry. On Mon, May 21, 2012 at 2:33 PM, Hui Zhang <[email protected]>wrote:
Hi,
It might be unclear. The error here is that the submat[1] does have a (1,2) entry, but MatSetValues at (1,2) gives an error message. GASM and ASM give the same error.
Thanks!
On May 21, 2012, at 9:11 PM, Hui Zhang wrote:
Sorry, I uses PCASMSetSortIndices for GASM. Now, I correct it and the output becomes the same as PCASM. See attached.
<GASM_test.zip>
On May 21, 2012, at 8:59 PM, Hui Zhang wrote:
Dmitry,
finally, I am able to make a simple program and reproduce the error. Since I had moved to petsc-dev (before I used petsc-3.2), the error is a little different.
Download the program attached and run the run.sh. You will see the output from PCASM. There is a submatrix (submat[1] on rank 0) which has a row like
row 1: (0, -1) (1, 0 + 0.4 i) (4, -1) (2, -1)
Note that the column 2 is placed in the last and the column numbers are not in the normal order. Following that, we can also see the error message like
[0]PETSC ERROR: New nonzero at (1,2) caused a malloc!
You can also try with GASM, use
mpirun -np 2 ./gasm_test -dd_type gasm -n 4 -px 1 -py 2 -sx 2 -sy 1
which partitions the domain to 1 * 2 and mapped to the two processors, and for each processor we further partition its local domain to 2 * 1 subdomains.
Then, you will see that the row 1 of submat[1] on rank 1 becomes
row 1: (0, -1) (1, 0 + 0.4 i) (3, -1) (6, -1)
which is totally wrong because from the IS for this overlapping subdomain and the big matrix A, row 1 should have non-zeros like the output from PCASM.
I guess the problem is due to that we set PCASMSetSortIndices to FALSE and something goes wrong in asm.c and gasm.c.
Thanks! Hui
<GASM_test.zip>
On Sun, May 20, 2012 at 5:06 AM, Hui Zhang <[email protected]>wrote:
I have the following MatView output:
row 0: (0, -0.0351719 - 0.638814 i) (1, -0.517586 - 0.617823 i) (4, -0.517586 - 0.617823 i) (1, -1.03517) (5, -0.508793)
Note that there are two values for the column 1, is this normal?
The above matrix is obtained from PCASMGetSubKSP(pc,subksp) and KSPGetOperators(subksp[0], ...).
This is odd. Can you explain a bit more what leads up to this, so we can try and reproduce the problem? Thanks. Dmitry.
Thanks!
On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling your modification subroutine), but not the subdomains or the subdomain solvers. If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures. As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
Dmitry.
On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <[email protected]>wrote:
I just have a question about reuse of PCASM or PCGASM. Suppose I have seted up the PCASM and related KSP and I solved one time. Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with different user context for the modifying routine.
What can I do for this task? Currently, I destroy the KSP and re-construct it. I guess even for PCASM I can re-use it because the partition of subdomains remain the same.
Thanks!
On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
Hui, There've been several changes to PCGASM ahead of the new release. Let me go back and see if it affected the convergence problem. Dmitry.
On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <[email protected]>wrote:
Hi Dmitry,
is there any news about PCGASM?
thanks, Hui
On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
Okay, thanks. I'll take a look.
Dmitry.
On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <[email protected]
wrote:
For reference, my results are attached.
asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes
thank you, Hui
On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang < [email protected]> wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang < [email protected]> wrote:
I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not.
Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor?
I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference.
That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <<[email protected]> [email protected]> wrote:
About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
Dmitry, thanks! For the moment, I can use the sorted IS and I just need to pass through user context some AO or ISLocalToGlobalMapping. Hope you will get the unsorted IS case work soon! Thanks! On May 22, 2012, at 8:04 PM, Dmitry Karpeev wrote:
Hui, There appears to be a bug in MatGetSubMatrices() which inserts submatrix column indices in the same order as they appeared in the original matrix. If a column IS has an inversion -- IS[2] = 8 and IS[6] = 4, as in your case -- then the submatrix column index 2 will appear after the submatrix column index 4, since that's how they were ordered in the original matrix (4 < 8, and the column indices were properly sorted). This violates the ordering of column indices, which prevents binary search from finding column index 2. That results in an apparent new nonzero insertion, which is in turn caught, since by default no new nonzeros are allowed.
I'm looking into this, and I'm not sure that there is an easy workaround for you at this time: if you sort the IS indices, it will violate the implicit local-to-global submatrix mapping you are using in your ModifySubMatrices().
Thanks. Dmitry.
On Mon, May 21, 2012 at 2:33 PM, Hui Zhang <[email protected]> wrote: Hi,
It might be unclear. The error here is that the submat[1] does have a (1,2) entry, but MatSetValues at (1,2) gives an error message. GASM and ASM give the same error.
Thanks!
On May 21, 2012, at 9:11 PM, Hui Zhang wrote:
Sorry, I uses PCASMSetSortIndices for GASM. Now, I correct it and the output becomes the same as PCASM. See attached.
<GASM_test.zip>
On May 21, 2012, at 8:59 PM, Hui Zhang wrote:
Dmitry,
finally, I am able to make a simple program and reproduce the error. Since I had moved to petsc-dev (before I used petsc-3.2), the error is a little different.
Download the program attached and run the run.sh. You will see the output from PCASM. There is a submatrix (submat[1] on rank 0) which has a row like
row 1: (0, -1) (1, 0 + 0.4 i) (4, -1) (2, -1)
Note that the column 2 is placed in the last and the column numbers are not in the normal order. Following that, we can also see the error message like
[0]PETSC ERROR: New nonzero at (1,2) caused a malloc!
You can also try with GASM, use
mpirun -np 2 ./gasm_test -dd_type gasm -n 4 -px 1 -py 2 -sx 2 -sy 1
which partitions the domain to 1 * 2 and mapped to the two processors, and for each processor we further partition its local domain to 2 * 1 subdomains.
Then, you will see that the row 1 of submat[1] on rank 1 becomes
row 1: (0, -1) (1, 0 + 0.4 i) (3, -1) (6, -1)
which is totally wrong because from the IS for this overlapping subdomain and the big matrix A, row 1 should have non-zeros like the output from PCASM.
I guess the problem is due to that we set PCASMSetSortIndices to FALSE and something goes wrong in asm.c and gasm.c.
Thanks! Hui
<GASM_test.zip>
On Sun, May 20, 2012 at 5:06 AM, Hui Zhang <[email protected]> wrote: I have the following MatView output:
row 0: (0, -0.0351719 - 0.638814 i) (1, -0.517586 - 0.617823 i) (4, -0.517586 - 0.617823 i) (1, -1.03517) (5, -0.508793)
Note that there are two values for the column 1, is this normal?
The above matrix is obtained from PCASMGetSubKSP(pc,subksp) and KSPGetOperators(subksp[0], ...).
This is odd. Can you explain a bit more what leads up to this, so we can try and reproduce the problem? Thanks. Dmitry.
Thanks!
On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote:
You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and recreate the matrices (including calling your modification subroutine), but not the subdomains or the subdomain solvers. If you just want to modify the submatrices, you can call PC(G)ASMGetSubmatrices() and modify the matrices it returns (in the same order as the subdomains were set). That's a bit of a hack, since you will essentially be modifying the PC's internal data structures. As long as you are careful, you should be okay, since you already effectively have the same type of access to the submatrices through the Modify callback.
Dmitry.
On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <[email protected]> wrote: I just have a question about reuse of PCASM or PCGASM. Suppose I have seted up the PCASM and related KSP and I solved one time. Next for the same linear system (matrix and RHS), I just want PCASM modify the submatrices (PCSetModifySubmatrices) in a different way, using the same routine for modifying but with different user context for the modifying routine.
What can I do for this task? Currently, I destroy the KSP and re-construct it. I guess even for PCASM I can re-use it because the partition of subdomains remain the same.
Thanks!
On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote:
Hui, There've been several changes to PCGASM ahead of the new release. Let me go back and see if it affected the convergence problem. Dmitry.
On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <[email protected]> wrote: Hi Dmitry,
is there any news about PCGASM?
thanks, Hui
On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote:
Okay, thanks. I'll take a look.
Dmitry.
On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <[email protected]> wrote: For reference, my results are attached.
asm1.txt for asm with 1 process, asm2.txt for asm with 2 processes, gasm1.txt for gasm with 1 process, (with the iteration numbers different from others) gasm2.txt for gasm with 2 processes
thank you, Hui
On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote:
On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <[email protected]> wrote:
On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote:
On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <[email protected]> wrote: I have a new problem: the results from ASM and GASM are different and it seems GASM has something wrong with SetModifySubMatrices. Numerical tests are with each subdomain supported only by one subdomain. There are no problems when I did not modify submatrices. But when I modify submatrices, there are problems with GASM but no problems with ASM.
For example, I use two subdomains. In the first case each subdomain is supported by one processor and there seems no problem with GASM. But when I use run my program with only one proc. so that it supports both of the two subdomains, the iteration number is different from the first case and is much larger. On the other hand ASM has no such problem.
Are the solutions the same? What problem are you solving?
Yes, the solutions are the same. That's why ASM gives the same results with one or two processors. But GASM did not. Sorry, I wasn't clear: ASM and GASM produced different solutions in the case of two domains per processor? I'm solving the Helmholtz equation. Maybe I can prepare a simpler example to show this difference. That would be helpful. Thanks.
Dmitry.
Dmitry.
On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote:
You should be able to. This behavior is the same as in PCASM, except in GASM the matrices live on subcommunicators. I am in transit right now, but I can take a closer look in Friday.
Dmitry
On Feb 15, 2012, at 8:07, Hui Zhang <[email protected]> wrote:
On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote:
Hi Dmitry,
thanks a lot! Currently, I'm not using ISColoring. Just comes another question on PCGASMSetModifySubMatrices(). The user provided function has the prototype
func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx);
I think the coloumns from the parameter 'col' are always the same as the rows from the parameter 'row'. Because PCGASMSetLocalSubdomains() only accepts index sets but not rows and columns. Has I misunderstood something?
As I tested, the row and col are always the same.
I have a new question. Am I allowed to SetLocalToGlobalMapping() for the submat's in the above func()?
thanks, Hui
thanks, Hui
On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote:
Yes, that's right. There is no good way to help the user assemble the subdomains at the moment beyond the 2D stuff. It is expected that they are generated from mesh subdomains. Each IS does carry the subdomains subcomm.
There is ISColoringToList() that is supposed to convert a "coloring" of indices to an array of ISs, each having the indices with the same color and the subcomm that supports that color. It is largely untested, though. You could try using it and give us feedback on any problems you encounter.
Dmitry.
On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <[email protected]> wrote: About PCGASMSetLocalSubdomains(), in the case of one subdomain supported by multiple processors, shall I always create the arguments 'is[s]' and 'is_local[s]' in a subcommunicator consisting of processors supporting the subdomain 's'?
The source code of PCGASMCreateSubdomains2D() seemingly does so.
Thanks, Hui
participants (4)
-
Barry Smith -
Dmitry Karpeev -
Dmitry Karpeev -
Hui Zhang