June 2016 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] MatMFFDSetType
by Barry Smith 26 Jun '16

26 Jun '16

> On Jun 26, 2016, at 8:30 PM, Adrian Croucher <a.croucher(a)auckland.ac.nz> wrote: > > hi > > Sorry, I realise now I was barking up the wrong tree in trying to use MatMFFDSetType(). > > What I'm actually looking for is just the equivalent of setting the command line option "-mat_fd_type ds" as default in my code- for an ordinary Jacobian matrix, not for a matrix-free method. > > The doc page on MatFDColoringSetFromOptions(): > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/MatFD/MatFDColo… > > mentions setting the fd type to 'wp' or 'ds', and links to pages about MATMFFD_WP and MATMFFD_DS, which is probably where I started going astray. > > > How do I do the equivalent of setting the option "-mat_fd_type ds" in my code? There is MatMFFDSetType() for MATMFFD, but doesn't seem to be anything like a plain MatFDSetType(). Yes, you are correct, there is no MatFDSetType() and in the SetFromOptions we totally cheat http://www.mcs.anl.gov/petsc/petsc-current/src/mat/matfd/fdmatrix.c.html#Ma… and just stick the value directly into the data structure. For now you can call PetscOptionsSetValue(NULL,"-mat_fd_type","ds") in your code (before you start using the MatFD stuff) and we'll add the routine to the master branch. Barry I didn't think anyone would care about the difference between ds and wp with MatFD. > > - Adrian > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > University of Auckland, New Zealand > email: > a.croucher(a)auckland.ac.nz > > tel: +64 (0)9 923 84611 >

1 0

Re: [petsc-dev] MatMFFDSetType
by Adrian Croucher 26 Jun '16

26 Jun '16

hi Sorry, I realise now I was barking up the wrong tree in trying to use MatMFFDSetType(). What I'm actually looking for is just the equivalent of setting the command line option "-mat_fd_type ds" as default in my code- for an ordinary Jacobian matrix, not for a matrix-free method. The doc page on MatFDColoringSetFromOptions(): http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/MatFD/MatFDColo… mentions setting the fd type to 'wp' or 'ds', and links to pages about MATMFFD_WP and MATMFFD_DS, which is probably where I started going astray. How do I do the equivalent of setting the option "-mat_fd_type ds" in my code? There is MatMFFDSetType() for MATMFFD, but doesn't seem to be anything like a plain MatFDSetType(). - Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher(a)auckland.ac.nz tel: +64 (0)9 923 84611

1 0

Re: [petsc-dev] Question about PCGASMSetSubdomains()
by Barry Smith 26 Jun '16

26 Jun '16

> On Jun 26, 2016, at 6:41 PM, Fande Kong <fdkong.jd(a)gmail.com> wrote: > > Hi Barry, > > There is no need to support osm->n>1 in GASM, I think. There should be only one local subdomain on each process. These subdomains are grouped to form a few multi-rank subdomains. A multi-rank subdomain is recognized using the sub-communcator. A multi-rank subdomain could be solved using superlu in parallel. The reason for doing those things is to improve the converge rate, and to fix the number of multi-rank subdomains when increasing the number of processor cores for the scalability. This is a very special case. Should this case be a different PC or a special case of PCGASM? Since you already have the code for much more complex configurations in GASM it seems a pity to throw away all that code? Barry > > In other words, we want to form a "large" subproblem for solving. But on the other hand, we allow more than one local subdomains to make subproblems "small". If people fall into the latter case, should just use ASM instead of GASM. If users want to group the ith (numbered in local) subdomains to form the ith multi-rank subdomains. They should just use the filedsplit preconditioner, where a GASM is applied to each field. > > How do you think? > > Fande Kong, > > On Sun, Jun 26, 2016 at 4:54 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > On Jun 26, 2016, at 5:48 PM, Fande Kong <fdkong.jd(a)gmail.com> wrote: > > > > Barry, > > > > Good questions. > > > > Yes, we are assuming the index sets on each process are somehow ordered. There is no way to check this. How to decompose the domain is determined by users, and they could choose any ways they want. > > > > But we could check this condition. The number of local subdomains should be the same on all processors, that is, osm->n should be the same value on all processors. > > But this is in that sub communicator. If different processes have the sub communicators in a different order then the communication for GASM will hang as some process is trying to communicate with the "wrong set" of other processes. Hmm, doesn't seem to be any easy way to detect this. > > Also can you document this requirement in the manual page for PCGASMSetSubdomains()? > > Thanks > > Barry > > > > > Fande, > > > > On Sun, Jun 26, 2016 at 4:31 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > > Fande, > > > > How does PCGASMSetSubdomains() know which subdomain from an index set on one process is connected to an index set on another processor? Are you assuming the index sets on each process are somehow ordered to make sure that the right parts on each process connect to form the entire domain? What if someone passed them in a different order on some process, is there anyway to error check that? > > > > Thanks > > > > Barry > > > > > >

1 0

Re: [petsc-dev] Question about PCGASMSetSubdomains()
by Fande Kong 26 Jun '16

26 Jun '16

Hi Barry, There is no need to support osm->n>1 in GASM, I think. There should be only one local subdomain on each process. These subdomains are grouped to form a few multi-rank subdomains. A multi-rank subdomain is recognized using the sub-communcator. A multi-rank subdomain could be solved using superlu in parallel. The reason for doing those things is to improve the converge rate, and to fix the number of multi-rank subdomains when increasing the number of processor cores for the scalability. In other words, we want to form a "large" subproblem for solving. But on the other hand, we allow more than one local subdomains to make subproblems "small". If people fall into the latter case, should just use ASM instead of GASM. If users want to group the ith (numbered in local) subdomains to form the ith multi-rank subdomains. They should just use the filedsplit preconditioner, where a GASM is applied to each field. How do you think? Fande Kong, On Sun, Jun 26, 2016 at 4:54 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > On Jun 26, 2016, at 5:48 PM, Fande Kong <fdkong.jd(a)gmail.com> wrote: > > > > Barry, > > > > Good questions. > > > > Yes, we are assuming the index sets on each process are somehow ordered. > There is no way to check this. How to decompose the domain is determined by > users, and they could choose any ways they want. > > > > But we could check this condition. The number of local subdomains > should be the same on all processors, that is, osm->n should be the same > value on all processors. > > But this is in that sub communicator. If different processes have the > sub communicators in a different order then the communication for GASM will > hang as some process is trying to communicate with the "wrong set" of other > processes. Hmm, doesn't seem to be any easy way to detect this. > > Also can you document this requirement in the manual page for > PCGASMSetSubdomains()? > > Thanks > > Barry > > > > > Fande, > > > > On Sun, Jun 26, 2016 at 4:31 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > > Fande, > > > > How does PCGASMSetSubdomains() know which subdomain from an index > set on one process is connected to an index set on another processor? Are > you assuming the index sets on each process are somehow ordered to make > sure that the right parts on each process connect to form the entire > domain? What if someone passed them in a different order on some process, > is there anyway to error check that? > > > > Thanks > > > > Barry > > > > > >

1 0

Re: [petsc-dev] Question about PCGASMSetSubdomains()
by Barry Smith 26 Jun '16

26 Jun '16

> On Jun 26, 2016, at 5:48 PM, Fande Kong <fdkong.jd(a)gmail.com> wrote: > > Barry, > > Good questions. > > Yes, we are assuming the index sets on each process are somehow ordered. There is no way to check this. How to decompose the domain is determined by users, and they could choose any ways they want. > > But we could check this condition. The number of local subdomains should be the same on all processors, that is, osm->n should be the same value on all processors. But this is in that sub communicator. If different processes have the sub communicators in a different order then the communication for GASM will hang as some process is trying to communicate with the "wrong set" of other processes. Hmm, doesn't seem to be any easy way to detect this. Also can you document this requirement in the manual page for PCGASMSetSubdomains()? Thanks Barry > > Fande, > > On Sun, Jun 26, 2016 at 4:31 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Fande, > > How does PCGASMSetSubdomains() know which subdomain from an index set on one process is connected to an index set on another processor? Are you assuming the index sets on each process are somehow ordered to make sure that the right parts on each process connect to form the entire domain? What if someone passed them in a different order on some process, is there anyway to error check that? > > Thanks > > Barry > >

1 0

Re: [petsc-dev] Question about PCGASMSetSubdomains()
by Fande Kong 26 Jun '16

26 Jun '16

Barry, Good questions. Yes, we are assuming the index sets on each process are somehow ordered. There is no way to check this. How to decompose the domain is determined by users, and they could choose any ways they want. But we could check this condition. The number of local subdomains should be the same on all processors, that is, osm->n should be the same value on all processors. Fande, On Sun, Jun 26, 2016 at 4:31 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Fande, > > How does PCGASMSetSubdomains() know which subdomain from an index set > on one process is connected to an index set on another processor? Are you > assuming the index sets on each process are somehow ordered to make sure > that the right parts on each process connect to form the entire domain? > What if someone passed them in a different order on some process, is there > anyway to error check that? > > Thanks > > Barry > >

1 0

Question about PCGASMSetSubdomains()
by Barry Smith 26 Jun '16

26 Jun '16

Fande, How does PCGASMSetSubdomains() know which subdomain from an index set on one process is connected to an index set on another processor? Are you assuming the index sets on each process are somehow ordered to make sure that the right parts on each process connect to form the entire domain? What if someone passed them in a different order on some process, is there anyway to error check that? Thanks Barry

1 0

Re: [petsc-dev] asm / gasm
by Barry Smith 26 Jun '16

26 Jun '16

I have changed the GAMG aggs support to use PCASM and not PCGASM. I don't see how it ever worked with PCGASM; very strange. Maybe it was written and tested with PCASM and then later someone changed to PCGASM and did not test it again. I was wrong when I said it would be difficult to change to PCASM, it was much simpler to change to PCASM then I thought; this is because I assumed it had been done correctly for PCGASM while in fact though it used PCGASM in the code it followed the model for what PCASM needed :-(. Anyways I have a branch barry/fix-gamg-asm-aggs that now works for my ex10 tests I will continue to clean up the branch, fixing naming styles and test with more difficult GAMG examples. And add proper nightly tests for this functionality which it never had before. In addition Fande is adding error checking to PCGASM so if you pass it badly formatted subdomain information (like was passed from GAMG) it will generate a very useful error message instead of just chugging along with gibberish. Barry Mark, my confusion came from that fact that a single MPI process owns each of the aggs; that is the list of degrees of freedom for each agg is all on one process. This is exactly what PCASM needs but NOT what PCGASM needs. > On Jun 26, 2016, at 2:12 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > On Sun, Jun 26, 2016 at 5:43 PM, Fande Kong <fdkong.jd(a)gmail.com> wrote: > Mark, > > Size of the index set is smaller than size of pmat in your case? > > I've added in the BC vertices into a dummy domain, which will just be a diagonal matrix, so there is complete "cover" now. It works in serial. > > It does not work in parallel. If I change the number of processor, for ex56 in ksp/kps/examples/tutorials, from 1 to 8 I get this error: > > mark/gamg-agg-gasm *= ~/Codes/petsc/src/ksp/ksp/examples/tutorials$ make runex56 > 1,4c1,198 > < Linear solve converged due to CONVERGED_RTOL iterations 8 > < Linear solve converged due to CONVERGED_RTOL iterations 8 > < Linear solve converged due to CONVERGED_RTOL iterations 8 > < [0]main |b-Ax|/|b|=1.940043e-04, |b|=4.969822e+00, emax=9.926090e-01 > --- > > [4]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [4]PETSC ERROR: Nonconforming object sizes > > [4]PETSC ERROR: Vector wrong size 192 for scatter 201 (scatter forward and vector from != ctx from size) > > > They should be the same. I do not think we ever considered the index set has a different size from that of Pmat. > > Fande, > > On Sun, Jun 26, 2016 at 7:09 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > GASM does assume the index set includes every equation in the matrix. It should probably check this as it has pmat. > > I guess I can add these BC vertices in. > > On Sun, Jun 26, 2016 at 11:20 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > Fande, > > An alternative debug path that may be simpler and direct, as it works with master: 'make runex56' in ksp/examples/tutorials in branch mark/gamg-agg-asm. > > This runs clean in valgrind (for me), I've added an ISView call to see the data that causes the error, it runs on one processor, uses a 3^3 cell grid (tiny), it exits cleanly with the error: > > .... > 87 189 > 88 190 > 89 191 > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Argument out of range > > [0]PETSC ERROR: Index 144 at 42 location greater than max 144 > [0]PETSC ERROR: #1 VecScatterCheckIndices_Private() line 39 in /Users/markadams/Codes/petsc/src/vec/vec/utils/vscat.c > [0]PETSC ERROR: #2 VecScatterCreate() line 1227 in /Users/markadams/Codes/petsc/src/vec/vec/utils/vscat.c > [0]PETSC ERROR: #3 PCSetUp_GASM() line 481 in /Users/markadams/Codes/petsc/src/ksp/pc/impls/gasm/gasm.c > .... > > This index 144 is very suspicious to me because there are 144 REAL dofs in this test, 192 "gross" dofs, minus (48) boundary conditions. This problem has just two aggregates with sizes 54 & 90 (=144). (I strip out the BC vertices.) > > Maybe GASM is getting confused because I do not give it domains that cover the entire mesh (I strip out BC vertices). GASM thinks there are 144 equations in this system when in fact there are 192. > > It looks to me like GASM is working a space stripped of BCs but it is using my indices that are in the full space. > > Mark > > > > >

1 0

Re: [petsc-dev] asm / gasm
by Mark Adams 26 Jun '16

26 Jun '16

On Sun, Jun 26, 2016 at 5:43 PM, Fande Kong <fdkong.jd(a)gmail.com> wrote: > Mark, > > Size of the index set is smaller than size of pmat in your case? > I've added in the BC vertices into a dummy domain, which will just be a diagonal matrix, so there is complete "cover" now. It works in serial. It does not work in parallel. If I change the number of processor, for ex56 in ksp/kps/examples/tutorials, from 1 to 8 I get this error: mark/gamg-agg-gasm *= ~/Codes/petsc/src/ksp/ksp/examples/tutorials$ make runex56 1,4c1,198 < Linear solve converged due to CONVERGED_RTOL iterations 8 < Linear solve converged due to CONVERGED_RTOL iterations 8 < Linear solve converged due to CONVERGED_RTOL iterations 8 < [0]main |b-Ax|/|b|=1.940043e-04, |b|=4.969822e+00, emax=9.926090e-01 --- > [4]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [4]PETSC ERROR: Nonconforming object sizes > [4]PETSC ERROR: Vector wrong size 192 for scatter 201 (scatter forward and vector from != ctx from size) > They should be the same. I do not think we ever considered the index set > has a different size from that of Pmat. > > Fande, > > On Sun, Jun 26, 2016 at 7:09 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > >> GASM does assume the index set includes every equation in the matrix. It >> should probably check this as it has pmat. >> >> I guess I can add these BC vertices in. >> >> On Sun, Jun 26, 2016 at 11:20 AM, Mark Adams <mfadams(a)lbl.gov> wrote: >> >>> Fande, >>> >>> An alternative debug path that may be simpler and direct, as it works >>> with master: 'make runex56' in ksp/examples/tutorials in >>> branch mark/gamg-agg-asm. >>> >>> This runs clean in valgrind (for me), I've added an ISView call to see >>> the data that causes the error, it runs on one processor, uses a 3^3 cell >>> grid (tiny), it exits cleanly with the error: >>> >>> .... >>> 87 189 >>> 88 190 >>> 89 191 >>> > [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> > [0]PETSC ERROR: Argument out of range >>> > [0]PETSC ERROR: Index 144 at 42 location greater than max 144 >>> [0]PETSC ERROR: #1 VecScatterCheckIndices_Private() line 39 in >>> /Users/markadams/Codes/petsc/src/vec/vec/utils/vscat.c >>> [0]PETSC ERROR: #2 VecScatterCreate() line 1227 in >>> /Users/markadams/Codes/petsc/src/vec/vec/utils/vscat.c >>> [0]PETSC ERROR: #3 PCSetUp_GASM() line 481 in >>> /Users/markadams/Codes/petsc/src/ksp/pc/impls/gasm/gasm.c >>> .... >>> >>> This index 144 is very suspicious to me because there are 144 REAL dofs >>> in this test, 192 "gross" dofs, minus (48) boundary conditions. This >>> problem has just two aggregates with sizes 54 & 90 (=144). (I strip out the >>> BC vertices.) >>> >>> Maybe GASM is getting confused because I do not give it domains that >>> cover the entire mesh (I strip out BC vertices). GASM thinks there are 144 >>> equations in this system when in fact there are 192. >>> >>> It looks to me like GASM is working a space stripped of BCs but it is >>> using my indices that are in the full space. >>> >>> Mark >>> >>> >>> >> >

1 0

Re: [petsc-dev] asm / gasm
by Fande Kong 26 Jun '16

26 Jun '16

Mark, Size of the index set is smaller than size of pmat in your case? They should be the same. I do not think we ever considered the index set has a different size from that of Pmat. Fande, On Sun, Jun 26, 2016 at 7:09 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > GASM does assume the index set includes every equation in the matrix. It > should probably check this as it has pmat. > > I guess I can add these BC vertices in. > > On Sun, Jun 26, 2016 at 11:20 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > >> Fande, >> >> An alternative debug path that may be simpler and direct, as it works >> with master: 'make runex56' in ksp/examples/tutorials in >> branch mark/gamg-agg-asm. >> >> This runs clean in valgrind (for me), I've added an ISView call to see >> the data that causes the error, it runs on one processor, uses a 3^3 cell >> grid (tiny), it exits cleanly with the error: >> >> .... >> 87 189 >> 88 190 >> 89 191 >> > [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> > [0]PETSC ERROR: Argument out of range >> > [0]PETSC ERROR: Index 144 at 42 location greater than max 144 >> [0]PETSC ERROR: #1 VecScatterCheckIndices_Private() line 39 in >> /Users/markadams/Codes/petsc/src/vec/vec/utils/vscat.c >> [0]PETSC ERROR: #2 VecScatterCreate() line 1227 in >> /Users/markadams/Codes/petsc/src/vec/vec/utils/vscat.c >> [0]PETSC ERROR: #3 PCSetUp_GASM() line 481 in >> /Users/markadams/Codes/petsc/src/ksp/pc/impls/gasm/gasm.c >> .... >> >> This index 144 is very suspicious to me because there are 144 REAL dofs >> in this test, 192 "gross" dofs, minus (48) boundary conditions. This >> problem has just two aggregates with sizes 54 & 90 (=144). (I strip out the >> BC vertices.) >> >> Maybe GASM is getting confused because I do not give it domains that >> cover the entire mesh (I strip out BC vertices). GASM thinks there are 144 >> equations in this system when in fact there are 192. >> >> It looks to me like GASM is working a space stripped of BCs but it is >> using my indices that are in the full space. >> >> Mark >> >> >> >

1 0