December 2018 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Implementing of a variable block size BILU preconditioner
by Jed Brown 04 Dec '18

04 Dec '18

Ali Reza Khaz'ali <arkhazali(a)cc.iut.ac.ir> writes: > Dear Jed, > > ILU with BAIJ works, and its performance in reducing the condition number is slightly better than PCVPBJACOBI. Thanks for your guidance. Is it ILU(0)? Did you need to turn enable shifts? Can you write out a small matrix that succeeds with BAIJ/ILU, but not with AIJ/ILU so we can compare? > Best wishes, > Ali > > -----Original Message----- > From: Jed Brown <jed(a)jedbrown.org> > Sent: Tuesday, December 04, 2018 9:40 PM > To: Ali Reza Khaz'ali <arkhazali(a)cc.iut.ac.ir>; 'Smith, Barry F.' <bsmith(a)mcs.anl.gov> > Cc: petsc-dev(a)mcs.anl.gov > Subject: RE: Re[2]: [petsc-dev] Implementing of a variable block size BILU preconditioner > > Ali Reza Khaz'ali <arkhazali(a)cc.iut.ac.ir> writes: > >> Dear Jed, >> >> Thanks for your kind answer. I thought Scalar BJACOBI does not need >> data from the other domains, but ILU does. > > There is no parallel ILU in PETSc. > > $ mpiexec -n 2 mpich-clang/tests/ksp/ksp/examples/tutorials/ex2 -pc_type ilu [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for possible LU and Cholesky solvers [0]PETSC ERROR: Could not locate a solver package. Perhaps you must ./configure with --download-<package> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.10.2-19-g217b8b62e2 GIT Date: 2018-10-17 10:34:59 +0200 [0]PETSC ERROR: mpich-clang/tests/ksp/ksp/examples/tutorials/ex2 on a mpich-clang named joule by jed Tue Dec 4 11:02:53 2018 [0]PETSC ERROR: Configure options --download-chaco --download-p4est --download-sundials --download-triangle --with-fc=0 --with-mpi-dir=/home/jed/usr/ccache/mpich-clang --with-visibility --with-x --with-yaml PETSC_ARCH=mpich-clang [0]PETSC ERROR: #1 MatGetFactor() line 4485 in /home/jed/petsc/src/mat/interface/matrix.c > [0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in /home/jed/petsc/src/ksp/pc/impls/factor/ilu/ilu.c > [0]PETSC ERROR: #3 PCSetUp() line 932 in /home/jed/petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #4 KSPSetUp() line 391 in /home/jed/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #5 KSPSolve() line 723 in /home/jed/petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: #6 main() line 201 in /home/jed/petsc/src/ksp/ksp/examples/tutorials/ex2.c > [0]PETSC ERROR: PETSc Option Table entries: > [0]PETSC ERROR: -malloc_test > [0]PETSC ERROR: -pc_type ilu > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- application called MPI_Abort(MPI_COMM_WORLD, 92) - process 0 > >> I have tested my code with scalar ILU. However, no KSP could converge. > > There are no guarantees. See src/ksp/pc/examples/tutorials/ex1.c which tests with Kershaw's matrix, a 4x4 sparse SPD matrix where incomplete factorization yields an indefinite preconditioner. > >> Also, there are no zeros on the diagonal, at least in the current >> cases that I am simulating them. However, I will recheck it. >> Additionally, I am going to do a limited test with the available BILU >> (ILU with BAIJ matrices) to see whether it can work if I keep my block >> sizes constant. > > Good. We should understand why that works or doesn't work before proceeding. > >> Since PCVPBJACOBI had a limited success, I feel it is going to work. >> >> Best wishes, Ali >> >> -----Original Message----- >> From: Jed Brown <jed(a)jedbrown.org> >> Sent: Tuesday, December 04, 2018 6:22 PM >> To: Alireza Khazali <arkhazali(a)cc.iut.ac.ir>; Smith, Barry F. >> <bsmith(a)mcs.anl.gov> >> Cc: petsc-dev(a)mcs.anl.gov >> Subject: Re: Re[2]: [petsc-dev] Implementing of a variable block size >> BILU preconditioner >> >> Alireza Khazali <arkhazali(a)cc.iut.ac.ir> writes: >> >>> Dear Barry, >>> >>> >>> Thanks for your kind answer. You are right. I will try to write my own block ILU with the properties I need. However, my primary problem is the matrix storage. As I understand, each process keeps a portion of a matrix and has access to that portion only. Additionally, I have not found any routine that enables the access of one process to the portion of the matrix that is saved on the memory of another process. However, some algorithms like ILU need such access, and despite spending much time investigating the code, I still do not understand how such a thing is done. >> >> Even scalar ILU in PETSc is used inside domain decomposition, such as block Jacobi or additive Schwarz. Hypre has a parallel ILU, but the parallel scalability is bad. This is pretty fundamental to ILU: it is not a good parallel algorithm. >> >>> I am a very experienced programmer in the field of numerical analysis, but PETSc is a very huge and complicated code. Therefore, I have to apologize for taking your precious time if you find the solution to my problem too obvious, but I will be really really grateful if you could give me a hint. >>> >>> >>> >>> >>> Dear Jed, >>> >>> >>> Thank you for your kind answer. I have a multi-component fluid flow >>> simulator, which produces valid results using a direct solver (like MKL DGESV). However, direct solvers are useless if the problem size increases, no other available combinations of PC/KSP could solve the system. The simulator must solve a system of nonlinear PDEs for each node, and the primary unknowns are pressure and fluid compositions. However, at some pressures, the fluid is vaporized/liquefied (A.K.A undergoes phase change), and the number of PDEs for that node is increased (phase equilibrium equations have to be solved for that node, too). Therefore, in the discretized then linearized system, we have a block of equations for each node, but the block size is variable, depending on the fluid phase status in that node. The block preconditioners can handle the problem but only if they are designed for such a variable size block matrix. Thanks to Barry, we have a variable block sized BJacobi preconditioner (PCVPBJACOBI), but it does not provide the required precision in a few cases, and more effective preconditioning is needed. Also, I have found that others may need such variable block size handling, as it can be found in PETSc mailing lists: >> >> I understand that you have variable sized blocks, but why not use scalar ILU? Is it failing due to zeros on the diagonal and you have evidence that blocking fixes that? If so, what ordering are you using for your fields? Putting the dual variable (often pressure) last often works. >> Note that incomplete factorization can break down even for SPD matrices. >> >> I keep asking on this point because block ILU(0) is algebraically equivalent to scalar ILU(0) on a matrix with the same nonzero pattern, modulo handling of singular blocks that is hard to achieve with scalar ILU. >> >>> https://lists.mcs.anl.gov/pipermail/petsc-users/2011-October/010491.h >>> t >>> ml >>> >>> https://lists.mcs.anl.gov/pipermail/petsc-users/2018-August/036028.ht >>> m >>> l >>> >>> Since I have always loved to contribute to open source projects, and PETSc helped me a lot in my other researches, I decided to add variable size block ILU preconditioner to PETSc. However, PETSc is too complicated, andI cannot accomplish such a task efficiently without help. >>> >>> >>> >>> >>> Many thanks, >>> >>> Ali >>> ----- Original Message ----- >>> >>> >>> From: Jed Brown (jed(a)jedbrown.org) >>> Date: 13/09/97 05:09 >>> To: Smith, Barry F. (bsmith(a)mcs.anl.gov) Ali Reza Khaz'ali >>> (arkhazali(a)cc.iut.ac.ir) >>> Cc: petsc-dev(a)mcs.anl.gov >>> Subject: Re: [petsc-dev] Implementing of a variable block size BILU >>> preconditioner >>> >>> >>> >>> >>> >>> "Smith, Barry F. via petsc-dev" <petsc-dev(a)mcs.anl.gov> writes: >>> >>>>> On Dec 3, 2018, at 4:49 PM, Ali Reza Khaz'ali <arkhazali(a)cc.iut.ac.ir> wrote: >>>>> >>>>> Hi, >>>>> >>>>> I think that the topic is more suited for PETSc-developers than its users; therefore, I move it to the dev list. >>>>> >>>>> Continuing the discussion on implementing a variable-block size BILU preconditioner, would it be possible to change the block size parameter (bs) on BAIJ format such that it can handle variable block sizes? (i.e., instead of it being a scalar, it can be an array). Although BILU does not necessarily require rectangular blocks, I think, it leads to less messy code. >>>> >>>> That is an alternative to using the AIJ format. The problem with this approach is you will need to write a lot of code for the variable block size BAIJ; MatSetValues_SeqVBAIJ, MatMult_SeqVBAIJ, etc etc. While if you reuse the AIJ you only need to write new factorization and solve routines (much less code). >>> >>> Sure, but the result isn't really different (modulo associativity) >>> from normal ILU applied to a block matrix (at least unless you start >>> considering fill with incomplete blocks). >>> >>> Ali, what are you hoping to achieve with variable block ILU? >>> >>>>> Also, being a newbie on PETSc code, I do not understand some parts of the code, especially distributed matrix storage and some of the implemented numerical algorithms. Is there any reference that I can use for this? >>>> >>>> There is a little discussion at the end chapters of the users manual, plus you should read the developers manual. But there is not a lot of detail except in the actual code. >>>> >>>> Barry >>>> >>>>> >>>>> >>>>> -- >>>>> Ali Reza Khaz’ali >>>>> Assistant Professor of Petroleum Engineering, Department of >>>>> Chemical Engineering Isfahan University of Technology Isfahan, Iran >>>>>

1 0

Re: [petsc-dev] Implementing of a variable block size BILU preconditioner
by Ali Reza Khaz'ali 04 Dec '18

04 Dec '18

Dear Jed, ILU with BAIJ works, and its performance in reducing the condition number is slightly better than PCVPBJACOBI. Thanks for your guidance. Best wishes, Ali -----Original Message----- From: Jed Brown <jed(a)jedbrown.org> Sent: Tuesday, December 04, 2018 9:40 PM To: Ali Reza Khaz'ali <arkhazali(a)cc.iut.ac.ir>; 'Smith, Barry F.' <bsmith(a)mcs.anl.gov> Cc: petsc-dev(a)mcs.anl.gov Subject: RE: Re[2]: [petsc-dev] Implementing of a variable block size BILU preconditioner Ali Reza Khaz'ali <arkhazali(a)cc.iut.ac.ir> writes: > Dear Jed, > > Thanks for your kind answer. I thought Scalar BJACOBI does not need > data from the other domains, but ILU does. There is no parallel ILU in PETSc. $ mpiexec -n 2 mpich-clang/tests/ksp/ksp/examples/tutorials/ex2 -pc_type ilu [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for possible LU and Cholesky solvers [0]PETSC ERROR: Could not locate a solver package. Perhaps you must ./configure with --download-<package> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.10.2-19-g217b8b62e2 GIT Date: 2018-10-17 10:34:59 +0200 [0]PETSC ERROR: mpich-clang/tests/ksp/ksp/examples/tutorials/ex2 on a mpich-clang named joule by jed Tue Dec 4 11:02:53 2018 [0]PETSC ERROR: Configure options --download-chaco --download-p4est --download-sundials --download-triangle --with-fc=0 --with-mpi-dir=/home/jed/usr/ccache/mpich-clang --with-visibility --with-x --with-yaml PETSC_ARCH=mpich-clang [0]PETSC ERROR: #1 MatGetFactor() line 4485 in /home/jed/petsc/src/mat/interface/matrix.c [0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in /home/jed/petsc/src/ksp/pc/impls/factor/ilu/ilu.c [0]PETSC ERROR: #3 PCSetUp() line 932 in /home/jed/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #4 KSPSetUp() line 391 in /home/jed/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #5 KSPSolve() line 723 in /home/jed/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #6 main() line 201 in /home/jed/petsc/src/ksp/ksp/examples/tutorials/ex2.c [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -malloc_test [0]PETSC ERROR: -pc_type ilu [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- application called MPI_Abort(MPI_COMM_WORLD, 92) - process 0 > I have tested my code with scalar ILU. However, no KSP could converge. There are no guarantees. See src/ksp/pc/examples/tutorials/ex1.c which tests with Kershaw's matrix, a 4x4 sparse SPD matrix where incomplete factorization yields an indefinite preconditioner. > Also, there are no zeros on the diagonal, at least in the current > cases that I am simulating them. However, I will recheck it. > Additionally, I am going to do a limited test with the available BILU > (ILU with BAIJ matrices) to see whether it can work if I keep my block > sizes constant. Good. We should understand why that works or doesn't work before proceeding. > Since PCVPBJACOBI had a limited success, I feel it is going to work. > > Best wishes, Ali > > -----Original Message----- > From: Jed Brown <jed(a)jedbrown.org> > Sent: Tuesday, December 04, 2018 6:22 PM > To: Alireza Khazali <arkhazali(a)cc.iut.ac.ir>; Smith, Barry F. > <bsmith(a)mcs.anl.gov> > Cc: petsc-dev(a)mcs.anl.gov > Subject: Re: Re[2]: [petsc-dev] Implementing of a variable block size > BILU preconditioner > > Alireza Khazali <arkhazali(a)cc.iut.ac.ir> writes: > >> Dear Barry, >> >> >> Thanks for your kind answer. You are right. I will try to write my own block ILU with the properties I need. However, my primary problem is the matrix storage. As I understand, each process keeps a portion of a matrix and has access to that portion only. Additionally, I have not found any routine that enables the access of one process to the portion of the matrix that is saved on the memory of another process. However, some algorithms like ILU need such access, and despite spending much time investigating the code, I still do not understand how such a thing is done. > > Even scalar ILU in PETSc is used inside domain decomposition, such as block Jacobi or additive Schwarz. Hypre has a parallel ILU, but the parallel scalability is bad. This is pretty fundamental to ILU: it is not a good parallel algorithm. > >> I am a very experienced programmer in the field of numerical analysis, but PETSc is a very huge and complicated code. Therefore, I have to apologize for taking your precious time if you find the solution to my problem too obvious, but I will be really really grateful if you could give me a hint. >> >> >> >> >> Dear Jed, >> >> >> Thank you for your kind answer. I have a multi-component fluid flow >> simulator, which produces valid results using a direct solver (like MKL DGESV). However, direct solvers are useless if the problem size increases, no other available combinations of PC/KSP could solve the system. The simulator must solve a system of nonlinear PDEs for each node, and the primary unknowns are pressure and fluid compositions. However, at some pressures, the fluid is vaporized/liquefied (A.K.A undergoes phase change), and the number of PDEs for that node is increased (phase equilibrium equations have to be solved for that node, too). Therefore, in the discretized then linearized system, we have a block of equations for each node, but the block size is variable, depending on the fluid phase status in that node. The block preconditioners can handle the problem but only if they are designed for such a variable size block matrix. Thanks to Barry, we have a variable block sized BJacobi preconditioner (PCVPBJACOBI), but it does not provide the required precision in a few cases, and more effective preconditioning is needed. Also, I have found that others may need such variable block size handling, as it can be found in PETSc mailing lists: > > I understand that you have variable sized blocks, but why not use scalar ILU? Is it failing due to zeros on the diagonal and you have evidence that blocking fixes that? If so, what ordering are you using for your fields? Putting the dual variable (often pressure) last often works. > Note that incomplete factorization can break down even for SPD matrices. > > I keep asking on this point because block ILU(0) is algebraically equivalent to scalar ILU(0) on a matrix with the same nonzero pattern, modulo handling of singular blocks that is hard to achieve with scalar ILU. > >> https://lists.mcs.anl.gov/pipermail/petsc-users/2011-October/010491.h >> t >> ml >> >> https://lists.mcs.anl.gov/pipermail/petsc-users/2018-August/036028.ht >> m >> l >> >> Since I have always loved to contribute to open source projects, and PETSc helped me a lot in my other researches, I decided to add variable size block ILU preconditioner to PETSc. However, PETSc is too complicated, andI cannot accomplish such a task efficiently without help. >> >> >> >> >> Many thanks, >> >> Ali >> ----- Original Message ----- >> >> >> From: Jed Brown (jed(a)jedbrown.org) >> Date: 13/09/97 05:09 >> To: Smith, Barry F. (bsmith(a)mcs.anl.gov) Ali Reza Khaz'ali >> (arkhazali(a)cc.iut.ac.ir) >> Cc: petsc-dev(a)mcs.anl.gov >> Subject: Re: [petsc-dev] Implementing of a variable block size BILU >> preconditioner >> >> >> >> >> >> "Smith, Barry F. via petsc-dev" <petsc-dev(a)mcs.anl.gov> writes: >> >>>> On Dec 3, 2018, at 4:49 PM, Ali Reza Khaz'ali <arkhazali(a)cc.iut.ac.ir> wrote: >>>> >>>> Hi, >>>> >>>> I think that the topic is more suited for PETSc-developers than its users; therefore, I move it to the dev list. >>>> >>>> Continuing the discussion on implementing a variable-block size BILU preconditioner, would it be possible to change the block size parameter (bs) on BAIJ format such that it can handle variable block sizes? (i.e., instead of it being a scalar, it can be an array). Although BILU does not necessarily require rectangular blocks, I think, it leads to less messy code. >>> >>> That is an alternative to using the AIJ format. The problem with this approach is you will need to write a lot of code for the variable block size BAIJ; MatSetValues_SeqVBAIJ, MatMult_SeqVBAIJ, etc etc. While if you reuse the AIJ you only need to write new factorization and solve routines (much less code). >> >> Sure, but the result isn't really different (modulo associativity) >> from normal ILU applied to a block matrix (at least unless you start >> considering fill with incomplete blocks). >> >> Ali, what are you hoping to achieve with variable block ILU? >> >>>> Also, being a newbie on PETSc code, I do not understand some parts of the code, especially distributed matrix storage and some of the implemented numerical algorithms. Is there any reference that I can use for this? >>> >>> There is a little discussion at the end chapters of the users manual, plus you should read the developers manual. But there is not a lot of detail except in the actual code. >>> >>> Barry >>> >>>> >>>> >>>> -- >>>> Ali Reza Khaz’ali >>>> Assistant Professor of Petroleum Engineering, Department of >>>> Chemical Engineering Isfahan University of Technology Isfahan, Iran >>>>

1 0

Re: [petsc-dev] Implementing of a variable block size BILU preconditioner
by Jed Brown 04 Dec '18

04 Dec '18

Ali Reza Khaz'ali <arkhazali(a)cc.iut.ac.ir> writes: > Dear Jed, > > Thanks for your kind answer. I thought Scalar BJACOBI does not need > data from the other domains, but ILU does. There is no parallel ILU in PETSc. $ mpiexec -n 2 mpich-clang/tests/ksp/ksp/examples/tutorials/ex2 -pc_type ilu [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for possible LU and Cholesky solvers [0]PETSC ERROR: Could not locate a solver package. Perhaps you must ./configure with --download-<package> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.10.2-19-g217b8b62e2 GIT Date: 2018-10-17 10:34:59 +0200 [0]PETSC ERROR: mpich-clang/tests/ksp/ksp/examples/tutorials/ex2 on a mpich-clang named joule by jed Tue Dec 4 11:02:53 2018 [0]PETSC ERROR: Configure options --download-chaco --download-p4est --download-sundials --download-triangle --with-fc=0 --with-mpi-dir=/home/jed/usr/ccache/mpich-clang --with-visibility --with-x --with-yaml PETSC_ARCH=mpich-clang [0]PETSC ERROR: #1 MatGetFactor() line 4485 in /home/jed/petsc/src/mat/interface/matrix.c [0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in /home/jed/petsc/src/ksp/pc/impls/factor/ilu/ilu.c [0]PETSC ERROR: #3 PCSetUp() line 932 in /home/jed/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #4 KSPSetUp() line 391 in /home/jed/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #5 KSPSolve() line 723 in /home/jed/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #6 main() line 201 in /home/jed/petsc/src/ksp/ksp/examples/tutorials/ex2.c [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -malloc_test [0]PETSC ERROR: -pc_type ilu [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- application called MPI_Abort(MPI_COMM_WORLD, 92) - process 0 > I have tested my code with scalar ILU. However, no KSP could > converge. There are no guarantees. See src/ksp/pc/examples/tutorials/ex1.c which tests with Kershaw's matrix, a 4x4 sparse SPD matrix where incomplete factorization yields an indefinite preconditioner. > Also, there are no zeros on the diagonal, at least in the current > cases that I am simulating them. However, I will recheck > it. Additionally, I am going to do a limited test with the available > BILU (ILU with BAIJ matrices) to see whether it can work if I keep my > block sizes constant. Good. We should understand why that works or doesn't work before proceeding. > Since PCVPBJACOBI had a limited success, I feel it is going to work. > > Best wishes, Ali > > -----Original Message----- > From: Jed Brown <jed(a)jedbrown.org> > Sent: Tuesday, December 04, 2018 6:22 PM > To: Alireza Khazali <arkhazali(a)cc.iut.ac.ir>; Smith, Barry F. <bsmith(a)mcs.anl.gov> > Cc: petsc-dev(a)mcs.anl.gov > Subject: Re: Re[2]: [petsc-dev] Implementing of a variable block size BILU preconditioner > > Alireza Khazali <arkhazali(a)cc.iut.ac.ir> writes: > >> Dear Barry, >> >> >> Thanks for your kind answer. You are right. I will try to write my own block ILU with the properties I need. However, my primary problem is the matrix storage. As I understand, each process keeps a portion of a matrix and has access to that portion only. Additionally, I have not found any routine that enables the access of one process to the portion of the matrix that is saved on the memory of another process. However, some algorithms like ILU need such access, and despite spending much time investigating the code, I still do not understand how such a thing is done. > > Even scalar ILU in PETSc is used inside domain decomposition, such as block Jacobi or additive Schwarz. Hypre has a parallel ILU, but the parallel scalability is bad. This is pretty fundamental to ILU: it is not a good parallel algorithm. > >> I am a very experienced programmer in the field of numerical analysis, but PETSc is a very huge and complicated code. Therefore, I have to apologize for taking your precious time if you find the solution to my problem too obvious, but I will be really really grateful if you could give me a hint. >> >> >> >> >> Dear Jed, >> >> >> Thank you for your kind answer. I have a multi-component fluid flow >> simulator, which produces valid results using a direct solver (like MKL DGESV). However, direct solvers are useless if the problem size increases, no other available combinations of PC/KSP could solve the system. The simulator must solve a system of nonlinear PDEs for each node, and the primary unknowns are pressure and fluid compositions. However, at some pressures, the fluid is vaporized/liquefied (A.K.A undergoes phase change), and the number of PDEs for that node is increased (phase equilibrium equations have to be solved for that node, too). Therefore, in the discretized then linearized system, we have a block of equations for each node, but the block size is variable, depending on the fluid phase status in that node. The block preconditioners can handle the problem but only if they are designed for such a variable size block matrix. Thanks to Barry, we have a variable block sized BJacobi preconditioner (PCVPBJACOBI), but it does not provide the required precision in a few cases, and more effective preconditioning is needed. Also, I have found that others may need such variable block size handling, as it can be found in PETSc mailing lists: > > I understand that you have variable sized blocks, but why not use scalar ILU? Is it failing due to zeros on the diagonal and you have evidence that blocking fixes that? If so, what ordering are you using for your fields? Putting the dual variable (often pressure) last often works. > Note that incomplete factorization can break down even for SPD matrices. > > I keep asking on this point because block ILU(0) is algebraically equivalent to scalar ILU(0) on a matrix with the same nonzero pattern, modulo handling of singular blocks that is hard to achieve with scalar ILU. > >> https://lists.mcs.anl.gov/pipermail/petsc-users/2011-October/010491.ht >> ml >> >> https://lists.mcs.anl.gov/pipermail/petsc-users/2018-August/036028.htm >> l >> >> Since I have always loved to contribute to open source projects, and PETSc helped me a lot in my other researches, I decided to add variable size block ILU preconditioner to PETSc. However, PETSc is too complicated, andI cannot accomplish such a task efficiently without help. >> >> >> >> >> Many thanks, >> >> Ali >> ----- Original Message ----- >> >> >> From: Jed Brown (jed(a)jedbrown.org) >> Date: 13/09/97 05:09 >> To: Smith, Barry F. (bsmith(a)mcs.anl.gov) Ali Reza Khaz'ali >> (arkhazali(a)cc.iut.ac.ir) >> Cc: petsc-dev(a)mcs.anl.gov >> Subject: Re: [petsc-dev] Implementing of a variable block size BILU >> preconditioner >> >> >> >> >> >> "Smith, Barry F. via petsc-dev" <petsc-dev(a)mcs.anl.gov> writes: >> >>>> On Dec 3, 2018, at 4:49 PM, Ali Reza Khaz'ali <arkhazali(a)cc.iut.ac.ir> wrote: >>>> >>>> Hi, >>>> >>>> I think that the topic is more suited for PETSc-developers than its users; therefore, I move it to the dev list. >>>> >>>> Continuing the discussion on implementing a variable-block size BILU preconditioner, would it be possible to change the block size parameter (bs) on BAIJ format such that it can handle variable block sizes? (i.e., instead of it being a scalar, it can be an array). Although BILU does not necessarily require rectangular blocks, I think, it leads to less messy code. >>> >>> That is an alternative to using the AIJ format. The problem with this approach is you will need to write a lot of code for the variable block size BAIJ; MatSetValues_SeqVBAIJ, MatMult_SeqVBAIJ, etc etc. While if you reuse the AIJ you only need to write new factorization and solve routines (much less code). >> >> Sure, but the result isn't really different (modulo associativity) >> from normal ILU applied to a block matrix (at least unless you start >> considering fill with incomplete blocks). >> >> Ali, what are you hoping to achieve with variable block ILU? >> >>>> Also, being a newbie on PETSc code, I do not understand some parts of the code, especially distributed matrix storage and some of the implemented numerical algorithms. Is there any reference that I can use for this? >>> >>> There is a little discussion at the end chapters of the users manual, plus you should read the developers manual. But there is not a lot of detail except in the actual code. >>> >>> Barry >>> >>>> >>>> >>>> -- >>>> Ali Reza Khaz’ali >>>> Assistant Professor of Petroleum Engineering, Department of Chemical >>>> Engineering Isfahan University of Technology Isfahan, Iran >>>>

1 0

Re: [petsc-dev] Implementing of a variable block size BILU preconditioner
by Ali Reza Khaz'ali 04 Dec '18

04 Dec '18

Dear Jed, Thanks for your kind answer. I thought Scalar BJACOBI does not need data from the other domains, but ILU does. I have tested my code with scalar ILU. However, no KSP could converge. Also, there are no zeros on the diagonal, at least in the current cases that I am simulating them. However, I will recheck it. Additionally, I am going to do a limited test with the available BILU (ILU with BAIJ matrices) to see whether it can work if I keep my block sizes constant. Since PCVPBJACOBI had a limited success, I feel it is going to work. Best wishes, Ali -----Original Message----- From: Jed Brown <jed(a)jedbrown.org> Sent: Tuesday, December 04, 2018 6:22 PM To: Alireza Khazali <arkhazali(a)cc.iut.ac.ir>; Smith, Barry F. <bsmith(a)mcs.anl.gov> Cc: petsc-dev(a)mcs.anl.gov Subject: Re: Re[2]: [petsc-dev] Implementing of a variable block size BILU preconditioner Alireza Khazali <arkhazali(a)cc.iut.ac.ir> writes: > Dear Barry, > > > Thanks for your kind answer. You are right. I will try to write my own block ILU with the properties I need. However, my primary problem is the matrix storage. As I understand, each process keeps a portion of a matrix and has access to that portion only. Additionally, I have not found any routine that enables the access of one process to the portion of the matrix that is saved on the memory of another process. However, some algorithms like ILU need such access, and despite spending much time investigating the code, I still do not understand how such a thing is done. Even scalar ILU in PETSc is used inside domain decomposition, such as block Jacobi or additive Schwarz. Hypre has a parallel ILU, but the parallel scalability is bad. This is pretty fundamental to ILU: it is not a good parallel algorithm. > I am a very experienced programmer in the field of numerical analysis, but PETSc is a very huge and complicated code. Therefore, I have to apologize for taking your precious time if you find the solution to my problem too obvious, but I will be really really grateful if you could give me a hint. > > > > > Dear Jed, > > > Thank you for your kind answer. I have a multi-component fluid flow > simulator, which produces valid results using a direct solver (like MKL DGESV). However, direct solvers are useless if the problem size increases, no other available combinations of PC/KSP could solve the system. The simulator must solve a system of nonlinear PDEs for each node, and the primary unknowns are pressure and fluid compositions. However, at some pressures, the fluid is vaporized/liquefied (A.K.A undergoes phase change), and the number of PDEs for that node is increased (phase equilibrium equations have to be solved for that node, too). Therefore, in the discretized then linearized system, we have a block of equations for each node, but the block size is variable, depending on the fluid phase status in that node. The block preconditioners can handle the problem but only if they are designed for such a variable size block matrix. Thanks to Barry, we have a variable block sized BJacobi preconditioner (PCVPBJACOBI), but it does not provide the required precision in a few cases, and more effective preconditioning is needed. Also, I have found that others may need such variable block size handling, as it can be found in PETSc mailing lists: I understand that you have variable sized blocks, but why not use scalar ILU? Is it failing due to zeros on the diagonal and you have evidence that blocking fixes that? If so, what ordering are you using for your fields? Putting the dual variable (often pressure) last often works. Note that incomplete factorization can break down even for SPD matrices. I keep asking on this point because block ILU(0) is algebraically equivalent to scalar ILU(0) on a matrix with the same nonzero pattern, modulo handling of singular blocks that is hard to achieve with scalar ILU. > https://lists.mcs.anl.gov/pipermail/petsc-users/2011-October/010491.ht > ml > > https://lists.mcs.anl.gov/pipermail/petsc-users/2018-August/036028.htm > l > > Since I have always loved to contribute to open source projects, and PETSc helped me a lot in my other researches, I decided to add variable size block ILU preconditioner to PETSc. However, PETSc is too complicated, andI cannot accomplish such a task efficiently without help. > > > > > Many thanks, > > Ali > ----- Original Message ----- > > > From: Jed Brown (jed(a)jedbrown.org) > Date: 13/09/97 05:09 > To: Smith, Barry F. (bsmith(a)mcs.anl.gov) Ali Reza Khaz'ali > (arkhazali(a)cc.iut.ac.ir) > Cc: petsc-dev(a)mcs.anl.gov > Subject: Re: [petsc-dev] Implementing of a variable block size BILU > preconditioner > > > > > > "Smith, Barry F. via petsc-dev" <petsc-dev(a)mcs.anl.gov> writes: > >>> On Dec 3, 2018, at 4:49 PM, Ali Reza Khaz'ali <arkhazali(a)cc.iut.ac.ir> wrote: >>> >>> Hi, >>> >>> I think that the topic is more suited for PETSc-developers than its users; therefore, I move it to the dev list. >>> >>> Continuing the discussion on implementing a variable-block size BILU preconditioner, would it be possible to change the block size parameter (bs) on BAIJ format such that it can handle variable block sizes? (i.e., instead of it being a scalar, it can be an array). Although BILU does not necessarily require rectangular blocks, I think, it leads to less messy code. >> >> That is an alternative to using the AIJ format. The problem with this approach is you will need to write a lot of code for the variable block size BAIJ; MatSetValues_SeqVBAIJ, MatMult_SeqVBAIJ, etc etc. While if you reuse the AIJ you only need to write new factorization and solve routines (much less code). > > Sure, but the result isn't really different (modulo associativity) > from normal ILU applied to a block matrix (at least unless you start > considering fill with incomplete blocks). > > Ali, what are you hoping to achieve with variable block ILU? > >>> Also, being a newbie on PETSc code, I do not understand some parts of the code, especially distributed matrix storage and some of the implemented numerical algorithms. Is there any reference that I can use for this? >> >> There is a little discussion at the end chapters of the users manual, plus you should read the developers manual. But there is not a lot of detail except in the actual code. >> >> Barry >> >>> >>> >>> -- >>> Ali Reza Khaz’ali >>> Assistant Professor of Petroleum Engineering, Department of Chemical >>> Engineering Isfahan University of Technology Isfahan, Iran >>>

1 0

Re: [petsc-dev] Implementing of a variable block size BILU preconditioner
by Jed Brown 04 Dec '18

04 Dec '18

Alireza Khazali <arkhazali(a)cc.iut.ac.ir> writes: > Dear Barry, > > > Thanks for your kind answer. You are right. I will try to write my own block ILU with the properties I need. However, my primary problem is the matrix storage. As I understand, each process keeps a portion of a matrix and has access to that portion only. Additionally, I have not found any routine that enables the access of one process to the portion of the matrix that is saved on the memory of another process. However, some algorithms like ILU need such access, and despite spending much time investigating the code, I still do not understand how such a thing is done. Even scalar ILU in PETSc is used inside domain decomposition, such as block Jacobi or additive Schwarz. Hypre has a parallel ILU, but the parallel scalability is bad. This is pretty fundamental to ILU: it is not a good parallel algorithm. > I am a very experienced programmer in the field of numerical analysis, but PETSc is a very huge and complicated code. Therefore, I have to apologize for taking your precious time if you find the solution to my problem too obvious, but I will be really really grateful if you could give me a hint. > > > > > Dear Jed, > > > Thank you for your kind answer. I have a multi-component fluid flow simulator, which produces valid results using a direct solver (like MKL DGESV). However, direct solvers are useless if the problem size increases, no other available combinations of PC/KSP could solve the system. The simulator must solve a system of nonlinear PDEs for each node, and the primary unknowns are pressure and fluid compositions. However, at some pressures, the fluid is vaporized/liquefied (A.K.A undergoes phase change), and the number of PDEs for that node is increased (phase equilibrium equations have to be solved for that node, too). Therefore, in the discretized then linearized system, we have a block of equations for each node, but the block size is variable, depending on the fluid phase status in that node. The block preconditioners can handle the problem but only if they are designed for such a variable > size block matrix. Thanks to Barry, we have a variable block sized BJacobi preconditioner (PCVPBJACOBI), but it does not provide the required precision in a few cases, and more effective preconditioning is needed. Also, I have found that others may need such variable block size handling, as it can be found in PETSc mailing lists: I understand that you have variable sized blocks, but why not use scalar ILU? Is it failing due to zeros on the diagonal and you have evidence that blocking fixes that? If so, what ordering are you using for your fields? Putting the dual variable (often pressure) last often works. Note that incomplete factorization can break down even for SPD matrices. I keep asking on this point because block ILU(0) is algebraically equivalent to scalar ILU(0) on a matrix with the same nonzero pattern, modulo handling of singular blocks that is hard to achieve with scalar ILU. > https://lists.mcs.anl.gov/pipermail/petsc-users/2011-October/010491.html > > https://lists.mcs.anl.gov/pipermail/petsc-users/2018-August/036028.html > > Since I have always loved to contribute to open source projects, and PETSc helped me a lot in my other researches, I decided to add variable size block ILU preconditioner to PETSc. However, PETSc is too complicated, andI cannot accomplish such a task efficiently without help. > > > > > Many thanks, > > Ali > ----- Original Message ----- > > > From: Jed Brown (jed(a)jedbrown.org) > Date: 13/09/97 05:09 > To: Smith, Barry F. (bsmith(a)mcs.anl.gov) Ali Reza Khaz'ali (arkhazali(a)cc.iut.ac.ir) > Cc: petsc-dev(a)mcs.anl.gov > Subject: Re: [petsc-dev] Implementing of a variable block size BILU preconditioner > > > > > > "Smith, Barry F. via petsc-dev" <petsc-dev(a)mcs.anl.gov> writes: > >>> On Dec 3, 2018, at 4:49 PM, Ali Reza Khaz'ali <arkhazali(a)cc.iut.ac.ir> wrote: >>> >>> Hi, >>> >>> I think that the topic is more suited for PETSc-developers than its users; therefore, I move it to the dev list. >>> >>> Continuing the discussion on implementing a variable-block size BILU preconditioner, would it be possible to change the block size parameter (bs) on BAIJ format such that it can handle variable block sizes? (i.e., instead of it being a scalar, it can be an array). Although BILU does not necessarily require rectangular blocks, I think, it leads to less messy code. >> >> That is an alternative to using the AIJ format. The problem with this approach is you will need to write a lot of code for the variable block size BAIJ; MatSetValues_SeqVBAIJ, MatMult_SeqVBAIJ, etc etc. While if you reuse the AIJ you only need to write new factorization and solve routines (much less code). > > Sure, but the result isn't really different (modulo associativity) from > normal ILU applied to a block matrix (at least unless you start > considering fill with incomplete blocks). > > Ali, what are you hoping to achieve with variable block ILU? > >>> Also, being a newbie on PETSc code, I do not understand some parts of the code, especially distributed matrix storage and some of the implemented numerical algorithms. Is there any reference that I can use for this? >> >> There is a little discussion at the end chapters of the users manual, plus you should read the developers manual. But there is not a lot of detail except in the actual code. >> >> Barry >> >>> >>> >>> -- >>> Ali Reza Khaz’ali >>> Assistant Professor of Petroleum Engineering, >>> Department of Chemical Engineering >>> Isfahan University of Technology >>> Isfahan, Iran >>>

1 0

Re: [petsc-dev] Implementing of a variable block size BILU preconditioner
by Alireza Khazali 04 Dec '18

04 Dec '18

Dear Barry, Thanks for your kind answer. You are right. I will try to write my own block ILU with the properties I need. However, my primary problem is the matrix storage. As I understand, each process keeps a portion of a matrix and has access to that portion only. Additionally, I have not found any routine that enables the access of one process to the portion of the matrix that is saved on the memory of another process. However, some algorithms like ILU need such access, and despite spending much time investigating the code, I still do not understand how such a thing is done. I am a very experienced programmer in the field of numerical analysis, but PETSc is a very huge and complicated code. Therefore, I have to apologize for taking your precious time if you find the solution to my problem too obvious, but I will be really really grateful if you could give me a hint. Dear Jed, Thank you for your kind answer. I have a multi-component fluid flow simulator, which produces valid results using a direct solver (like MKL DGESV). However, direct solvers are useless if the problem size increases, no other available combinations of PC/KSP could solve the system. The simulator must solve a system of nonlinear PDEs for each node, and the primary unknowns are pressure and fluid compositions. However, at some pressures, the fluid is vaporized/liquefied (A.K.A undergoes phase change), and the number of PDEs for that node is increased (phase equilibrium equations have to be solved for that node, too). Therefore, in the discretized then linearized system, we have a block of equations for each node, but the block size is variable, depending on the fluid phase status in that node. The block preconditioners can handle the problem but only if they are designed for such a variable size block matrix. Thanks to Barry, we have a variable block sized BJacobi preconditioner (PCVPBJACOBI), but it does not provide the required precision in a few cases, and more effective preconditioning is needed. Also, I have found that others may need such variable block size handling, as it can be found in PETSc mailing lists: https://lists.mcs.anl.gov/pipermail/petsc-users/2011-October/010491.html https://lists.mcs.anl.gov/pipermail/petsc-users/2018-August/036028.html Since I have always loved to contribute to open source projects, and PETSc helped me a lot in my other researches, I decided to add variable size block ILU preconditioner to PETSc. However, PETSc is too complicated, andI cannot accomplish such a task efficiently without help. Many thanks, Ali ----- Original Message ----- From: Jed Brown (jed(a)jedbrown.org) Date: 13/09/97 05:09 To: Smith, Barry F. (bsmith(a)mcs.anl.gov) Ali Reza Khaz'ali (arkhazali(a)cc.iut.ac.ir) Cc: petsc-dev(a)mcs.anl.gov Subject: Re: [petsc-dev] Implementing of a variable block size BILU preconditioner "Smith, Barry F. via petsc-dev" <petsc-dev(a)mcs.anl.gov> writes: >> On Dec 3, 2018, at 4:49 PM, Ali Reza Khaz'ali <arkhazali(a)cc.iut.ac.ir> wrote: >> >> Hi, >> >> I think that the topic is more suited for PETSc-developers than its users; therefore, I move it to the dev list. >> >> Continuing the discussion on implementing a variable-block size BILU preconditioner, would it be possible to change the block size parameter (bs) on BAIJ format such that it can handle variable block sizes? (i.e., instead of it being a scalar, it can be an array). Although BILU does not necessarily require rectangular blocks, I think, it leads to less messy code. > > That is an alternative to using the AIJ format. The problem with this approach is you will need to write a lot of code for the variable block size BAIJ; MatSetValues_SeqVBAIJ, MatMult_SeqVBAIJ, etc etc. While if you reuse the AIJ you only need to write new factorization and solve routines (much less code). Sure, but the result isn't really different (modulo associativity) from normal ILU applied to a block matrix (at least unless you start considering fill with incomplete blocks). Ali, what are you hoping to achieve with variable block ILU? >> Also, being a newbie on PETSc code, I do not understand some parts of the code, especially distributed matrix storage and some of the implemented numerical algorithms. Is there any reference that I can use for this? > > There is a little discussion at the end chapters of the users manual, plus you should read the developers manual. But there is not a lot of detail except in the actual code. > > Barry > >> >> >> -- >> Ali Reza Khaz’ali >> Assistant Professor of Petroleum Engineering, >> Department of Chemical Engineering >> Isfahan University of Technology >> Isfahan, Iran >>

1 0

Re: [petsc-dev] configure issues with new MKL and --with-mkl_sparse_optimize=0
by Matthew Knepley 03 Dec '18

03 Dec '18

On Mon, Dec 3, 2018 at 8:35 PM Mills, Richard Tran via petsc-dev < petsc-dev(a)mcs.anl.gov> wrote: > Sorry, everyone, for how long it took me to have time to get back to this. > I tried the change I suggested in my previous message, but configure fails > with > > > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > details): > > ------------------------------------------------------------------------------- > Did not find package MKL_SPARSE_OPTIMIZE needed by mkl_sparse_sp2m. > Enable the package using --with-mkl_sparse_optimize > > ******************************************************************************* > > Does this happen because I have specified that this is a "look for by > default" package? What I'd like to happen is that this package be looked > for my default, in fact, but if MKL_SPARSE_OPTIMIZE is not present or > disabled, then I'd like the test for mkl_sparse_sp2m to fail gracefully and > configure to continue. That is: I don't want the fact that I am looking for > this package automatically to imply that it HAS to be there. Is there any > easy way to get this behavior? > Put it in the optional packages list. Then check explicitly that it was found in the sp2m configure method. Thanks, Matt > --Richard > > On 11/27/18 2:15 PM, Smith, Barry F. wrote: > > On Nov 26, 2018, at 6:12 PM, Mills, Richard Tran via petsc-dev <petsc-dev(a)mcs.anl.gov> <petsc-dev(a)mcs.anl.gov> wrote: > > Hi Stefano, > > Apologies for the slow reply; I was out for the US Thanksgiving Holiday. > > You've found an error in the logic for the MKL configure tests: Your MKL does indeed have mkl_sparse_sp2m(), but since this depends on having mkl_sparse_optimize() and you have specified '--with-mkl_sparse_optimize=0', PETSC_HAVE_MKL_SPARSE_SP2M either ought to not be defined, or in my AIJMKL code I should only enable the SP2M code if PETSC_HAVE_MKL_SPARSE_OPTIMIZE is defined. I think I like the former option better. For those more familiar with BuildSystem than I am, I think I can do this by putting the following in setupDependencies() in mkl_sparse_sp2m.py: > > self.mkl_sparse_optimize = framework.require('config.packages.mkl_sparse_optimize', self) > self.deps = [self.blasLapack,self.mkl_sparse_optimize] > > Looks ok to me. Make a pull request with Satish as a reviewer. > > Barry > > > Is that all that is required? > > --Richard > > On 11/21/18 10:59 PM, Stefano Zampini wrote: > > Richard, > > I just noticed that PETSc master does not build with the options > > '--with-mkl_pardiso-dir=/soft/com/packages/intel/18/u3/mkl', > '--with-mkl_sparse_optimize=0', > > This is on frog at MCS, but it will be the same on other machines as the macros configuration > > PETSC_HAVE_MKL_SPARSE_OPTIMIZE not defined > PETSC_HAVE_MKL_SPARSE_SP2M defined > > does not seem to be supported. > > /nfs2/szampini/src/petsc/src/mat/impls/aij/seq/aijmkl/aijmkl.c(797): error: struct "<unnamed>" has no field "csrA" > csrA = a->csrA; > > Could you please take a look? Why is the mkl_sparse_sp2m package flagged as lookforbydefault in the configure scripts? > > Thanks > -- > Stefano > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] Implementing of a variable block size BILU preconditioner
by Jed Brown 03 Dec '18

03 Dec '18

"Smith, Barry F. via petsc-dev" <petsc-dev(a)mcs.anl.gov> writes: >> On Dec 3, 2018, at 4:49 PM, Ali Reza Khaz'ali <arkhazali(a)cc.iut.ac.ir> wrote: >> >> Hi, >> >> I think that the topic is more suited for PETSc-developers than its users; therefore, I move it to the dev list. >> >> Continuing the discussion on implementing a variable-block size BILU preconditioner, would it be possible to change the block size parameter (bs) on BAIJ format such that it can handle variable block sizes? (i.e., instead of it being a scalar, it can be an array). Although BILU does not necessarily require rectangular blocks, I think, it leads to less messy code. > > That is an alternative to using the AIJ format. The problem with this approach is you will need to write a lot of code for the variable block size BAIJ; MatSetValues_SeqVBAIJ, MatMult_SeqVBAIJ, etc etc. While if you reuse the AIJ you only need to write new factorization and solve routines (much less code). Sure, but the result isn't really different (modulo associativity) from normal ILU applied to a block matrix (at least unless you start considering fill with incomplete blocks). Ali, what are you hoping to achieve with variable block ILU? >> Also, being a newbie on PETSc code, I do not understand some parts of the code, especially distributed matrix storage and some of the implemented numerical algorithms. Is there any reference that I can use for this? > > There is a little discussion at the end chapters of the users manual, plus you should read the developers manual. But there is not a lot of detail except in the actual code. > > Barry > >> >> >> -- >> Ali Reza Khaz’ali >> Assistant Professor of Petroleum Engineering, >> Department of Chemical Engineering >> Isfahan University of Technology >> Isfahan, Iran >>

1 0

Re: [petsc-dev] configure issues with new MKL and --with-mkl_sparse_optimize=0
by Mills, Richard Tran 03 Dec '18

03 Dec '18

Sorry, everyone, for how long it took me to have time to get back to this. I tried the change I suggested in my previous message, but configure fails with ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Did not find package MKL_SPARSE_OPTIMIZE needed by mkl_sparse_sp2m. Enable the package using --with-mkl_sparse_optimize ******************************************************************************* Does this happen because I have specified that this is a "look for by default" package? What I'd like to happen is that this package be looked for my default, in fact, but if MKL_SPARSE_OPTIMIZE is not present or disabled, then I'd like the test for mkl_sparse_sp2m to fail gracefully and configure to continue. That is: I don't want the fact that I am looking for this package automatically to imply that it HAS to be there. Is there any easy way to get this behavior? --Richard On 11/27/18 2:15 PM, Smith, Barry F. wrote: On Nov 26, 2018, at 6:12 PM, Mills, Richard Tran via petsc-dev <petsc-dev(a)mcs.anl.gov><mailto:[email protected]> wrote: Hi Stefano, Apologies for the slow reply; I was out for the US Thanksgiving Holiday. You've found an error in the logic for the MKL configure tests: Your MKL does indeed have mkl_sparse_sp2m(), but since this depends on having mkl_sparse_optimize() and you have specified '--with-mkl_sparse_optimize=0', PETSC_HAVE_MKL_SPARSE_SP2M either ought to not be defined, or in my AIJMKL code I should only enable the SP2M code if PETSC_HAVE_MKL_SPARSE_OPTIMIZE is defined. I think I like the former option better. For those more familiar with BuildSystem than I am, I think I can do this by putting the following in setupDependencies() in mkl_sparse_sp2m.py: self.mkl_sparse_optimize = framework.require('config.packages.mkl_sparse_optimize', self) self.deps = [self.blasLapack,self.mkl_sparse_optimize] Looks ok to me. Make a pull request with Satish as a reviewer. Barry Is that all that is required? --Richard On 11/21/18 10:59 PM, Stefano Zampini wrote: Richard, I just noticed that PETSc master does not build with the options '--with-mkl_pardiso-dir=/soft/com/packages/intel/18/u3/mkl', '--with-mkl_sparse_optimize=0', This is on frog at MCS, but it will be the same on other machines as the macros configuration PETSC_HAVE_MKL_SPARSE_OPTIMIZE not defined PETSC_HAVE_MKL_SPARSE_SP2M defined does not seem to be supported. /nfs2/szampini/src/petsc/src/mat/impls/aij/seq/aijmkl/aijmkl.c(797): error: struct "<unnamed>" has no field "csrA" csrA = a->csrA; Could you please take a look? Why is the mkl_sparse_sp2m package flagged as lookforbydefault in the configure scripts? Thanks -- Stefano

1 0

Re: [petsc-dev] Implementing of a variable block size BILU preconditioner
by Smith, Barry F. 03 Dec '18

03 Dec '18

> On Dec 3, 2018, at 4:49 PM, Ali Reza Khaz'ali <arkhazali(a)cc.iut.ac.ir> wrote: > > Hi, > > I think that the topic is more suited for PETSc-developers than its users; therefore, I move it to the dev list. > > Continuing the discussion on implementing a variable-block size BILU preconditioner, would it be possible to change the block size parameter (bs) on BAIJ format such that it can handle variable block sizes? (i.e., instead of it being a scalar, it can be an array). Although BILU does not necessarily require rectangular blocks, I think, it leads to less messy code. That is an alternative to using the AIJ format. The problem with this approach is you will need to write a lot of code for the variable block size BAIJ; MatSetValues_SeqVBAIJ, MatMult_SeqVBAIJ, etc etc. While if you reuse the AIJ you only need to write new factorization and solve routines (much less code). > > Also, being a newbie on PETSc code, I do not understand some parts of the code, especially distributed matrix storage and some of the implemented numerical algorithms. Is there any reference that I can use for this? There is a little discussion at the end chapters of the users manual, plus you should read the developers manual. But there is not a lot of detail except in the actual code. Barry > > > -- > Ali Reza Khaz’ali > Assistant Professor of Petroleum Engineering, > Department of Chemical Engineering > Isfahan University of Technology > Isfahan, Iran >

1 0