Dear Barry,

Thanks for your kind answer. You are right. I will try to write my own block ILU with the properties I need. However, my primary problem is the matrix storage. As I understand, each process keeps a portion of a matrix and has access to that portion only. Additionally, I have not found any routine that enables the access of one process to the portion of the matrix that is saved on the memory of another process. However, some algorithms like ILU need such access, and despite spending much time investigating the code, I still do not understand how such a thing is done.

I am a very experienced programmer in the field of numerical analysis, but PETSc is a very huge and complicated code. Therefore, I have to apologize for taking your precious time if you find the solution to my problem too obvious, but I will be really really grateful if you could give me a hint.

Dear Jed,

Thank you for your kind answer. I have a multi-component fluid flow simulator, which produces valid results using a direct solver (like MKL DGESV). However, direct solvers are useless if the problem size increases, no other available combinations of PC/KSP could solve the system. The simulator must solve a system of nonlinear PDEs for each node, and the primary unknowns are pressure and fluid compositions. However, at some pressures, the fluid is vaporized/liquefied (A.K.A undergoes phase change), and the number of PDEs for that node is increased (phase equilibrium equations have to be solved for that node, too). Therefore, in the discretized then linearized system, we have a block of equations for each node, but the block size is variable, depending on the fluid phase status in that node. The block preconditioners can handle the problem but only if they are designed for such a variable size block matrix. Thanks to Barry, we have a variable block sized BJacobi preconditioner (PCVPBJACOBI), but it does not provide the required precision in a few cases, and more effective preconditioning is needed. Also, I have found that others may need such variable block size handling, as it can be found in PETSc mailing lists:

https://lists.mcs.anl.gov/pipermail/petsc-users/2011-October/010491.html

https://lists.mcs.anl.gov/pipermail/petsc-users/2018-August/036028.html

Since I have always loved to contribute to open source projects, and PETSc helped me a lot in my other researches, I decided to add variable size block ILU preconditioner to PETSc. However, PETSc is too complicated, andI cannot accomplish such a task efficiently without help.

Many thanks,

Ali

From: Jed Brown (jed@jedbrown.org)
Date: 13/09/97 05:09
To: Smith, Barry F. (bsmith@mcs.anl.gov), Ali Reza Khaz'ali (arkhazali@cc.iut.ac.ir)
Cc: petsc-dev@mcs.anl.gov
Subject: Re: [petsc-dev] Implementing of a variable block size BILU preconditioner

"Smith, Barry F. via petsc-dev" <petsc-dev@mcs.anl.gov> writes:

>> On Dec 3, 2018, at 4:49 PM, Ali Reza Khaz'ali <arkhazali@cc.iut.ac.ir> wrote:
>>
>> Hi,
>>
>> I think that the topic is more suited for PETSc-developers than its users; therefore, I move it to the dev list.
>>
>> Continuing the discussion on implementing a variable-block size BILU preconditioner, would it be possible to change the block size parameter (bs) on BAIJ format such that it can handle variable block sizes? (i.e., instead of it being a scalar, it can be an array). Although BILU does not necessarily require rectangular blocks, I think, it leads to less messy code.
>
> That is an alternative to using the AIJ format. The problem with this approach is you will need to write a lot of code for the variable block size BAIJ; MatSetValues_SeqVBAIJ, MatMult_SeqVBAIJ, etc etc. While if you reuse the AIJ you only need to write new factorization and solve routines (much less code).

Sure, but the result isn't really different (modulo associativity) from
normal ILU applied to a block matrix (at least unless you start
considering fill with incomplete blocks).

Ali, what are you hoping to achieve with variable block ILU?

>> Also, being a newbie on PETSc code, I do not understand some parts of the code, especially distributed matrix storage and some of the implemented numerical algorithms. Is there any reference that I can use for this?
>
> There is a little discussion at the end chapters of the users manual, plus you should read the developers manual. But there is not a lot of detail except in the actual code.
>
> Barry
>
>>
>>
>> --
>> Ali Reza Khaz'ali
>> Assistant Professor of Petroleum Engineering,
>> Department of Chemical Engineering
>> Isfahan University of Technology
>> Isfahan, Iran
>>