On Mon, Apr 17, 2023 at 6:00 AM Edoardo alinovi <[email protected]> wrote:

Hey Matt,

Thanks for the help. Here is the error:

[0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Not for unassembled vector, did you call VecAssemblyBegin()/VecAssemblyEnd()? [0]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc! [0]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_p_ksp_rtol value: 0.10000000000000000E-0001 source: code [0]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_p_pc_type value: hypre source: code [0]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_u_ksp_rtol value: 0.10000000000000000E-0001 source: code [0]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_u_pc_type value: bjacobi source: code [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.19.0, unknown [0]PETSC ERROR: flubio_coupled on a arch-gnu named betelgeuse by edo Mon Apr 17 12:05:28 2023 [0]PETSC ERROR: Configure options PETSC_ARCH=arch-gnu FOPTFLAGS=-O3 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 -with-debugging=no -with-mpi-dir=/home/edo/user_software_repository/openmpi-4.1.2/build -download-fblaslapack=1 -download-superlu_dist -download-mumps -download-hypre -download-metis -download-parmetis -download-scalapack --download-ml -download-slepc -download-spai -download-fftw [0]PETSC ERROR: #1 MatMult() at /home/edo/user_software_repository/petsc/src/mat/interface/matrix.c:2557 [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Object is in wrong state [1]PETSC ERROR: Not for unassembled vector, did you call VecAssemblyBegin()/VecAssemblyEnd()? [1]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc! [1]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_p_ksp_rtol value: 0.10000000000000000E-0001 source: code [1]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_p_pc_type value: hypre source: code [1]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_u_ksp_rtol value: 0.10000000000000000E-0001 source: code [1]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_u_pc_type value: bjacobi source: code [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.19.0, unknown [1]PETSC ERROR: flubio_coupled on a arch-gnu named betelgeuse by edo Mon Apr 17 12:05:28 2023 [1]PETSC ERROR: Configure options PETSC_ARCH=arch-gnu FOPTFLAGS=-O3 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 -with-debugging=no -with-mpi-dir=/home/edo/user_software_repository/openmpi-4.1.2/build -download-fblaslapack=1 -download-superlu_dist -download-mumps -download-hypre -download-metis -download-parmetis -download-scalapack --download-ml -download-slepc -download-spai -download-fftw [1]PETSC ERROR: #1 MatMult() at /home/edo/user_software_repository/petsc/src/mat/interface/matrix.c:2557 [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [2]PETSC ERROR: Object is in wrong state [2]PETSC ERROR: Not for unassembled vector, did you call VecAssemblyBegin()/VecAssemblyEnd()? [2]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc! [2]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_p_ksp_rtol value: 0.10000000000000000E-0001 source: code [2]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_p_pc_type value: hypre source: code [2]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_u_ksp_rtol value: 0.10000000000000000E-0001 source: code [2]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_u_pc_type value: bjacobi source: code [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [3]PETSC ERROR: Object is in wrong state [3]PETSC ERROR: Not for unassembled vector, did you call VecAssemblyBegin()/VecAssemblyEnd()? [3]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc! [3]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_p_ksp_rtol value: 0.10000000000000000E-0001 source: code [3]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_p_pc_type value: hypre source: code [3]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_u_ksp_rtol value: 0.10000000000000000E-0001 source: code [3]PETSC ERROR: Option left: name:-UPeqn_fieldsplit_u_pc_type value: bjacobi source: code See https://petsc.org/release/faq/ for trouble shooting. [2]PETSC ERROR: Petsc Release Version 3.19.0, unknown [2]PETSC ERROR: flubio_coupled on a arch-gnu named betelgeuse by edo Mon Apr 17 12:05:28 2023 [2]PETSC ERROR: Configure options PETSC_ARCH=arch-gnu FOPTFLAGS=-O3 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 -with-debugging=no -with-mpi-dir=/home/edo/user_software_repository/openmpi-4.1.2/build -download-fblaslapack=1 -download-superlu_dist -download-mumps -download-hypre -download-metis -download-parmetis -download-scalapack --download-ml -download-slepc -download-spai -download-fftw [3]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [3]PETSC ERROR: Petsc Release Version 3.19.0, unknown [3]PETSC ERROR: flubio_coupled on a arch-gnu named betelgeuse by edo Mon Apr 17 12:05:28 2023 [3]PETSC ERROR: Configure options PETSC_ARCH=arch-gnu FOPTFLAGS=-O3 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 -with-debugging=no -with-mpi-dir=/home/edo/user_software_repository/openmpi-4.1.2/build -download-fblaslapack=1 -download-superlu_dist -download-mumps -download-hypre -download-metis -download-parmetis -download-scalapack --download-ml -download-slepc -download-spai -download-fftw [2]PETSC ERROR: #1 MatMult() at /home/edo/user_software_repository/petsc/src/mat/interface/matrix.c:2557 [3]PETSC ERROR: #1 MatMult() at /home/edo/user_software_repository/petsc/src/mat/interface/matrix.c:2557

Here the way I am doing:

! Duplicate vector call VecDuplicate(this%rhs, x, ierr) call VecZeroEntries(x, ierr)

call VecDuplicate(this%rhs, vres, ierr) ! Set the current solution call VecSetValues(x, numberOfElements, mesh%cellGlobalAddr-1, field%phi(1:numberOfElements,iComp), INSERT_VALUES, ierr)

After VecSetValues(), you must call VecAssemblyBegin/End(). Thanks, Matt

call flubioStopMsg('HELLO')

cheers

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Thanks Matt, your always there when you need <3

Do you mean the solver I am messing around? XD

Sure thing, the solver I am working on is this one: https://gitlab.com/alie89/flubio-code-fvm. It is a 3D, collocated, unstructured, finite volume solver for incompressibility NS. I can run steady, unsteady and I can use SIMPLE, PISO and Factional step method (both explicit and fully implicit momentum). I can also solve for turbulence (k-omega, BSL, SST, Spalart-Allmars, LES). I have also implemented some kind of Immersed boundary (2D/3D) that I need to resume at some point. Hot topic of the moment, I am developing a fully coupled pressure based solver using field-split. What I have now is working ok, I have validated it on a lot of 2D problems and going on with 3D right now. If all the tests are passed, I'll focus on tuning the field splitting which looks to be a quite interesting topic! Flubio is a project I have been carrying on since PhD days. The implementation is 99% on my shoulders, despite the fact I am collaborating with some people around. I am coding evenings and weekends/free time, it gives me a lot of satisfaction and also a lot of insights!

Hi Matt, thanks for sharing the literature. Would you suggest any monolitic approach for the mpiaij/mpibaij matrix instead of fieldsplit? I did some blind search using gamg/hypre and they look terribile. I guess i am missing a trick, probaly they are not the way to go? Thanks! Il Lun 17 Apr 2023, 13:50 Matthew Knepley <[email protected]> ha scritto:

On Mon, Apr 17, 2023 at 6:37 AM Edoardo alinovi <[email protected]> wrote:

...

Sure thing, the solver I am working on is this one: https://gitlab.com/alie89/flubio-code-fvm.

It is a 3D, collocated, unstructured, finite volume solver for incompressibility NS. I can run steady, unsteady and I can use SIMPLE, PISO and Factional step method (both explicit and fully implicit momentum). I can also solve for turbulence (k-omega, BSL, SST, Spalart-Allmars, LES). I have also implemented some kind of Immersed boundary (2D/3D) that I need to resume at some point.

Hot topic of the moment, I am developing a fully coupled pressure based solver using field-split. What I have now is working ok, I have validated it on a lot of 2D problems and going on with 3D right now. If all the tests are passed, I'll focus on tuning the field splitting which looks to be a quite interesting topic!

I think a very good discussion of the issues from the point of view of FEM is

https://arxiv.org/abs/1810.03315

There should be a similar analysis from the FVM side, although it might not be possible to find a pressure discretization compatible with the FVM velocity for this purpose.

Thanks,

Matt

...

Flubio is a project I have been carrying on since PhD days. The implementation is 99% on my shoulders, despite the fact I am collaborating with some people around. I am coding evenings and weekends/free time, it gives me a lot of satisfaction and also a lot of insights!

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Hi Matt, I have some more questions on the fieldsplit saga :) I am running a 1M cell ahmed body case using the following options: "solver": "fgmres", "preconditioner": "fieldsplit", "absTol": 1e-6, "relTol": 0.0, "options":{ "pc_fieldsplit_type": "multiplicative", "fieldsplit_u_pc_type": "ml", "fieldsplit_p_pc_type": "ml", "fieldsplit_u_ksp_type": "preonly", "fieldsplit_p_ksp_type": "preonly", "fieldsplit_u_ksp_rtol": 1e-2, "fieldsplit_p_ksp_rtol": 1e-2, } I have run the case using -ksp_monitor_true_residual and this is the results I am getting: Residual norms for UPeqn_ solve. 0 KSP unpreconditioned resid norm 5.003190920461e+00 true resid norm 5.003190920461e+00 ||r(i)||/||b|| 4.993739374163e-03 1 KSP unpreconditioned resid norm 4.959389845148e+00 true resid norm 4.959389845148e+00 ||r(i)||/||b|| 4.950021043622e-03 2 KSP unpreconditioned resid norm 4.793097370730e+00 true resid norm 4.793097370730e+00 ||r(i)||/||b|| 4.784042712927e-03 3 KSP unpreconditioned resid norm 4.187770916162e+00 true resid norm 4.187770916162e+00 ||r(i)||/||b|| 4.179859782782e-03 4 KSP unpreconditioned resid norm 3.099045576565e+00 true resid norm 3.099045576565e+00 ||r(i)||/||b|| 3.093191158212e-03 5 KSP unpreconditioned resid norm 2.072551338956e+00 true resid norm 2.072551338956e+00 ||r(i)||/||b|| 2.068636074628e-03 6 KSP unpreconditioned resid norm 1.414678932482e+00 true resid norm 1.414678932482e+00 ||r(i)||/||b|| 1.412006457328e-03 7 KSP unpreconditioned resid norm 1.006854855789e+00 true resid norm 1.006854855789e+00 ||r(i)||/||b|| 1.004952802592e-03 8 KSP unpreconditioned resid norm 7.332800358083e-01 true resid norm 7.332800358084e-01 ||r(i)||/||b|| 7.318947938062e-04 9 KSP unpreconditioned resid norm 5.406076142092e-01 true resid norm 5.406076142093e-01 ||r(i)||/||b|| 5.395863503846e-04 10 KSP unpreconditioned resid norm 4.037336888099e-01 true resid norm 4.037336888100e-01 ||r(i)||/||b|| 4.029709940193e-04 11 KSP unpreconditioned resid norm 3.041388930530e-01 true resid norm 3.041388930530e-01 ||r(i)||/||b|| 3.035643431559e-04 12 KSP unpreconditioned resid norm 2.299815364065e-01 true resid norm 2.299815364066e-01 ||r(i)||/||b|| 2.295470774436e-04 13 KSP unpreconditioned resid norm 1.739866268817e-01 true resid norm 1.739866268817e-01 ||r(i)||/||b|| 1.736579481074e-04 14 KSP unpreconditioned resid norm 1.317133652074e-01 true resid norm 1.317133652074e-01 ||r(i)||/||b|| 1.314645450067e-04 15 KSP unpreconditioned resid norm 9.966247212017e-02 true resid norm 9.966247212019e-02 ||r(i)||/||b|| 9.947419937897e-05 16 KSP unpreconditioned resid norm 7.531138284402e-02 true resid norm 7.531138284404e-02 ||r(i)||/||b|| 7.516911183479e-05 17 KSP unpreconditioned resid norm 5.646770286889e-02 true resid norm 5.646770286889e-02 ||r(i)||/||b|| 5.636102952452e-05 18 KSP unpreconditioned resid norm 4.225114444838e-02 true resid norm 4.225114444838e-02 ||r(i)||/||b|| 4.217132765661e-05 19 KSP unpreconditioned resid norm 3.160393382046e-02 true resid norm 3.160393382046e-02 ||r(i)||/||b|| 3.154423071329e-05 20 KSP unpreconditioned resid norm 2.366499890335e-02 true resid norm 2.366499890334e-02 ||r(i)||/||b|| 2.362029326721e-05 21 KSP unpreconditioned resid norm 1.759504138840e-02 true resid norm 1.759504138839e-02 ||r(i)||/||b|| 1.756180253124e-05 22 KSP unpreconditioned resid norm 1.309326628086e-02 true resid norm 1.309326628085e-02 ||r(i)||/||b|| 1.306853174355e-05 23 KSP unpreconditioned resid norm 9.710513089816e-03 true resid norm 9.710513089808e-03 ||r(i)||/||b|| 9.692168923954e-06 24 KSP unpreconditioned resid norm 7.171039760236e-03 true resid norm 7.171039760227e-03 ||r(i)||/||b|| 7.157492922743e-06 25 KSP unpreconditioned resid norm 5.277221846963e-03 true resid norm 5.277221846949e-03 ||r(i)||/||b|| 5.267252627824e-06 26 KSP unpreconditioned resid norm 3.906960734588e-03 true resid norm 3.906960734575e-03 ||r(i)||/||b|| 3.899580080737e-06 27 KSP unpreconditioned resid norm 2.896843259283e-03 true resid norm 2.896843259273e-03 ||r(i)||/||b|| 2.891370822059e-06 28 KSP unpreconditioned resid norm 2.140269358580e-03 true resid norm 2.140269358568e-03 ||r(i)||/||b|| 2.136226167881e-06 29 KSP unpreconditioned resid norm 1.585513255966e-03 true resid norm 1.585513255956e-03 ||r(i)||/||b|| 1.582518057055e-06 30 KSP unpreconditioned resid norm 1.173839272299e-03 true resid norm 1.173839272299e-03 ||r(i)||/||b|| 1.171621768229e-06 31 KSP unpreconditioned resid norm 8.777233482545e-04 true resid norm 8.777233482544e-04 ||r(i)||/||b|| 8.760652378615e-07 32 KSP unpreconditioned resid norm 6.546689191353e-04 true resid norm 6.546689191370e-04 ||r(i)||/||b|| 6.534321816833e-07 33 KSP unpreconditioned resid norm 4.973281362004e-04 true resid norm 4.973281362015e-04 ||r(i)||/||b|| 4.963886317973e-07 34 KSP unpreconditioned resid norm 3.775325682448e-04 true resid norm 3.775325682480e-04 ||r(i)||/||b|| 3.768193700902e-07 35 KSP unpreconditioned resid norm 2.908052735383e-04 true resid norm 2.908052735387e-04 ||r(i)||/||b|| 2.902559122310e-07 36 KSP unpreconditioned resid norm 2.239556213185e-04 true resid norm 2.239556213218e-04 ||r(i)||/||b|| 2.235325459370e-07 37 KSP unpreconditioned resid norm 1.698373081988e-04 true resid norm 1.698373081996e-04 ||r(i)||/||b|| 1.695164679184e-07 38 KSP unpreconditioned resid norm 1.277467123301e-04 true resid norm 1.277467123333e-04 ||r(i)||/||b|| 1.275053855510e-07 39 KSP unpreconditioned resid norm 9.506326626848e-05 true resid norm 9.506326626983e-05 ||r(i)||/||b|| 9.488368190523e-08 40 KSP unpreconditioned resid norm 7.223958235163e-05 true resid norm 7.223958235336e-05 ||r(i)||/||b|| 7.210311429367e-08 41 KSP unpreconditioned resid norm 5.509671615415e-05 true resid norm 5.509671615512e-05 ||r(i)||/||b|| 5.499263274677e-08 42 KSP unpreconditioned resid norm 4.189229263778e-05 true resid norm 4.189229263744e-05 ||r(i)||/||b|| 4.181315375394e-08 43 KSP unpreconditioned resid norm 3.067856645608e-05 true resid norm 3.067856645894e-05 ||r(i)||/||b|| 3.062061146665e-08 44 KSP unpreconditioned resid norm 2.340298386078e-05 true resid norm 2.340298386081e-05 ||r(i)||/||b|| 2.335877319825e-08 45 KSP unpreconditioned resid norm 1.791143784234e-05 true resid norm 1.791143784276e-05 ||r(i)||/||b|| 1.787760127991e-08 46 KSP unpreconditioned resid norm 1.355654057227e-05 true resid norm 1.355654057087e-05 ||r(i)||/||b|| 1.353093086041e-08 47 KSP unpreconditioned resid norm 1.020861247518e-05 true resid norm 1.020861247732e-05 ||r(i)||/||b|| 1.018932735009e-08 48 KSP unpreconditioned resid norm 7.642335784085e-06 true resid norm 7.642335784452e-06 ||r(i)||/||b|| 7.627898619923e-09 49 KSP unpreconditioned resid norm 5.874756954976e-06 true resid norm 5.874756956850e-06 ||r(i)||/||b|| 5.863658931960e-09 50 KSP unpreconditioned resid norm 4.512356844825e-06 true resid norm 4.512356846552e-06 ||r(i)||/||b|| 4.503832536701e-09 51 KSP unpreconditioned resid norm 3.438985239280e-06 true resid norm 3.438985240743e-06 ||r(i)||/||b|| 3.432488641125e-09 52 KSP unpreconditioned resid norm 2.655998139390e-06 true resid norm 2.655998140374e-06 ||r(i)||/||b|| 2.650980684556e-09 53 KSP unpreconditioned resid norm 2.051081181832e-06 true resid norm 2.051081181929e-06 ||r(i)||/||b|| 2.047206476954e-09 54 KSP unpreconditioned resid norm 1.581756364000e-06 true resid norm 1.581756364725e-06 ||r(i)||/||b|| 1.578768262982e-09 55 KSP unpreconditioned resid norm 1.207420527415e-06 true resid norm 1.207420527996e-06 ||r(i)||/||b|| 1.205139585453e-09 56 KSP unpreconditioned resid norm 9.377914349033e-07 true resid norm 9.377914347157e-07 ||r(i)||/||b|| 9.360198494806e-10 I have the feeling I am self-inflicting some overhead not using relative tolerance (if I set 1e-3 I converge in 8 iters vs 56...), however I would like to ask the following question:

From your point of view, is this a reasonably convergence history?

Thank you!

To expand on what Matt said slightly. When you have a preconditioner based on (possibly nested) sub solves one generally "tunes" the solves to minimize time to solution. We recommend doing this by first using very accurate subsolves (when possible using direct solves inside); this tells us the "best reasonably expected" convergence rate of the outermost solve. One then "backs off" on the accuracy of the inner solves (making them less accurate), monitoring how this "backing off" affects the convergence rate of the outermost solve (one will often allow the convergence of the outer-most solve to get "a bit worse" in the hope that the since the cost of the inner solves will be much cheaper (with the less accuracy) the cost of the extra outer iterations will not make the entire solve time longer. Deciding how much to "back-off" on each inner solve is mostly determined by monitoring the time for convergence of the entire problem (but monitoring number of iterations also provides important information, for example if the number of outer iterations suddenly shots up you know you are probably undersolving a subproblem. While doing this process, it is important to work with "production" size problems since for PDEs the convergence rates are often very affected by the problem size. One should start with smaller problems to get a feeling for the tolerances needed but usually final tuning needs to be done for large problems. And, sadly, if one changes some parameters in the system a very different tuning may be optimal. Barry

On May 5, 2023, at 5:13 AM, Edoardo alinovi <[email protected]> wrote:

Hi Matt,

I have some more questions on the fieldsplit saga :)

I am running a 1M cell ahmed body case using the following options:

"solver": "fgmres", "preconditioner": "fieldsplit", "absTol": 1e-6, "relTol": 0.0,

"options":{ "pc_fieldsplit_type": "multiplicative", "fieldsplit_u_pc_type": "ml", "fieldsplit_p_pc_type": "ml", "fieldsplit_u_ksp_type": "preonly", "fieldsplit_p_ksp_type": "preonly", "fieldsplit_u_ksp_rtol": 1e-2, "fieldsplit_p_ksp_rtol": 1e-2, }

I have run the case using -ksp_monitor_true_residual and this is the results I am getting:

Residual norms for UPeqn_ solve. 0 KSP unpreconditioned resid norm 5.003190920461e+00 true resid norm 5.003190920461e+00 ||r(i)||/||b|| 4.993739374163e-03 1 KSP unpreconditioned resid norm 4.959389845148e+00 true resid norm 4.959389845148e+00 ||r(i)||/||b|| 4.950021043622e-03 2 KSP unpreconditioned resid norm 4.793097370730e+00 true resid norm 4.793097370730e+00 ||r(i)||/||b|| 4.784042712927e-03 3 KSP unpreconditioned resid norm 4.187770916162e+00 true resid norm 4.187770916162e+00 ||r(i)||/||b|| 4.179859782782e-03 4 KSP unpreconditioned resid norm 3.099045576565e+00 true resid norm 3.099045576565e+00 ||r(i)||/||b|| 3.093191158212e-03 5 KSP unpreconditioned resid norm 2.072551338956e+00 true resid norm 2.072551338956e+00 ||r(i)||/||b|| 2.068636074628e-03 6 KSP unpreconditioned resid norm 1.414678932482e+00 true resid norm 1.414678932482e+00 ||r(i)||/||b|| 1.412006457328e-03 7 KSP unpreconditioned resid norm 1.006854855789e+00 true resid norm 1.006854855789e+00 ||r(i)||/||b|| 1.004952802592e-03 8 KSP unpreconditioned resid norm 7.332800358083e-01 true resid norm 7.332800358084e-01 ||r(i)||/||b|| 7.318947938062e-04 9 KSP unpreconditioned resid norm 5.406076142092e-01 true resid norm 5.406076142093e-01 ||r(i)||/||b|| 5.395863503846e-04 10 KSP unpreconditioned resid norm 4.037336888099e-01 true resid norm 4.037336888100e-01 ||r(i)||/||b|| 4.029709940193e-04 11 KSP unpreconditioned resid norm 3.041388930530e-01 true resid norm 3.041388930530e-01 ||r(i)||/||b|| 3.035643431559e-04 12 KSP unpreconditioned resid norm 2.299815364065e-01 true resid norm 2.299815364066e-01 ||r(i)||/||b|| 2.295470774436e-04 13 KSP unpreconditioned resid norm 1.739866268817e-01 true resid norm 1.739866268817e-01 ||r(i)||/||b|| 1.736579481074e-04 14 KSP unpreconditioned resid norm 1.317133652074e-01 true resid norm 1.317133652074e-01 ||r(i)||/||b|| 1.314645450067e-04 15 KSP unpreconditioned resid norm 9.966247212017e-02 true resid norm 9.966247212019e-02 ||r(i)||/||b|| 9.947419937897e-05 16 KSP unpreconditioned resid norm 7.531138284402e-02 true resid norm 7.531138284404e-02 ||r(i)||/||b|| 7.516911183479e-05 17 KSP unpreconditioned resid norm 5.646770286889e-02 true resid norm 5.646770286889e-02 ||r(i)||/||b|| 5.636102952452e-05 18 KSP unpreconditioned resid norm 4.225114444838e-02 true resid norm 4.225114444838e-02 ||r(i)||/||b|| 4.217132765661e-05 19 KSP unpreconditioned resid norm 3.160393382046e-02 true resid norm 3.160393382046e-02 ||r(i)||/||b|| 3.154423071329e-05 20 KSP unpreconditioned resid norm 2.366499890335e-02 true resid norm 2.366499890334e-02 ||r(i)||/||b|| 2.362029326721e-05 21 KSP unpreconditioned resid norm 1.759504138840e-02 true resid norm 1.759504138839e-02 ||r(i)||/||b|| 1.756180253124e-05 22 KSP unpreconditioned resid norm 1.309326628086e-02 true resid norm 1.309326628085e-02 ||r(i)||/||b|| 1.306853174355e-05 23 KSP unpreconditioned resid norm 9.710513089816e-03 true resid norm 9.710513089808e-03 ||r(i)||/||b|| 9.692168923954e-06 24 KSP unpreconditioned resid norm 7.171039760236e-03 true resid norm 7.171039760227e-03 ||r(i)||/||b|| 7.157492922743e-06 25 KSP unpreconditioned resid norm 5.277221846963e-03 true resid norm 5.277221846949e-03 ||r(i)||/||b|| 5.267252627824e-06 26 KSP unpreconditioned resid norm 3.906960734588e-03 true resid norm 3.906960734575e-03 ||r(i)||/||b|| 3.899580080737e-06 27 KSP unpreconditioned resid norm 2.896843259283e-03 true resid norm 2.896843259273e-03 ||r(i)||/||b|| 2.891370822059e-06 28 KSP unpreconditioned resid norm 2.140269358580e-03 true resid norm 2.140269358568e-03 ||r(i)||/||b|| 2.136226167881e-06 29 KSP unpreconditioned resid norm 1.585513255966e-03 true resid norm 1.585513255956e-03 ||r(i)||/||b|| 1.582518057055e-06 30 KSP unpreconditioned resid norm 1.173839272299e-03 true resid norm 1.173839272299e-03 ||r(i)||/||b|| 1.171621768229e-06 31 KSP unpreconditioned resid norm 8.777233482545e-04 true resid norm 8.777233482544e-04 ||r(i)||/||b|| 8.760652378615e-07 32 KSP unpreconditioned resid norm 6.546689191353e-04 true resid norm 6.546689191370e-04 ||r(i)||/||b|| 6.534321816833e-07 33 KSP unpreconditioned resid norm 4.973281362004e-04 true resid norm 4.973281362015e-04 ||r(i)||/||b|| 4.963886317973e-07 34 KSP unpreconditioned resid norm 3.775325682448e-04 true resid norm 3.775325682480e-04 ||r(i)||/||b|| 3.768193700902e-07 35 KSP unpreconditioned resid norm 2.908052735383e-04 true resid norm 2.908052735387e-04 ||r(i)||/||b|| 2.902559122310e-07 36 KSP unpreconditioned resid norm 2.239556213185e-04 true resid norm 2.239556213218e-04 ||r(i)||/||b|| 2.235325459370e-07 37 KSP unpreconditioned resid norm 1.698373081988e-04 true resid norm 1.698373081996e-04 ||r(i)||/||b|| 1.695164679184e-07 38 KSP unpreconditioned resid norm 1.277467123301e-04 true resid norm 1.277467123333e-04 ||r(i)||/||b|| 1.275053855510e-07 39 KSP unpreconditioned resid norm 9.506326626848e-05 true resid norm 9.506326626983e-05 ||r(i)||/||b|| 9.488368190523e-08 40 KSP unpreconditioned resid norm 7.223958235163e-05 true resid norm 7.223958235336e-05 ||r(i)||/||b|| 7.210311429367e-08 41 KSP unpreconditioned resid norm 5.509671615415e-05 true resid norm 5.509671615512e-05 ||r(i)||/||b|| 5.499263274677e-08 42 KSP unpreconditioned resid norm 4.189229263778e-05 true resid norm 4.189229263744e-05 ||r(i)||/||b|| 4.181315375394e-08 43 KSP unpreconditioned resid norm 3.067856645608e-05 true resid norm 3.067856645894e-05 ||r(i)||/||b|| 3.062061146665e-08 44 KSP unpreconditioned resid norm 2.340298386078e-05 true resid norm 2.340298386081e-05 ||r(i)||/||b|| 2.335877319825e-08 45 KSP unpreconditioned resid norm 1.791143784234e-05 true resid norm 1.791143784276e-05 ||r(i)||/||b|| 1.787760127991e-08 46 KSP unpreconditioned resid norm 1.355654057227e-05 true resid norm 1.355654057087e-05 ||r(i)||/||b|| 1.353093086041e-08 47 KSP unpreconditioned resid norm 1.020861247518e-05 true resid norm 1.020861247732e-05 ||r(i)||/||b|| 1.018932735009e-08 48 KSP unpreconditioned resid norm 7.642335784085e-06 true resid norm 7.642335784452e-06 ||r(i)||/||b|| 7.627898619923e-09 49 KSP unpreconditioned resid norm 5.874756954976e-06 true resid norm 5.874756956850e-06 ||r(i)||/||b|| 5.863658931960e-09 50 KSP unpreconditioned resid norm 4.512356844825e-06 true resid norm 4.512356846552e-06 ||r(i)||/||b|| 4.503832536701e-09 51 KSP unpreconditioned resid norm 3.438985239280e-06 true resid norm 3.438985240743e-06 ||r(i)||/||b|| 3.432488641125e-09 52 KSP unpreconditioned resid norm 2.655998139390e-06 true resid norm 2.655998140374e-06 ||r(i)||/||b|| 2.650980684556e-09 53 KSP unpreconditioned resid norm 2.051081181832e-06 true resid norm 2.051081181929e-06 ||r(i)||/||b|| 2.047206476954e-09 54 KSP unpreconditioned resid norm 1.581756364000e-06 true resid norm 1.581756364725e-06 ||r(i)||/||b|| 1.578768262982e-09 55 KSP unpreconditioned resid norm 1.207420527415e-06 true resid norm 1.207420527996e-06 ||r(i)||/||b|| 1.205139585453e-09 56 KSP unpreconditioned resid norm 9.377914349033e-07 true resid norm 9.377914347157e-07 ||r(i)||/||b|| 9.360198494806e-10

I have the feeling I am self-inflicting some overhead not using relative tolerance (if I set 1e-3 I converge in 8 iters vs 56...), however I would like to ask the following question:

From your point of view, is this a reasonably convergence history?

Thank you!

Edoardo alinovi <[email protected]> writes:

Hello Barry,

Welcome to the party! Thank you guys for your precious suggestions, they are really helpful!

It's been a while since I am messing around and I have tested many combinations. Schur + selfp is the best preconditioner, it converges within 5 iters using gmres for inner solvers but it is not very fast and sometimes multiplicative it's a better option as the inner iterations looks lighter. If I tune the relative tolerance I get a huge speed up, but I am a bit less confident about the results. The funny thing is that the default relative tolerance if commercial CFD solver is huge, very often 0.1 -.-

This depends on time step size and tolerances for quantities of interest.

If I may borrow your brain guys for a while, I would like to ask your opinion about the multigrid they use in this paper: https://www.aub.edu.lb/msfea/research/Documents/CFD-P18.pdf. At some point they say: " The algorithm used in this work is a combination of the ILU(0) [28] algorithm with an additive corrective multigrid method [29] ", 29: https://www.tandfonline.com/doi/abs/10.1080/10407788608913491

It looks like this is a coupled solver with collocated dofs and thus I don't know how they stabilize in the incompressible limit (an inf-sup requirement is needed for stability). (This is based on a cursory read.)