Limiting the number of vectors allocated at a time by fgmres etc.
Hello, I've been profiling the memory usage of my solver and it looks like a huge number (roughly half) of allocations are from KSPFGMRESGetNewVectors(). I read through the source code and it looks like these vectors are allocated ten at a time (FGMRES_DELTA_DIRECTIONS) in a couple of places inside that KSP. Is there a way to change this value? If not - how hard would it be to add an API to set a different initial value for that? These vectors take up a lot of memory and I would rather just one at a time. Best, David Wells
On Tue, Feb 10, 2026 at 5:32 PM Wells, David via petsc-users < [email protected]> wrote:
Hello,
I've been profiling the memory usage of my solver and it looks like a huge number (roughly half) of allocations are from KSPFGMRESGetNewVectors(). I read through the source code and it looks like these vectors are allocated ten at a time (FGMRES_DELTA_DIRECTIONS) in a couple of places inside that KSP. Is there a way to change this value?
We could add an option to change this delta. Actually theory suggests that a constant is not optimal, but rather we should double the number each time. I would also be willing to code that.
If not - how hard would it be to add an API to set a different initial value for that? These vectors take up a lot of memory and I would rather just one at a time.
I cannot understand precisely what is happening here. You specify a restart size when you setup the KSP. It allocates that many vecs (roughly). Why are there reallocations? Do you increase the restart size during the iteration? Thanks, Matt
Best, David Wells
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >
Hi Matt, Thanks for the quick response!
I cannot understand precisely what is happening here. You specify a restart size when you setup the KSP. It allocates that many vecs (roughly). Why are there reallocations? Do you increase the restart size during the iteration?
I don't believe there are any reallocations (I didn't write this solver, but I don't see any calls which set the restart size or any other relevant parameter [1]): as far as I can tell, the solver just allocates a lot of vectors. I'm working off of traces computed by heaptrack, which is my only insight into how this works. The allocations come from KSPCreateVecs(), which is called by 1. KSPFGMRESGetNewVectors() (for about 1.7 GB [2] of memory) 2. KSPSetUp_GMRES() (for about 300 MB of memory) 3. KSPSetUp_FGMRES() (for about 264 MB of memory) 4. KSPSetWorkVecs() (for about 236 MB of memory) Is there some relevant set of monitoring flags I can set which will show me how many vectors I allocate or use? That would also help. Best, David [1] This is IBAMR's PETScKrylovLinearSolver. [2] This is half the total memory we use for side-centered data vectors. ________________________________ From: Matthew Knepley <[email protected]> Sent: Tuesday, February 10, 2026 6:28 PM To: Wells, David <[email protected]> Cc: [email protected] <[email protected]> Subject: Re: [petsc-users] Limiting the number of vectors allocated at a time by fgmres etc. On Tue, Feb 10, 2026 at 5:32 PM Wells, David via petsc-users <[email protected]<mailto:[email protected]>> wrote: Hello, I've been profiling the memory usage of my solver and it looks like a huge number (roughly half) of allocations are from KSPFGMRESGetNewVectors(). I read through the source code and it looks like these vectors are allocated ten at a time (FGMRES_DELTA_DIRECTIONS) in a couple of places inside that KSP. Is there a way to change this value? We could add an option to change this delta. Actually theory suggests that a constant is not optimal, but rather we should double the number each time. I would also be willing to code that. If not - how hard would it be to add an API to set a different initial value for that? These vectors take up a lot of memory and I would rather just one at a time. I cannot understand precisely what is happening here. You specify a restart size when you setup the KSP. It allocates that many vecs (roughly). Why are there reallocations? Do you increase the restart size during the iteration? Thanks, Matt Best, David Wells -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfs... >
1) For a fixed restart (say of 30) FGMRES needs 60 vectors, while GMRES only needs 30. This is a big disadvantage of FGMRES over GMRES. 2) By default PETSc GMRES uses a restart of 30 meaning it keeps 30 previous Krylov vectors (and FGMRES needs 60 vectors). You can use a smaller restart with KSPGMRESSetRestart or -ksp_gmres_restart to need less memory (of course the convergence may get far worse or not depending on the problem. 3) When GMRES (or FGMRES) starts up it does not immediately allocate all 30 (or whatever) restart vectors because it may be that GMRES only takes 15 steps so why allocate all of them? Instead it allocates a chunk at a time GMRES_DELTA_DIRECTIONS which is 10 when it uses up the 10 it allocates another 10 (if needed) etc until it gets to the restart You can force GMRES to allocate all 30 (or whatever) initially instead of the chunk of a time approach by using “KSPGMRESSetPreAllocateVectors() or -ksp_gmres_preallocate This will prevent confusion about why more vectors are allocated later and why they are not all allocated when the solve starts. 4) PETSc’s GMRES tries to use BLAS 2 operations for MDot() and MAXPY (the orthogonalization in GMRES). It can only use the BLAS 2 on vector chunks that are allocated together. By preallocating all the vectors at the beginning one gets a single chunk and hence more efficient orthogonalization; this is more important on GPUs. For CPUs whether you have 10 or 30 vectors together doesn’t matter much at all. I hope this clarifies why you are seeing the memory allocations. Note that these are NOT “reallocations” in the sense of KSPGMRES allocating more memory and then copying something into the new memory and freeing the old. They are just allocations of new memory which will then be used. Barry
On Feb 10, 2026, at 9:04 PM, Wells, David via petsc-users <[email protected]> wrote:
Hi Matt,
Thanks for the quick response!
I cannot understand precisely what is happening here. You specify a restart size when you setup the KSP. It allocates that many vecs (roughly). Why are there reallocations? Do you increase the restart size during the iteration?
I don't believe there are any reallocations (I didn't write this solver, but I don't see any calls which set the restart size or any other relevant parameter [1]): as far as I can tell, the solver just allocates a lot of vectors. I'm working off of traces computed by heaptrack, which is my only insight into how this works. The allocations come from KSPCreateVecs(), which is called by 1. KSPFGMRESGetNewVectors() (for about 1.7 GB [2] of memory) 2. KSPSetUp_GMRES() (for about 300 MB of memory) 3. KSPSetUp_FGMRES() (for about 264 MB of memory) 4. KSPSetWorkVecs() (for about 236 MB of memory)
Is there some relevant set of monitoring flags I can set which will show me how many vectors I allocate or use? That would also help.
Best, David
[1] This is IBAMR's PETScKrylovLinearSolver. [2] This is half the total memory we use for side-centered data vectors. From: Matthew Knepley <[email protected]> Sent: Tuesday, February 10, 2026 6:28 PM To: Wells, David <[email protected]> Cc: [email protected] <[email protected]> Subject: Re: [petsc-users] Limiting the number of vectors allocated at a time by fgmres etc.
On Tue, Feb 10, 2026 at 5:32 PM Wells, David via petsc-users <[email protected] <mailto:[email protected]>> wrote: Hello,
I've been profiling the memory usage of my solver and it looks like a huge number (roughly half) of allocations are from KSPFGMRESGetNewVectors(). I read through the source code and it looks like these vectors are allocated ten at a time (FGMRES_DELTA_DIRECTIONS) in a couple of places inside that KSP. Is there a way to change this value?
We could add an option to change this delta. Actually theory suggests that a constant is not optimal, but rather we should double the number each time. I would also be willing to code that.
If not - how hard would it be to add an API to set a different initial value for that? These vectors take up a lot of memory and I would rather just one at a time.
I cannot understand precisely what is happening here. You specify a restart size when you setup the KSP. It allocates that many vecs (roughly). Why are there reallocations? Do you increase the restart size during the iteration?
Thanks,
Matt
Best, David Wells
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aBTZ4oaySeEsyg3U98-DZfMmoj0Wg8RUNEzlPdQoEnZ6pfCuP3cZSF7ib3bqpT7GBGVct81F2vjmak_Zva9875mpIc0$>
Hi Barry, Sorry for the slow response - yes, that makes sense, and it is a sensible default to have. I appreciate the writeup. Best, David ________________________________ From: Barry Smith <[email protected]> Sent: Tuesday, February 10, 2026 9:32 PM To: Wells, David <[email protected]> Cc: Matthew Knepley <[email protected]>; [email protected] <[email protected]> Subject: Re: [petsc-users] Limiting the number of vectors allocated at a time by fgmres etc. You don't often get email from [email protected]. Learn why this is important<https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!... > 1) For a fixed restart (say of 30) FGMRES needs 60 vectors, while GMRES only needs 30. This is a big disadvantage of FGMRES over GMRES. 2) By default PETSc GMRES uses a restart of 30 meaning it keeps 30 previous Krylov vectors (and FGMRES needs 60 vectors). You can use a smaller restart with KSPGMRESSetRestart or -ksp_gmres_restart to need less memory (of course the convergence may get far worse or not depending on the problem. 3) When GMRES (or FGMRES) starts up it does not immediately allocate all 30 (or whatever) restart vectors because it may be that GMRES only takes 15 steps so why allocate all of them? Instead it allocates a chunk at a time GMRES_DELTA_DIRECTIONS which is 10 when it uses up the 10 it allocates another 10 (if needed) etc until it gets to the restart You can force GMRES to allocate all 30 (or whatever) initially instead of the chunk of a time approach by using “KSPGMRESSetPreAllocateVectors() or -ksp_gmres_preallocate This will prevent confusion about why more vectors are allocated later and why they are not all allocated when the solve starts. 4) PETSc’s GMRES tries to use BLAS 2 operations for MDot() and MAXPY (the orthogonalization in GMRES). It can only use the BLAS 2 on vector chunks that are allocated together. By preallocating all the vectors at the beginning one gets a single chunk and hence more efficient orthogonalization; this is more important on GPUs. For CPUs whether you have 10 or 30 vectors together doesn’t matter much at all. I hope this clarifies why you are seeing the memory allocations. Note that these are NOT “reallocations” in the sense of KSPGMRES allocating more memory and then copying something into the new memory and freeing the old. They are just allocations of new memory which will then be used. Barry On Feb 10, 2026, at 9:04 PM, Wells, David via petsc-users <[email protected]> wrote: Hi Matt, Thanks for the quick response!
I cannot understand precisely what is happening here. You specify a restart size when you setup the KSP. It allocates that many vecs (roughly). Why are there reallocations? Do you increase the restart size during the iteration?
I don't believe there are any reallocations (I didn't write this solver, but I don't see any calls which set the restart size or any other relevant parameter [1]): as far as I can tell, the solver just allocates a lot of vectors. I'm working off of traces computed by heaptrack, which is my only insight into how this works. The allocations come from KSPCreateVecs(), which is called by 1. KSPFGMRESGetNewVectors() (for about 1.7 GB [2] of memory) 2. KSPSetUp_GMRES() (for about 300 MB of memory) 3. KSPSetUp_FGMRES() (for about 264 MB of memory) 4. KSPSetWorkVecs() (for about 236 MB of memory) Is there some relevant set of monitoring flags I can set which will show me how many vectors I allocate or use? That would also help. Best, David [1] This is IBAMR's PETScKrylovLinearSolver. [2] This is half the total memory we use for side-centered data vectors. ________________________________ From: Matthew Knepley <[email protected]> Sent: Tuesday, February 10, 2026 6:28 PM To: Wells, David <[email protected]> Cc: [email protected] <[email protected]> Subject: Re: [petsc-users] Limiting the number of vectors allocated at a time by fgmres etc. On Tue, Feb 10, 2026 at 5:32 PM Wells, David via petsc-users <[email protected]<mailto:[email protected]>> wrote: Hello, I've been profiling the memory usage of my solver and it looks like a huge number (roughly half) of allocations are from KSPFGMRESGetNewVectors(). I read through the source code and it looks like these vectors are allocated ten at a time (FGMRES_DELTA_DIRECTIONS) in a couple of places inside that KSP. Is there a way to change this value? We could add an option to change this delta. Actually theory suggests that a constant is not optimal, but rather we should double the number each time. I would also be willing to code that. If not - how hard would it be to add an API to set a different initial value for that? These vectors take up a lot of memory and I would rather just one at a time. I cannot understand precisely what is happening here. You specify a restart size when you setup the KSP. It allocates that many vecs (roughly). Why are there reallocations? Do you increase the restart size during the iteration? Thanks, Matt Best, David Wells -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf... <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aBTZ4oaySeEsyg3U98-DZfMmoj0Wg8RUNEzlPdQoEnZ6pfCuP3cZSF7ib3bqpT7GBGVct81F2vjmak_Zva9875mpIc0$>
participants (3)
-
Barry Smith -
Matthew Knepley -
Wells, David