It is not surprising. BCGS uses less memory for the Krylov vectors, but that might be a small fraction of the total memory used (considering your matrix and GAMG). FGMRES(30) needs 60 work vectors (2 per iteration). If you're using a linear (non-iterative) preconditioner, then you don't need a flexible method -- plain GMRES should be fine. FGMRES uses the unpreconditioned norm, which you can also get via -ksp_type gmres -ksp_norm_type unpreconditioned.

This classic paper shows that for any class of nonsymmetric Krylov method, there are matrices in which that method outperforms every other method by at least sqrt(N).

https://epubs.siam.org/doi/10.1137/0613049

Marco Cisternino <marco.cisternino@optimad.it> writes:

> Good Morning,
> I usually solve a non-symmetric discretization of the Poisson equation using GAMG+FGMRES.
> In the last days I tried to use BCGS in place of FGMRES, still using GAMG as preconditioner.
> No problem in finding the solution but I'm experiencing something I didn't expect.
> The test case is a 25 millions cells domain with Dirichlet and Neumann boundary conditions.
> Both the solvers are able to solve the problem with an increasing number of MPI processes, but:
>
>   *   FGMRES is about 25% faster than BCGS for all the processes number
>   *   Both solvers have the same scalability from 48 to 384 processes
>   *   Both solvers almost use the same amount of memory (FGMRES use a restart=30)
> Am I wrong expecting less memory consumption and more performance from BCGS with respect to FGMRES?
> Thank you in advance for any help.
>
> Best regards,
> Marco Cisternino