Hypre freezing with Mat type mpiaij and Vec type mpi
Dear all, I have a code running both on GPU and CPU. This code has both cuda kernels and calls to PETSc KSP and related PC. In particular, I am trying to perform tests with Hypre BoomerAMG both on CPU and GPU. In order to do that, on CPU I am running the code with -mat_type mpiaij and -vec_type mpi, while on GPU I am using respectively aijcusparse and cuda. The configuration ran for PETSc (version is 3.20) is ./configure PETSC_ARCH=arch-linux-cuda --with-cuda --download-mumps --download-hypre --with-debugging=0 --download-scalapack --download-parmetis --download-metis --download-fblaslapack=1 --download-mpich --download-make --download-cmake My problem is that when I try to run my code on GPU it works well, while on CPU with mat_type mpiaij and -vec_type mpi it works regularly until the call to Hypre, then freezes (I have to kill it myself), while with GAMG it works on CPU with the same configuration (and the same code, just PC is changed). On another machine running PETSc version 3.17 everything worked smoothly with the same code and the same configuration, also on Hypre. Do you have any insights on what is happening? Best regards, Edoardo
On Fri, Jan 5, 2024 at 5:40 AM Edoardo Centofanti < [email protected]> wrote:
Dear all,
I have a code running both on GPU and CPU. This code has both cuda kernels and calls to PETSc KSP and related PC. In particular, I am trying to perform tests with Hypre BoomerAMG both on CPU and GPU. In order to do that, on CPU I am running the code with -mat_type mpiaij and -vec_type mpi, while on GPU I am using respectively aijcusparse and cuda.
The configuration ran for PETSc (version is 3.20) is ./configure PETSC_ARCH=arch-linux-cuda --with-cuda --download-mumps --download-hypre --with-debugging=0 --download-scalapack --download-parmetis --download-metis --download-fblaslapack=1 --download-mpich --download-make --download-cmake
My problem is that when I try to run my code on GPU it works well, while on CPU with mat_type mpiaij and -vec_type mpi it works regularly until the call to Hypre, then freezes (I have to kill it myself), while with GAMG it works on CPU with the same configuration (and the same code, just PC is changed). On another machine running PETSc version 3.17 everything worked smoothly with the same code and the same configuration, also on Hypre.
Can you reproduce this error on this machine with 3.20? If yes you do a git bisect to find the commit that causes this. That would be a good start. A stack trace would be helpful. You can run this in a debugger and see where you are hung. A GUI debugger is best for this (DDT or Totalview) but a command line debugger is fine if you can do this in a serial run (Control-C will stop the code and give you a prompt and you can then see the stack). Mark
Do you have any insights on what is happening?
Best regards, Edoardo
On Fri, Jan 5, 2024 at 4:40 AM Edoardo Centofanti < [email protected]> wrote:
Dear all,
I have a code running both on GPU and CPU. This code has both cuda kernels and calls to PETSc KSP and related PC. In particular, I am trying to perform tests with Hypre BoomerAMG both on CPU and GPU. In order to do that, on CPU I am running the code with -mat_type mpiaij and -vec_type mpi, while on GPU I am using respectively aijcusparse and cuda.
The configuration ran for PETSc (version is 3.20) is ./configure PETSC_ARCH=arch-linux-cuda --with-cuda --download-mumps --download-hypre --with-debugging=0 --download-scalapack --download-parmetis --download-metis --download-fblaslapack=1 --download-mpich --download-make --download-cmake
My problem is that when I try to run my code on GPU it works well, while on CPU with mat_type mpiaij and -vec_type mpi it works regularly until the call to Hypre, then freezes (I have to kill it myself), while with GAMG it works on CPU with the same configuration (and the same code, just PC is changed).
Probably because if hypre is configured with GPU support, even petsc is running on CPU (with -mat_type mpiaij and -vec_type mpi), hypre is still trying to run on GPU, causing the hanging problem. The petsc/hypre interface is not able to support both CPU/GPU with the same build.
On another machine running PETSc version 3.17 everything worked smoothly with the same code and the same configuration, also on Hypre.
I guess with petsc-3.17, you also used a different hypre version (downloaded by petsc).
Do you have any insights on what is happening?
Best regards, Edoardo
Il giorno ven 5 gen 2024 alle ore 16:21 Junchao Zhang < [email protected]> ha scritto:
On Fri, Jan 5, 2024 at 4:40 AM Edoardo Centofanti < [email protected]> wrote:
Dear all,
I have a code running both on GPU and CPU. This code has both cuda kernels and calls to PETSc KSP and related PC. In particular, I am trying to perform tests with Hypre BoomerAMG both on CPU and GPU. In order to do that, on CPU I am running the code with -mat_type mpiaij and -vec_type mpi, while on GPU I am using respectively aijcusparse and cuda.
The configuration ran for PETSc (version is 3.20) is ./configure PETSC_ARCH=arch-linux-cuda --with-cuda --download-mumps --download-hypre --with-debugging=0 --download-scalapack --download-parmetis --download-metis --download-fblaslapack=1 --download-mpich --download-make --download-cmake
My problem is that when I try to run my code on GPU it works well, while on CPU with mat_type mpiaij and -vec_type mpi it works regularly until the call to Hypre, then freezes (I have to kill it myself), while with GAMG it works on CPU with the same configuration (and the same code, just PC is changed).
Probably because if hypre is configured with GPU support, even petsc is running on CPU (with -mat_type mpiaij and -vec_type mpi), hypre is still trying to run on GPU, causing the hanging problem. The petsc/hypre interface is not able to support both CPU/GPU with the same build.
On another machine running PETSc version 3.17 everything worked smoothly with the same code and the same configuration, also on Hypre.
I guess with petsc-3.17, you also used a different hypre version (downloaded by petsc).
Yes, this is exactly what I concluded after a bit of debugging and recompiling. In the new version it seems that CPU/GPU with the same build is not supported anymore. Since the CUDA kernels I have in my code do not interfere with Hypre, the only workaround I found was removing kernels from my code and performing an only-CPU run. It slows down a bit overall, but nothing dramatic. Anyway, in another test with version 3.18 I encountered the same problem, but the program crashes and throws "Invalid options" error, while in 3.20 it just hangs. Thanks, Edoardo
Do you have any insights on what is happening?
Best regards, Edoardo
On Fri, Jan 5, 2024 at 9:33 AM Edoardo Centofanti < [email protected]> wrote:
Il giorno ven 5 gen 2024 alle ore 16:21 Junchao Zhang < [email protected]> ha scritto:
On Fri, Jan 5, 2024 at 4:40 AM Edoardo Centofanti < [email protected]> wrote:
Dear all,
I have a code running both on GPU and CPU. This code has both cuda kernels and calls to PETSc KSP and related PC. In particular, I am trying to perform tests with Hypre BoomerAMG both on CPU and GPU. In order to do that, on CPU I am running the code with -mat_type mpiaij and -vec_type mpi, while on GPU I am using respectively aijcusparse and cuda.
The configuration ran for PETSc (version is 3.20) is ./configure PETSC_ARCH=arch-linux-cuda --with-cuda --download-mumps --download-hypre --with-debugging=0 --download-scalapack --download-parmetis --download-metis --download-fblaslapack=1 --download-mpich --download-make --download-cmake
My problem is that when I try to run my code on GPU it works well, while on CPU with mat_type mpiaij and -vec_type mpi it works regularly until the call to Hypre, then freezes (I have to kill it myself), while with GAMG it works on CPU with the same configuration (and the same code, just PC is changed).
Probably because if hypre is configured with GPU support, even petsc is running on CPU (with -mat_type mpiaij and -vec_type mpi), hypre is still trying to run on GPU, causing the hanging problem. The petsc/hypre interface is not able to support both CPU/GPU with the same build.
On another machine running PETSc version 3.17 everything worked smoothly with the same code and the same configuration, also on Hypre.
I guess with petsc-3.17, you also used a different hypre version (downloaded by petsc).
Yes, this is exactly what I concluded after a bit of debugging and recompiling. In the new version it seems that CPU/GPU with the same build is not supported anymore. Since the CUDA kernels I have in my code do not interfere with Hypre, the only workaround I found was removing kernels from my code and performing an only-CPU run. It slows down a bit overall, but nothing dramatic. Anyway, in another test with version 3.18 I encountered the same problem, but the program crashes and throws "Invalid options" error, while in 3.20 it just hangs.
Improving the petsc/hypre interface to support both CPU/GPU is WIP.
Thanks, Edoardo
Do you have any insights on what is happening?
Best regards, Edoardo
participants (3)
-
Edoardo Centofanti -
Junchao Zhang -
Mark Adams