Kokkos with OpenMP
I am getting this Kokkos build failure with OpenMP. Also, can you build Kokkos with OMP and Cuda? I know you can't have two execution spaces but can you use Kokkos-Cuda with OMP in the same executable? Thanks, Mark
Mark,
On Jan 17, 2021, at 3:08 PM, Mark Adams <[email protected]> wrote:
I am getting this Kokkos build failure with OpenMP.
Here is the problem. You set --CXXOPTFLAGS="-g -O0 -fPIC -fopenmp" to indicate how to compile for OpenMP but when ./configure checks the installed KokkosKernel include files it uses CXXPPFLAGS. (Since you provided the flag CXXOPTFLAGS ./configure never determines the flag to put into CXXPPFLAGS. It is more desirable if you do not provide -fPIC -fopenmp. ./configure will figure them out for you. But yes it should still work. I don't know how to fix configure to handle this case cleanly. We could not use the CXXPPFLAGS but always use the compiler flags for checking includes, this would change a 20 year old decision, since CXXPPFLAGS is for the preprocess :-(. Please remove the -fPIC -fopenmp from your ./configure options and see if it succeeds. (see below for the answer to your second question). ----- TESTING: checkInclude from config.headers(config/BuildSystem/config/headers.py:86) Checks if a particular include file can be found along particular include paths Checking for header files ['KokkosBlas.hpp', 'KokkosSparse_CrsMatrix.hpp'] in ['/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-omp/include'] Checking include with compiler flags var CXXPPFLAGS ['/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-omp/include', '/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-omp/include'] Preprocessing source: #include "confdefs.h" #include "conffix.h" #include <KokkosBlas.hpp> Executing: mpicxx -E -I/tmp/petsc-jEwQ8b/config.setCompilers -I/tmp/petsc-jEwQ8b/config.packages.kokkos -I/tmp/petsc-jEwQ8b/config.headers -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-omp/include -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-omp/include /tmp/petsc-jEwQ8b/config.headers/conftest.cc Possible ERROR while running preprocessor: exit code 1 stdout: ----
Also, can you build Kokkos with OMP and Cuda? I know you can't have two execution spaces but can you use Kokkos-Cuda with OMP in the same executable?
In theory you can do this but Barry enforced a packages design decision (for the sake of less configure options;) that does not allow it by default. Edit kokkos.py and remove the lines. if self.openmp.found + pthreadfound + self.cuda.found > 1: raise RuntimeError("Kokkos only supports a single parallel system during its configuration") if self.openmp.found: args.append('-DKokkos_ENABLE_OPENMP=ON') self.system = 'OpenMP' I guess we will need to add an option like --with-kokkos-device=CUDA or OpenMP or prethread or HIP to allow the user to set a particular device when several are possible. Stefano pushed for this long ago but I resisted. Barry
Thanks, Mark <configure.log>
Here is the problem. You set --CXXOPTFLAGS="-g -O0 -fPIC -fopenmp" to indicate how to compile for OpenMP but when ./configure checks the installed KokkosKernel include files it uses CXXPPFLAGS. (Since you provided the flag CXXOPTFLAGS ./configure never determines the flag to put into CXXPPFLAGS.
It is more desirable if you do not provide -fPIC -fopenmp. ./configure will figure them out for you.
:(
Mark, Looks like you tried to do two things at once, you removed the fopenmp and put back the --with-cuda. Like I said if you want PETSc to build OpenMP and CUDA you need to remove this line (and some others) from kokkos.py if self.openmp.found + pthreadfound + self.cuda.found > 1: raise RuntimeError("Kokkos only supports a single parallel system during its configuration")
On Jan 17, 2021, at 9:39 PM, Mark Adams <[email protected]> wrote:
Here is the problem. You set --CXXOPTFLAGS="-g -O0 -fPIC -fopenmp" to indicate how to compile for OpenMP but when ./configure checks the installed KokkosKernel include files it uses CXXPPFLAGS. (Since you provided the flag CXXOPTFLAGS ./configure never determines the flag to put into CXXPPFLAGS.
It is more desirable if you do not provide -fPIC -fopenmp. ./configure will figure them out for you.
:( <configure.log>
OK, this running and using OMP Fieldsplit/additive. Kokkos does not have any solvers hooked up yet, but I have added an LU interface that just uses AIJ's LU. Junchao, I asked Kokkos about adding LU and they said that it is on the todo list. Until they do that I can't use it for the paper, but if you want to work on integrating solvers into aijkokkos I can do that and test with GMRES/Jacobi. Thanks, Mark On Mon, Jan 18, 2021 at 1:00 AM Barry Smith <[email protected]> wrote:
Mark,
Looks like you tried to do two things at once, you removed the fopenmp and put back the --with-cuda.
Like I said if you want PETSc to build OpenMP and CUDA you need to remove this line (and some others) from kokkos.py
if self.openmp.found + pthreadfound + self.cuda.found > 1: raise RuntimeError("Kokkos only supports a single parallel system during its configuration")
On Jan 17, 2021, at 9:39 PM, Mark Adams <[email protected]> wrote:
Here is the problem. You set --CXXOPTFLAGS="-g -O0 -fPIC -fopenmp" to indicate how to compile for OpenMP but when ./configure checks the installed KokkosKernel include files it uses CXXPPFLAGS. (Since you provided the flag CXXOPTFLAGS ./configure never determines the flag to put into CXXPPFLAGS.
It is more desirable if you do not provide -fPIC -fopenmp. ./configure will figure them out for you.
:( <configure.log>
Mark, I hope I can take up the Kokkos work in early February once we have the ecp-tpds paper done. --Junchao Zhang On Mon, Jan 18, 2021 at 8:16 AM Mark Adams <[email protected]> wrote:
OK, this running and using OMP Fieldsplit/additive.
Kokkos does not have any solvers hooked up yet, but I have added an LU interface that just uses AIJ's LU.
Junchao, I asked Kokkos about adding LU and they said that it is on the todo list. Until they do that I can't use it for the paper, but if you want to work on integrating solvers into aijkokkos I can do that and test with GMRES/Jacobi.
Thanks, Mark
On Mon, Jan 18, 2021 at 1:00 AM Barry Smith <[email protected]> wrote:
Mark,
Looks like you tried to do two things at once, you removed the fopenmp and put back the --with-cuda.
Like I said if you want PETSc to build OpenMP and CUDA you need to remove this line (and some others) from kokkos.py
if self.openmp.found + pthreadfound + self.cuda.found > 1: raise RuntimeError("Kokkos only supports a single parallel system during its configuration")
On Jan 17, 2021, at 9:39 PM, Mark Adams <[email protected]> wrote:
Here is the problem. You set --CXXOPTFLAGS="-g -O0 -fPIC -fopenmp" to indicate how to compile for OpenMP but when ./configure checks the installed KokkosKernel include files it uses CXXPPFLAGS. (Since you provided the flag CXXOPTFLAGS ./configure never determines the flag to put into CXXPPFLAGS.
It is more desirable if you do not provide -fPIC -fopenmp. ./configure will figure them out for you.
:( <configure.log>
That would be great and early Feb would be perfect. Not having LU in Kokkos is not perfect but having a GPU ILU would be much much better than the current CPU LU. I am now suspecting that I am not running the job correctly for OMP at NERSc. I am going to read the docs more carefully. On Tue, Jan 19, 2021 at 10:27 AM Junchao Zhang <[email protected]> wrote:
Mark, I hope I can take up the Kokkos work in early February once we have the ecp-tpds paper done. --Junchao Zhang
On Mon, Jan 18, 2021 at 8:16 AM Mark Adams <[email protected]> wrote:
OK, this running and using OMP Fieldsplit/additive.
Kokkos does not have any solvers hooked up yet, but I have added an LU interface that just uses AIJ's LU.
Junchao, I asked Kokkos about adding LU and they said that it is on the todo list. Until they do that I can't use it for the paper, but if you want to work on integrating solvers into aijkokkos I can do that and test with GMRES/Jacobi.
Thanks, Mark
On Mon, Jan 18, 2021 at 1:00 AM Barry Smith <[email protected]> wrote:
Mark,
Looks like you tried to do two things at once, you removed the fopenmp and put back the --with-cuda.
Like I said if you want PETSc to build OpenMP and CUDA you need to remove this line (and some others) from kokkos.py
if self.openmp.found + pthreadfound + self.cuda.found > 1: raise RuntimeError("Kokkos only supports a single parallel system during its configuration")
On Jan 17, 2021, at 9:39 PM, Mark Adams <[email protected]> wrote:
Here is the problem. You set --CXXOPTFLAGS="-g -O0 -fPIC -fopenmp" to indicate how to compile for OpenMP but when ./configure checks the installed KokkosKernel include files it uses CXXPPFLAGS. (Since you provided the flag CXXOPTFLAGS ./configure never determines the flag to put into CXXPPFLAGS.
It is more desirable if you do not provide -fPIC -fopenmp. ./configure will figure them out for you.
:( <configure.log>
participants (3)
-
Barry Smith -
Junchao Zhang -
Mark Adams