Using multiple MPI ranks with COO interface crashes in some cases
In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use the COO interface for preallocating and setting values in the Jacobian matrix. I have found that with some of our test cases, using more than one MPI rank results in a crash. Way down in the preconditioner code in petsc a Mat gets computed that has "null" for the "productsymbolic" member of its "ops". It's pretty far removed from where we compute the Jacobian entries, so I haven't been able (so far) to track it back to an error in my code. I'd appreciate some help with this from someone who is more familiar with the petsc guts so we can figure out what I'm doing wrong. (I'm assuming it's a bug in Xolotl.) Note that this is using the kokkos backend for Mat and Vec in petsc, but with a serial-only build of kokkos and kokkos-kernels. So, it's a CPU-only multiple MPI rank run. Here's a paste of the error output showing the relevant parts of the call stack: [ERROR] [0]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [1]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [0]PETSC ERROR: [ERROR] No method productsymbolic for Mat of type (null) [ERROR] No method productsymbolic for Mat of type (null) [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] See https://petsc.org/release/faq/ for trouble shooting. [ERROR] See https://petsc.org/release/faq/ for trouble shooting. [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] PetscSolver::solve: TSSolve failed. [ERROR] PetscSolver::solve: TSSolve failed. Aborting. Aborting. Thanks for the help, Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory
Mat of type (null) Either the entire matrix (header) data structure has gotten corrupted or the matrix type was never set. Can you run with valgrind to see if there is any memory corruption?
On Nov 14, 2022, at 1:24 PM, Fackler, Philip via petsc-users <[email protected]> wrote:
In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use the COO interface for preallocating and setting values in the Jacobian matrix. I have found that with some of our test cases, using more than one MPI rank results in a crash. Way down in the preconditioner code in petsc a Mat gets computed that has "null" for the "productsymbolic" member of its "ops". It's pretty far removed from where we compute the Jacobian entries, so I haven't been able (so far) to track it back to an error in my code. I'd appreciate some help with this from someone who is more familiar with the petsc guts so we can figure out what I'm doing wrong. (I'm assuming it's a bug in Xolotl.)
Note that this is using the kokkos backend for Mat and Vec in petsc, but with a serial-only build of kokkos and kokkos-kernels. So, it's a CPU-only multiple MPI rank run.
Here's a paste of the error output showing the relevant parts of the call stack:
[ERROR] [0]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [1]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [0]PETSC ERROR: [ERROR] No method productsymbolic for Mat of type (null) [ERROR] No method productsymbolic for Mat of type (null) [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] See https://petsc.org/release/faq/ for trouble shooting. [ERROR] See https://petsc.org/release/faq/ for trouble shooting. [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] PetscSolver::solve: TSSolve failed. [ERROR] PetscSolver::solve: TSSolve failed. Aborting. Aborting.
Thanks for the help,
Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory
Hi, Philip, Can you tell me instructions to build Xolotl to reproduce the error? --Junchao Zhang On Mon, Nov 14, 2022 at 12:24 PM Fackler, Philip via petsc-users < [email protected]> wrote:
In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use the COO interface for preallocating and setting values in the Jacobian matrix. I have found that with some of our test cases, using more than one MPI rank results in a crash. Way down in the preconditioner code in petsc a Mat gets computed that has "null" for the "productsymbolic" member of its "ops". It's pretty far removed from where we compute the Jacobian entries, so I haven't been able (so far) to track it back to an error in my code. I'd appreciate some help with this from someone who is more familiar with the petsc guts so we can figure out what I'm doing wrong. (I'm assuming it's a bug in Xolotl.)
Note that this is using the kokkos backend for Mat and Vec in petsc, but with a serial-only build of kokkos and kokkos-kernels. So, it's a CPU-only multiple MPI rank run.
Here's a paste of the error output showing the relevant parts of the call stack:
[ERROR] [0]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [1]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [0]PETSC ERROR: [ERROR] No method productsymbolic for Mat of type (null) [ERROR] No method productsymbolic for Mat of type (null) [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] See https://petsc.org/release/faq/ for trouble shooting. [ERROR] See https://petsc.org/release/faq/ for trouble shooting. [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] PetscSolver::solve: TSSolve failed. [ERROR] PetscSolver::solve: TSSolve failed. Aborting. Aborting.
Thanks for the help,
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory*
I built petsc with: $ ./configure PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-debugging=0 --prefix=$HOME/build/petsc/debug/install --with-64-bit-indices --with-shared-libraries --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --download-kokkos --download-kokkos-kernels $ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug all $ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug install Then I build xolotl in a separate build directory (after checking out the "feature-petsc-kokkos" branch) with: $ cmake -DCMAKE_BUILD_TYPE=Debug -DKokkos_DIR=$HOME/build/petsc/debug/install -DPETSC_DIR=$HOME/build/petsc/debug/install <xolotl-src> $ make -j4 SystemTester Then, from the xolotl build directory, run (for example): $ mpirun -n 2 ./test/system/SystemTester -t System/NE_4 -- -v Note that this test case will use the parameter file '<xolotl-src>/benchmarks/params_system_NE_4.txt' which has the command-line arguments for petsc in its "petscArgs=..." line. If you look at '<xolotl-src>/test/system/SystemTester.cpp' all the system test cases follow the same naming convention with their corresponding parameter files under '<xolotl-src>/benchmarks'. The failure happens with the NE_4 case (which is 2D) and the PSI_3 case (which is 1D). Let me know if this is still unclear. Thanks, Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory ________________________________ From: Junchao Zhang <[email protected]> Sent: Tuesday, November 15, 2022 00:16 To: Fackler, Philip <[email protected]> Cc: [email protected] <[email protected]>; Blondel, Sophie <[email protected]> Subject: [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases Hi, Philip, Can you tell me instructions to build Xolotl to reproduce the error? --Junchao Zhang On Mon, Nov 14, 2022 at 12:24 PM Fackler, Philip via petsc-users <[email protected]<mailto:[email protected]>> wrote: In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use the COO interface for preallocating and setting values in the Jacobian matrix. I have found that with some of our test cases, using more than one MPI rank results in a crash. Way down in the preconditioner code in petsc a Mat gets computed that has "null" for the "productsymbolic" member of its "ops". It's pretty far removed from where we compute the Jacobian entries, so I haven't been able (so far) to track it back to an error in my code. I'd appreciate some help with this from someone who is more familiar with the petsc guts so we can figure out what I'm doing wrong. (I'm assuming it's a bug in Xolotl.) Note that this is using the kokkos backend for Mat and Vec in petsc, but with a serial-only build of kokkos and kokkos-kernels. So, it's a CPU-only multiple MPI rank run. Here's a paste of the error output showing the relevant parts of the call stack: [ERROR] [0]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [1]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [0]PETSC ERROR: [ERROR] No method productsymbolic for Mat of type (null) [ERROR] No method productsymbolic for Mat of type (null) [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] PetscSolver::solve: TSSolve failed. [ERROR] PetscSolver::solve: TSSolve failed. Aborting. Aborting. Thanks for the help, Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory
Junchao, this is the same problem that I have been having right? On Tue, Nov 15, 2022 at 11:56 AM Fackler, Philip via petsc-users < [email protected]> wrote:
I built petsc with:
$ ./configure PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-debugging=0 --prefix=$HOME/build/petsc/debug/install --with-64-bit-indices --with-shared-libraries --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --download-kokkos --download-kokkos-kernels
$ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug all
$ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug install
Then I build xolotl in a separate build directory (after checking out the "feature-petsc-kokkos" branch) with:
$ cmake -DCMAKE_BUILD_TYPE=Debug -DKokkos_DIR=$HOME/build/petsc/debug/install -DPETSC_DIR=$HOME/build/petsc/debug/install <xolotl-src>
$ make -j4 SystemTester
Then, from the xolotl build directory, run (for example):
$ mpirun -n 2 ./test/system/SystemTester -t System/NE_4 -- -v
Note that this test case will use the parameter file '<xolotl-src>/benchmarks/params_system_NE_4.txt' which has the command-line arguments for petsc in its "petscArgs=..." line. If you look at '<xolotl-src>/test/system/SystemTester.cpp' all the system test cases follow the same naming convention with their corresponding parameter files under '<xolotl-src>/benchmarks'.
The failure happens with the NE_4 case (which is 2D) and the PSI_3 case (which is 1D).
Let me know if this is still unclear.
Thanks,
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory* ------------------------------ *From:* Junchao Zhang <[email protected]> *Sent:* Tuesday, November 15, 2022 00:16 *To:* Fackler, Philip <[email protected]> *Cc:* [email protected] <[email protected]>; Blondel, Sophie < [email protected]> *Subject:* [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases
Hi, Philip, Can you tell me instructions to build Xolotl to reproduce the error? --Junchao Zhang
On Mon, Nov 14, 2022 at 12:24 PM Fackler, Philip via petsc-users < [email protected]> wrote:
In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use the COO interface for preallocating and setting values in the Jacobian matrix. I have found that with some of our test cases, using more than one MPI rank results in a crash. Way down in the preconditioner code in petsc a Mat gets computed that has "null" for the "productsymbolic" member of its "ops". It's pretty far removed from where we compute the Jacobian entries, so I haven't been able (so far) to track it back to an error in my code. I'd appreciate some help with this from someone who is more familiar with the petsc guts so we can figure out what I'm doing wrong. (I'm assuming it's a bug in Xolotl.)
Note that this is using the kokkos backend for Mat and Vec in petsc, but with a serial-only build of kokkos and kokkos-kernels. So, it's a CPU-only multiple MPI rank run.
Here's a paste of the error output showing the relevant parts of the call stack:
[ERROR] [0]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [1]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [0]PETSC ERROR: [ERROR] No method productsymbolic for Mat of type (null) [ERROR] No method productsymbolic for Mat of type (null) [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] PetscSolver::solve: TSSolve failed. [ERROR] PetscSolver::solve: TSSolve failed. Aborting. Aborting.
Thanks for the help,
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory*
Mark, Do you have a reproducer using petsc examples? On Tue, Nov 15, 2022, 12:49 PM Mark Adams <[email protected]> wrote:
Junchao, this is the same problem that I have been having right?
On Tue, Nov 15, 2022 at 11:56 AM Fackler, Philip via petsc-users < [email protected]> wrote:
I built petsc with:
$ ./configure PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-debugging=0 --prefix=$HOME/build/petsc/debug/install --with-64-bit-indices --with-shared-libraries --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --download-kokkos --download-kokkos-kernels
$ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug all
$ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug install
Then I build xolotl in a separate build directory (after checking out the "feature-petsc-kokkos" branch) with:
$ cmake -DCMAKE_BUILD_TYPE=Debug -DKokkos_DIR=$HOME/build/petsc/debug/install -DPETSC_DIR=$HOME/build/petsc/debug/install <xolotl-src>
$ make -j4 SystemTester
Then, from the xolotl build directory, run (for example):
$ mpirun -n 2 ./test/system/SystemTester -t System/NE_4 -- -v
Note that this test case will use the parameter file '<xolotl-src>/benchmarks/params_system_NE_4.txt' which has the command-line arguments for petsc in its "petscArgs=..." line. If you look at '<xolotl-src>/test/system/SystemTester.cpp' all the system test cases follow the same naming convention with their corresponding parameter files under '<xolotl-src>/benchmarks'.
The failure happens with the NE_4 case (which is 2D) and the PSI_3 case (which is 1D).
Let me know if this is still unclear.
Thanks,
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory* ------------------------------ *From:* Junchao Zhang <[email protected]> *Sent:* Tuesday, November 15, 2022 00:16 *To:* Fackler, Philip <[email protected]> *Cc:* [email protected] <[email protected]>; Blondel, Sophie <[email protected]> *Subject:* [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases
Hi, Philip, Can you tell me instructions to build Xolotl to reproduce the error? --Junchao Zhang
On Mon, Nov 14, 2022 at 12:24 PM Fackler, Philip via petsc-users < [email protected]> wrote:
In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use the COO interface for preallocating and setting values in the Jacobian matrix. I have found that with some of our test cases, using more than one MPI rank results in a crash. Way down in the preconditioner code in petsc a Mat gets computed that has "null" for the "productsymbolic" member of its "ops". It's pretty far removed from where we compute the Jacobian entries, so I haven't been able (so far) to track it back to an error in my code. I'd appreciate some help with this from someone who is more familiar with the petsc guts so we can figure out what I'm doing wrong. (I'm assuming it's a bug in Xolotl.)
Note that this is using the kokkos backend for Mat and Vec in petsc, but with a serial-only build of kokkos and kokkos-kernels. So, it's a CPU-only multiple MPI rank run.
Here's a paste of the error output showing the relevant parts of the call stack:
[ERROR] [0]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [1]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [0]PETSC ERROR: [ERROR] No method productsymbolic for Mat of type (null) [ERROR] No method productsymbolic for Mat of type (null) [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] PetscSolver::solve: TSSolve failed. [ERROR] PetscSolver::solve: TSSolve failed. Aborting. Aborting.
Thanks for the help,
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory*
I can not build right now on Crusher or Perlmutter but I saw this on both. Here is an example output using src/snes/tests/ex13.c using the appended .petscrc This uses 64 processors and the 8 processor case worked. This has been semi-nondertminisitc for me. (and I have attached my current Perlmutter problem) Hope this helps, Mark -dm_plex_simplex 0 -dm_plex_dim 3 -dm_plex_box_lower 0,0,0 -dm_plex_box_upper 1,1,1 -petscpartitioner_simple_process_grid 2,2,2 -potential_petscspace_degree 2 -snes_max_it 1 -ksp_max_it 200 -ksp_type cg -ksp_rtol 1.e-12 -ksp_norm_type unpreconditioned -snes_rtol 1.e-8 #-pc_type gamg #-pc_gamg_type agg #-pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 100 -pc_gamg_process_eq_limit 400 -pc_gamg_reuse_interpolation true #-snes_monitor #-ksp_monitor_short -ksp_converged_reason #-ksp_view #-snes_converged_reason #-mg_levels_ksp_max_it 2 -mg_levels_ksp_type chebyshev #-mg_levels_ksp_type richardson #-mg_levels_ksp_richardson_scale 0.8 -mg_levels_pc_type jacobi -pc_gamg_esteig_ksp_type cg -pc_gamg_esteig_ksp_max_it 10 -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -dm_distribute -petscpartitioner_type simple -pc_gamg_repartition false -pc_gamg_coarse_grid_layout_type compact -pc_gamg_threshold 0.01 #-pc_gamg_threshold_scale .5 -pc_gamg_aggressive_coarsening 1 #-check_pointer_intensity 0 -snes_type ksponly #-mg_coarse_sub_pc_factor_mat_solver_type cusparse #-info :pc #-use_gpu_aware_mpi 1 -options_left #-malloc_debug -benchmark_it 10 #-pc_gamg_use_parallel_coarse_grid_solver #-mg_coarse_pc_type jacobi #-mg_coarse_ksp_type cg #-mg_coarse_ksp_rtol 1.e-2 #-mat_cusparse_transgen -snes_lag_jacobian -2 On Tue, Nov 15, 2022 at 3:42 PM Junchao Zhang <[email protected]> wrote:
Mark, Do you have a reproducer using petsc examples?
On Tue, Nov 15, 2022, 12:49 PM Mark Adams <[email protected]> wrote:
Junchao, this is the same problem that I have been having right?
On Tue, Nov 15, 2022 at 11:56 AM Fackler, Philip via petsc-users < [email protected]> wrote:
I built petsc with:
$ ./configure PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-debugging=0 --prefix=$HOME/build/petsc/debug/install --with-64-bit-indices --with-shared-libraries --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --download-kokkos --download-kokkos-kernels
$ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug all
$ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug install
Then I build xolotl in a separate build directory (after checking out the "feature-petsc-kokkos" branch) with:
$ cmake -DCMAKE_BUILD_TYPE=Debug -DKokkos_DIR=$HOME/build/petsc/debug/install -DPETSC_DIR=$HOME/build/petsc/debug/install <xolotl-src>
$ make -j4 SystemTester
Then, from the xolotl build directory, run (for example):
$ mpirun -n 2 ./test/system/SystemTester -t System/NE_4 -- -v
Note that this test case will use the parameter file '<xolotl-src>/benchmarks/params_system_NE_4.txt' which has the command-line arguments for petsc in its "petscArgs=..." line. If you look at '<xolotl-src>/test/system/SystemTester.cpp' all the system test cases follow the same naming convention with their corresponding parameter files under '<xolotl-src>/benchmarks'.
The failure happens with the NE_4 case (which is 2D) and the PSI_3 case (which is 1D).
Let me know if this is still unclear.
Thanks,
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory* ------------------------------ *From:* Junchao Zhang <[email protected]> *Sent:* Tuesday, November 15, 2022 00:16 *To:* Fackler, Philip <[email protected]> *Cc:* [email protected] <[email protected]>; Blondel, Sophie <[email protected]> *Subject:* [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases
Hi, Philip, Can you tell me instructions to build Xolotl to reproduce the error? --Junchao Zhang
On Mon, Nov 14, 2022 at 12:24 PM Fackler, Philip via petsc-users < [email protected]> wrote:
In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use the COO interface for preallocating and setting values in the Jacobian matrix. I have found that with some of our test cases, using more than one MPI rank results in a crash. Way down in the preconditioner code in petsc a Mat gets computed that has "null" for the "productsymbolic" member of its "ops". It's pretty far removed from where we compute the Jacobian entries, so I haven't been able (so far) to track it back to an error in my code. I'd appreciate some help with this from someone who is more familiar with the petsc guts so we can figure out what I'm doing wrong. (I'm assuming it's a bug in Xolotl.)
Note that this is using the kokkos backend for Mat and Vec in petsc, but with a serial-only build of kokkos and kokkos-kernels. So, it's a CPU-only multiple MPI rank run.
Here's a paste of the error output showing the relevant parts of the call stack:
[ERROR] [0]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [1]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [0]PETSC ERROR: [ERROR] No method productsymbolic for Mat of type (null) [ERROR] No method productsymbolic for Mat of type (null) [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] PetscSolver::solve: TSSolve failed. [ERROR] PetscSolver::solve: TSSolve failed. Aborting. Aborting.
Thanks for the help,
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory*
I am able to reproduce this on Crusher with 8 processors. Junchao, did you want me to use --download-kokkos-commit=origin/develop ? On Wed, Nov 16, 2022 at 8:05 AM Mark Adams <[email protected]> wrote:
I can not build right now on Crusher or Perlmutter but I saw this on both.
Here is an example output using src/snes/tests/ex13.c using the appended .petscrc This uses 64 processors and the 8 processor case worked. This has been semi-nondertminisitc for me.
(and I have attached my current Perlmutter problem)
Hope this helps, Mark
-dm_plex_simplex 0 -dm_plex_dim 3 -dm_plex_box_lower 0,0,0 -dm_plex_box_upper 1,1,1 -petscpartitioner_simple_process_grid 2,2,2 -potential_petscspace_degree 2 -snes_max_it 1 -ksp_max_it 200 -ksp_type cg -ksp_rtol 1.e-12 -ksp_norm_type unpreconditioned -snes_rtol 1.e-8 #-pc_type gamg #-pc_gamg_type agg #-pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 100 -pc_gamg_process_eq_limit 400 -pc_gamg_reuse_interpolation true #-snes_monitor #-ksp_monitor_short -ksp_converged_reason #-ksp_view #-snes_converged_reason #-mg_levels_ksp_max_it 2 -mg_levels_ksp_type chebyshev #-mg_levels_ksp_type richardson #-mg_levels_ksp_richardson_scale 0.8 -mg_levels_pc_type jacobi -pc_gamg_esteig_ksp_type cg -pc_gamg_esteig_ksp_max_it 10 -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -dm_distribute -petscpartitioner_type simple -pc_gamg_repartition false -pc_gamg_coarse_grid_layout_type compact -pc_gamg_threshold 0.01 #-pc_gamg_threshold_scale .5 -pc_gamg_aggressive_coarsening 1 #-check_pointer_intensity 0 -snes_type ksponly #-mg_coarse_sub_pc_factor_mat_solver_type cusparse #-info :pc #-use_gpu_aware_mpi 1 -options_left #-malloc_debug -benchmark_it 10 #-pc_gamg_use_parallel_coarse_grid_solver #-mg_coarse_pc_type jacobi #-mg_coarse_ksp_type cg #-mg_coarse_ksp_rtol 1.e-2 #-mat_cusparse_transgen -snes_lag_jacobian -2
On Tue, Nov 15, 2022 at 3:42 PM Junchao Zhang <[email protected]> wrote:
Mark, Do you have a reproducer using petsc examples?
On Tue, Nov 15, 2022, 12:49 PM Mark Adams <[email protected]> wrote:
Junchao, this is the same problem that I have been having right?
On Tue, Nov 15, 2022 at 11:56 AM Fackler, Philip via petsc-users < [email protected]> wrote:
I built petsc with:
$ ./configure PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-debugging=0 --prefix=$HOME/build/petsc/debug/install --with-64-bit-indices --with-shared-libraries --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --download-kokkos --download-kokkos-kernels
$ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug all
$ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug install
Then I build xolotl in a separate build directory (after checking out the "feature-petsc-kokkos" branch) with:
$ cmake -DCMAKE_BUILD_TYPE=Debug -DKokkos_DIR=$HOME/build/petsc/debug/install -DPETSC_DIR=$HOME/build/petsc/debug/install <xolotl-src>
$ make -j4 SystemTester
Then, from the xolotl build directory, run (for example):
$ mpirun -n 2 ./test/system/SystemTester -t System/NE_4 -- -v
Note that this test case will use the parameter file '<xolotl-src>/benchmarks/params_system_NE_4.txt' which has the command-line arguments for petsc in its "petscArgs=..." line. If you look at '<xolotl-src>/test/system/SystemTester.cpp' all the system test cases follow the same naming convention with their corresponding parameter files under '<xolotl-src>/benchmarks'.
The failure happens with the NE_4 case (which is 2D) and the PSI_3 case (which is 1D).
Let me know if this is still unclear.
Thanks,
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory* ------------------------------ *From:* Junchao Zhang <[email protected]> *Sent:* Tuesday, November 15, 2022 00:16 *To:* Fackler, Philip <[email protected]> *Cc:* [email protected] <[email protected]>; Blondel, Sophie <[email protected]> *Subject:* [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases
Hi, Philip, Can you tell me instructions to build Xolotl to reproduce the error? --Junchao Zhang
On Mon, Nov 14, 2022 at 12:24 PM Fackler, Philip via petsc-users < [email protected]> wrote:
In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use the COO interface for preallocating and setting values in the Jacobian matrix. I have found that with some of our test cases, using more than one MPI rank results in a crash. Way down in the preconditioner code in petsc a Mat gets computed that has "null" for the "productsymbolic" member of its "ops". It's pretty far removed from where we compute the Jacobian entries, so I haven't been able (so far) to track it back to an error in my code. I'd appreciate some help with this from someone who is more familiar with the petsc guts so we can figure out what I'm doing wrong. (I'm assuming it's a bug in Xolotl.)
Note that this is using the kokkos backend for Mat and Vec in petsc, but with a serial-only build of kokkos and kokkos-kernels. So, it's a CPU-only multiple MPI rank run.
Here's a paste of the error output showing the relevant parts of the call stack:
[ERROR] [0]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [1]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [0]PETSC ERROR: [ERROR] No method productsymbolic for Mat of type (null) [ERROR] No method productsymbolic for Mat of type (null) [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] PetscSolver::solve: TSSolve failed. [ERROR] PetscSolver::solve: TSSolve failed. Aborting. Aborting.
Thanks for the help,
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory*
Junchao, I tried with the Kokkos dev branch and get this with 8 processes (and the .petscrc file that I sent/appended): srun -n8 -N1 --gpus-per-task=1 --gpu-bind=closest ../ex13 -dm_plex_box_faces 2,2,2 -petscpartitioner_simple_process_grid 2,2,2 -dm_plex_box_upper 1,1,1 -petscpartitioner_simple_node_grid 1,1,1 -dm_refine 4 -dm_view -log_tracexxx -log_view -dm_mat_type aijkokkos -dm_vec_type kokkos DM Object: box 8 MPI processes type: plex box in 3 dimensions: Number of 0-cells per rank: 4913 4913 4913 4913 4913 4913 4913 4913 Number of 1-cells per rank: 13872 13872 13872 13872 13872 13872 13872 13872 Number of 2-cells per rank: 13056 13056 13056 13056 13056 13056 13056 13056 Number of 3-cells per rank: 4096 4096 4096 4096 4096 4096 4096 4096 Labels: celltype: 4 strata with value/size (0 (4913), 1 (13872), 4 (13056), 7 (4096)) depth: 4 strata with value/size (0 (4913), 1 (13872), 2 (13056), 3 (4096)) marker: 1 strata with value/size (1 (3169)) Face Sets: 3 strata with value/size (1 (961), 3 (961), 6 (961)) Number equations N = 250047 [5]PETSC ERROR: ------------------------------------------------------------------------ [5]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [5]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and https://petsc.org/release/faq/ [5]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [5]PETSC ERROR: The line numbers in the error traceback are not always exact. [5]PETSC ERROR: #1 MPI function [5]PETSC ERROR: #2 PetscSFLinkWaitRequests_MPI() at /gpfs/alpine/csc314/scratch/adams/petsc/src/vec/is/sf/impls/basic/sfmpi.c:53 [5]PETSC ERROR: #3 PetscSFLinkFinishCommunication() at /gpfs/alpine/csc314/scratch/adams/petsc/include/../src/vec/is/sf/impls/basic/sfpack.h:277 [5]PETSC ERROR: #4 PetscSFBcastEnd_Basic() at /gpfs/alpine/csc314/scratch/adams/petsc/src/vec/is/sf/impls/basic/sfbasic.c:205 [5]PETSC ERROR: #5 PetscSFBcastEnd() at /gpfs/alpine/csc314/scratch/adams/petsc/src/vec/is/sf/interface/sf.c:1477 [5]PETSC ERROR: #6 DMGlobalToLocalEnd() at /gpfs/alpine/csc314/scratch/adams/petsc/src/dm/interface/dm.c:2849 [5]PETSC ERROR: #7 SNESComputeFunction_DMLocal() at /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/utils/dmlocalsnes.c:65 [5]PETSC ERROR: #8 SNES callback function [5]PETSC ERROR: #9 SNESComputeFunction() at /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/interface/snes.c:2436 [5]PETSC ERROR: #10 SNESSolve_KSPONLY() at /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/impls/ksponly/ksponly.c:27 [5]PETSC ERROR: #11 SNESSolve() at /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/interface/snes.c:4690 [5]PETSC ERROR: #12 main() at ex13.c:178 MPICH ERROR [Rank 5] [job id 213404.0] [Wed Nov 16 18:30:49 2022] [crusher011] - Abort(59) (rank 5 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 5 On Wed, Nov 16, 2022 at 2:51 PM Mark Adams <[email protected]> wrote:
I am able to reproduce this on Crusher with 8 processors.
Junchao, did you want me to use --download-kokkos-commit=origin/develop ?
On Wed, Nov 16, 2022 at 8:05 AM Mark Adams <[email protected]> wrote:
I can not build right now on Crusher or Perlmutter but I saw this on both.
Here is an example output using src/snes/tests/ex13.c using the appended .petscrc This uses 64 processors and the 8 processor case worked. This has been semi-nondertminisitc for me.
(and I have attached my current Perlmutter problem)
Hope this helps, Mark
-dm_plex_simplex 0 -dm_plex_dim 3 -dm_plex_box_lower 0,0,0 -dm_plex_box_upper 1,1,1 -petscpartitioner_simple_process_grid 2,2,2 -potential_petscspace_degree 2 -snes_max_it 1 -ksp_max_it 200 -ksp_type cg -ksp_rtol 1.e-12 -ksp_norm_type unpreconditioned -snes_rtol 1.e-8 #-pc_type gamg #-pc_gamg_type agg #-pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 100 -pc_gamg_process_eq_limit 400 -pc_gamg_reuse_interpolation true #-snes_monitor #-ksp_monitor_short -ksp_converged_reason #-ksp_view #-snes_converged_reason #-mg_levels_ksp_max_it 2 -mg_levels_ksp_type chebyshev #-mg_levels_ksp_type richardson #-mg_levels_ksp_richardson_scale 0.8 -mg_levels_pc_type jacobi -pc_gamg_esteig_ksp_type cg -pc_gamg_esteig_ksp_max_it 10 -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -dm_distribute -petscpartitioner_type simple -pc_gamg_repartition false -pc_gamg_coarse_grid_layout_type compact -pc_gamg_threshold 0.01 #-pc_gamg_threshold_scale .5 -pc_gamg_aggressive_coarsening 1 #-check_pointer_intensity 0 -snes_type ksponly #-mg_coarse_sub_pc_factor_mat_solver_type cusparse #-info :pc #-use_gpu_aware_mpi 1 -options_left #-malloc_debug -benchmark_it 10 #-pc_gamg_use_parallel_coarse_grid_solver #-mg_coarse_pc_type jacobi #-mg_coarse_ksp_type cg #-mg_coarse_ksp_rtol 1.e-2 #-mat_cusparse_transgen -snes_lag_jacobian -2
On Tue, Nov 15, 2022 at 3:42 PM Junchao Zhang <[email protected]> wrote:
Mark, Do you have a reproducer using petsc examples?
On Tue, Nov 15, 2022, 12:49 PM Mark Adams <[email protected]> wrote:
Junchao, this is the same problem that I have been having right?
On Tue, Nov 15, 2022 at 11:56 AM Fackler, Philip via petsc-users < [email protected]> wrote:
I built petsc with:
$ ./configure PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-debugging=0 --prefix=$HOME/build/petsc/debug/install --with-64-bit-indices --with-shared-libraries --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --download-kokkos --download-kokkos-kernels
$ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug all
$ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug install
Then I build xolotl in a separate build directory (after checking out the "feature-petsc-kokkos" branch) with:
$ cmake -DCMAKE_BUILD_TYPE=Debug -DKokkos_DIR=$HOME/build/petsc/debug/install -DPETSC_DIR=$HOME/build/petsc/debug/install <xolotl-src>
$ make -j4 SystemTester
Then, from the xolotl build directory, run (for example):
$ mpirun -n 2 ./test/system/SystemTester -t System/NE_4 -- -v
Note that this test case will use the parameter file '<xolotl-src>/benchmarks/params_system_NE_4.txt' which has the command-line arguments for petsc in its "petscArgs=..." line. If you look at '<xolotl-src>/test/system/SystemTester.cpp' all the system test cases follow the same naming convention with their corresponding parameter files under '<xolotl-src>/benchmarks'.
The failure happens with the NE_4 case (which is 2D) and the PSI_3 case (which is 1D).
Let me know if this is still unclear.
Thanks,
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory* ------------------------------ *From:* Junchao Zhang <[email protected]> *Sent:* Tuesday, November 15, 2022 00:16 *To:* Fackler, Philip <[email protected]> *Cc:* [email protected] <[email protected]>; Blondel, Sophie <[email protected]> *Subject:* [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases
Hi, Philip, Can you tell me instructions to build Xolotl to reproduce the error? --Junchao Zhang
On Mon, Nov 14, 2022 at 12:24 PM Fackler, Philip via petsc-users < [email protected]> wrote:
In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use the COO interface for preallocating and setting values in the Jacobian matrix. I have found that with some of our test cases, using more than one MPI rank results in a crash. Way down in the preconditioner code in petsc a Mat gets computed that has "null" for the "productsymbolic" member of its "ops". It's pretty far removed from where we compute the Jacobian entries, so I haven't been able (so far) to track it back to an error in my code. I'd appreciate some help with this from someone who is more familiar with the petsc guts so we can figure out what I'm doing wrong. (I'm assuming it's a bug in Xolotl.)
Note that this is using the kokkos backend for Mat and Vec in petsc, but with a serial-only build of kokkos and kokkos-kernels. So, it's a CPU-only multiple MPI rank run.
Here's a paste of the error output showing the relevant parts of the call stack:
[ERROR] [0]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [1]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [0]PETSC ERROR: [ERROR] No method productsymbolic for Mat of type (null) [ERROR] No method productsymbolic for Mat of type (null) [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] PetscSolver::solve: TSSolve failed. [ERROR] PetscSolver::solve: TSSolve failed. Aborting. Aborting.
Thanks for the help,
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory*
Hi, Mark, On Perlmutter, I have export MPICH_GPU_SUPPORT_ENABLED=1 module load cudatoolkit module load PrgEnv-gnu module load craype-accel-nvidia80 $ module list Currently Loaded Modules: 1) craype-x86-milan 4) perftools-base/22.06.0 7) xalt/2.10.2 10) Nsight-Systems/2022.2.1 13) cray-dsmml/0.2.2 16) PrgEnv-gnu/8.3.3 2) libfabric/1.15.0.0 5) xpmem/2.4.4-2.3_13.8__gff0e1d9.shasta 8) gpu/1.0 11) cudatoolkit/11.7 14) cray-mpich/8.1.17 17) craype-accel-nvidia80 3) craype-network-ofi 6) gcc/11.2.0 9) Nsight-Compute/2022.1.1 12) craype/2.7.16 15) cray-libsci/ 21.08.1.2 18) cmake/3.22.0 And my petsc configure is as follows and I can build petsc on it. As I knew, the only problem on perlmutter is KK failed to find TPL. Turning off TPL is a workaround. '--with-debugging', '--with-cc=cc', '--with-cxx=CC', '--with-fc=ftn', '--download-sowing-cc=cc', # cc might be nvc '--CFLAGS=-g -O0', '--FFLAGS=-g -O0', '--CXXFLAGS=-g -O0', '--with-cuda', '--with-cudac=nvcc', '--download-kokkos', '--download-kokkos-kernels', '--download-kokkos-commit=origin/develop', '--download-kokkos-kernels-commit=origin/develop', '--with-kokkos-kernels-tpl=0', --Junchao Zhang On Wed, Nov 16, 2022 at 7:05 AM Mark Adams <[email protected]> wrote:
I can not build right now on Crusher or Perlmutter but I saw this on both.
Here is an example output using src/snes/tests/ex13.c using the appended .petscrc This uses 64 processors and the 8 processor case worked. This has been semi-nondertminisitc for me.
(and I have attached my current Perlmutter problem)
Hope this helps, Mark
-dm_plex_simplex 0 -dm_plex_dim 3 -dm_plex_box_lower 0,0,0 -dm_plex_box_upper 1,1,1 -petscpartitioner_simple_process_grid 2,2,2 -potential_petscspace_degree 2 -snes_max_it 1 -ksp_max_it 200 -ksp_type cg -ksp_rtol 1.e-12 -ksp_norm_type unpreconditioned -snes_rtol 1.e-8 #-pc_type gamg #-pc_gamg_type agg #-pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 100 -pc_gamg_process_eq_limit 400 -pc_gamg_reuse_interpolation true #-snes_monitor #-ksp_monitor_short -ksp_converged_reason #-ksp_view #-snes_converged_reason #-mg_levels_ksp_max_it 2 -mg_levels_ksp_type chebyshev #-mg_levels_ksp_type richardson #-mg_levels_ksp_richardson_scale 0.8 -mg_levels_pc_type jacobi -pc_gamg_esteig_ksp_type cg -pc_gamg_esteig_ksp_max_it 10 -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05 -dm_distribute -petscpartitioner_type simple -pc_gamg_repartition false -pc_gamg_coarse_grid_layout_type compact -pc_gamg_threshold 0.01 #-pc_gamg_threshold_scale .5 -pc_gamg_aggressive_coarsening 1 #-check_pointer_intensity 0 -snes_type ksponly #-mg_coarse_sub_pc_factor_mat_solver_type cusparse #-info :pc #-use_gpu_aware_mpi 1 -options_left #-malloc_debug -benchmark_it 10 #-pc_gamg_use_parallel_coarse_grid_solver #-mg_coarse_pc_type jacobi #-mg_coarse_ksp_type cg #-mg_coarse_ksp_rtol 1.e-2 #-mat_cusparse_transgen -snes_lag_jacobian -2
On Tue, Nov 15, 2022 at 3:42 PM Junchao Zhang <[email protected]> wrote:
Mark, Do you have a reproducer using petsc examples?
On Tue, Nov 15, 2022, 12:49 PM Mark Adams <[email protected]> wrote:
Junchao, this is the same problem that I have been having right?
On Tue, Nov 15, 2022 at 11:56 AM Fackler, Philip via petsc-users < [email protected]> wrote:
I built petsc with:
$ ./configure PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-debugging=0 --prefix=$HOME/build/petsc/debug/install --with-64-bit-indices --with-shared-libraries --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --download-kokkos --download-kokkos-kernels
$ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug all
$ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug install
Then I build xolotl in a separate build directory (after checking out the "feature-petsc-kokkos" branch) with:
$ cmake -DCMAKE_BUILD_TYPE=Debug -DKokkos_DIR=$HOME/build/petsc/debug/install -DPETSC_DIR=$HOME/build/petsc/debug/install <xolotl-src>
$ make -j4 SystemTester
Then, from the xolotl build directory, run (for example):
$ mpirun -n 2 ./test/system/SystemTester -t System/NE_4 -- -v
Note that this test case will use the parameter file '<xolotl-src>/benchmarks/params_system_NE_4.txt' which has the command-line arguments for petsc in its "petscArgs=..." line. If you look at '<xolotl-src>/test/system/SystemTester.cpp' all the system test cases follow the same naming convention with their corresponding parameter files under '<xolotl-src>/benchmarks'.
The failure happens with the NE_4 case (which is 2D) and the PSI_3 case (which is 1D).
Let me know if this is still unclear.
Thanks,
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory* ------------------------------ *From:* Junchao Zhang <[email protected]> *Sent:* Tuesday, November 15, 2022 00:16 *To:* Fackler, Philip <[email protected]> *Cc:* [email protected] <[email protected]>; Blondel, Sophie <[email protected]> *Subject:* [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases
Hi, Philip, Can you tell me instructions to build Xolotl to reproduce the error? --Junchao Zhang
On Mon, Nov 14, 2022 at 12:24 PM Fackler, Philip via petsc-users < [email protected]> wrote:
In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use the COO interface for preallocating and setting values in the Jacobian matrix. I have found that with some of our test cases, using more than one MPI rank results in a crash. Way down in the preconditioner code in petsc a Mat gets computed that has "null" for the "productsymbolic" member of its "ops". It's pretty far removed from where we compute the Jacobian entries, so I haven't been able (so far) to track it back to an error in my code. I'd appreciate some help with this from someone who is more familiar with the petsc guts so we can figure out what I'm doing wrong. (I'm assuming it's a bug in Xolotl.)
Note that this is using the kokkos backend for Mat and Vec in petsc, but with a serial-only build of kokkos and kokkos-kernels. So, it's a CPU-only multiple MPI rank run.
Here's a paste of the error output showing the relevant parts of the call stack:
[ERROR] [0]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [1]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [0]PETSC ERROR: [ERROR] No method productsymbolic for Mat of type (null) [ERROR] No method productsymbolic for Mat of type (null) [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] PetscSolver::solve: TSSolve failed. [ERROR] PetscSolver::solve: TSSolve failed. Aborting. Aborting.
Thanks for the help,
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory*
On Tue, Nov 15, 2022 at 10:55 AM Fackler, Philip <[email protected]> wrote:
I built petsc with:
$ ./configure PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-debugging=0 --prefix=$HOME/build/petsc/debug/install --with-64-bit-indices --with-shared-libraries --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --download-kokkos --download-kokkos-kernels
$ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug all
$ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug install
Then I build xolotl in a separate build directory (after checking out the "feature-petsc-kokkos" branch) with:
$ cmake -DCMAKE_BUILD_TYPE=Debug -DKokkos_DIR=$HOME/build/petsc/debug/install -DPETSC_DIR=$HOME/build/petsc/debug/install <xolotl-src>
$ make -j4 SystemTester
Hi, Philip, I tried multiple times and still failed at building xolotl. I installed boost-1.74 and HDF5, and used gcc-11.3. make -j4 SystemTester ... [ 9%] Building CXX object xolotl/core/CMakeFiles/xolotlCore.dir/src/diffusion/DiffusionHandler.cpp.o /home/jczhang/xolotl/xolotl/core/src/diffusion/DiffusionHandler.cpp(55): error: no instance of overloaded function "std::vector<_Tp, _Alloc>::push_back [with _Tp=xolotl::core::RowColPair, _Alloc=std::allocator<xolotl::core::RowColPair>]" matches the argument list argument types are: ({...}) object type is: std::vector<xolotl::core::RowColPair, std::allocator<xolotl::core::RowColPair>> 1 error detected in the compilation of "/home/jczhang/xolotl/xolotl/core/src/diffusion/DiffusionHandler.cpp".
Then, from the xolotl build directory, run (for example):
$ mpirun -n 2 ./test/system/SystemTester -t System/NE_4 -- -v
Note that this test case will use the parameter file '<xolotl-src>/benchmarks/params_system_NE_4.txt' which has the command-line arguments for petsc in its "petscArgs=..." line. If you look at '<xolotl-src>/test/system/SystemTester.cpp' all the system test cases follow the same naming convention with their corresponding parameter files under '<xolotl-src>/benchmarks'.
The failure happens with the NE_4 case (which is 2D) and the PSI_3 case (which is 1D).
Let me know if this is still unclear.
Thanks,
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory* ------------------------------ *From:* Junchao Zhang <[email protected]> *Sent:* Tuesday, November 15, 2022 00:16 *To:* Fackler, Philip <[email protected]> *Cc:* [email protected] <[email protected]>; Blondel, Sophie < [email protected]> *Subject:* [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases
Hi, Philip, Can you tell me instructions to build Xolotl to reproduce the error? --Junchao Zhang
On Mon, Nov 14, 2022 at 12:24 PM Fackler, Philip via petsc-users < [email protected]> wrote:
In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use the COO interface for preallocating and setting values in the Jacobian matrix. I have found that with some of our test cases, using more than one MPI rank results in a crash. Way down in the preconditioner code in petsc a Mat gets computed that has "null" for the "productsymbolic" member of its "ops". It's pretty far removed from where we compute the Jacobian entries, so I haven't been able (so far) to track it back to an error in my code. I'd appreciate some help with this from someone who is more familiar with the petsc guts so we can figure out what I'm doing wrong. (I'm assuming it's a bug in Xolotl.)
Note that this is using the kokkos backend for Mat and Vec in petsc, but with a serial-only build of kokkos and kokkos-kernels. So, it's a CPU-only multiple MPI rank run.
Here's a paste of the error output showing the relevant parts of the call stack:
[ERROR] [0]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [1]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [0]PETSC ERROR: [ERROR] No method productsymbolic for Mat of type (null) [ERROR] No method productsymbolic for Mat of type (null) [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] PetscSolver::solve: TSSolve failed. [ERROR] PetscSolver::solve: TSSolve failed. Aborting. Aborting.
Thanks for the help,
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory*
Not sure why. I'm using the same compiler. But you can try constructing the object explicitly on that line: idPairs.push_back(core::RowColPair{i, i}); Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory ________________________________ From: Junchao Zhang <[email protected]> Sent: Sunday, November 20, 2022 13:25 To: Fackler, Philip <[email protected]> Cc: [email protected] <[email protected]>; Blondel, Sophie <[email protected]> Subject: Re: [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases On Tue, Nov 15, 2022 at 10:55 AM Fackler, Philip <[email protected]<mailto:[email protected]>> wrote: I built petsc with: $ ./configure PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-debugging=0 --prefix=$HOME/build/petsc/debug/install --with-64-bit-indices --with-shared-libraries --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --download-kokkos --download-kokkos-kernels $ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug all $ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug install Then I build xolotl in a separate build directory (after checking out the "feature-petsc-kokkos" branch) with: $ cmake -DCMAKE_BUILD_TYPE=Debug -DKokkos_DIR=$HOME/build/petsc/debug/install -DPETSC_DIR=$HOME/build/petsc/debug/install <xolotl-src> $ make -j4 SystemTester Hi, Philip, I tried multiple times and still failed at building xolotl. I installed boost-1.74 and HDF5, and used gcc-11.3. make -j4 SystemTester ... [ 9%] Building CXX object xolotl/core/CMakeFiles/xolotlCore.dir/src/diffusion/DiffusionHandler.cpp.o /home/jczhang/xolotl/xolotl/core/src/diffusion/DiffusionHandler.cpp(55): error: no instance of overloaded function "std::vector<_Tp, _Alloc>::push_back [with _Tp=xolotl::core::RowColPair, _Alloc=std::allocator<xolotl::core::RowColPair>]" matches the argument list argument types are: ({...}) object type is: std::vector<xolotl::core::RowColPair, std::allocator<xolotl::core::RowColPair>> 1 error detected in the compilation of "/home/jczhang/xolotl/xolotl/core/src/diffusion/DiffusionHandler.cpp". Then, from the xolotl build directory, run (for example): $ mpirun -n 2 ./test/system/SystemTester -t System/NE_4 -- -v Note that this test case will use the parameter file '<xolotl-src>/benchmarks/params_system_NE_4.txt' which has the command-line arguments for petsc in its "petscArgs=..." line. If you look at '<xolotl-src>/test/system/SystemTester.cpp' all the system test cases follow the same naming convention with their corresponding parameter files under '<xolotl-src>/benchmarks'. The failure happens with the NE_4 case (which is 2D) and the PSI_3 case (which is 1D). Let me know if this is still unclear. Thanks, Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory ________________________________ From: Junchao Zhang <[email protected]<mailto:[email protected]>> Sent: Tuesday, November 15, 2022 00:16 To: Fackler, Philip <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>>; Blondel, Sophie <[email protected]<mailto:[email protected]>> Subject: [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases Hi, Philip, Can you tell me instructions to build Xolotl to reproduce the error? --Junchao Zhang On Mon, Nov 14, 2022 at 12:24 PM Fackler, Philip via petsc-users <[email protected]<mailto:[email protected]>> wrote: In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use the COO interface for preallocating and setting values in the Jacobian matrix. I have found that with some of our test cases, using more than one MPI rank results in a crash. Way down in the preconditioner code in petsc a Mat gets computed that has "null" for the "productsymbolic" member of its "ops". It's pretty far removed from where we compute the Jacobian entries, so I haven't been able (so far) to track it back to an error in my code. I'd appreciate some help with this from someone who is more familiar with the petsc guts so we can figure out what I'm doing wrong. (I'm assuming it's a bug in Xolotl.) Note that this is using the kokkos backend for Mat and Vec in petsc, but with a serial-only build of kokkos and kokkos-kernels. So, it's a CPU-only multiple MPI rank run. Here's a paste of the error output showing the relevant parts of the call stack: [ERROR] [0]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [1]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [0]PETSC ERROR: [ERROR] No method productsymbolic for Mat of type (null) [ERROR] No method productsymbolic for Mat of type (null) [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] PetscSolver::solve: TSSolve failed. [ERROR] PetscSolver::solve: TSSolve failed. Aborting. Aborting. Thanks for the help, Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory
On Mon, Nov 21, 2022 at 9:31 AM Fackler, Philip <[email protected]> wrote:
Not sure why. I'm using the same compiler. But you can try constructing the object explicitly on that line:
idPairs.push_back(core::RowColPair{i, i});
WIth your change, I continued but met another error: /home/jczhang/xolotl/test/core/diffusion/Diffusion2DHandlerTester.cpp(79): error: class "xolotl::core::diffusion::Diffusion2DHandler" has no member "initializeOFill" it seems all these problems are related to the branch * feature-petsc-kokkos, *instead of the compiler etc. When I switched to origin/stable, I could build xolotl.
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory* ------------------------------ *From:* Junchao Zhang <[email protected]> *Sent:* Sunday, November 20, 2022 13:25 *To:* Fackler, Philip <[email protected]> *Cc:* [email protected] <[email protected]>; Blondel, Sophie < [email protected]> *Subject:* Re: [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases
On Tue, Nov 15, 2022 at 10:55 AM Fackler, Philip <[email protected]> wrote:
I built petsc with:
$ ./configure PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-debugging=0 --prefix=$HOME/build/petsc/debug/install --with-64-bit-indices --with-shared-libraries --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --download-kokkos --download-kokkos-kernels
$ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug all
$ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug install
Then I build xolotl in a separate build directory (after checking out the "feature-petsc-kokkos" branch) with:
$ cmake -DCMAKE_BUILD_TYPE=Debug -DKokkos_DIR=$HOME/build/petsc/debug/install -DPETSC_DIR=$HOME/build/petsc/debug/install <xolotl-src>
$ make -j4 SystemTester
Hi, Philip, I tried multiple times and still failed at building xolotl. I installed boost-1.74 and HDF5, and used gcc-11.3.
make -j4 SystemTester ... [ 9%] Building CXX object xolotl/core/CMakeFiles/xolotlCore.dir/src/diffusion/DiffusionHandler.cpp.o /home/jczhang/xolotl/xolotl/core/src/diffusion/DiffusionHandler.cpp(55): error: no instance of overloaded function "std::vector<_Tp, _Alloc>::push_back [with _Tp=xolotl::core::RowColPair, _Alloc=std::allocator<xolotl::core::RowColPair>]" matches the argument list argument types are: ({...}) object type is: std::vector<xolotl::core::RowColPair, std::allocator<xolotl::core::RowColPair>>
1 error detected in the compilation of "/home/jczhang/xolotl/xolotl/core/src/diffusion/DiffusionHandler.cpp".
Then, from the xolotl build directory, run (for example):
$ mpirun -n 2 ./test/system/SystemTester -t System/NE_4 -- -v
Note that this test case will use the parameter file '<xolotl-src>/benchmarks/params_system_NE_4.txt' which has the command-line arguments for petsc in its "petscArgs=..." line. If you look at '<xolotl-src>/test/system/SystemTester.cpp' all the system test cases follow the same naming convention with their corresponding parameter files under '<xolotl-src>/benchmarks'.
The failure happens with the NE_4 case (which is 2D) and the PSI_3 case (which is 1D).
Let me know if this is still unclear.
Thanks,
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory* ------------------------------ *From:* Junchao Zhang <[email protected]> *Sent:* Tuesday, November 15, 2022 00:16 *To:* Fackler, Philip <[email protected]> *Cc:* [email protected] <[email protected]>; Blondel, Sophie < [email protected]> *Subject:* [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases
Hi, Philip, Can you tell me instructions to build Xolotl to reproduce the error? --Junchao Zhang
On Mon, Nov 14, 2022 at 12:24 PM Fackler, Philip via petsc-users < [email protected]> wrote:
In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use the COO interface for preallocating and setting values in the Jacobian matrix. I have found that with some of our test cases, using more than one MPI rank results in a crash. Way down in the preconditioner code in petsc a Mat gets computed that has "null" for the "productsymbolic" member of its "ops". It's pretty far removed from where we compute the Jacobian entries, so I haven't been able (so far) to track it back to an error in my code. I'd appreciate some help with this from someone who is more familiar with the petsc guts so we can figure out what I'm doing wrong. (I'm assuming it's a bug in Xolotl.)
Note that this is using the kokkos backend for Mat and Vec in petsc, but with a serial-only build of kokkos and kokkos-kernels. So, it's a CPU-only multiple MPI rank run.
Here's a paste of the error output showing the relevant parts of the call stack:
[ERROR] [0]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [1]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [0]PETSC ERROR: [ERROR] No method productsymbolic for Mat of type (null) [ERROR] No method productsymbolic for Mat of type (null) [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] PetscSolver::solve: TSSolve failed. [ERROR] PetscSolver::solve: TSSolve failed. Aborting. Aborting.
Thanks for the help,
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory*
Yes, that one is. I haven't updated the tests. So just build the SystemTester? target or the xolotl? target. Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory ________________________________ From: Junchao Zhang <[email protected]> Sent: Monday, November 21, 2022 15:36 To: Fackler, Philip <[email protected]> Cc: [email protected] <[email protected]>; Blondel, Sophie <[email protected]> Subject: Re: [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases On Mon, Nov 21, 2022 at 9:31 AM Fackler, Philip <[email protected]<mailto:[email protected]>> wrote: Not sure why. I'm using the same compiler. But you can try constructing the object explicitly on that line: idPairs.push_back(core::RowColPair{i, i}); WIth your change, I continued but met another error: /home/jczhang/xolotl/test/core/diffusion/Diffusion2DHandlerTester.cpp(79): error: class "xolotl::core::diffusion::Diffusion2DHandler" has no member "initializeOFill" it seems all these problems are related to the branch feature-petsc-kokkos, instead of the compiler etc. When I switched to origin/stable, I could build xolotl. Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory ________________________________ From: Junchao Zhang <[email protected]<mailto:[email protected]>> Sent: Sunday, November 20, 2022 13:25 To: Fackler, Philip <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>>; Blondel, Sophie <[email protected]<mailto:[email protected]>> Subject: Re: [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases On Tue, Nov 15, 2022 at 10:55 AM Fackler, Philip <[email protected]<mailto:[email protected]>> wrote: I built petsc with: $ ./configure PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-debugging=0 --prefix=$HOME/build/petsc/debug/install --with-64-bit-indices --with-shared-libraries --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --download-kokkos --download-kokkos-kernels $ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug all $ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug install Then I build xolotl in a separate build directory (after checking out the "feature-petsc-kokkos" branch) with: $ cmake -DCMAKE_BUILD_TYPE=Debug -DKokkos_DIR=$HOME/build/petsc/debug/install -DPETSC_DIR=$HOME/build/petsc/debug/install <xolotl-src> $ make -j4 SystemTester Hi, Philip, I tried multiple times and still failed at building xolotl. I installed boost-1.74 and HDF5, and used gcc-11.3. make -j4 SystemTester ... [ 9%] Building CXX object xolotl/core/CMakeFiles/xolotlCore.dir/src/diffusion/DiffusionHandler.cpp.o /home/jczhang/xolotl/xolotl/core/src/diffusion/DiffusionHandler.cpp(55): error: no instance of overloaded function "std::vector<_Tp, _Alloc>::push_back [with _Tp=xolotl::core::RowColPair, _Alloc=std::allocator<xolotl::core::RowColPair>]" matches the argument list argument types are: ({...}) object type is: std::vector<xolotl::core::RowColPair, std::allocator<xolotl::core::RowColPair>> 1 error detected in the compilation of "/home/jczhang/xolotl/xolotl/core/src/diffusion/DiffusionHandler.cpp". Then, from the xolotl build directory, run (for example): $ mpirun -n 2 ./test/system/SystemTester -t System/NE_4 -- -v Note that this test case will use the parameter file '<xolotl-src>/benchmarks/params_system_NE_4.txt' which has the command-line arguments for petsc in its "petscArgs=..." line. If you look at '<xolotl-src>/test/system/SystemTester.cpp' all the system test cases follow the same naming convention with their corresponding parameter files under '<xolotl-src>/benchmarks'. The failure happens with the NE_4 case (which is 2D) and the PSI_3 case (which is 1D). Let me know if this is still unclear. Thanks, Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory ________________________________ From: Junchao Zhang <[email protected]<mailto:[email protected]>> Sent: Tuesday, November 15, 2022 00:16 To: Fackler, Philip <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>>; Blondel, Sophie <[email protected]<mailto:[email protected]>> Subject: [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases Hi, Philip, Can you tell me instructions to build Xolotl to reproduce the error? --Junchao Zhang On Mon, Nov 14, 2022 at 12:24 PM Fackler, Philip via petsc-users <[email protected]<mailto:[email protected]>> wrote: In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use the COO interface for preallocating and setting values in the Jacobian matrix. I have found that with some of our test cases, using more than one MPI rank results in a crash. Way down in the preconditioner code in petsc a Mat gets computed that has "null" for the "productsymbolic" member of its "ops". It's pretty far removed from where we compute the Jacobian entries, so I haven't been able (so far) to track it back to an error in my code. I'd appreciate some help with this from someone who is more familiar with the petsc guts so we can figure out what I'm doing wrong. (I'm assuming it's a bug in Xolotl.) Note that this is using the kokkos backend for Mat and Vec in petsc, but with a serial-only build of kokkos and kokkos-kernels. So, it's a CPU-only multiple MPI rank run. Here's a paste of the error output showing the relevant parts of the call stack: [ERROR] [0]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [1]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [0]PETSC ERROR: [ERROR] No method productsymbolic for Mat of type (null) [ERROR] No method productsymbolic for Mat of type (null) [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] PetscSolver::solve: TSSolve failed. [ERROR] PetscSolver::solve: TSSolve failed. Aborting. Aborting. Thanks for the help, Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory
On Tue, Nov 22, 2022 at 10:14 AM Fackler, Philip <[email protected]> wrote:
Yes, that one is. I haven't updated the tests. So just build the SystemTester target or the xolotl target.
OK, I see. I reproduced the petsc error and am looking into it. Thanks a lot.
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory* ------------------------------ *From:* Junchao Zhang <[email protected]> *Sent:* Monday, November 21, 2022 15:36 *To:* Fackler, Philip <[email protected]> *Cc:* [email protected] <[email protected]>; Blondel, Sophie < [email protected]> *Subject:* Re: [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases
On Mon, Nov 21, 2022 at 9:31 AM Fackler, Philip <[email protected]> wrote:
Not sure why. I'm using the same compiler. But you can try constructing the object explicitly on that line:
idPairs.push_back(core::RowColPair{i, i});
WIth your change, I continued but met another error:
/home/jczhang/xolotl/test/core/diffusion/Diffusion2DHandlerTester.cpp(79): error: class "xolotl::core::diffusion::Diffusion2DHandler" has no member "initializeOFill"
it seems all these problems are related to the branch * feature-petsc-kokkos, *instead of the compiler etc. When I switched to origin/stable, I could build xolotl.
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory* ------------------------------ *From:* Junchao Zhang <[email protected]> *Sent:* Sunday, November 20, 2022 13:25 *To:* Fackler, Philip <[email protected]> *Cc:* [email protected] <[email protected]>; Blondel, Sophie < [email protected]> *Subject:* Re: [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases
On Tue, Nov 15, 2022 at 10:55 AM Fackler, Philip <[email protected]> wrote:
I built petsc with:
$ ./configure PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-debugging=0 --prefix=$HOME/build/petsc/debug/install --with-64-bit-indices --with-shared-libraries --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --download-kokkos --download-kokkos-kernels
$ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug all
$ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug install
Then I build xolotl in a separate build directory (after checking out the "feature-petsc-kokkos" branch) with:
$ cmake -DCMAKE_BUILD_TYPE=Debug -DKokkos_DIR=$HOME/build/petsc/debug/install -DPETSC_DIR=$HOME/build/petsc/debug/install <xolotl-src>
$ make -j4 SystemTester
Hi, Philip, I tried multiple times and still failed at building xolotl. I installed boost-1.74 and HDF5, and used gcc-11.3.
make -j4 SystemTester ... [ 9%] Building CXX object xolotl/core/CMakeFiles/xolotlCore.dir/src/diffusion/DiffusionHandler.cpp.o /home/jczhang/xolotl/xolotl/core/src/diffusion/DiffusionHandler.cpp(55): error: no instance of overloaded function "std::vector<_Tp, _Alloc>::push_back [with _Tp=xolotl::core::RowColPair, _Alloc=std::allocator<xolotl::core::RowColPair>]" matches the argument list argument types are: ({...}) object type is: std::vector<xolotl::core::RowColPair, std::allocator<xolotl::core::RowColPair>>
1 error detected in the compilation of "/home/jczhang/xolotl/xolotl/core/src/diffusion/DiffusionHandler.cpp".
Then, from the xolotl build directory, run (for example):
$ mpirun -n 2 ./test/system/SystemTester -t System/NE_4 -- -v
Note that this test case will use the parameter file '<xolotl-src>/benchmarks/params_system_NE_4.txt' which has the command-line arguments for petsc in its "petscArgs=..." line. If you look at '<xolotl-src>/test/system/SystemTester.cpp' all the system test cases follow the same naming convention with their corresponding parameter files under '<xolotl-src>/benchmarks'.
The failure happens with the NE_4 case (which is 2D) and the PSI_3 case (which is 1D).
Let me know if this is still unclear.
Thanks,
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory* ------------------------------ *From:* Junchao Zhang <[email protected]> *Sent:* Tuesday, November 15, 2022 00:16 *To:* Fackler, Philip <[email protected]> *Cc:* [email protected] <[email protected]>; Blondel, Sophie < [email protected]> *Subject:* [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases
Hi, Philip, Can you tell me instructions to build Xolotl to reproduce the error? --Junchao Zhang
On Mon, Nov 14, 2022 at 12:24 PM Fackler, Philip via petsc-users < [email protected]> wrote:
In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use the COO interface for preallocating and setting values in the Jacobian matrix. I have found that with some of our test cases, using more than one MPI rank results in a crash. Way down in the preconditioner code in petsc a Mat gets computed that has "null" for the "productsymbolic" member of its "ops". It's pretty far removed from where we compute the Jacobian entries, so I haven't been able (so far) to track it back to an error in my code. I'd appreciate some help with this from someone who is more familiar with the petsc guts so we can figure out what I'm doing wrong. (I'm assuming it's a bug in Xolotl.)
Note that this is using the kokkos backend for Mat and Vec in petsc, but with a serial-only build of kokkos and kokkos-kernels. So, it's a CPU-only multiple MPI rank run.
Here's a paste of the error output showing the relevant parts of the call stack:
[ERROR] [0]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [1]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [0]PETSC ERROR: [ERROR] No method productsymbolic for Mat of type (null) [ERROR] No method productsymbolic for Mat of type (null) [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] PetscSolver::solve: TSSolve failed. [ERROR] PetscSolver::solve: TSSolve failed. Aborting. Aborting.
Thanks for the help,
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory*
Great! Thank you! Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory ________________________________ From: Junchao Zhang <[email protected]> Sent: Tuesday, November 22, 2022 12:02 To: Fackler, Philip <[email protected]> Cc: [email protected] <[email protected]>; Blondel, Sophie <[email protected]> Subject: Re: [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases On Tue, Nov 22, 2022 at 10:14 AM Fackler, Philip <[email protected]<mailto:[email protected]>> wrote: Yes, that one is. I haven't updated the tests. So just build the SystemTester target or the xolotl target. OK, I see. I reproduced the petsc error and am looking into it. Thanks a lot. Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory ________________________________ From: Junchao Zhang <[email protected]<mailto:[email protected]>> Sent: Monday, November 21, 2022 15:36 To: Fackler, Philip <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>>; Blondel, Sophie <[email protected]<mailto:[email protected]>> Subject: Re: [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases On Mon, Nov 21, 2022 at 9:31 AM Fackler, Philip <[email protected]<mailto:[email protected]>> wrote: Not sure why. I'm using the same compiler. But you can try constructing the object explicitly on that line: idPairs.push_back(core::RowColPair{i, i}); WIth your change, I continued but met another error: /home/jczhang/xolotl/test/core/diffusion/Diffusion2DHandlerTester.cpp(79): error: class "xolotl::core::diffusion::Diffusion2DHandler" has no member "initializeOFill" it seems all these problems are related to the branch feature-petsc-kokkos, instead of the compiler etc. When I switched to origin/stable, I could build xolotl. Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory ________________________________ From: Junchao Zhang <[email protected]<mailto:[email protected]>> Sent: Sunday, November 20, 2022 13:25 To: Fackler, Philip <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>>; Blondel, Sophie <[email protected]<mailto:[email protected]>> Subject: Re: [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases On Tue, Nov 15, 2022 at 10:55 AM Fackler, Philip <[email protected]<mailto:[email protected]>> wrote: I built petsc with: $ ./configure PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-debugging=0 --prefix=$HOME/build/petsc/debug/install --with-64-bit-indices --with-shared-libraries --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --download-kokkos --download-kokkos-kernels $ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug all $ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug install Then I build xolotl in a separate build directory (after checking out the "feature-petsc-kokkos" branch) with: $ cmake -DCMAKE_BUILD_TYPE=Debug -DKokkos_DIR=$HOME/build/petsc/debug/install -DPETSC_DIR=$HOME/build/petsc/debug/install <xolotl-src> $ make -j4 SystemTester Hi, Philip, I tried multiple times and still failed at building xolotl. I installed boost-1.74 and HDF5, and used gcc-11.3. make -j4 SystemTester ... [ 9%] Building CXX object xolotl/core/CMakeFiles/xolotlCore.dir/src/diffusion/DiffusionHandler.cpp.o /home/jczhang/xolotl/xolotl/core/src/diffusion/DiffusionHandler.cpp(55): error: no instance of overloaded function "std::vector<_Tp, _Alloc>::push_back [with _Tp=xolotl::core::RowColPair, _Alloc=std::allocator<xolotl::core::RowColPair>]" matches the argument list argument types are: ({...}) object type is: std::vector<xolotl::core::RowColPair, std::allocator<xolotl::core::RowColPair>> 1 error detected in the compilation of "/home/jczhang/xolotl/xolotl/core/src/diffusion/DiffusionHandler.cpp". Then, from the xolotl build directory, run (for example): $ mpirun -n 2 ./test/system/SystemTester -t System/NE_4 -- -v Note that this test case will use the parameter file '<xolotl-src>/benchmarks/params_system_NE_4.txt' which has the command-line arguments for petsc in its "petscArgs=..." line. If you look at '<xolotl-src>/test/system/SystemTester.cpp' all the system test cases follow the same naming convention with their corresponding parameter files under '<xolotl-src>/benchmarks'. The failure happens with the NE_4 case (which is 2D) and the PSI_3 case (which is 1D). Let me know if this is still unclear. Thanks, Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory ________________________________ From: Junchao Zhang <[email protected]<mailto:[email protected]>> Sent: Tuesday, November 15, 2022 00:16 To: Fackler, Philip <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>>; Blondel, Sophie <[email protected]<mailto:[email protected]>> Subject: [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases Hi, Philip, Can you tell me instructions to build Xolotl to reproduce the error? --Junchao Zhang On Mon, Nov 14, 2022 at 12:24 PM Fackler, Philip via petsc-users <[email protected]<mailto:[email protected]>> wrote: In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use the COO interface for preallocating and setting values in the Jacobian matrix. I have found that with some of our test cases, using more than one MPI rank results in a crash. Way down in the preconditioner code in petsc a Mat gets computed that has "null" for the "productsymbolic" member of its "ops". It's pretty far removed from where we compute the Jacobian entries, so I haven't been able (so far) to track it back to an error in my code. I'd appreciate some help with this from someone who is more familiar with the petsc guts so we can figure out what I'm doing wrong. (I'm assuming it's a bug in Xolotl.) Note that this is using the kokkos backend for Mat and Vec in petsc, but with a serial-only build of kokkos and kokkos-kernels. So, it's a CPU-only multiple MPI rank run. Here's a paste of the error output showing the relevant parts of the call stack: [ERROR] [0]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [1]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [0]PETSC ERROR: [ERROR] No method productsymbolic for Mat of type (null) [ERROR] No method productsymbolic for Mat of type (null) [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] PetscSolver::solve: TSSolve failed. [ERROR] PetscSolver::solve: TSSolve failed. Aborting. Aborting. Thanks for the help, Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory
Hi, Philip, The petsc bug is fixed in https://gitlab.com/petsc/petsc/-/merge_requests/5892, which is now in petsc/release and will be merged to petsc/main Thanks. --Junchao Zhang On Tue, Nov 22, 2022 at 11:56 AM Fackler, Philip <[email protected]> wrote:
Great! Thank you!
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory* ------------------------------ *From:* Junchao Zhang <[email protected]> *Sent:* Tuesday, November 22, 2022 12:02 *To:* Fackler, Philip <[email protected]> *Cc:* [email protected] <[email protected]>; Blondel, Sophie < [email protected]> *Subject:* Re: [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases
On Tue, Nov 22, 2022 at 10:14 AM Fackler, Philip <[email protected]> wrote:
Yes, that one is. I haven't updated the tests. So just build the SystemTester target or the xolotl target.
OK, I see. I reproduced the petsc error and am looking into it. Thanks a lot.
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory* ------------------------------ *From:* Junchao Zhang <[email protected]> *Sent:* Monday, November 21, 2022 15:36 *To:* Fackler, Philip <[email protected]> *Cc:* [email protected] <[email protected]>; Blondel, Sophie < [email protected]> *Subject:* Re: [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases
On Mon, Nov 21, 2022 at 9:31 AM Fackler, Philip <[email protected]> wrote:
Not sure why. I'm using the same compiler. But you can try constructing the object explicitly on that line:
idPairs.push_back(core::RowColPair{i, i});
WIth your change, I continued but met another error:
/home/jczhang/xolotl/test/core/diffusion/Diffusion2DHandlerTester.cpp(79): error: class "xolotl::core::diffusion::Diffusion2DHandler" has no member "initializeOFill"
it seems all these problems are related to the branch * feature-petsc-kokkos, *instead of the compiler etc. When I switched to origin/stable, I could build xolotl.
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory* ------------------------------ *From:* Junchao Zhang <[email protected]> *Sent:* Sunday, November 20, 2022 13:25 *To:* Fackler, Philip <[email protected]> *Cc:* [email protected] <[email protected]>; Blondel, Sophie < [email protected]> *Subject:* Re: [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases
On Tue, Nov 15, 2022 at 10:55 AM Fackler, Philip <[email protected]> wrote:
I built petsc with:
$ ./configure PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-debugging=0 --prefix=$HOME/build/petsc/debug/install --with-64-bit-indices --with-shared-libraries --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --download-kokkos --download-kokkos-kernels
$ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug all
$ make PETSC_DIR=$PWD PETSC_ARCH=arch-kokkos-serial-debug install
Then I build xolotl in a separate build directory (after checking out the "feature-petsc-kokkos" branch) with:
$ cmake -DCMAKE_BUILD_TYPE=Debug -DKokkos_DIR=$HOME/build/petsc/debug/install -DPETSC_DIR=$HOME/build/petsc/debug/install <xolotl-src>
$ make -j4 SystemTester
Hi, Philip, I tried multiple times and still failed at building xolotl. I installed boost-1.74 and HDF5, and used gcc-11.3.
make -j4 SystemTester ... [ 9%] Building CXX object xolotl/core/CMakeFiles/xolotlCore.dir/src/diffusion/DiffusionHandler.cpp.o /home/jczhang/xolotl/xolotl/core/src/diffusion/DiffusionHandler.cpp(55): error: no instance of overloaded function "std::vector<_Tp, _Alloc>::push_back [with _Tp=xolotl::core::RowColPair, _Alloc=std::allocator<xolotl::core::RowColPair>]" matches the argument list argument types are: ({...}) object type is: std::vector<xolotl::core::RowColPair, std::allocator<xolotl::core::RowColPair>>
1 error detected in the compilation of "/home/jczhang/xolotl/xolotl/core/src/diffusion/DiffusionHandler.cpp".
Then, from the xolotl build directory, run (for example):
$ mpirun -n 2 ./test/system/SystemTester -t System/NE_4 -- -v
Note that this test case will use the parameter file '<xolotl-src>/benchmarks/params_system_NE_4.txt' which has the command-line arguments for petsc in its "petscArgs=..." line. If you look at '<xolotl-src>/test/system/SystemTester.cpp' all the system test cases follow the same naming convention with their corresponding parameter files under '<xolotl-src>/benchmarks'.
The failure happens with the NE_4 case (which is 2D) and the PSI_3 case (which is 1D).
Let me know if this is still unclear.
Thanks,
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory* ------------------------------ *From:* Junchao Zhang <[email protected]> *Sent:* Tuesday, November 15, 2022 00:16 *To:* Fackler, Philip <[email protected]> *Cc:* [email protected] <[email protected]>; Blondel, Sophie < [email protected]> *Subject:* [EXTERNAL] Re: [petsc-users] Using multiple MPI ranks with COO interface crashes in some cases
Hi, Philip, Can you tell me instructions to build Xolotl to reproduce the error? --Junchao Zhang
On Mon, Nov 14, 2022 at 12:24 PM Fackler, Philip via petsc-users < [email protected]> wrote:
In Xolotl's "feature-petsc-kokkos" branch, I have moved our code to use the COO interface for preallocating and setting values in the Jacobian matrix. I have found that with some of our test cases, using more than one MPI rank results in a crash. Way down in the preconditioner code in petsc a Mat gets computed that has "null" for the "productsymbolic" member of its "ops". It's pretty far removed from where we compute the Jacobian entries, so I haven't been able (so far) to track it back to an error in my code. I'd appreciate some help with this from someone who is more familiar with the petsc guts so we can figure out what I'm doing wrong. (I'm assuming it's a bug in Xolotl.)
Note that this is using the kokkos backend for Mat and Vec in petsc, but with a serial-only build of kokkos and kokkos-kernels. So, it's a CPU-only multiple MPI rank run.
Here's a paste of the error output showing the relevant parts of the call stack:
[ERROR] [0]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] --------------------- Error Message -------------------------------------------------------------- [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [1]PETSC ERROR: [ERROR] No support for this operation for this object type [ERROR] [0]PETSC ERROR: [ERROR] No method productsymbolic for Mat of type (null) [ERROR] No method productsymbolic for Mat of type (null) [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] See hxxps://petsc.org/release/faq/ for trouble shooting. [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] Petsc Development GIT revision: v3.18.1-115-gdca010e0e9a GIT Date: 2022-10-28 14:39:41 +0000 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] Unknown Name on a named PC0115427 by 4pf Mon Nov 14 13:22:01 2022 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] Configure options PETSC_DIR=/home/4pf/repos/petsc PETSC_ARCH=arch-kokkos-serial-debug --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --with-cudac=0 --prefix=/home/4pf/build/petsc/serial-debug/install --with-64-bit-indices --with-shared-libraries --with-kokkos-dir=/home/4pf/build/kokkos/serial/install --with-kokkos-kernels-dir=/home/4pf/build/kokkos-kernels/serial/install [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] #1 MatProductSymbolic_MPIAIJKokkos_AB() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:918 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] #2 MatProductSymbolic_MPIAIJKokkos() at /home/4pf/repos/petsc/src/mat/impls/aij/mpi/kokkos/mpiaijkok.kokkos.cxx:1138 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] #3 MatProductSymbolic() at /home/4pf/repos/petsc/src/mat/interface/matproduct.c:793 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] #4 MatProduct_Private() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9820 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] #5 MatMatMult() at /home/4pf/repos/petsc/src/mat/interface/matrix.c:9897 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] #6 PCGAMGOptProlongator_AGG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/agg.c:769 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] #7 PCSetUp_GAMG() at /home/4pf/repos/petsc/src/ksp/pc/impls/gamg/gamg.c:639 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] #8 PCSetUp() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:994 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] #9 KSPSetUp() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:406 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] #10 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:825 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #11 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] #12 PCApply_FieldSplit() at /home/4pf/repos/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:1246 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] #13 PCApply() at /home/4pf/repos/petsc/src/ksp/pc/interface/precon.c:441 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] #14 KSP_PCApply() at /home/4pf/repos/petsc/include/petsc/private/kspimpl.h:380 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] #15 KSPFGMRESCycle() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:152 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] #16 KSPSolve_FGMRES() at /home/4pf/repos/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c:273 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] #17 KSPSolve_Private() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:899 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] #18 KSPSolve() at /home/4pf/repos/petsc/src/ksp/ksp/interface/itfunc.c:1071 [ERROR] [0]PETSC ERROR: [ERROR] [1]PETSC ERROR: [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] #19 SNESSolve_NEWTONLS() at /home/4pf/repos/petsc/src/snes/impls/ls/ls.c:210 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] #20 SNESSolve() at /home/4pf/repos/petsc/src/snes/interface/snes.c:4689 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] #21 TSStep_ARKIMEX() at /home/4pf/repos/petsc/src/ts/impls/arkimex/arkimex.c:791 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] #22 TSStep() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3445 [ERROR] [1]PETSC ERROR: [ERROR] [0]PETSC ERROR: [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] #23 TSSolve() at /home/4pf/repos/petsc/src/ts/interface/ts.c:3836 [ERROR] PetscSolver::solve: TSSolve failed. [ERROR] PetscSolver::solve: TSSolve failed. Aborting. Aborting.
Thanks for the help,
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory*
participants (4)
-
Barry Smith -
Fackler, Philip -
Junchao Zhang -
Mark Adams