Dear PETSc developers, I use Libmesh with PETSc for my FEM simulation. The program works when using MATSEQCUSP. However, When I test MATMPICUSP with 2 GPU cards, I met the following errors: " [1]PETSC ERROR: --------------------- Error Message ------------------------------------ [1]PETSC ERROR: Argument out of range! [1]PETSC ERROR: New nonzero at (1,5) caused a malloc! [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Petsc Development HG revision: b4086d236fb35071ea565635e24ea99d70deeaac HG Date: Sat Jan 21 21:50:39 2012 -0600 [1]PETSC ERROR: See docs/changes/index.html for recent updates. [1]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [1]PETSC ERROR: See docs/index.html for manual pages. [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: /work/01820/ylu/libmesh_svn01232012_petscme/examples/myproj_sp1/myproj-opt on a linux named c300-202.ls4.tacc.utexas.edu by ylu Fri Feb 10 13:52:39 2012 [1]PETSC ERROR: Libraries linked from /work/01820/ylu/petsc-dev01222012_real/linux/lib [1]PETSC ERROR: Configure run at Thu Feb 9 10:42:41 2012 [1]PETSC ERROR: Configure options --with-clanguage=C++ --with-debugging=1 --with-shared-libraries=1 --with-mpi-compilers=1 --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl/lib/em64t/ --with-cuda=1 --with-cusp=1 --with-thrust=1 --with-cuda-dir=/opt/apps/cuda/4.0/cuda/ --with-cusp-dir=/work/01820/ylu --with-valgrind-dir=/opt/apps/valgrind/3.6.0/ [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: MatSetValues_MPIAIJ() line 505 in src/mat/impls/aij/mpi/mpiaij.c [1]PETSC ERROR: MatSetValues() line 1119 in src/mat/interface/matrix.c [1]PETSC ERROR: add_matrix() line 468 in "unknowndirectory/"src/numerics/petsc_matrix.C application called MPI_Abort(comm=0x84000004, 63) - process 1 [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: New nonzero at (2,54) caused a malloc! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development HG revision: b4086d236fb35071ea565635e24ea99d70deeaac HG Date: Sat Jan 21 21:50:39 2012 -0600 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /work/01820/ylu/libmesh_svn01232012_petscme/examples/myproj_sp1/myproj-opt on a linux named c300-202.ls4.tacc.utexas.edu by ylu Fri Feb 10 13:52:39 2012 [0]PETSC ERROR: Libraries linked from /work/01820/ylu/petsc-dev01222012_real/linux/lib [0]PETSC ERROR: Configure run at Thu Feb 9 10:42:41 2012 [0]PETSC ERROR: Configure options --with-clanguage=C++ --with-debugging=1 --with-shared-libraries=1 --with-mpi-compilers=1 --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl/lib/em64t/ --with-cuda=1 --with-cusp=1 --with-thrust=1 --with-cuda-dir=/opt/apps/cuda/4.0/cuda/ --with-cusp-dir=/work/01820/ylu --with-valgrind-dir=/opt/apps/valgrind/3.6.0/ [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatSetValues_MPIAIJ() line 505 in src/mat/impls/aij/mpi/mpiaij.c [0]PETSC ERROR: MatSetValues() line 1119 in src/mat/interface/matrix.c [0]PETSC ERROR: add_matrix() line 468 in "unknowndirectory/"src/numerics/petsc_matrix.C application called MPI_Abort(comm=0xC4000000, 63) - process 0 " It seem MatSetValues sets some values out of range. In parallel FEM simulation, I partition the mesh into 2 submeshes. Test is performed with 2GPU cards/2 CPU cores. Each CPU is responsible for one submesh for matrix assembly (I think MatSetValues should set the values within the submesh, otherwise it should generate errors like the above). However, I don't understand why there are errors. Is anything wrong? Thank you very much, Yujie