Thought I sent you this...will change the order of MatSetOption to see if that helps

I ran it and get that error which I have already sent a ticket to OLCF:

CUSPARSE ERROR (code = 11, insufficient resources) at csr_spgemm_device_cusparse.c:128

Here's my petscrc

#do not use -mat_view with hypre-cuda if running on gpu
#-mat_view :A.m:ascii_matlab
-help
-proj_mac_pc_type hypre
-proj_mac_pc_hypre_type boomeramg
-proj_mac_pc_hypre_boomeramg_no_CF
-proj_mac_pc_hypre_boomeramg_agg_nl 0
-proj_mac_pc_hypre_boomeramg_coarsen_type PMIS
-proj_mac_pc_hypre_boomeramg_interp_type ext+i
-proj_mac_pc_hypre_boomeramg_print_statistics
-proj_mac_pc_hypre_boomeramg_relax_type_all l1scaled-Jacobi
-proj_mac_ksp_type gmres
-proj_mac_ksp_max_it 50
-proj_mac_ksp_rtol 1.e-12
-proj_mac_ksp_atol 1.e-30
-mat_type hypre
-use_gpu_aware_mpi 0

-log_view
-history PETSc.history
-visc_ksp_rtol 1.e-12
-visc_pc_type jacobi
-visc_ksp_type gmres
-visc_ksp_max_it 50
-diff_ksp_rtol 1.e-6
-diff_pc_type jacobi

-diff_ksp_max_it 50
-proj_mac_ksp_converged_reason
-visc_ksp_converged_reason
-diff_ksp_converged_reason
-proj_mac_ksp_norm_type unpreconditioned
-diff_ksp_norm_type unpreconditioned
-visc_ksp_norm_type unpreconditioned

And here's my code:

      ierr = MatSetSizes(m_mat,NN,NN,PETSC_DECIDE,PETSC_DECIDE);CHKERRQ(ierr);
      ierr = MatSetBlockSize(m_mat,nc);CHKERRQ(ierr);
      ierr = MatSetType(m_mat,MATAIJ);CHKERRQ(ierr);
      // ierr = MatSetOption(m_mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) ;CHKERRQ(ierr);
      ierr = MatSetFromOptions( m_mat ); CHKERRQ(ierr);
      ierr = MatSeqAIJSetPreallocation(m_mat,nnzrow, d_nnz);CHKERRQ(ierr);
      ierr = MatMPIAIJSetPreallocation(m_mat,nnzrow, d_nnz, nnzrow/2, o_nnz);CHKERRQ(ierr);
      ierr = MatSetOption(m_mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE) ;CHKERRQ(ierr);

#if defined(PETSC_HAVE_HYPRE)
  ierr = MatHYPRESetPreallocation(m_mat,nnzrow, d_nnz, nnzrow/2, o_nnz);CHKERRQ(ierr);
#endif

On Wed, Jan 26, 2022 at 6:18 PM Mark Adams <mfadams@lbl.gov> wrote:

On Wed, Jan 26, 2022 at 7:43 PM David Trebotich <dptrebotich@lbl.gov> wrote:

Can you confirm with me on the settings in .petscrc for Summit with -pc_type hypre?  We were using

-mat_type aijcusparse

which worked a few months ago and now is not working.

I don't know the difference between cusparse, aijcusparse and hypre as -mat_type

 

cusparse is not a matrix type. The other two are and they both should work.

 

I tested the builds that I just sent in another email (eg, PETSC_DIR=/gpfs/alpine/world-shared/geo127/petsc/arch-summit-dbg-gcc-cuda PETSC_ARCH="")

$ make PETSC_DIR=$PWD PETSC_ARCH=arch-summit-opt-gnu-hypre-cuda -f gmakefile test search='ksp_ksp_tutorials-ex55_hypre_device'
Using MAKEFLAGS: -- search=ksp_ksp_tutorials-ex55_hypre_device PETSC_ARCH=arch-summit-opt-gnu-hypre-cuda PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc2
          CC arch-summit-opt-gnu-hypre-cuda/tests/ksp/ksp/tutorials/ex55.o
     CLINKER arch-summit-opt-gnu-hypre-cuda/tests/ksp/ksp/tutorials/ex55
        TEST arch-summit-opt-gnu-hypre-cuda/tests/counts/ksp_ksp_tutorials-ex55_hypre_device.counts
 ok ksp_ksp_tutorials-ex55_hypre_device
 ok diff-ksp_ksp_tutorials-ex55_hypre_device

So this work. In this file (attached) you will see an example of a construction a matrix that we have gone over before:

  /* create stiffness matrix */
  ierr = MatCreate(comm,&Amat);CHKERRQ(ierr);
  ierr = MatSetSizes(Amat,m,m,M,M);CHKERRQ(ierr);
  ierr = MatSetType(Amat,MATAIJ);CHKERRQ(ierr);
  ierr = MatSetOption(Amat,MAT_SPD,PETSC_TRUE);CHKERRQ(ierr);
  ierr = MatSetFromOptions(Amat);CHKERRQ(ierr);
  ierr = MatSetBlockSize(Amat,2);CHKERRQ(ierr);
  ierr = MatSeqAIJSetPreallocation(Amat,18,NULL);CHKERRQ(ierr);
  ierr = MatMPIAIJSetPreallocation(Amat,18,NULL,18,NULL);CHKERRQ(ierr);
#if defined(PETSC_HAVE_HYPRE)
  ierr = MatHYPRESetPreallocation(Amat,18,NULL,18,NULL);CHKERRQ(ierr);
#endif

At the end of the file you will what is executed with this test for "hypre_device":

   # command line options match GPU defaults
   test:
      suffix: hypre_device
      nsize: 4
      requires: hypre !complex
      args: -mat_type hypre -ksp_view -ne 29 -alpha 1.e-3 -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg -ksp_monitor_short -pc_hypre_boomeramg_relax_type_all l1scaled-Jacobi -pc_hypre_boomeramg_interp_type ext+i -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_no_CF

All you need is -mat_type hypre and -pc_type hypre. You could also add these hypre arguments.

If this is not working please send me a description of the problem, like any error output on your screen and your petsc.history file.

 

--

----------------------
David Trebotich
Lawrence Berkeley National Laboratory
Computational Research Division
Applied Numerical Algorithms Group
treb@lbl.gov
(510) 486-5984 office
(510) 384-6868 mobile