On Sat, Feb 11, 2012 at 10:58, recrusader <recrusader@gmail.com> wrote:
    "    ierr = MatCreateMPIAIJ (libMesh::COMM_WORLD,
                              
  m_local, n_local,
                                m_global, n_global,
                                PETSC_NULL, (int*) &n_nz[0],
                                PETSC_NULL, (int*) &n_oz[0], &_mat);
             CHKERRABORT(libMesh::COMM_WORLD,ierr);

      MatSetOption(_mat,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE); //by Yujie
      std::cout<<"MatSetOption"<<std::endl;"

I run the same codes in CPU and GPU modes (the same parameters except that GPU uses '-vec_type mpicusp -mat_type mpiaijcusp'). I can find "MatSetOption" output from both the modes. Does that mean that the codes set the options for both the modes?

Libmesh is calling MatSetFromOptions() after MatCreateMPIAIJ() which means the preallocation information will be lost if the type is changed. The code should be written differently

  ierr = MatCreate(comm,A);CHKERRQ(ierr);
  ierr = MatSetSizes(*A,m,n,M,N);CHKERRQ(ierr);
  ierr = MPI_Comm_size(comm,&size);CHKERRQ(ierr);
  ierr = MatSetType(*A,MATAIJ);CHKERRQ(ierr);
  ierr = MatSetOptionsPrefix(*A,optional_prefix);CHKERRQ(ierr);
  ierr = MatSetFromOptions(*A);CHKERRQ(ierr);
  ierr = MatMPIAIJSetPreallocation(*A,d_nz,d_nnz,o_nz,o_nnz);CHKERRQ(ierr);
  ierr = MatSeqAIJSetPreallocation(*A,d_nz,d_nnz);CHKERRQ(ierr);

I can talk to the libmesh developers about making this change.