On Tue, Jul 21, 2020 at 10:45 AM Stefano Zampini <stefano.zampini@gmail.com> wrote:

On Jul 21, 2020, at 1:32 PM, Pierpaolo Minelli <pierpaolo.minelli@cnr.it> wrote:

Hi,

I have asked to compile a Petsc Version updated and with 64bit indices.
Now I have Version 3.13.3 and these are the configure options used:

#!/bin/python
if __name__ == '__main__':
  import sys
  import os
  sys.path.insert(0, os.path.abspath('config'))
  import configure
  configure_options = [
  '--CC=mpiicc',
  '--CXX=mpiicpc',
  '--download-hypre',
  '--download-metis',
  '--download-mumps=yes',
  '--download-parmetis',
  '--download-scalapack',
  '--download-superlu_dist',
  '--known-64-bit-blas-indices',
  '--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary',
  '--with-64-bit-indices=1',
  '--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl',
  '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none',
  '--with-debugging=0',
  '--with-fortran-interfaces=1',
  '--with-fortran=1',
  'FC=mpiifort',
  'PETSC_ARCH=arch-linux2-c-opt',
  ]
  configure.petsc_configure(configure_options)

Now, I receive an error on hypre:

forrtl: error (78): process killed (SIGTERM)
Image   PC   Routine   Line   Source
libHYPRE-2.18.2.s  00002B33CF465D3F  for__signal_handl     Unknown  Unknown
libpthread-2.17.s  00002B33D5BFD370  Unknown     Unknown  Unknown
libpthread-2.17.s  00002B33D5BF96D3  pthread_cond_wait     Unknown  Unknown
libiomp5.so   00002B33DBA14E07  Unknown     Unknown  Unknown
libiomp5.so   00002B33DB98810C  Unknown     Unknown  Unknown
libiomp5.so   00002B33DB990578  Unknown     Unknown  Unknown
libiomp5.so   00002B33DB9D9659  Unknown     Unknown  Unknown
libiomp5.so   00002B33DB9D8C39  Unknown     Unknown  Unknown
libiomp5.so   00002B33DB993BCE  __kmpc_fork_call   Unknown  Unknown
PIC_3D     00000000004071C0  Unknown     Unknown  Unknown
PIC_3D     0000000000490299  Unknown     Unknown  Unknown
PIC_3D     0000000000492C17  Unknown     Unknown  Unknown
PIC_3D     000000000040562E  Unknown     Unknown  Unknown
libc-2.17.so     00002B33DC5BEB35  __libc_start_main     Unknown  Unknown
PIC_3D     0000000000405539  Unknown     Unknown  Unknown

Is it possible
that I need to ask also to compile hypre with an option for 64bit indices?

These configure options compile hypre with 64bit indices support.
It should work just fine. Can you run a very small case of your code to confirm?

Is it possible to instruct this inside Petsc configure?
Alternatively, is it possible to use a different multigrid PC inside PETSc that accept 64bit indices?

Thanks in advance

Pierpaolo

Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini <stefano.zampini@gmail.com> ha scritto:

You need a version of PETSc compiled with 64bit indices, since the message indicates the number of dofs in this case is larger the INT_MAX
2501×3401×1601 = 13617947501

I also suggest you upgrade to a newer version, 3.8.3 is quite old as the error message reports

Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli <pierpaolo.minelli@cnr.it> ha scritto:
Hi,

I am trying to solve a Poisson equation on this grid:

Nx = 2501
Ny = 3401
Nz = 1601

I received this error:

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Overflow in integer operation: http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices
[0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 bit indices
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017
[0]PETSC ERROR: /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a arch-linux2-c-opt named r129c09s02 by pminelli Tu
e May 26 20:16:34 2020
[0]PETSC ERROR: Configure options --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary CC=mpiicc FC=mpiifort CXX=mpiicpc
F77=mpiifort F90=mpiifort --with-debugging=0 --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl --with-fortran=1
--with-fortran-interfaces=1 --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe-
2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes --download-hypre --download-superlu_dist --download-parmetis --downlo
ad-metis
[0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/
impls/da/da3.c
[0]PETSC ERROR: #2 DMSetUp_DA() line 25 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl
s/da/dareg.c
[0]PETSC ERROR: #3 DMSetUp() line 720 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf
ace/dm.c
forrtl: error (76): Abort trap signal

I am on an HPC facility and after I loaded PETSC module, I have seen that it is configured with INTEGER size = 32

I solve my problem with these options and it works perfectly with smaller grids:

-dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 -ksp_type richardson

Is it possible to overcome this if I ask them to install a version with INTEGER SIZE = 64?
Alternatively, is it possible to overcome this using intel compiler options?

Thanks in advance

Pierpaolo Minelli

--
Stefano