On Tue, 21 Jul 2020 at 12:32, Pierpaolo Minelli <[email protected]> wrote:

Hi,

I have asked to compile a Petsc Version updated and with 64bit indices. Now I have Version 3.13.3 and these are the configure options used:

#!/bin/python if __name__ == '__main__': import sys import os sys.path.insert(0, os.path.abspath('config')) import configure configure_options = [ '--CC=mpiicc', '--CXX=mpiicpc', '--download-hypre', '--download-metis', '--download-mumps=yes', '--download-parmetis', '--download-scalapack', '--download-superlu_dist', '--known-64-bit-blas-indices',

'--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary', '--with-64-bit-indices=1',

'--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl', '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none', '--with-debugging=0', '--with-fortran-interfaces=1', '--with-fortran=1', 'FC=mpiifort', 'PETSC_ARCH=arch-linux2-c-opt', ] configure.petsc_configure(configure_options)

Now, I receive an error on hypre:

forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl Unknown Unknown libpthread-2.17.s 00002B33D5BFD370 Unknown Unknown Unknown libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait Unknown Unknown libiomp5.so 00002B33DBA14E07 Unknown Unknown Unknown libiomp5.so 00002B33DB98810C Unknown Unknown Unknown libiomp5.so 00002B33DB990578 Unknown Unknown Unknown libiomp5.so 00002B33DB9D9659 Unknown Unknown Unknown libiomp5.so 00002B33DB9D8C39 Unknown Unknown Unknown libiomp5.so 00002B33DB993BCE __kmpc_fork_call Unknown Unknown PIC_3D 00000000004071C0 Unknown Unknown Unknown PIC_3D 0000000000490299 Unknown Unknown Unknown PIC_3D 0000000000492C17 Unknown Unknown Unknown PIC_3D 000000000040562E Unknown Unknown Unknown libc-2.17.so 00002B33DC5BEB35 __libc_start_main Unknown Unknown PIC_3D 0000000000405539 Unknown Unknown Unknown

Is it possible that I need to ask also to compile hypre with an option for 64bit indices? Is it possible to instruct this inside Petsc configure? Alternatively, is it possible to use a different multigrid PC inside PETSc that accept 64bit indices?

You can use -pc_type gamg All native PETSc implementations support 64bit indices.

Thanks in advance

Pierpaolo

Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini < [email protected]> ha scritto:

You need a version of PETSc compiled with 64bit indices, since the message indicates the number of dofs in this case is larger the INT_MAX 2501×3401×1601 = 13617947501

I also suggest you upgrade to a newer version, 3.8.3 is quite old as the error message reports

Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli < [email protected]> ha scritto:

...

Hi,

I am trying to solve a Poisson equation on this grid:

Nx = 2501 Ny = 3401 Nz = 1601

I received this error:

[0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Overflow in integer operation: http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 bit indices [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 [0]PETSC ERROR: /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a arch-linux2-c-opt named r129c09s02 by pminelli Tu e May 26 20:16:34 2020 [0]PETSC ERROR: Configure options --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary CC=mpiicc FC=mpiifort CXX=mpiicpc F77=mpiifort F90=mpiifort --with-debugging=0 --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl --with-fortran=1 --with-fortran-interfaces=1 --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe- 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes --download-hypre --download-superlu_dist --download-parmetis --downlo ad-metis [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/ impls/da/da3.c [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl s/da/dareg.c [0]PETSC ERROR: #3 DMSetUp() line 720 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf ace/dm.c forrtl: error (76): Abort trap signal

I am on an HPC facility and after I loaded PETSC module, I have seen that it is configured with INTEGER size = 32

I solve my problem with these options and it works perfectly with smaller grids:

-dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 -ksp_type richardson

Is it possible to overcome this if I ask them to install a version with INTEGER SIZE = 64? Alternatively, is it possible to overcome this using intel compiler options?

Thanks in advance

Pierpaolo Minelli

-- Stefano

On Tue, Jul 21, 2020 at 9:35 AM Pierpaolo Minelli <[email protected]> wrote:

Thanks for your reply. As I wrote before, I use these settings:

-dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 -ksp_type richardson

Is there a way to emulate this features also with GAMG?

Smoothers: You have complete control here -mg_levels_pc_type sor (the default is Chebyshev which you could also try) Coarsening: This is much different in agglomeration AMG. There is a discussion here: https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThr... https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetSqu... Interpolation: This is built-in for agglomeration AMG.

It would be better to use only native Petsc implementations, but these settings, up to single precision indexing for integers, gave me optimal performances. For this reason I asked also, if it was possible to configure hypre (inside Petsc) with 64bit integers.

Yes. That happened when you reconfigured for 64 bits. You may have encountered a Hypre bug. Thanks, Matt

Pierpaolo

Il giorno 21 lug 2020, alle ore 13:36, Dave May <[email protected]> ha scritto:

On Tue, 21 Jul 2020 at 12:32, Pierpaolo Minelli <[email protected]> wrote:

...

Hi,

I have asked to compile a Petsc Version updated and with 64bit indices. Now I have Version 3.13.3 and these are the configure options used:

#!/bin/python if __name__ == '__main__': import sys import os sys.path.insert(0, os.path.abspath('config')) import configure configure_options = [ '--CC=mpiicc', '--CXX=mpiicpc', '--download-hypre', '--download-metis', '--download-mumps=yes', '--download-parmetis', '--download-scalapack', '--download-superlu_dist', '--known-64-bit-blas-indices',

'--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary', '--with-64-bit-indices=1',

'--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl', '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none', '--with-debugging=0', '--with-fortran-interfaces=1', '--with-fortran=1', 'FC=mpiifort', 'PETSC_ARCH=arch-linux2-c-opt', ] configure.petsc_configure(configure_options)

Now, I receive an error on hypre:

forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl Unknown Unknown libpthread-2.17.s 00002B33D5BFD370 Unknown Unknown Unknown libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait Unknown Unknown libiomp5.so 00002B33DBA14E07 Unknown Unknown Unknown libiomp5.so 00002B33DB98810C Unknown Unknown Unknown libiomp5.so 00002B33DB990578 Unknown Unknown Unknown libiomp5.so 00002B33DB9D9659 Unknown Unknown Unknown libiomp5.so 00002B33DB9D8C39 Unknown Unknown Unknown libiomp5.so 00002B33DB993BCE __kmpc_fork_call Unknown Unknown PIC_3D 00000000004071C0 Unknown Unknown Unknown PIC_3D 0000000000490299 Unknown Unknown Unknown PIC_3D 0000000000492C17 Unknown Unknown Unknown PIC_3D 000000000040562E Unknown Unknown Unknown libc-2.17.so 00002B33DC5BEB35 __libc_start_main Unknown Unknown PIC_3D 0000000000405539 Unknown Unknown Unknown

Is it possible that I need to ask also to compile hypre with an option for 64bit indices? Is it possible to instruct this inside Petsc configure? Alternatively, is it possible to use a different multigrid PC inside PETSc that accept 64bit indices?

You can use -pc_type gamg All native PETSc implementations support 64bit indices.

...

Thanks in advance

Pierpaolo

Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini < [email protected]> ha scritto:

You need a version of PETSc compiled with 64bit indices, since the message indicates the number of dofs in this case is larger the INT_MAX 2501×3401×1601 = 13617947501

I also suggest you upgrade to a newer version, 3.8.3 is quite old as the error message reports

Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli < [email protected]> ha scritto:

...

Hi,

I am trying to solve a Poisson equation on this grid:

Nx = 2501 Ny = 3401 Nz = 1601

I received this error:

[0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Overflow in integer operation: http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 bit indices [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 [0]PETSC ERROR: /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a arch-linux2-c-opt named r129c09s02 by pminelli Tu e May 26 20:16:34 2020 [0]PETSC ERROR: Configure options --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary CC=mpiicc FC=mpiifort CXX=mpiicpc F77=mpiifort F90=mpiifort --with-debugging=0 --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl --with-fortran=1 --with-fortran-interfaces=1 --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe- 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes --download-hypre --download-superlu_dist --download-parmetis --downlo ad-metis [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/ impls/da/da3.c [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl s/da/dareg.c [0]PETSC ERROR: #3 DMSetUp() line 720 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf ace/dm.c forrtl: error (76): Abort trap signal

I am on an HPC facility and after I loaded PETSC module, I have seen that it is configured with INTEGER size = 32

I solve my problem with these options and it works perfectly with smaller grids:

-dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 -ksp_type richardson

Is it possible to overcome this if I ask them to install a version with INTEGER SIZE = 64? Alternatively, is it possible to overcome this using intel compiler options?

Thanks in advance

Pierpaolo Minelli

-- Stefano

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

On Tue, Jul 21, 2020 at 9:46 AM Matthew Knepley <[email protected]> wrote:

On Tue, Jul 21, 2020 at 9:35 AM Pierpaolo Minelli < [email protected]> wrote:

...

Thanks for your reply. As I wrote before, I use these settings:

-dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 -ksp_type richardson

Is there a way to emulate this features also with GAMG?

Smoothers: You have complete control here

-mg_levels_pc_type sor (the default is Chebyshev which you could also try)

And you set the KSP type. You have -ksp_type richardson above but that is not used for Hypre. It is for GAMG. Chebyshev is a ksp type (-ksp_type chebyshev). Hypre is very good on Poisson. THe grid complexity (cost per iteration) can be high but the convergence rate will be better than GAMG. But, you should be able to get hypre to work.

Coarsening: This is much different in agglomeration AMG. There is a discussion here:

https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThr...

https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetSqu...

Interpolation: This is built-in for agglomeration AMG.

...

It would be better to use only native Petsc implementations, but these settings, up to single precision indexing for integers, gave me optimal performances. For this reason I asked also, if it was possible to configure hypre (inside Petsc) with 64bit integers.

Yes. That happened when you reconfigured for 64 bits. You may have encountered a Hypre bug.

Thanks,

Matt

...

Pierpaolo

Il giorno 21 lug 2020, alle ore 13:36, Dave May <[email protected]> ha scritto:

On Tue, 21 Jul 2020 at 12:32, Pierpaolo Minelli <[email protected]> wrote:

...

Hi,

I have asked to compile a Petsc Version updated and with 64bit indices. Now I have Version 3.13.3 and these are the configure options used:

#!/bin/python if __name__ == '__main__': import sys import os sys.path.insert(0, os.path.abspath('config')) import configure configure_options = [ '--CC=mpiicc', '--CXX=mpiicpc', '--download-hypre', '--download-metis', '--download-mumps=yes', '--download-parmetis', '--download-scalapack', '--download-superlu_dist', '--known-64-bit-blas-indices',

'--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary', '--with-64-bit-indices=1',

'--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl', '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none', '--with-debugging=0', '--with-fortran-interfaces=1', '--with-fortran=1', 'FC=mpiifort', 'PETSC_ARCH=arch-linux2-c-opt', ] configure.petsc_configure(configure_options)

Now, I receive an error on hypre:

forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl Unknown Unknown libpthread-2.17.s 00002B33D5BFD370 Unknown Unknown Unknown libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait Unknown Unknown libiomp5.so 00002B33DBA14E07 Unknown Unknown Unknown libiomp5.so 00002B33DB98810C Unknown Unknown Unknown libiomp5.so 00002B33DB990578 Unknown Unknown Unknown libiomp5.so 00002B33DB9D9659 Unknown Unknown Unknown libiomp5.so 00002B33DB9D8C39 Unknown Unknown Unknown libiomp5.so 00002B33DB993BCE __kmpc_fork_call Unknown Unknown PIC_3D 00000000004071C0 Unknown Unknown Unknown PIC_3D 0000000000490299 Unknown Unknown Unknown PIC_3D 0000000000492C17 Unknown Unknown Unknown PIC_3D 000000000040562E Unknown Unknown Unknown libc-2.17.so 00002B33DC5BEB35 __libc_start_main Unknown Unknown PIC_3D 0000000000405539 Unknown Unknown Unknown

Is it possible that I need to ask also to compile hypre with an option for 64bit indices? Is it possible to instruct this inside Petsc configure? Alternatively, is it possible to use a different multigrid PC inside PETSc that accept 64bit indices?

You can use -pc_type gamg All native PETSc implementations support 64bit indices.

...

Thanks in advance

Pierpaolo

Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini < [email protected]> ha scritto:

You need a version of PETSc compiled with 64bit indices, since the message indicates the number of dofs in this case is larger the INT_MAX 2501×3401×1601 = 13617947501

I also suggest you upgrade to a newer version, 3.8.3 is quite old as the error message reports

Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli < [email protected]> ha scritto:

...

Hi,

I am trying to solve a Poisson equation on this grid:

Nx = 2501 Ny = 3401 Nz = 1601

I received this error:

[0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Overflow in integer operation: http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 bit indices [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 [0]PETSC ERROR: /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a arch-linux2-c-opt named r129c09s02 by pminelli Tu e May 26 20:16:34 2020 [0]PETSC ERROR: Configure options --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary CC=mpiicc FC=mpiifort CXX=mpiicpc F77=mpiifort F90=mpiifort --with-debugging=0 --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl --with-fortran=1 --with-fortran-interfaces=1 --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe- 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes --download-hypre --download-superlu_dist --download-parmetis --downlo ad-metis [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/ impls/da/da3.c [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl s/da/dareg.c [0]PETSC ERROR: #3 DMSetUp() line 720 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf ace/dm.c forrtl: error (76): Abort trap signal

I am on an HPC facility and after I loaded PETSC module, I have seen that it is configured with INTEGER size = 32

I solve my problem with these options and it works perfectly with smaller grids:

-dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 -ksp_type richardson

Is it possible to overcome this if I ask them to install a version with INTEGER SIZE = 64? Alternatively, is it possible to overcome this using intel compiler options?

Thanks in advance

Pierpaolo Minelli

-- Stefano

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

On Tue, Jul 21, 2020 at 12:06 PM Pierpaolo Minelli <[email protected]> wrote:

Il giorno 21 lug 2020, alle ore 16:56, Mark Adams <[email protected]> ha scritto:

On Tue, Jul 21, 2020 at 9:46 AM Matthew Knepley <[email protected]> wrote:

...

On Tue, Jul 21, 2020 at 9:35 AM Pierpaolo Minelli < [email protected]> wrote:

...

Thanks for your reply. As I wrote before, I use these settings:

-dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 -ksp_type richardson

Is there a way to emulate this features also with GAMG?

Smoothers: You have complete control here

-mg_levels_pc_type sor (the default is Chebyshev which you could also try)

And you set the KSP type. You have -ksp_type richardson above but that is not used for Hypre. It is for GAMG. Chebyshev is a ksp type (-ksp_type chebyshev).

Hypre is very good on Poisson. THe grid complexity (cost per iteration) can be high but the convergence rate will be better than GAMG.

But, you should be able to get hypre to work.

Yes it is very good for Poisson, and on a smaller case, at the beginning of my code development, I have tried Hypre, ML, and GAMG (without adding more options I have to admit) and hypre was faster without losing in precision and accurateness of results (I have checked them with -ksp_monitor_true_residual). I left -kps_type Richardson instead of default gmres only because from residuals it seems more accurate.

Whoops, I made a mistake. I was talking about the smoother. So -mg_levels_pc_type sor -mg_levels_ksp_type chebyshev

So first, i will try again to see if hypre (with integer 64bit) is able to work on a smaller case as suggested by Stefano. Then I will investigate GAMG options and I will give you a feedback. The problem is that I need 64bit integers because of my problem size so I have to follow both path, but I hope that I will be able to continue to use hypre.

Thanks

Pierpaolo

...

Coarsening: This is much different in agglomeration AMG. There is a discussion here:

https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThr...

https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetSqu...

Interpolation: This is built-in for agglomeration AMG.

...

It would be better to use only native Petsc implementations, but these settings, up to single precision indexing for integers, gave me optimal performances. For this reason I asked also, if it was possible to configure hypre (inside Petsc) with 64bit integers.

Yes. That happened when you reconfigured for 64 bits. You may have encountered a Hypre bug.

Thanks,

Matt

...

Pierpaolo

Il giorno 21 lug 2020, alle ore 13:36, Dave May <[email protected]> ha scritto:

On Tue, 21 Jul 2020 at 12:32, Pierpaolo Minelli < [email protected]> wrote:

...

Hi,

I have asked to compile a Petsc Version updated and with 64bit indices. Now I have Version 3.13.3 and these are the configure options used:

#!/bin/python if __name__ == '__main__': import sys import os sys.path.insert(0, os.path.abspath('config')) import configure configure_options = [ '--CC=mpiicc', '--CXX=mpiicpc', '--download-hypre', '--download-metis', '--download-mumps=yes', '--download-parmetis', '--download-scalapack', '--download-superlu_dist', '--known-64-bit-blas-indices',

'--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary', '--with-64-bit-indices=1',

'--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl', '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none', '--with-debugging=0', '--with-fortran-interfaces=1', '--with-fortran=1', 'FC=mpiifort', 'PETSC_ARCH=arch-linux2-c-opt', ] configure.petsc_configure(configure_options)

Now, I receive an error on hypre:

forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl Unknown Unknown libpthread-2.17.s 00002B33D5BFD370 Unknown Unknown Unknown libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait Unknown Unknown libiomp5.so 00002B33DBA14E07 Unknown Unknown Unknown libiomp5.so 00002B33DB98810C Unknown Unknown Unknown libiomp5.so 00002B33DB990578 Unknown Unknown Unknown libiomp5.so 00002B33DB9D9659 Unknown Unknown Unknown libiomp5.so 00002B33DB9D8C39 Unknown Unknown Unknown libiomp5.so 00002B33DB993BCE __kmpc_fork_call Unknown Unknown PIC_3D 00000000004071C0 Unknown Unknown Unknown PIC_3D 0000000000490299 Unknown Unknown Unknown PIC_3D 0000000000492C17 Unknown Unknown Unknown PIC_3D 000000000040562E Unknown Unknown Unknown libc-2.17.so 00002B33DC5BEB35 __libc_start_main Unknown Unknown PIC_3D 0000000000405539 Unknown Unknown Unknown

Is it possible that I need to ask also to compile hypre with an option for 64bit indices? Is it possible to instruct this inside Petsc configure? Alternatively, is it possible to use a different multigrid PC inside PETSc that accept 64bit indices?

You can use -pc_type gamg All native PETSc implementations support 64bit indices.

...

Thanks in advance

Pierpaolo

Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini < [email protected]> ha scritto:

You need a version of PETSc compiled with 64bit indices, since the message indicates the number of dofs in this case is larger the INT_MAX 2501×3401×1601 = 13617947501

I also suggest you upgrade to a newer version, 3.8.3 is quite old as the error message reports

Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli < [email protected]> ha scritto:

...

Hi,

I am trying to solve a Poisson equation on this grid:

Nx = 2501 Ny = 3401 Nz = 1601

I received this error:

[0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Overflow in integer operation: http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 bit indices [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 [0]PETSC ERROR: /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a arch-linux2-c-opt named r129c09s02 by pminelli Tu e May 26 20:16:34 2020 [0]PETSC ERROR: Configure options --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary CC=mpiicc FC=mpiifort CXX=mpiicpc F77=mpiifort F90=mpiifort --with-debugging=0 --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl --with-fortran=1 --with-fortran-interfaces=1 --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe- 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes --download-hypre --download-superlu_dist --download-parmetis --downlo ad-metis [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/ impls/da/da3.c [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl s/da/dareg.c [0]PETSC ERROR: #3 DMSetUp() line 720 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf ace/dm.c forrtl: error (76): Abort trap signal

I am on an HPC facility and after I loaded PETSC module, I have seen that it is configured with INTEGER size = 32

I solve my problem with these options and it works perfectly with smaller grids:

-dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 -ksp_type richardson

Is it possible to overcome this if I ask them to install a version with INTEGER SIZE = 64? Alternatively, is it possible to overcome this using intel compiler options?

Thanks in advance

Pierpaolo Minelli

-- Stefano

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

This also looks like it could be some sort of library mismatch. You might try deleting your architecture directory and start over. This PETSc's "make realclean" On Tue, Jul 21, 2020 at 10:45 AM Stefano Zampini <[email protected]> wrote:

On Jul 21, 2020, at 1:32 PM, Pierpaolo Minelli <[email protected]> wrote:

Hi,

I have asked to compile a Petsc Version updated and with 64bit indices. Now I have Version 3.13.3 and these are the configure options used:

#!/bin/python if __name__ == '__main__': import sys import os sys.path.insert(0, os.path.abspath('config')) import configure configure_options = [ '--CC=mpiicc', '--CXX=mpiicpc', '--download-hypre', '--download-metis', '--download-mumps=yes', '--download-parmetis', '--download-scalapack', '--download-superlu_dist', '--known-64-bit-blas-indices',

'--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary', '--with-64-bit-indices=1',

'--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl', '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none', '--with-debugging=0', '--with-fortran-interfaces=1', '--with-fortran=1', 'FC=mpiifort', 'PETSC_ARCH=arch-linux2-c-opt', ] configure.petsc_configure(configure_options)

Now, I receive an error on hypre:

forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl Unknown Unknown libpthread-2.17.s 00002B33D5BFD370 Unknown Unknown Unknown libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait Unknown Unknown libiomp5.so 00002B33DBA14E07 Unknown Unknown Unknown libiomp5.so 00002B33DB98810C Unknown Unknown Unknown libiomp5.so 00002B33DB990578 Unknown Unknown Unknown libiomp5.so 00002B33DB9D9659 Unknown Unknown Unknown libiomp5.so 00002B33DB9D8C39 Unknown Unknown Unknown libiomp5.so 00002B33DB993BCE __kmpc_fork_call Unknown Unknown PIC_3D 00000000004071C0 Unknown Unknown Unknown PIC_3D 0000000000490299 Unknown Unknown Unknown PIC_3D 0000000000492C17 Unknown Unknown Unknown PIC_3D 000000000040562E Unknown Unknown Unknown libc-2.17.so 00002B33DC5BEB35 __libc_start_main Unknown Unknown PIC_3D 0000000000405539 Unknown Unknown Unknown

Is it possible

that I need to ask also to compile hypre with an option for 64bit indices?

These configure options compile hypre with 64bit indices support. It should work just fine. Can you run a very small case of your code to confirm?

Is it possible to instruct this inside Petsc configure? Alternatively, is it possible to use a different multigrid PC inside PETSc that accept 64bit indices?

Thanks in advance

Pierpaolo

Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini < [email protected]> ha scritto:

You need a version of PETSc compiled with 64bit indices, since the message indicates the number of dofs in this case is larger the INT_MAX 2501×3401×1601 = 13617947501

I also suggest you upgrade to a newer version, 3.8.3 is quite old as the error message reports

Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli < [email protected]> ha scritto:

...

Hi,

I am trying to solve a Poisson equation on this grid:

Nx = 2501 Ny = 3401 Nz = 1601

I received this error:

[0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Overflow in integer operation: http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 bit indices [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 [0]PETSC ERROR: /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a arch-linux2-c-opt named r129c09s02 by pminelli Tu e May 26 20:16:34 2020 [0]PETSC ERROR: Configure options --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary CC=mpiicc FC=mpiifort CXX=mpiicpc F77=mpiifort F90=mpiifort --with-debugging=0 --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl --with-fortran=1 --with-fortran-interfaces=1 --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe- 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes --download-hypre --download-superlu_dist --download-parmetis --downlo ad-metis [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/ impls/da/da3.c [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl s/da/dareg.c [0]PETSC ERROR: #3 DMSetUp() line 720 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf ace/dm.c forrtl: error (76): Abort trap signal

I am on an HPC facility and after I loaded PETSC module, I have seen that it is configured with INTEGER size = 32

I solve my problem with these options and it works perfectly with smaller grids:

-dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 -ksp_type richardson

Is it possible to overcome this if I ask them to install a version with INTEGER SIZE = 64? Alternatively, is it possible to overcome this using intel compiler options?

Thanks in advance

Pierpaolo Minelli

-- Stefano

On Tue, Jul 21, 2020 at 12:11 PM Pierpaolo Minelli <[email protected]> wrote:

Il giorno 21 lug 2020, alle ore 16:58, Mark Adams <[email protected]> ha scritto:

This also looks like it could be some sort of library mismatch. You might try deleting your architecture directory and start over. This PETSc's "make realclean"

I hope this is not the case, because I am working on CINECA HPC Facility (Italy) and in this center I need to load modules for each software I need. I asked Cineca support to compile a version of Petsc with 64 bit integers and all that external packages and after they have done it, I loaded directly this new module, so the older version (3.8.x with integer in single precision) is not involved at all. At least I hope…

You can run a PETSc test. Download PETSc and go into the PETSc directory. On an execution node or batch script run 'make check' with PETSC_ARCH and PETSC_DIR set as usual (ie, not $PWD). THis should run a few tests including a hypre test. If it works, then there may be a problem with how you are linking your program. If it fails then then ask the Facility to help you.

Thanks

Pierpaolo

On Tue, Jul 21, 2020 at 10:45 AM Stefano Zampini < [email protected]> wrote:

...

On Jul 21, 2020, at 1:32 PM, Pierpaolo Minelli <[email protected]> wrote:

Hi,

I have asked to compile a Petsc Version updated and with 64bit indices. Now I have Version 3.13.3 and these are the configure options used:

#!/bin/python if __name__ == '__main__': import sys import os sys.path.insert(0, os.path.abspath('config')) import configure configure_options = [ '--CC=mpiicc', '--CXX=mpiicpc', '--download-hypre', '--download-metis', '--download-mumps=yes', '--download-parmetis', '--download-scalapack', '--download-superlu_dist', '--known-64-bit-blas-indices',

'--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary', '--with-64-bit-indices=1',

'--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl', '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none', '--with-debugging=0', '--with-fortran-interfaces=1', '--with-fortran=1', 'FC=mpiifort', 'PETSC_ARCH=arch-linux2-c-opt', ] configure.petsc_configure(configure_options)

Now, I receive an error on hypre:

forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl Unknown Unknown libpthread-2.17.s 00002B33D5BFD370 Unknown Unknown Unknown libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait Unknown Unknown libiomp5.so 00002B33DBA14E07 Unknown Unknown Unknown libiomp5.so 00002B33DB98810C Unknown Unknown Unknown libiomp5.so 00002B33DB990578 Unknown Unknown Unknown libiomp5.so 00002B33DB9D9659 Unknown Unknown Unknown libiomp5.so 00002B33DB9D8C39 Unknown Unknown Unknown libiomp5.so 00002B33DB993BCE __kmpc_fork_call Unknown Unknown PIC_3D 00000000004071C0 Unknown Unknown Unknown PIC_3D 0000000000490299 Unknown Unknown Unknown PIC_3D 0000000000492C17 Unknown Unknown Unknown PIC_3D 000000000040562E Unknown Unknown Unknown libc-2.17.so 00002B33DC5BEB35 __libc_start_main Unknown Unknown PIC_3D 0000000000405539 Unknown Unknown Unknown

Is it possible

that I need to ask also to compile hypre with an option for 64bit indices?

These configure options compile hypre with 64bit indices support. It should work just fine. Can you run a very small case of your code to confirm?

Is it possible to instruct this inside Petsc configure? Alternatively, is it possible to use a different multigrid PC inside PETSc that accept 64bit indices?

Thanks in advance

Pierpaolo

Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini < [email protected]> ha scritto:

You need a version of PETSc compiled with 64bit indices, since the message indicates the number of dofs in this case is larger the INT_MAX 2501×3401×1601 = 13617947501

I also suggest you upgrade to a newer version, 3.8.3 is quite old as the error message reports

Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli < [email protected]> ha scritto:

...

Hi,

I am trying to solve a Poisson equation on this grid:

Nx = 2501 Ny = 3401 Nz = 1601

I received this error:

[0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Overflow in integer operation: http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 bit indices [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 [0]PETSC ERROR: /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a arch-linux2-c-opt named r129c09s02 by pminelli Tu e May 26 20:16:34 2020 [0]PETSC ERROR: Configure options --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary CC=mpiicc FC=mpiifort CXX=mpiicpc F77=mpiifort F90=mpiifort --with-debugging=0 --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl --with-fortran=1 --with-fortran-interfaces=1 --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe- 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes --download-hypre --download-superlu_dist --download-parmetis --downlo ad-metis [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/ impls/da/da3.c [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl s/da/dareg.c [0]PETSC ERROR: #3 DMSetUp() line 720 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf ace/dm.c forrtl: error (76): Abort trap signal

I am on an HPC facility and after I loaded PETSC module, I have seen that it is configured with INTEGER size = 32

I solve my problem with these options and it works perfectly with smaller grids:

-dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 -ksp_type richardson

Is it possible to overcome this if I ask them to install a version with INTEGER SIZE = 64? Alternatively, is it possible to overcome this using intel compiler options?

Thanks in advance

Pierpaolo Minelli

-- Stefano

Hi, I want to thank you for all your useful suggestions, but luckily fo me it was non a problem related to Petsc as I wrongly supposed reading this error. There was a problem on GPFS filesystem that CINECA support solved and also a problem related to my code that was writing too much files when using 512 nodes. Once these two problem were solved I was able to continue to use Hypre without any problem. Sorry for the trouble and thanks again for your suggestions. Pierpaolo

Il giorno 22 lug 2020, alle ore 02:51, Barry Smith <[email protected]> ha scritto:

I assume PIC_3D is your code and you are using OpenMP?

Are you calling hypre from inside your OpenMP parallelism? From inside PIC_3D?

The SIGTERM is confusing to me. Are you using signals in any way? Usually a sigterm comes outside a process not process or thread crash.

I assume for__signal_handl... is a Fortran signal handler

forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl Unknown Unknown libpthread-2.17.s 00002B33D5BFD370 Unknown Unknown Unknown libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait Unknown Unknown libiomp5.so 00002B33DBA14E07 Unknown Unknown Unknown libiomp5.so 00002B33DB98810C Unknown Unknown Unknown libiomp5.so 00002B33DB990578 Unknown Unknown Unknown libiomp5.so 00002B33DB9D9659 Unknown Unknown Unknown libiomp5.so 00002B33DB9D8C39 Unknown Unknown Unknown libiomp5.so 00002B33DB993BCE __kmpc_fork_call Unknown Unknown PIC_3D 00000000004071C0 Unknown Unknown Unknown PIC_3D 0000000000490299 Unknown Unknown Unknown PIC_3D 0000000000492C17 Unknown Unknown Unknown PIC_3D 000000000040562E Unknown Unknown Unknown libc-2.17.so 00002B33DC5BEB35 __libc_start_main Unknown Unknown PIC_3D 0000000000405539 Unknown Unknown Unknown

...

On Jul 21, 2020, at 6:32 AM, Pierpaolo Minelli <[email protected] <mailto:[email protected]>> wrote:

Hi,

I have asked to compile a Petsc Version updated and with 64bit indices. Now I have Version 3.13.3 and these are the configure options used:

#!/bin/python if __name__ == '__main__': import sys import os sys.path.insert(0, os.path.abspath('config')) import configure configure_options = [ '--CC=mpiicc', '--CXX=mpiicpc', '--download-hypre', '--download-metis', '--download-mumps=yes', '--download-parmetis', '--download-scalapack', '--download-superlu_dist', '--known-64-bit-blas-indices', '--prefix=/cineca/prod/opt/libraries/petsc/3.13.3_int64/intelmpi--2018--binary', '--with-64-bit-indices=1', '--with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl', '--with-cmake-dir=/cineca/prod/opt/tools/cmake/3.12.0/none', '--with-debugging=0', '--with-fortran-interfaces=1', '--with-fortran=1', 'FC=mpiifort', 'PETSC_ARCH=arch-linux2-c-opt', ] configure.petsc_configure(configure_options)

Now, I receive an error on hypre:

forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libHYPRE-2.18.2.s 00002B33CF465D3F for__signal_handl Unknown Unknown libpthread-2.17.s 00002B33D5BFD370 Unknown Unknown Unknown libpthread-2.17.s 00002B33D5BF96D3 pthread_cond_wait Unknown Unknown libiomp5.so 00002B33DBA14E07 Unknown Unknown Unknown libiomp5.so 00002B33DB98810C Unknown Unknown Unknown libiomp5.so 00002B33DB990578 Unknown Unknown Unknown libiomp5.so 00002B33DB9D9659 Unknown Unknown Unknown libiomp5.so 00002B33DB9D8C39 Unknown Unknown Unknown libiomp5.so 00002B33DB993BCE __kmpc_fork_call Unknown Unknown PIC_3D 00000000004071C0 Unknown Unknown Unknown PIC_3D 0000000000490299 Unknown Unknown Unknown PIC_3D 0000000000492C17 Unknown Unknown Unknown PIC_3D 000000000040562E Unknown Unknown Unknown libc-2.17.so 00002B33DC5BEB35 __libc_start_main Unknown Unknown PIC_3D 0000000000405539 Unknown Unknown Unknown

Is it possible that I need to ask also to compile hypre with an option for 64bit indices? Is it possible to instruct this inside Petsc configure? Alternatively, is it possible to use a different multigrid PC inside PETSc that accept 64bit indices?

Thanks in advance

Pierpaolo

...

Il giorno 27 mag 2020, alle ore 11:26, Stefano Zampini <[email protected] <mailto:[email protected]>> ha scritto:

You need a version of PETSc compiled with 64bit indices, since the message indicates the number of dofs in this case is larger the INT_MAX 2501×3401×1601 = 13617947501

I also suggest you upgrade to a newer version, 3.8.3 is quite old as the error message reports

Il giorno mer 27 mag 2020 alle ore 11:50 Pierpaolo Minelli <[email protected] <mailto:[email protected]>> ha scritto: Hi,

I am trying to solve a Poisson equation on this grid:

Nx = 2501 Ny = 3401 Nz = 1601

I received this error:

[0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Overflow in integer operation: http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices <http://www.mcs.anl.gov/petsc/documentation/faq.html#64-bit-indices> [0]PETSC ERROR: Mesh of 2501 by 3401 by 1 (dof) is too large for 32 bit indices [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017 [0]PETSC ERROR: /marconi_scratch/userexternal/pminelli/PIC3D/2500_3400_1600/./PIC_3D on a arch-linux2-c-opt named r129c09s02 by pminelli Tu e May 26 20:16:34 2020 [0]PETSC ERROR: Configure options --prefix=/cineca/prod/opt/libraries/petsc/3.8.3/intelmpi--2018--binary CC=mpiicc FC=mpiifort CXX=mpiicpc F77=mpiifort F90=mpiifort --with-debugging=0 --with-blaslapack-dir=/cineca/prod/opt/compilers/intel/pe-xe-2018/binary/mkl --with-fortran=1 --with-fortran-interfaces=1 --with-cmake-dir=/cineca/prod/opt/tools/cmake/3.5.2/none --with-mpi-dir=/cineca/prod/opt/compilers/intel/pe-xe- 2018/binary/impi/2018.4.274 --download-scalapack --download-mumps=yes --download-hypre --download-superlu_dist --download-parmetis --downlo ad-metis [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 218 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/ impls/da/da3.c [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/impl s/da/dareg.c [0]PETSC ERROR: #3 DMSetUp() line 720 in /marconi/prod/build/libraries/petsc/3.8.3/intelmpi--2018--binary/BA_WORK/petsc-3.8.3/src/dm/interf ace/dm.c forrtl: error (76): Abort trap signal

I am on an HPC facility and after I loaded PETSC module, I have seen that it is configured with INTEGER size = 32

I solve my problem with these options and it works perfectly with smaller grids:

-dm_mat_type hypre -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -pc_hypre_boomeramg_coarsen_type PMIS -pc_hypre_boomeramg_interp_type FF1 -ksp_type richardson

Is it possible to overcome this if I ask them to install a version with INTEGER SIZE = 64? Alternatively, is it possible to overcome this using intel compiler options?

Thanks in advance

Pierpaolo Minelli

-- Stefano