Large MATMPIAIJ - 32bit integer overflow in nz value
Dear PETSc developers I am creating a very large parallel sparse matrix (MATMPIAIJ) with PETSc. I write this matrix to disk. The number of non-zeros exceeds the maximum number a 32-bit integer can hold. When I read the matrix from disk i get an error because there was an overflow in the nz number. (see petsc-3.18.4/src/mat/impls/aij/seq/aij.c:4977) Obviously I could compile PETSc with 64bit integers (--with-64-bit-indices). But I wanted to ask if there is another way. Because the total number of nonzeros nz is the only numer that exceeds the 32bit limit. It would not be efficient to use 64bit integers everywhere just because of this single number. This how I configured PETSc: ./configure --download-fblaslapack --download-hpddm --download-hypre --with-debugging=0 COPTFLAGS='-O3 -march=native -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 -march=native -mtune=native' --with-scalar-type=real (--with-mpi-dir=/home/benno/Libraries/openMPI) Kind Regards Benno ________________________________ Hochschule Luzern Technik & Architektur Institute for Mechanical Engineering and Energy Technology Competence Center Fluid Mechanics and Numerical Methods Benno Fleischli MSc in Mechanical Engineering / BSc in Electrical Engineering Wissenschaftlicher Mitarbeiter [email protected]<https://webmail.hslu.ch/owa/redir.aspx?C=rM5K5xPd_kKJmyojcJXaG18BaXM9wM8IqSslYJQen5H6HtdiNS2KtWNFL89yfRdRtW-Oa6xCXTk.&URL=mailto%3akatrina.kemp%40hslu.ch>
On Wed, May 17, 2023 at 9:02 AM Fleischli Benno HSLU T&A < [email protected]> wrote:
Dear PETSc developers
I am creating a very large parallel sparse matrix (MATMPIAIJ) with PETSc. I write this matrix to disk. The number of non-zeros exceeds the maximum number a 32-bit integer can hold. When I read the matrix from disk i get an error because there was an overflow in the nz number. (see petsc-3.18.4/src/mat/impls/aij/seq/aij.c:4977)
Obviously I could compile PETSc with 64bit integers (--with-64-bit-indices). But I wanted to ask if there is another way. Because the total number of nonzeros nz is the only numer that exceeds the 32bit limit. It would not be efficient to use 64bit integers everywhere just because of this single number.
Integers tend to be a small part of the overall storage. How much does it increase the maximum overall storage for your problem? Thanks, Matt
This how I configured PETSc:
./configure --download-fblaslapack --download-hpddm --download-hypre --with-debugging=0 COPTFLAGS='-O3 -march=native -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 -march=native -mtune=native' --with-scalar-type=real (--with-mpi-dir=/home/benno/Libraries/openMPI)
Kind Regards
Benno
________________________________
*Hochschule Luzern Technik & Architektur* Institute for Mechanical Engineering and Energy Technology
Competence Center Fluid Mechanics and Numerical Methods
*Benno Fleischli* MSc in Mechanical Engineering / BSc in Electrical Engineering
Wissenschaftlicher Mitarbeiter
[email protected] <https://webmail.hslu.ch/owa/redir.aspx?C=rM5K5xPd_kKJmyojcJXaG18BaXM9wM8IqSslYJQen5H6HtdiNS2KtWNFL89yfRdRtW-Oa6xCXTk.&URL=mailto%3akatrina.kemp%40hslu.ch>
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
Yeah, this is silly. The check is just a "sanity-check" on the data in the file. We store redundant information in the matrix header in the file, header[3] is the total number of nonzeros in the matrix. When nz is too large, the correct value cannot fit in the header. Changing the file format would be a breaking change meaning old files could no longer be read (MatLoad() can still load binary files saved in 1995) which would be bad. I think the easiest way to handle this is to provide an option to avoid the unneeded (and doesn't work) sanity check. In other words, something like -mat_load_ignore_nz turns off the sanity check in MatLoad_MPIAIJ_Binary(). Is that something you could try in an MR or would you prefer we do it? Barry
On May 17, 2023, at 8:55 AM, Fleischli Benno HSLU T&A <[email protected]> wrote:
Dear PETSc developers
I am creating a very large parallel sparse matrix (MATMPIAIJ) with PETSc. I write this matrix to disk. The number of non-zeros exceeds the maximum number a 32-bit integer can hold. When I read the matrix from disk i get an error because there was an overflow in the nz number. (see petsc-3.18.4/src/mat/impls/aij/seq/aij.c:4977)
Obviously I could compile PETSc with 64bit integers (--with-64-bit-indices). But I wanted to ask if there is another way. Because the total number of nonzeros nz is the only numer that exceeds the 32bit limit. It would not be efficient to use 64bit integers everywhere just because of this single number.
This how I configured PETSc:
./configure --download-fblaslapack --download-hpddm --download-hypre --with-debugging=0 COPTFLAGS='-O3 -march=native -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 -march=native -mtune=native' --with-scalar-type=real (--with-mpi-dir=/home/benno/Libraries/openMPI)
Kind Regards
Benno
________________________________ Hochschule Luzern Technik & Architektur Institute for Mechanical Engineering and Energy Technology Competence Center Fluid Mechanics and Numerical Methods
Benno Fleischli MSc in Mechanical Engineering / BSc in Electrical Engineering Wissenschaftlicher Mitarbeiter [email protected] <https://webmail.hslu.ch/owa/redir.aspx?C=rM5K5xPd_kKJmyojcJXaG18BaXM9wM8IqSslYJQen5H6HtdiNS2KtWNFL89yfRdRtW-Oa6xCXTk.&URL=mailto%3akatrina.kemp%40hslu.ch>
participants (3)
-
Barry Smith -
Fleischli Benno HSLU T&A -
Matthew Knepley