March 2021 - petsc-dev - mailman3.cels.anl.gov

problem registering a new solver
by Mark Adams 02 Mar '21

02 Mar '21

I am trying to add a band solver to PETSc (later to be moved to Cuda and Kokkos) and I have started by adding some types and a copy of the current LU as placeholders. I register with: ierr = MatSolverTypeRegister(MATSOLVERPETSC, MATSEQAIJ, MAT_FACTOR_LUBAND,MatGetFactor_seqaij_petsc);CHKERRQ(ierr); And that is about all that I do that can have any effect at this point. I add a switch in MatGetFactor_seqaij_petsc on (ftype == MAT_FACTOR_LUBAND) to set the symbolic factorization method (*B)->ops->lufactorsymbolic = MatLUBandFactorSymbolic_SeqAIJ; But this is not called (I verified this) because I don't know how to get MatGetFactor_seqaij_petsc to receive ftype == MAT_FACTOR_LUBAND. (I need help on this to) Anyway, I would hope that these changes would not do anything but I get an error (appended). It is failing in MatSolverTypeDestroy on this second PetscFree: while (inext) { ierr = PetscFree(inext->mtype);CHKERRQ(ierr); iprev = inext; inext = inext->next; ierr = PetscFree(iprev);CHKERRQ(ierr); } I tried to clone LU here, but I clearly missed something. Any ideas? And I just made an MR for this if you want to look at the code. Thanks, Mark ... Number of SNES iterations = 2 [0]PETSC ERROR: PetscTrFreeDefault() called from MatSolverTypeDestroy() line 4513 in /Users/markadams/Codes/petsc/src/mat/interface/matrix.c [0]PETSC ERROR: Block [id=0(48)] at address 0x7fb7fb8f7620 is corrupted (probably write past end of array) [0]PETSC ERROR: Block allocated in MatSolverTypeRegister() line 4382 in /Users/markadams/Codes/petsc/src/mat/interface/matrix.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Memory corruption: https://www.mcs.anl.gov/petsc/documentation/installation.html#valgrind [0]PETSC ERROR: Corrupted memory [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.14.4-739-g575f931ef8 GIT Date: 2021-03-02 13:38:55 -0500 [0]PETSC ERROR: ./ex19 on a arch-macosx-gnu-g named MarksMac-302.local by markadams Tue Mar 2 15:28:41 2021 [0]PETSC ERROR: Configure options --with-mpi-dir=/usr/local/Cellar/mpich/3.3.2_1 COPTFLAGS="-g -O0" CXXOPTFLAGS="-g -O0" --download-metis=1 --download-parmetis=1 --download-kokkos=1 --download-kokkos-kernels=1 --download-p4est=1 --with-zlib=1 --download-superlu_dist --download-superlu --with-make-np=4 --download-hdf5=1 -with-cuda=0 --with-x=0 --with-debugging=1 PETSC_ARCH=arch-macosx-gnu-g --with-64-bit-indices=0 --with-openmp=0 --with-ctable=0 [0]PETSC ERROR: #1 PetscTrFreeDefault() line 310 in /Users/markadams/Codes/petsc/src/sys/memory/mtr.c [0]PETSC ERROR: #2 MatSolverTypeDestroy() line 4513 in /Users/markadams/Codes/petsc/src/mat/interface/matrix.c [0]PETSC ERROR: #3 MatFinalizePackage() line 57 in /Users/markadams/Codes/petsc/src/mat/interface/dlregismat.c [0]PETSC ERROR: #4 PetscRegisterFinalizeAll() line 389 in /Users/markadams/Codes/petsc/src/sys/objects/destroy.c [0]PETSC ERROR: #5 PetscFinalize() line 1474 in /Users/markadams/Codes/petsc/src/sys/objects/pinit.c

1 0

Petsc "make test" have more failures for --with-openmp=1
by Eric Chamberland 02 Mar '21

02 Mar '21

Hi, It all started when I wanted to test PETSC/CUDA compatibility for our code. I had to activate --with-openmp to configure with --with-cuda=1 successfully. I then saw that PETSC_HAVE_OPENMP is used at least in MUMPS (and some other places). So, I configured and tested petsc with openmp activated, without CUDA. The first thing I see is that our code CI pipelines now fails for many tests. After looking deeper, it seems that PETSc itself fails many tests when I activate openmp! Here are all the configurations I have results for, after/before activating OpenMP for PETSc: ============================================================================== ============================================================================== For petsc/master + OpenMPI 4.0.4 + MKL 2019.4.243: With OpenMP=1 https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.03.02.02h00m02s_m… https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.03.02.02h00m02s_c… # ------------- # Summary # ------------- # FAILED snes_tutorials-ex12_quad_hpddm_reuse_baij diff-ksp_ksp_tests-ex33_superlu_dist_2 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 ksp_ksp_tutorials-ex50_tut_2 diff-ksp_ksp_tests-ex33_superlu_dist diff-snes_tutorials-ex56_hypre snes_tutorials-ex17_3d_q3_trig_elas snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij ksp_ksp_tutorials-ex5_superlu_dist_3 ksp_ksp_tutorials-ex5f_superlu_dist snes_tutorials-ex12_tri_parmetis_hpddm_baij diff-snes_tutorials-ex19_tut_3 mat_tests-ex242_3 snes_tutorials-ex17_3d_q3_trig_vlap ksp_ksp_tutorials-ex5f_superlu_dist_3 snes_tutorials-ex19_superlu_dist diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre diff-ksp_ksp_tutorials-ex49_hypre_nullspace ts_tutorials-ex18_p1p1_xper_ref ts_tutorials-ex18_p1p1_xyper_ref snes_tutorials-ex19_superlu_dist_2 ksp_ksp_tutorials-ex5_superlu_dist_2 diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre ksp_ksp_tutorials-ex64_1 ksp_ksp_tutorials-ex5_superlu_dist ksp_ksp_tutorials-ex5f_superlu_dist_2 # success 8275/10003 tests (82.7%) #*failed 33/10003* tests (0.3%) With OpenMP=0 https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.02.26.02h00m16s_m… https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.02.26.02h00m16s_c… # ------------- # Summary # ------------- # FAILED tao_constrained_tutorials-tomographyADMM_6 snes_tutorials-ex17_3d_q3_trig_elas mat_tests-ex242_3 snes_tutorials-ex17_3d_q3_trig_vlap tao_leastsquares_tutorials-tomography_1 tao_constrained_tutorials-tomographyADMM_5 # success 8262/9983 tests (82.8%) #*failed 6/9983* tests (0.1%) ============================================================================== ============================================================================== For OpenMPI 3.1.x/master: With OpenMP=1: https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.03.01.22h00m01s_make_test.l… https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.03.01.22h00m01s_configure.l… # ------------- # Summary # ------------- # FAILED mat_tests-ex242_3 mat_tests-ex242_2 diff-mat_tests-ex219f_1 diff-dm_tutorials-ex11f90_1 ksp_ksp_tutorials-ex5_superlu_dist_3 diff-ksp_ksp_tutorials-ex49_hypre_nullspace ksp_ksp_tutorials-ex5f_superlu_dist_3 snes_tutorials-ex17_3d_q3_trig_vlap diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre diff-snes_tutorials-ex19_tut_3 diff-snes_tutorials-ex56_hypre diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre tao_leastsquares_tutorials-tomography_1 tao_constrained_tutorials-tomographyADMM_4 tao_constrained_tutorials-tomographyADMM_6 diff-tao_constrained_tutorials-toyf_1 # success 8142/9765 tests (83.4%) #*failed 16/9765* tests (0.2%) With OpenMP=0: https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.02.28.22h00m02s_make_test.l… https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.02.28.22h00m02s_configure.l… # ------------- # Summary # ------------- # FAILED mat_tests-ex242_3 mat_tests-ex242_2 diff-mat_tests-ex219f_1 diff-dm_tutorials-ex11f90_1 ksp_ksp_tutorials-ex56_2 snes_tutorials-ex17_3d_q3_trig_vlap tao_leastsquares_tutorials-tomography_1 tao_constrained_tutorials-tomographyADMM_4 diff-tao_constrained_tutorials-toyf_1 # success 8151/9767 tests (83.5%) #*failed 9/9767* tests (0.1%) ============================================================================== ============================================================================== For OpenMPI 4.0.x/master: With OpenMP=1: https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.03.01.20h00m01s_make_test.l… https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.03.01.20h00m01s_configure.l… # FAILED snes_tutorials-ex17_3d_q3_trig_elas snes_tutorials-ex19_hypre ksp_ksp_tutorials-ex56_2 tao_leastsquares_tutorials-tomography_1 tao_constrained_tutorials-tomographyADMM_5 mat_tests-ex242_3 ksp_ksp_tutorials-ex55_hypre ksp_ksp_tutorials-ex5_superlu_dist_2 tao_constrained_tutorials-tomographyADMM_6 snes_tutorials-ex56_hypre snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre ksp_ksp_tutorials-ex5f_superlu_dist_3 ksp_ksp_tutorials-ex34_hyprestruct diff-ksp_ksp_tutorials-ex49_hypre_nullspace snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre ksp_ksp_tutorials-ex5f_superlu_dist ksp_ksp_tutorials-ex5f_superlu_dist_2 ksp_ksp_tutorials-ex5_superlu_dist snes_tutorials-ex19_tut_3 snes_tutorials-ex19_superlu_dist ksp_ksp_tutorials-ex50_tut_2 snes_tutorials-ex17_3d_q3_trig_vlap ksp_ksp_tutorials-ex5_superlu_dist_3 snes_tutorials-ex19_superlu_dist_2 tao_constrained_tutorials-tomographyADMM_4 ts_tutorials-ex26_2 # success 8125/9753 tests (83.3%) #*failed 26/9753* tests (0.3%) With OpenMP=0 https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.02.28.20h00m04s_make_test.l… https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.02.28.20h00m04s_configure.l… # FAILED mat_tests-ex242_3 # success 8174/9777 tests (83.6%) #*failed 1/9777* tests (0.0%) ============================================================================== ============================================================================== Is that known and normal? In all cases, I am using MKL and I suspect it may come from there... :/ I also saw a second problem, "make test" fails to compile petsc examples on older versions of MKL (but that's less important for me, I just upgraded to OneAPI to avoid this, but you may want to know): https://giref.ulaval.ca/~cmpgiref/dernier_ompi/2021.03.02.02h16m01s_make_te… https://giref.ulaval.ca/~cmpgiref/dernier_ompi/2021.03.02.02h16m01s_configu… Thanks, Eric -- Eric Chamberland, ing., M. Ing Professionnel de recherche GIREF/Université Laval (418) 656-2131 poste 41 22 42

1 0

Re: [petsc-dev] Infinite loop in A*B
by Zhang, Hong 01 Mar '21

01 Mar '21

Pierre, I pushed a fix in branch hzhang/fix-matmatmult_aij_dense/release. https://gitlab.com/petsc/petsc/-/merge_requests/3667 [https://assets.gitlab-static.net/uploads/-/system/project/avatar/13882401/P…]<https://gitlab.com/petsc/petsc/-/merge_requests/3667> bugfix for MatMatMultSymbolic_MPIAIJ_MPIDense() when Bbn1 = 0. (!3667) · Merge Requests · PETSc / petsc<https://gitlab.com/petsc/petsc/-/merge_requests/3667> Reported-by: Pierre Jolivet pierre(a)joliv.et Bb (column block size) cannot be zero; it leads to infinite loop in MatMatMultNumeric_MPIAIJ_MPIDense() with n=0 gitlab.com Give it a try. Let me know if the bug is not fixed. Your code is very helpful in debugging. Hong ________________________________ From: Pierre Jolivet <pierre(a)joliv.et> Sent: Monday, March 1, 2021 12:51 AM To: Zhang, Hong <hzhang(a)mcs.anl.gov> Cc: For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov> Subject: Re: [petsc-dev] Infinite loop in A*B On 1 Mar 2021, at 6:29 AM, Zhang, Hong <hzhang(a)mcs.anl.gov<mailto:[email protected]>> wrote: Pierre, This is a bug in MatMatMultSymbolic_MPIAIJ_MPIDense() during optimization of block column size of B. Run your code with '-matmatmult_Bbn 1', the infinite loop should not occur. Thanks Hong, I can confirm this option makes the more complex use case run smoothly as well. I'll try to figure out a fix tomorrow. Great. Thanks, Pierre Hong ________________________________ From: Zhang, Hong <hzhang(a)mcs.anl.gov<mailto:[email protected]>> Sent: Sunday, February 28, 2021 11:05 PM To: Pierre Jolivet <pierre(a)joliv.et<mailto:[email protected]>>; For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>>; Zhang, Hong <hzhang(a)mcs.anl.gov<mailto:[email protected]>> Subject: Re: [petsc-dev] Infinite loop in A*B The infinite loop in MatMatMultNumeric_MPIAIJ_MPIDense() for (i=0; i<BN; i+=n) { } is cause by n=contents->workB->cmap->n=0 (line 590 in mpimatmatmult.c) Hong ________________________________ From: petsc-dev <petsc-dev-bounces(a)mcs.anl.gov<mailto:[email protected]>> on behalf of Zhang, Hong via petsc-dev <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> Sent: Sunday, February 28, 2021 10:33 PM To: Pierre Jolivet <pierre(a)joliv.et<mailto:[email protected]>>; For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> Subject: Re: [petsc-dev] Infinite loop in A*B I can reproduce the hang with mpiexec -n 2 ./matmatmult It seems in an infinite loop of calling MatDensePlaceArray() from #0 MatDensePlaceArray (mat=0xda5c50, array=0xd15e60) at /home/hongsu/soft/petsc/src/mat/impls/dense/mpi/mpidense.c:2047 #1 0x00007fa0d13bf4f7 in MatDenseGetSubMatrix_SeqDense (A=0xcfb2b0, cbegin=0, cend=0, v=0xd90370) at /home/hongsu/soft/petsc/src/mat/impls/dense/seq/dense.c:2997 #2 0x00007fa0d13c574e in MatDenseGetSubMatrix (A=0xcfb2b0, cbegin=0, cend=0, v=0xd90370) at /home/hongsu/soft/petsc/src/mat/impls/dense/seq/dense.c:3371 #3 0x00007fa0d13db5ce in MatDenseGetSubMatrix_MPIDense (A=0xca5250, cbegin=0, cend=0, v=0x7ffe87d41de0) at /home/hongsu/soft/petsc/src/mat/impls/dense/mpi/mpidense.c:1835 #4 0x00007fa0d13c574e in MatDenseGetSubMatrix (A=0xca5250, cbegin=0, cend=0, v=0x7ffe87d41de0) at /home/hongsu/soft/petsc/src/mat/impls/dense/seq/dense.c:3371 #5 0x00007fa0d179c2fa in MatMatMultNumeric_MPIAIJ_MPIDense (A=0xc55490, B=0xca5250, C=0xd282b0) at /home/hongsu/soft/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c:593 #6 0x00007fa0d1181331 in MatProductNumeric_AB (mat=0xd282b0) at /home/hongsu/soft/petsc/src/mat/interface/matproduct.c:567 #7 0x00007fa0d1182c14 in MatProductNumeric (mat=0xd282b0) at /home/hongsu/soft/petsc/src/mat/interface/matproduct.c:679 #8 0x00007fa0d115ef69 in MatProduct_Private (A=0xc55490, B=0xca5250, scall=MAT_INITIAL_MATRIX, fill=-2, ptype=MATPRODUCT_AB, C=0x7ffe87d42018) at /home/hongsu/soft/petsc/src/mat/interface/matrix.c:9405 ---Type <return> to continue, or q <return> to quit--- #9 0x00007fa0d115f274 in MatMatMult (A=0xc55490, B=0xca5250, scall=MAT_INITIAL_MATRIX, fill=-2, C=0x7ffe87d42018) at /home/hongsu/soft/petsc/src/mat/interface/matrix.c:9445 #10 0x000000000040130a in main (argc=2, argv=0x7ffe87d42108) at ex1.c:20 I'll try to figure out what is going on. If anyone has a clue, please help. The above stack comes from 'release' branch. Hong ________________________________ From: petsc-dev <petsc-dev-bounces(a)mcs.anl.gov<mailto:[email protected]>> on behalf of Pierre Jolivet <pierre(a)joliv.et<mailto:[email protected]>> Sent: Sunday, February 28, 2021 4:17 PM To: For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> Subject: [petsc-dev] Infinite loop in A*B Hello, The following MWE loops indefinitely for MPI_Comm_size in {2; 3}. Nothing fancy, just MatAIJ and MatDense. The problem is either in MatMPIDenseScatter() or MatMatMultSymbolic_MPIAIJ_MPIDense(), I believe, so if someone familiar with those routines can figure out a hot fix, I’m all ears. I could of course switch to a MatMult(), but the same infinite loop happens in another more complex code with A = rows=8, cols=35212 B = rows=35212, cols=9 So I’ll need a fix eventually. Thanks, Pierre

1 0

Re: [petsc-dev] Add Flags To CFLAGS In Configure
by Jacob Faibussowitsch 01 Mar '21

01 Mar '21

Alternatively have configure recognize the “+=“ operator? Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) Cell: (312) 694-3391 > On Mar 1, 2021, at 13:39, Satish Balay <balay(a)mcs.anl.gov> wrote: > > Yeah - we don't have the 'add-to-defaults' feature. Currently its either 'use defaults' or 'replace defaults' > > Perhaps we need EXTRA_CFLAGS etc interface.. > > Satish > > On Mon, 1 Mar 2021, Jacob Faibussowitsch wrote: > >> Hello All, >> >> Is there a configure option to add flags to whatever configure picks as your CFLAGS/COPTFLAGS? Something like --CFLAGS+=“-my-flag1 -my-flag2”. >> >> I often find myself wanting to add additional debug flags or warning flags, but so far the only method that as worked has been to run configure once, copy the CFLAGS, and reconfigure with my flags added to the end of the list. >> >> Best regards, >> >> Jacob Faibussowitsch >> (Jacob Fai - booss - oh - vitch) >> Cell: (312) 694-3391 >> >>

1 0

Re: [petsc-dev] Add Flags To CFLAGS In Configure
by Satish Balay 01 Mar '21

01 Mar '21

We have defaults for COPTFLAGS - and user can override them [but not add to them] Satish On Mon, 1 Mar 2021, Matthew Knepley wrote: > This is what COPTFLAGS is intended for. > > Thanks, > > matt > > On Mon, Mar 1, 2021 at 2:35 PM Jacob Faibussowitsch <jacob.fai(a)gmail.com> > wrote: > > > Hello All, > > > > Is there a configure option to add flags to whatever configure picks as > > your CFLAGS/COPTFLAGS? Something like --CFLAGS+=“-my-flag1 -my-flag2”. > > > > I often find myself wanting to add additional debug flags or warning > > flags, but so far the only method that as worked has been to run configure > > once, copy the CFLAGS, and reconfigure with my flags added to the end of > > the list. > > > > Best regards, > > > > Jacob Faibussowitsch > > (Jacob Fai - booss - oh - vitch) > > Cell: (312) 694-3391 > > > > > >

1 0

Re: [petsc-dev] Add Flags To CFLAGS In Configure
by Satish Balay 01 Mar '21

01 Mar '21

Yeah - we don't have the 'add-to-defaults' feature. Currently its either 'use defaults' or 'replace defaults' Perhaps we need EXTRA_CFLAGS etc interface.. Satish On Mon, 1 Mar 2021, Jacob Faibussowitsch wrote: > Hello All, > > Is there a configure option to add flags to whatever configure picks as your CFLAGS/COPTFLAGS? Something like --CFLAGS+=“-my-flag1 -my-flag2”. > > I often find myself wanting to add additional debug flags or warning flags, but so far the only method that as worked has been to run configure once, copy the CFLAGS, and reconfigure with my flags added to the end of the list. > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > Cell: (312) 694-3391 > >

1 0

Re: [petsc-dev] Add Flags To CFLAGS In Configure
by Matthew Knepley 01 Mar '21

01 Mar '21

This is what COPTFLAGS is intended for. Thanks, matt On Mon, Mar 1, 2021 at 2:35 PM Jacob Faibussowitsch <jacob.fai(a)gmail.com> wrote: > Hello All, > > Is there a configure option to add flags to whatever configure picks as > your CFLAGS/COPTFLAGS? Something like --CFLAGS+=“-my-flag1 -my-flag2”. > > I often find myself wanting to add additional debug flags or warning > flags, but so far the only method that as worked has been to run configure > once, copy the CFLAGS, and reconfigure with my flags added to the end of > the list. > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > Cell: (312) 694-3391 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Add Flags To CFLAGS In Configure
by Jacob Faibussowitsch 01 Mar '21

01 Mar '21

Hello All, Is there a configure option to add flags to whatever configure picks as your CFLAGS/COPTFLAGS? Something like --CFLAGS+=“-my-flag1 -my-flag2”. I often find myself wanting to add additional debug flags or warning flags, but so far the only method that as worked has been to run configure once, copy the CFLAGS, and reconfigure with my flags added to the end of the list. Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) Cell: (312) 694-3391

1 0

Re: [petsc-dev] Infinite loop in A*B
by Pierre Jolivet 01 Mar '21

01 Mar '21

> On 1 Mar 2021, at 6:29 AM, Zhang, Hong <hzhang(a)mcs.anl.gov> wrote: > > Pierre, > This is a bug in MatMatMultSymbolic_MPIAIJ_MPIDense() during optimization of block column size of B. Run your code with > '-matmatmult_Bbn 1', the infinite loop should not occur. Thanks Hong, I can confirm this option makes the more complex use case run smoothly as well. > I'll try to figure out a fix tomorrow. Great. Thanks, Pierre > Hong > From: Zhang, Hong <hzhang(a)mcs.anl.gov> > Sent: Sunday, February 28, 2021 11:05 PM > To: Pierre Jolivet <pierre(a)joliv.et>; For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov>; Zhang, Hong <hzhang(a)mcs.anl.gov> > Subject: Re: [petsc-dev] Infinite loop in A*B > > The infinite loop in MatMatMultNumeric_MPIAIJ_MPIDense() > for (i=0; i<BN; i+=n) { > } > is cause by n=contents->workB->cmap->n=0 (line 590 in mpimatmatmult.c) > Hong > From: petsc-dev <petsc-dev-bounces(a)mcs.anl.gov> on behalf of Zhang, Hong via petsc-dev <petsc-dev(a)mcs.anl.gov> > Sent: Sunday, February 28, 2021 10:33 PM > To: Pierre Jolivet <pierre(a)joliv.et>; For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov> > Subject: Re: [petsc-dev] Infinite loop in A*B > > I can reproduce the hang with > mpiexec -n 2 ./matmatmult > > It seems in an infinite loop of calling MatDensePlaceArray() from > > #0 MatDensePlaceArray (mat=0xda5c50, array=0xd15e60) > at /home/hongsu/soft/petsc/src/mat/impls/dense/mpi/mpidense.c:2047 > #1 0x00007fa0d13bf4f7 in MatDenseGetSubMatrix_SeqDense (A=0xcfb2b0, cbegin=0, > cend=0, v=0xd90370) > at /home/hongsu/soft/petsc/src/mat/impls/dense/seq/dense.c:2997 > #2 0x00007fa0d13c574e in MatDenseGetSubMatrix (A=0xcfb2b0, cbegin=0, cend=0, > v=0xd90370) at /home/hongsu/soft/petsc/src/mat/impls/dense/seq/dense.c:3371 > #3 0x00007fa0d13db5ce in MatDenseGetSubMatrix_MPIDense (A=0xca5250, cbegin=0, > cend=0, v=0x7ffe87d41de0) > at /home/hongsu/soft/petsc/src/mat/impls/dense/mpi/mpidense.c:1835 > #4 0x00007fa0d13c574e in MatDenseGetSubMatrix (A=0xca5250, cbegin=0, cend=0, > v=0x7ffe87d41de0) > at /home/hongsu/soft/petsc/src/mat/impls/dense/seq/dense.c:3371 > #5 0x00007fa0d179c2fa in MatMatMultNumeric_MPIAIJ_MPIDense (A=0xc55490, > B=0xca5250, C=0xd282b0) > at /home/hongsu/soft/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c:593 > #6 0x00007fa0d1181331 in MatProductNumeric_AB (mat=0xd282b0) > at /home/hongsu/soft/petsc/src/mat/interface/matproduct.c:567 > #7 0x00007fa0d1182c14 in MatProductNumeric (mat=0xd282b0) > at /home/hongsu/soft/petsc/src/mat/interface/matproduct.c:679 > #8 0x00007fa0d115ef69 in MatProduct_Private (A=0xc55490, B=0xca5250, > scall=MAT_INITIAL_MATRIX, fill=-2, ptype=MATPRODUCT_AB, C=0x7ffe87d42018) > at /home/hongsu/soft/petsc/src/mat/interface/matrix.c:9405 > ---Type <return> to continue, or q <return> to quit--- > #9 0x00007fa0d115f274 in MatMatMult (A=0xc55490, B=0xca5250, scall=MAT_INITIAL_MATRIX, fill=-2, > C=0x7ffe87d42018) at /home/hongsu/soft/petsc/src/mat/interface/matrix.c:9445 > #10 0x000000000040130a in main (argc=2, argv=0x7ffe87d42108) at ex1.c:20 > > I'll try to figure out what is going on. If anyone has a clue, please help. The above stack comes from 'release' branch. > Hong > From: petsc-dev <petsc-dev-bounces(a)mcs.anl.gov> on behalf of Pierre Jolivet <pierre(a)joliv.et> > Sent: Sunday, February 28, 2021 4:17 PM > To: For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov> > Subject: [petsc-dev] Infinite loop in A*B > > Hello, > The following MWE loops indefinitely for MPI_Comm_size in {2; 3}. > Nothing fancy, just MatAIJ and MatDense. > The problem is either in MatMPIDenseScatter() or MatMatMultSymbolic_MPIAIJ_MPIDense(), I believe, so if someone familiar with those routines can figure out a hot fix, I’m all ears. > I could of course switch to a MatMult(), but the same infinite loop happens in another more complex code with > A = rows=8, cols=35212 > B = rows=35212, cols=9 > So I’ll need a fix eventually. > > Thanks, > Pierre

1 0

Re: [petsc-dev] Infinite loop in A*B
by Zhang, Hong 28 Feb '21

28 Feb '21

Pierre, This is a bug in MatMatMultSymbolic_MPIAIJ_MPIDense() during optimization of block column size of B. Run your code with '-matmatmult_Bbn 1', the infinite loop should not occur. I'll try to figure out a fix tomorrow. Hong ________________________________ From: Zhang, Hong <hzhang(a)mcs.anl.gov> Sent: Sunday, February 28, 2021 11:05 PM To: Pierre Jolivet <pierre(a)joliv.et>; For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov>; Zhang, Hong <hzhang(a)mcs.anl.gov> Subject: Re: [petsc-dev] Infinite loop in A*B The infinite loop in MatMatMultNumeric_MPIAIJ_MPIDense() for (i=0; i<BN; i+=n) { } is cause by n=contents->workB->cmap->n=0 (line 590 in mpimatmatmult.c) Hong ________________________________ From: petsc-dev <petsc-dev-bounces(a)mcs.anl.gov> on behalf of Zhang, Hong via petsc-dev <petsc-dev(a)mcs.anl.gov> Sent: Sunday, February 28, 2021 10:33 PM To: Pierre Jolivet <pierre(a)joliv.et>; For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov> Subject: Re: [petsc-dev] Infinite loop in A*B I can reproduce the hang with mpiexec -n 2 ./matmatmult It seems in an infinite loop of calling MatDensePlaceArray() from #0 MatDensePlaceArray (mat=0xda5c50, array=0xd15e60) at /home/hongsu/soft/petsc/src/mat/impls/dense/mpi/mpidense.c:2047 #1 0x00007fa0d13bf4f7 in MatDenseGetSubMatrix_SeqDense (A=0xcfb2b0, cbegin=0, cend=0, v=0xd90370) at /home/hongsu/soft/petsc/src/mat/impls/dense/seq/dense.c:2997 #2 0x00007fa0d13c574e in MatDenseGetSubMatrix (A=0xcfb2b0, cbegin=0, cend=0, v=0xd90370) at /home/hongsu/soft/petsc/src/mat/impls/dense/seq/dense.c:3371 #3 0x00007fa0d13db5ce in MatDenseGetSubMatrix_MPIDense (A=0xca5250, cbegin=0, cend=0, v=0x7ffe87d41de0) at /home/hongsu/soft/petsc/src/mat/impls/dense/mpi/mpidense.c:1835 #4 0x00007fa0d13c574e in MatDenseGetSubMatrix (A=0xca5250, cbegin=0, cend=0, v=0x7ffe87d41de0) at /home/hongsu/soft/petsc/src/mat/impls/dense/seq/dense.c:3371 #5 0x00007fa0d179c2fa in MatMatMultNumeric_MPIAIJ_MPIDense (A=0xc55490, B=0xca5250, C=0xd282b0) at /home/hongsu/soft/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c:593 #6 0x00007fa0d1181331 in MatProductNumeric_AB (mat=0xd282b0) at /home/hongsu/soft/petsc/src/mat/interface/matproduct.c:567 #7 0x00007fa0d1182c14 in MatProductNumeric (mat=0xd282b0) at /home/hongsu/soft/petsc/src/mat/interface/matproduct.c:679 #8 0x00007fa0d115ef69 in MatProduct_Private (A=0xc55490, B=0xca5250, scall=MAT_INITIAL_MATRIX, fill=-2, ptype=MATPRODUCT_AB, C=0x7ffe87d42018) at /home/hongsu/soft/petsc/src/mat/interface/matrix.c:9405 ---Type <return> to continue, or q <return> to quit--- #9 0x00007fa0d115f274 in MatMatMult (A=0xc55490, B=0xca5250, scall=MAT_INITIAL_MATRIX, fill=-2, C=0x7ffe87d42018) at /home/hongsu/soft/petsc/src/mat/interface/matrix.c:9445 #10 0x000000000040130a in main (argc=2, argv=0x7ffe87d42108) at ex1.c:20 I'll try to figure out what is going on. If anyone has a clue, please help. The above stack comes from 'release' branch. Hong ________________________________ From: petsc-dev <petsc-dev-bounces(a)mcs.anl.gov> on behalf of Pierre Jolivet <pierre(a)joliv.et> Sent: Sunday, February 28, 2021 4:17 PM To: For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov> Subject: [petsc-dev] Infinite loop in A*B Hello, The following MWE loops indefinitely for MPI_Comm_size in {2; 3}. Nothing fancy, just MatAIJ and MatDense. The problem is either in MatMPIDenseScatter() or MatMatMultSymbolic_MPIAIJ_MPIDense(), I believe, so if someone familiar with those routines can figure out a hot fix, I’m all ears. I could of course switch to a MatMult(), but the same infinite loop happens in another more complex code with A = rows=8, cols=35212 B = rows=35212, cols=9 So I’ll need a fix eventually. Thanks, Pierre

1 0