March 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Lawrence Mitchell 19 Mar '21

19 Mar '21

> On 19 Mar 2021, at 14:21, Jed Brown <jed(a)jedbrown.org> wrote: > > Notice how the permutations are contained within the vertices {0, ..., 8}, edges {9, ..., 24}, and cells {25, ..., 32}. I would like to get rid of that restriction, but you've said it would have significant non-local consequences so I haven't tried. What we do, and I'm not sure if this is exactly what you want, is the following. Use DMPlexGetOrdering to compute permutatations of points with RCM. As you say, this is stratified. Then, we traverse cells in this RCM ordering and compute a permutation of the full plex chart greedily. Pseudo-code: permutation = empty(pstart - pend) i = 0 for cell in reordered_cells: points = plex.getclosure(cell) for point in points: if not seen(point): permutation[i] = point i += 1 Now when we create a section, we say PetscSectionSetPermutation(sec, permutation) And then when the section offsets are calculated, the points are visited and numbered in the order provided by the permutation. This seems to work, here are the sparsity patterns for a Q3 mass matrix on an unstructured quad mesh (from gmsh). One with RCM ordering applied, the other without). I guess the only issue is now one needs to always remember to traverse cells in the reordered_cells order (although if you call DMPlexPermute that is automatic?). Cheers, Lawrence

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Jed Brown 19 Mar '21

19 Mar '21

Matthew Knepley <knepley(a)gmail.com> writes: > On Fri, Mar 19, 2021 at 7:30 AM Lawrence Mitchell <wence(a)gmx.li> wrote: > >> > On 19 Mar 2021, at 03:51, Jed Brown <jed(a)jedbrown.org> wrote: >> > >> > It's a notable weakness of DMPlex that it does not apply such an >> ordering of dofs and I've complained to Matt about it many times over the >> years, but any blame rests solely with me for not carving out time to >> implement it here. >> >> I think there is a way to do this with plex (at least we do it), like so: >> >> DMPlexGetOrdering(dm, MATORDERINGRCM, NULL, &isperm); >> >> you might need to then invert this ordering. >> >> Now when you make a section you either say >> >> PetscSectionSetPermutation(section, isperm); >> > > You can do that if you only want to permute the dofs, but usually I want to > reorder everything so that any Section comes out right > > DMPlexPermute() Last we talked, and all that's tested, this can only permute within strata, not across strata. That would incur the "basically point Jacobi" problem for naive OpenMP smoothers highlighted in Allison's paper and (what I care about more) the memory locality impact is similar to moving from interlaced fields to segregated fields. diff --git i/src/dm/impls/plex/tests/ex10.c w/src/dm/impls/plex/tests/ex10.c index eed9aa878e..cfe35dc1a6 100644 --- i/src/dm/impls/plex/tests/ex10.c +++ w/src/dm/impls/plex/tests/ex10.c @@ -80,6 +80,7 @@ PetscErrorCode TestReordering(DM dm, AppCtx *user) PetscFunctionBegin; ierr = DMPlexGetOrdering(dm, order, NULL, &perm);CHKERRQ(ierr); + ierr = ISView(perm, NULL);CHKERRQ(ierr); ierr = DMPlexPermute(dm, perm, &pdm);CHKERRQ(ierr); ierr = PetscObjectSetOptionsPrefix((PetscObject) pdm, "perm_");CHKERRQ(ierr); ierr = DMSetFromOptions(pdm);CHKERRQ(ierr); $ $PETSC_ARCH/tests/dm/impls/plex/tests/ex10 -dim 2 -num_dof 1,1,1 -perm_dm_view IS Object: 1 MPI processes type: general Number of indices in set 33 0 0 1 2 2 1 3 5 4 3 5 7 6 6 7 4 8 8 9 9 10 13 11 10 12 11 13 15 14 12 15 14 16 16 17 17 18 18 19 19 20 22 21 23 22 21 23 20 24 24 25 28 26 29 27 25 28 30 29 31 30 32 31 26 32 27 DM Object: (perm_) 1 MPI processes type: plex DM_0x564eba4a45d0_1 in 2 dimensions: 0-cells: 9 1-cells: 16 2-cells: 8 Labels: celltype: 3 strata with value/size (0 (9), 3 (8), 1 (16)) depth: 3 strata with value/size (0 (9), 1 (16), 2 (8)) marker: 1 strata with value/size (1 (16)) Face Sets: 1 strata with value/size (1 (8)) Field Field_0: adjacency FEM Ordering method rcm reduced bandwidth from 51 to 51 Notice how the permutations are contained within the vertices {0, ..., 8}, edges {9, ..., 24}, and cells {25, ..., 32}. I would like to get rid of that restriction, but you've said it would have significant non-local consequences so I haven't tried.

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Matthew Knepley 19 Mar '21

19 Mar '21

On Fri, Mar 19, 2021 at 7:30 AM Lawrence Mitchell <wence(a)gmx.li> wrote: > > On 19 Mar 2021, at 03:51, Jed Brown <jed(a)jedbrown.org> wrote: > > > > It's a notable weakness of DMPlex that it does not apply such an > ordering of dofs and I've complained to Matt about it many times over the > years, but any blame rests solely with me for not carving out time to > implement it here. > > I think there is a way to do this with plex (at least we do it), like so: > > DMPlexGetOrdering(dm, MATORDERINGRCM, NULL, &isperm); > > you might need to then invert this ordering. > > Now when you make a section you either say > > PetscSectionSetPermutation(section, isperm); > You can do that if you only want to permute the dofs, but usually I want to reorder everything so that any Section comes out right DMPlexPermute() Thanks, Matt > before PetscSectionSetup(section); > > or, if you're using DMPlexCreateSection, pass in the isperm > > This gives RCM-based ordering for the dof numbering of the section. > > Cheers, > > Lawrence -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Matthew Knepley 19 Mar '21

19 Mar '21

On Thu, Mar 18, 2021 at 11:51 PM Jed Brown <jed(a)jedbrown.org> wrote: > Note that this is specific to the node numbering, and that node numbering > tends to produce poor results even for MatMult due to poor cache reuse of > the vector. It's good practice after partitioning to use a > locality-preserving ordering of dofs on a process (e.g., RCM if you use > MatOrdering). This was shown in the PETSc-FUN3D papers circa 1999 and has > been confirmed multiple times over the years by various members of this > list (including me). I believe FEniCS and libMesh now do this by default > (or at least have an option) and it was shown to perform better. It's a > notable weakness of DMPlex that it does not apply such an ordering of dofs > and I've complained to Matt about it many times over the years, but any > blame rests solely with me for not carving out time to implement it here. > Jesus. Of course Plex can do this. It is the default for PyLith. Less complaining, more looking. Matt > Better SGS/SOR smoothing factors with simple OpenMP partitioning is an > additional bonus, though I'm not a fan of using OpenMP in this way. > > Eric Chamberland <Eric.Chamberland(a)giref.ulaval.ca> writes: > > > Hi, > > > > For the knowledge of readers, I just read section 7.3 here: > > > > > https://www.researchgate.net/publication/220411740_Multigrid_Smoothers_for_… > > > > And it is explained why multi-threading gives a poor result with the > > Hybrid−SGS smoother... > > > > Eric > > > > > > On 2021-03-15 2:50 p.m., Barry Smith wrote: > >> > >> I posted some information at the issue. > >> > >> IMHO it is likely a bug in one or more of hypre's smoothers that > >> use OpenMP. We have never tested them before (and likely hypre has not > >> tested all the combinations) and so would not have seen the bug. > >> Hopefully they can just fix it. > >> > >> Barry > >> > >> I got the problem to occur with ex56 with 2 MPI ranks and 4 OpenMP > >> threads, if I used less than 4 threads it did not generate an > >> indefinite preconditioner. > >> > >> > >>> On Mar 14, 2021, at 1:18 PM, Eric Chamberland > >>> <Eric.Chamberland(a)giref.ulaval.ca > >>> <mailto:[email protected]>> wrote: > >>> > >>> Done: > >>> > >>> https://github.com/hypre-space/hypre/issues/303 > >>> > >>> Maybe I will need some help about PETSc to answer their questions... > >>> > >>> Eric > >>> > >>> On 2021-03-14 3:44 a.m., Stefano Zampini wrote: > >>>> Eric > >>>> > >>>> You should report these HYPRE issues upstream > >>>> https://github.com/hypre-space/hypre/issues > >>>> <https://github.com/hypre-space/hypre/issues> > >>>> > >>>> > >>>>> On Mar 14, 2021, at 3:44 AM, Eric Chamberland > >>>>> <Eric.Chamberland(a)giref.ulaval.ca > >>>>> <mailto:[email protected]>> wrote: > >>>>> > >>>>> For us it clearly creates problems in real computations... > >>>>> > >>>>> I understand the need to have clean test for PETSc, but for me, it > >>>>> reveals that hypre isn't usable with more than one thread for now... > >>>>> > >>>>> Another solution: force single-threaded configuration for hypre > >>>>> until this is fixed? > >>>>> > >>>>> Eric > >>>>> > >>>>> On 2021-03-13 8:50 a.m., Pierre Jolivet wrote: > >>>>>> -pc_hypre_boomeramg_relax_type_all Jacobi => > >>>>>> Linear solve did not converge due to DIVERGED_INDEFINITE_PC > >>>>>> iterations 3 > >>>>>> -pc_hypre_boomeramg_relax_type_all l1scaled-Jacobi => > >>>>>> OK, independently of the architecture it seems (Eric Docker image > >>>>>> with 1 or 2 threads or my macOS), but contraction factor is higher > >>>>>> Linear solve converged due to CONVERGED_RTOL iterations 8 > >>>>>> Linear solve converged due to CONVERGED_RTOL iterations 24 > >>>>>> Linear solve converged due to CONVERGED_RTOL iterations 26 > >>>>>> v. currently > >>>>>> Linear solve converged due to CONVERGED_RTOL iterations 7 > >>>>>> Linear solve converged due to CONVERGED_RTOL iterations 9 > >>>>>> Linear solve converged due to CONVERGED_RTOL iterations 10 > >>>>>> > >>>>>> Do we change this? Or should we force OMP_NUM_THREADS=1 for make > test? > >>>>>> > >>>>>> Thanks, > >>>>>> Pierre > >>>>>> > >>>>>>> On 13 Mar 2021, at 2:26 PM, Mark Adams <mfadams(a)lbl.gov > >>>>>>> <mailto:[email protected]>> wrote: > >>>>>>> > >>>>>>> Hypre uses a multiplicative smoother by default. It has a > >>>>>>> chebyshev smoother. That with a Jacobi PC should be thread > >>>>>>> invariant. > >>>>>>> Mark > >>>>>>> > >>>>>>> On Sat, Mar 13, 2021 at 8:18 AM Pierre Jolivet <pierre(a)joliv.et > >>>>>>> <mailto:[email protected]>> wrote: > >>>>>>> > >>>>>>> > >>>>>>>> On 13 Mar 2021, at 9:17 AM, Pierre Jolivet <pierre(a)joliv.et > >>>>>>>> <mailto:[email protected]>> wrote: > >>>>>>>> > >>>>>>>> Hello Eric, > >>>>>>>> I’ve made an “interesting” discovery, so I’ll put back the > >>>>>>>> list in c/c. > >>>>>>>> It appears the following snippet of code which uses > >>>>>>>> Allreduce() + lambda function + MPI_IN_PLACE is: > >>>>>>>> - Valgrind-clean with MPICH; > >>>>>>>> - Valgrind-clean with OpenMPI 4.0.5; > >>>>>>>> - not Valgrind-clean with OpenMPI 4.1.0. > >>>>>>>> I’m not sure who is to blame here, I’ll need to look at the > >>>>>>>> MPI specification for what is required by the implementors > >>>>>>>> and users in that case. > >>>>>>>> > >>>>>>>> In the meantime, I’ll do the following: > >>>>>>>> - update config/BuildSystem/config/packages/OpenMPI.py to > >>>>>>>> use OpenMPI 4.1.0, see if any other error appears; > >>>>>>>> - provide a hotfix to bypass the segfaults; > >>>>>>> > >>>>>>> I can confirm that splitting the single Allreduce with my own > >>>>>>> MPI_Op into two Allreduce with MAX and BAND fixes the > >>>>>>> segfaults with OpenMPI (*). > >>>>>>> > >>>>>>>> - look at the hypre issue and whether they should be > >>>>>>>> deferred to the hypre team. > >>>>>>> > >>>>>>> I don’t know if there is something wrong in hypre threading > >>>>>>> or if it’s just a side effect of threading, but it seems that > >>>>>>> the number of threads has a drastic effect on the quality of > >>>>>>> the PC. > >>>>>>> By default, it looks that there are two threads per process > >>>>>>> with your Docker image. > >>>>>>> If I force OMP_NUM_THREADS=1, then I get the same convergence > >>>>>>> as in the output file. > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Pierre > >>>>>>> > >>>>>>> (*) https://gitlab.com/petsc/petsc/-/merge_requests/3712 > >>>>>>> <https://gitlab.com/petsc/petsc/-/merge_requests/3712> > >>>>>>> > >>>>>>>> Thank you for the Docker files, they were really useful. > >>>>>>>> If you want to avoid oversubscription failures, you can edit > >>>>>>>> the file /opt/openmpi-4.1.0/etc/openmpi-default-hostfile and > >>>>>>>> append the line: > >>>>>>>> localhost slots=12 > >>>>>>>> If you want to increase the timeout limit of PETSc test > >>>>>>>> suite for each test, you can add the extra flag in your > >>>>>>>> command line TIMEOUT=180 (default is 60, units are seconds). > >>>>>>>> > >>>>>>>> Thanks, I’ll ping you on GitLab when I’ve got something > >>>>>>>> ready for you to try, > >>>>>>>> Pierre > >>>>>>>> > >>>>>>>> <ompi.cxx> > >>>>>>>> > >>>>>>>>> On 12 Mar 2021, at 8:54 PM, Eric Chamberland > >>>>>>>>> <Eric.Chamberland(a)giref.ulaval.ca > >>>>>>>>> <mailto:[email protected]>> wrote: > >>>>>>>>> > >>>>>>>>> Hi Pierre, > >>>>>>>>> > >>>>>>>>> I now have a docker container reproducing the problems here. > >>>>>>>>> > >>>>>>>>> Actually, if I look at > >>>>>>>>> snes_tutorials-ex12_quad_singular_hpddm it fails like this: > >>>>>>>>> > >>>>>>>>> not ok snes_tutorials-ex12_quad_singular_hpddm # Error code: > 59 > >>>>>>>>> # Initial guess > >>>>>>>>> # L_2 Error: 0.00803099 > >>>>>>>>> # Initial Residual > >>>>>>>>> # L_2 Residual: 1.09057 > >>>>>>>>> # Au - b = Au + F(0) > >>>>>>>>> # Linear L_2 Residual: 1.09057 > >>>>>>>>> # [d470c54ce086:14127] Read -1, expected 4096, errno = 1 > >>>>>>>>> # [d470c54ce086:14128] Read -1, expected 4096, errno = 1 > >>>>>>>>> # [d470c54ce086:14129] Read -1, expected 4096, errno = 1 > >>>>>>>>> # [3]PETSC ERROR: > >>>>>>>>> > ------------------------------------------------------------------------ > >>>>>>>>> # [3]PETSC ERROR: Caught signal number 11 SEGV: > >>>>>>>>> Segmentation Violation, probably memory access out of range > >>>>>>>>> # [3]PETSC ERROR: Try option -start_in_debugger or > >>>>>>>>> -on_error_attach_debugger > >>>>>>>>> # [3]PETSC ERROR: or see > >>>>>>>>> > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > >>>>>>>>> < > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind> > >>>>>>>>> # [3]PETSC ERROR: or try http://valgrind.org > >>>>>>>>> <http://valgrind.org/> on GNU/linux and Apple Mac OS X to > >>>>>>>>> find memory corruption errors > >>>>>>>>> # [3]PETSC ERROR: likely location of problem given in stack > >>>>>>>>> below > >>>>>>>>> # [3]PETSC ERROR: --------------------- Stack Frames > >>>>>>>>> ------------------------------------ > >>>>>>>>> # [3]PETSC ERROR: Note: The EXACT line numbers in the stack > >>>>>>>>> are not available, > >>>>>>>>> # [3]PETSC ERROR: INSTEAD the line number of the start of > >>>>>>>>> the function > >>>>>>>>> # [3]PETSC ERROR: is given. > >>>>>>>>> # [3]PETSC ERROR: [3] buildTwo line 987 > >>>>>>>>> /opt/petsc-main/include/HPDDM_schwarz.hpp > >>>>>>>>> # [3]PETSC ERROR: [3] next line 1130 > >>>>>>>>> /opt/petsc-main/include/HPDDM_schwarz.hpp > >>>>>>>>> # [3]PETSC ERROR: --------------------- Error Message > >>>>>>>>> > -------------------------------------------------------------- > >>>>>>>>> # [3]PETSC ERROR: Signal received > >>>>>>>>> # [3]PETSC ERROR: [0]PETSC ERROR: > >>>>>>>>> > ------------------------------------------------------------------------ > >>>>>>>>> > >>>>>>>>> also ex12_quad_hpddm_reuse_baij fails with a lot more "Read > >>>>>>>>> -1, expected ..." which I don't know where they come from...? > >>>>>>>>> > >>>>>>>>> Hypre (like in diff-snes_tutorials-ex56_hypre) is also > >>>>>>>>> having DIVERGED_INDEFINITE_PC failures... > >>>>>>>>> > >>>>>>>>> Please see the 3 attached docker files: > >>>>>>>>> > >>>>>>>>> 1) fedora_mkl_and_devtools : the DockerFile which install > >>>>>>>>> fedore 33 with gnu compilers and MKL and everything to > develop. > >>>>>>>>> > >>>>>>>>> 2) openmpi: the DockerFile to bluid OpenMPI > >>>>>>>>> > >>>>>>>>> 3) petsc: The las DockerFile that build/install and test > PETSc > >>>>>>>>> > >>>>>>>>> I build the 3 like this: > >>>>>>>>> > >>>>>>>>> docker build -t fedora_mkl_and_devtools -f > >>>>>>>>> fedora_mkl_and_devtools . > >>>>>>>>> > >>>>>>>>> docker build -t openmpi -f openmpi . > >>>>>>>>> > >>>>>>>>> docker build -t petsc -f petsc . > >>>>>>>>> > >>>>>>>>> Disclaimer: I am not a docker expert, so I may do things > >>>>>>>>> that are not docker-stat-of-the-art but I am opened to > >>>>>>>>> suggestions... ;) > >>>>>>>>> > >>>>>>>>> I have just ran it on my portable (long) which have not > >>>>>>>>> enough cores, so many more tests failed (should force > >>>>>>>>> --oversubscribe but don't know how to). I will relaunch on > >>>>>>>>> my workstation in a few minutes. > >>>>>>>>> > >>>>>>>>> I will now test your branch! (sorry for the delay). > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> > >>>>>>>>> Eric > >>>>>>>>> > >>>>>>>>> On 2021-03-11 9:03 a.m., Eric Chamberland wrote: > >>>>>>>>>> > >>>>>>>>>> Hi Pierre, > >>>>>>>>>> > >>>>>>>>>> ok, that's interesting! > >>>>>>>>>> > >>>>>>>>>> I will try to build a docker image until tomorrow and give > >>>>>>>>>> you the exact recipe to reproduce the bugs. > >>>>>>>>>> > >>>>>>>>>> Eric > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On 2021-03-11 2:46 a.m., Pierre Jolivet wrote: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> On 11 Mar 2021, at 6:16 AM, Barry Smith > >>>>>>>>>>>> <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Eric, > >>>>>>>>>>>> > >>>>>>>>>>>> Sorry about not being more immediate. We still have > >>>>>>>>>>>> this in our active email so you don't need to submit > >>>>>>>>>>>> individual issues. We'll try to get to them as soon as > >>>>>>>>>>>> we can. > >>>>>>>>>>> > >>>>>>>>>>> Indeed, I’m still trying to figure this out. > >>>>>>>>>>> I realized that some of my configure flags were different > >>>>>>>>>>> than yours, e.g., no --with-memalign. > >>>>>>>>>>> I’ve also added SuperLU_DIST to my installation. > >>>>>>>>>>> Still, I can’t reproduce any issue. > >>>>>>>>>>> I will continue looking into this, it appears I’m seeing > >>>>>>>>>>> some valgrind errors, but I don’t know if this is some > >>>>>>>>>>> side effect of OpenMPI not being valgrind-clean (last > >>>>>>>>>>> time I checked, there was no error with MPICH). > >>>>>>>>>>> > >>>>>>>>>>> Thank you for your patience, > >>>>>>>>>>> Pierre > >>>>>>>>>>> > >>>>>>>>>>> /usr/bin/gmake -f gmakefile test test-fail=1 > >>>>>>>>>>> Using MAKEFLAGS: test-fail=1 > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_baij.counts > >>>>>>>>>>> ok snes_tutorials-ex12_quad_hpddm_reuse_baij > >>>>>>>>>>> ok diff-snes_tutorials-ex12_quad_hpddm_reuse_baij > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist_2.counts > >>>>>>>>>>> ok ksp_ksp_tests-ex33_superlu_dist_2 > >>>>>>>>>>> ok diff-ksp_ksp_tests-ex33_superlu_dist_2 > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex49_superlu_dist.counts > >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 > >>>>>>>>>>> ok > diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 > >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 > >>>>>>>>>>> ok > diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 > >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 > >>>>>>>>>>> ok > diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 > >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 > >>>>>>>>>>> ok > diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 > >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 > >>>>>>>>>>> ok > diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 > >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 > >>>>>>>>>>> ok > diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 > >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 > >>>>>>>>>>> ok > diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 > >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 > >>>>>>>>>>> ok > diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex50_tut_2.counts > >>>>>>>>>>> ok ksp_ksp_tutorials-ex50_tut_2 > >>>>>>>>>>> ok diff-ksp_ksp_tutorials-ex50_tut_2 > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist.counts > >>>>>>>>>>> ok ksp_ksp_tests-ex33_superlu_dist > >>>>>>>>>>> ok diff-ksp_ksp_tests-ex33_superlu_dist > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_hypre.counts > >>>>>>>>>>> ok snes_tutorials-ex56_hypre > >>>>>>>>>>> ok diff-snes_tutorials-ex56_hypre > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex56_2.counts > >>>>>>>>>>> ok ksp_ksp_tutorials-ex56_2 > >>>>>>>>>>> ok diff-ksp_ksp_tutorials-ex56_2 > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_elas.counts > >>>>>>>>>>> ok snes_tutorials-ex17_3d_q3_trig_elas > >>>>>>>>>>> ok diff-snes_tutorials-ex17_3d_q3_trig_elas > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij.counts > >>>>>>>>>>> ok snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij > >>>>>>>>>>> ok > diff-snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist_3.counts > >>>>>>>>>>> not ok ksp_ksp_tutorials-ex5_superlu_dist_3 # Error code: 1 > >>>>>>>>>>> #srun: error: Unable to create step for job 1426755: More > >>>>>>>>>>> processors requested than permitted > >>>>>>>>>>> ok ksp_ksp_tutorials-ex5_superlu_dist_3 # SKIP Command > >>>>>>>>>>> failed so no diff > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist.counts > >>>>>>>>>>> ok ksp_ksp_tutorials-ex5f_superlu_dist # SKIP Fortran > >>>>>>>>>>> required for this test > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_tri_parmetis_hpddm_baij.counts > >>>>>>>>>>> ok snes_tutorials-ex12_tri_parmetis_hpddm_baij > >>>>>>>>>>> ok diff-snes_tutorials-ex12_tri_parmetis_hpddm_baij > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_tut_3.counts > >>>>>>>>>>> ok snes_tutorials-ex19_tut_3 > >>>>>>>>>>> ok diff-snes_tutorials-ex19_tut_3 > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_vlap.counts > >>>>>>>>>>> ok snes_tutorials-ex17_3d_q3_trig_vlap > >>>>>>>>>>> ok diff-snes_tutorials-ex17_3d_q3_trig_vlap > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist_3.counts > >>>>>>>>>>> ok ksp_ksp_tutorials-ex5f_superlu_dist_3 # SKIP Fortran > >>>>>>>>>>> required for this test > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_superlu_dist.counts > >>>>>>>>>>> ok snes_tutorials-ex19_superlu_dist > >>>>>>>>>>> ok diff-snes_tutorials-ex19_superlu_dist > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre.counts > >>>>>>>>>>> ok > >>>>>>>>>>> > snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre > >>>>>>>>>>> ok > >>>>>>>>>>> > diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex49_hypre_nullspace.counts > >>>>>>>>>>> ok ksp_ksp_tutorials-ex49_hypre_nullspace > >>>>>>>>>>> ok diff-ksp_ksp_tutorials-ex49_hypre_nullspace > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_superlu_dist_2.counts > >>>>>>>>>>> ok snes_tutorials-ex19_superlu_dist_2 > >>>>>>>>>>> ok diff-snes_tutorials-ex19_superlu_dist_2 > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist_2.counts > >>>>>>>>>>> not ok ksp_ksp_tutorials-ex5_superlu_dist_2 # Error code: 1 > >>>>>>>>>>> #srun: error: Unable to create step for job 1426755: More > >>>>>>>>>>> processors requested than permitted > >>>>>>>>>>> ok ksp_ksp_tutorials-ex5_superlu_dist_2 # SKIP Command > >>>>>>>>>>> failed so no diff > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre.counts > >>>>>>>>>>> ok > >>>>>>>>>>> > snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre > >>>>>>>>>>> ok > >>>>>>>>>>> > diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex64_1.counts > >>>>>>>>>>> ok ksp_ksp_tutorials-ex64_1 > >>>>>>>>>>> ok diff-ksp_ksp_tutorials-ex64_1 > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist.counts > >>>>>>>>>>> not ok ksp_ksp_tutorials-ex5_superlu_dist # Error code: 1 > >>>>>>>>>>> #srun: error: Unable to create step for job 1426755: More > >>>>>>>>>>> processors requested than permitted > >>>>>>>>>>> ok ksp_ksp_tutorials-ex5_superlu_dist # SKIP Command > >>>>>>>>>>> failed so no diff > >>>>>>>>>>> TEST > >>>>>>>>>>> > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist_2.counts > >>>>>>>>>>> ok ksp_ksp_tutorials-ex5f_superlu_dist_2 # SKIP Fortran > >>>>>>>>>>> required for this test > >>>>>>>>>>> > >>>>>>>>>>>> Barry > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> On Mar 10, 2021, at 11:03 PM, Eric Chamberland > >>>>>>>>>>>>> <Eric.Chamberland(a)giref.ulaval.ca > >>>>>>>>>>>>> <mailto:[email protected]>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Barry, > >>>>>>>>>>>>> > >>>>>>>>>>>>> to get a some follow up on --with-openmp=1 failures, > >>>>>>>>>>>>> shall I open gitlab issues for: > >>>>>>>>>>>>> > >>>>>>>>>>>>> a) all hypre failures giving DIVERGED_INDEFINITE_PC > >>>>>>>>>>>>> > >>>>>>>>>>>>> b) all superlu_dist failures giving different results > >>>>>>>>>>>>> with initia and "Exceeded timeout limit of 60 s" > >>>>>>>>>>>>> > >>>>>>>>>>>>> c) hpddm failures "free(): invalid next size (fast)" > >>>>>>>>>>>>> and "Segmentation Violation" > >>>>>>>>>>>>> > >>>>>>>>>>>>> d) all tao's "Exceeded timeout limit of 60 s" > >>>>>>>>>>>>> > >>>>>>>>>>>>> I don't see how I could do all these debugging by > myself... > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>> > >>>>>>>>>>>>> Eric > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> Eric Chamberland, ing., M. Ing > >>>>>>>>>> Professionnel de recherche > >>>>>>>>>> GIREF/Université Laval > >>>>>>>>>> (418) 656-2131 poste 41 22 42 > >>>>>>>>> -- > >>>>>>>>> Eric Chamberland, ing., M. Ing > >>>>>>>>> Professionnel de recherche > >>>>>>>>> GIREF/Université Laval > >>>>>>>>> (418) 656-2131 poste 41 22 42 > >>>>>>>>> <fedora_mkl_and_devtools.txt><openmpi.txt><petsc.txt> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> -- > >>>>> Eric Chamberland, ing., M. Ing > >>>>> Professionnel de recherche > >>>>> GIREF/Université Laval > >>>>> (418) 656-2131 poste 41 22 42 > >>>> > >>> -- > >>> Eric Chamberland, ing., M. Ing > >>> Professionnel de recherche > >>> GIREF/Université Laval > >>> (418) 656-2131 poste 41 22 42 > >> > > -- > > Eric Chamberland, ing., M. Ing > > Professionnel de recherche > > GIREF/Université Laval > > (418) 656-2131 poste 41 22 42 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Pierre Jolivet 19 Mar '21

19 Mar '21

There is a fix there: https://gitlab.com/petsc/petsc/-/commit/ea41da7ae9ca74379a24f049c09adf372dd… <https://gitlab.com/petsc/petsc/-/commit/ea41da7ae9ca74379a24f049c09adf372dd…> It’s a spin-off of https://gitlab.com/petsc/petsc/-/commit/83f9b43b15ff2b6a8a4ff6970cc325feb78… <https://gitlab.com/petsc/petsc/-/commit/83f9b43b15ff2b6a8a4ff6970cc325feb78…> I don’t think it’s OK to have different behaviors between PCASM/PCGASM/PCBJacobi_Multiblock on one side and PCBJacobi_Singleblock on the other. I’m also not sure what was the rationale for setting the same prefix for the SubMats and the Mat, as it makes it very hard to scope options properly, I think. I can make a MR if you are OK with this change. Thanks, Pierre $ mpirun -n 1 src/ksp/ksp/tests/ex60 -ksp_view -pc_type bjacobi […] Mat Object: (sub_) 1 MPI processes type: seqaij rows=64, cols=64 […] $ mpirun -n 4 src/snes/tutorials/ex56 -cells 2,2,1 -max_conv_its 2 -petscspace_degree 2 -snes_max_it 2 -ksp_max_it 100 -ksp_type cg -ksp_rtol 1.e-10 -ksp_norm_type unpreconditioned -snes_rtol 1.e-10 -pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1 -pc_gamg_coarse_eq_limit 10 -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1 -pc_gamg_threshold 0.05 -pc_gamg_threshold_scale .0 -ksp_converged_reason -snes_monitor_short -ksp_monitor_short -snes_converged_reason -use_mat_nearnullspace true -mg_levels_ksp_max_it 2 -mg_levels_ksp_type chebyshev -mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.1 -mg_levels_pc_type jacobi -petscpartitioner_type simple -matptap_via scalable -ex56_dm_view -snes_view -mg_coarse_sub_pc_factor_mat_solver_type mumps -mg_coarse_sub_mat_mumps_icntl_11 1 […] ICNTL(11) (error analysis): 1 […] linear system matrix = precond matrix: Mat Object: (mg_coarse_sub_) 1 MPI processes type: seqaij rows=6, cols=6, bs=6 […] > On 19 Mar 2021, at 12:01 PM, Pierre Jolivet <pierre(a)joliv.et> wrote: > > > >> On 19 Mar 2021, at 5:00 AM, Barry Smith <bsmith(a)petsc.dev> wrote: >> >> >> Eric, >> >>> -Options_ProjectionL2_0mg_coarse_sub_mat_mumps_icntl_24 value: 1 >> >> If an option is skipped it is often due to the exact string name used with the option. I see your KSP option is Options_ProjectionL2_0mg_coarse_sub_mat_mumps_icntl_24 but then below I see >> >>> Coarse grid solver -- level ------------------------------- >>> KSP Object: (Options_ProjectionL2_0mg_coarse_) 1 MPI processes >>> type: preonly >> >> That is, it seems to be looking for an option without the sub. This is normal. For the coarsest level of multigrid it uses coarse otherwise it uses levels. Sub is used for block methods in PETSc such as block Jacobi methods but that doesn't seem to apply in your run with one process. It is possible since the run is on one process without a method such as block Jacobi the sub is just not relevant. > > There is something wrong with the option, IMHO, Eric is right in thinking that he should prepend sub_. > The prefix is not being propagated properly, I’ll investigate. > Here is a simple reproducer: > $ mpirun -n 1 src/ksp/ksp/tests/ex60 -ksp_view -pc_type bjacobi // KO > […] > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > […] > $ mpirun -n 1 src/ksp/ksp/tests/ex60 -ksp_view -pc_type asm // OK > […] > linear system matrix = precond matrix: > Mat Object: (sub_) 1 MPI processes > type: seqaij > […] > $ mpirun -n 1 src/ksp/ksp/tests/ex60 -ksp_view -pc_type gasm // OK > […] > linear system matrix = precond matrix: > Mat Object: (sub_) 1 MPI processes > type: seqaij > […] > $ mpirun -n 4 src/ksp/ksp/tests/ex60 -ksp_view -pc_type bjacobi -pc_bjacobi_blocks 1 // OK > […] > linear system matrix = precond matrix: > Mat Object: (sub_) 4 MPI processes > type: mpiaij > […] > > Eric, in the meantime, you can just put the MUMPS options in the global scope, i.e., -mat_mumps_icntl_24 1, but this will apply to all unprefixed MUMPS instances. > > Thanks, > Pierre > >> You can run any code with your current options and with -help and then grep for particular options that you may wish to add. Or you can run with -ts/snes/ksp_view to see the option prefixes needed for each inner solve. >> >> I am not sure how to make the code bullet proof in your situation. ideally it would explain why your options don't work but I am not sure if that is possible. >> >> Barry >> >> >> >> >> >>> On Mar 18, 2021, at 8:46 PM, Eric Chamberland <Eric.Chamberland(a)giref.ulaval.ca> wrote: >>> >>> Hi again, >>> >>> ok, just saw that some matrices have lines of "0" in case of 3D hermite DOFs (ex: du/dz derivatives) when used into a 2D plane mesh... >>> >>> So, my last problem about hypre smoother is "normal". >>> >>> However, just to play with one of this matrix, I tried to do a "LU" with mumps icntl_24 option activated on the global system: fine it works. >>> >>> Then I tried to switche to GAMG with mumps for the coarse_sub level, but it seems my icntl_24 option is then ignored and I don't know why... >>> >>> See my KSP: >>> >>> KSP Object: (Options_ProjectionL2_0) 1 MPI processes >>> type: bcgs >>> maximum iterations=10000, initial guess is zero >>> tolerances: relative=1e-15, absolute=1e-15, divergence=1e+12 >>> left preconditioning >>> using PRECONDITIONED norm type for convergence test >>> PC Object: (Options_ProjectionL2_0) 1 MPI processes >>> type: gamg >>> type is MULTIPLICATIVE, levels=2 cycles=v >>> Cycles per PCApply=1 >>> Using externally compute Galerkin coarse grid matrices >>> GAMG specific options >>> Threshold for dropping small values in graph on each level = >>> Threshold scaling factor for each level not specified = 1. >>> AGG specific options >>> Symmetric graph false >>> Number of levels to square graph 1 >>> Number smoothing steps 1 >>> Complexity: grid = 1.09756 >>> Coarse grid solver -- level ------------------------------- >>> KSP Object: (Options_ProjectionL2_0mg_coarse_) 1 MPI processes >>> type: preonly >>> maximum iterations=10000, initial guess is zero >>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000. >>> left preconditioning >>> using NONE norm type for convergence test >>> PC Object: (Options_ProjectionL2_0mg_coarse_) 1 MPI processes >>> type: bjacobi >>> number of blocks = 1 >>> Local solver is the same for all blocks, as in the following KSP and PC objects on rank 0: >>> KSP Object: (Options_ProjectionL2_0mg_coarse_sub_) 1 MPI processes >>> type: preonly >>> maximum iterations=1, initial guess is zero >>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000. >>> left preconditioning >>> using NONE norm type for convergence test >>> PC Object: (Options_ProjectionL2_0mg_coarse_sub_) 1 MPI processes >>> type: lu >>> out-of-place factorization >>> tolerance for zero pivot 2.22045e-14 >>> using diagonal shift on blocks to prevent zero pivot [INBLOCKS] >>> matrix ordering: nd >>> factor fill ratio given 0., needed 0. >>> Factored matrix follows: >>> Mat Object: 1 MPI processes >>> type: mumps >>> rows=8, cols=8 >>> package used to perform factorization: mumps >>> total: nonzeros=64, allocated nonzeros=64 >>> MUMPS run parameters: >>> SYM (matrix type): 0 >>> PAR (host participation): 1 >>> ICNTL(1) (output for error): 6 >>> ICNTL(2) (output of diagnostic msg): 0 >>> ICNTL(3) (output for global info): 0 >>> ICNTL(4) (level of printing): 0 >>> ICNTL(5) (input mat struct): 0 >>> ICNTL(6) (matrix prescaling): 7 >>> ICNTL(7) (sequential matrix ordering):7 >>> ICNTL(8) (scaling strategy): 77 >>> ICNTL(10) (max num of refinements): 0 >>> ICNTL(11) (error analysis): 0 >>> ICNTL(12) (efficiency control): 1 >>> ICNTL(13) (sequential factorization of the root node): 0 >>> ICNTL(14) (percentage of estimated workspace increase): 20 >>> ICNTL(18) (input mat struct): 0 >>> ICNTL(19) (Schur complement info): 0 >>> ICNTL(20) (RHS sparse pattern): 0 >>> ICNTL(21) (solution struct): 0 >>> ICNTL(22) (in-core/out-of-core facility): 0 >>> ICNTL(23) (max size of memory can be allocated locally):0 >>> ICNTL(24) (detection of null pivot rows): 0 >>> ICNTL(25) (computation of a null space basis): 0 >>> ICNTL(26) (Schur options for RHS or solution): 0 >>> ICNTL(27) (blocking size for multiple RHS): -32 >>> ICNTL(28) (use parallel or sequential ordering): 1 >>> ICNTL(29) (parallel ordering): 0 >>> ICNTL(30) (user-specified set of entries in inv(A)): 0 >>> ICNTL(31) (factors is discarded in the solve phase): 0 >>> ICNTL(33) (compute determinant): 0 >>> ICNTL(35) (activate BLR based factorization): 0 >>> ICNTL(36) (choice of BLR factorization variant): 0 >>> ICNTL(38) (estimated compression rate of LU factors): 333 >>> CNTL(1) (relative pivoting threshold): 0.01 >>> CNTL(2) (stopping criterion of refinement): 1.49012e-08 >>> CNTL(3) (absolute pivoting threshold): 0. >>> CNTL(4) (value of static pivoting): -1. >>> CNTL(5) (fixation for null pivots): 0. >>> CNTL(7) (dropping parameter for BLR): 0. >>> RINFO(1) (local estimated flops for the elimination after analysis): >>> [0] 308. >>> RINFO(2) (local estimated flops for the assembly after factorization): >>> [0] 0. >>> RINFO(3) (local estimated flops for the elimination after factorization): >>> [0] 0. >>> INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): >>> [0] 0 >>> INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): >>> [0] 0 >>> INFO(23) (num of pivots eliminated on this processor after factorization): >>> [0] 6 >>> RINFOG(1) (global estimated flops for the elimination after analysis): 308. >>> RINFOG(2) (global estimated flops for the assembly after factorization): 0. >>> RINFOG(3) (global estimated flops for the elimination after factorization): 0. >>> (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0.,0.)*(2^0) >>> INFOG(3) (estimated real workspace for factors on all processors after analysis): 64 >>> INFOG(4) (estimated integer workspace for factors on all processors after analysis): 35 >>> INFOG(5) (estimated maximum front size in the complete tree): 8 >>> INFOG(6) (number of nodes in the complete tree): 1 >>> INFOG(7) (ordering option effectively use after analysis): 2 >>> INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 100 >>> INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 64 >>> INFOG(10) (total integer space store the matrix factors after factorization): 35 >>> INFOG(11) (order of largest frontal matrix after factorization): 8 >>> INFOG(12) (number of off-diagonal pivots): 0 >>> INFOG(13) (number of delayed pivots after factorization): 0 >>> INFOG(14) (number of memory compress after factorization): 0 >>> INFOG(15) (number of steps of iterative refinement after solution): 0 >>> INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 0 >>> INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 0 >>> INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 0 >>> INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 0 >>> INFOG(20) (estimated number of entries in the factors): 64 >>> INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 0 >>> INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 0 >>> INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 >>> INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 >>> INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 >>> INFOG(28) (after factorization: number of null pivots encountered): 0 >>> INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 0 >>> INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 0, 0 >>> INFOG(32) (after analysis: type of analysis done): 1 >>> INFOG(33) (value used for ICNTL(8)): 7 >>> INFOG(34) (exponent of the determinant if determinant is requested): 0 >>> INFOG(35) (after factorization: number of entries taking into account BLR factor compression - sum over all processors): 0 >>> INFOG(36) (after analysis: estimated size of all MUMPS internal data for running BLR in-core - value on the most memory consuming processor): 0 >>> INFOG(37) (after analysis: estimated size of all MUMPS internal data for running BLR in-core - sum over all processors): 0 >>> INFOG(38) (after analysis: estimated size of all MUMPS internal data for running BLR out-of-core - value on the most memory consuming processor): 0 >>> INFOG(39) (after analysis: estimated size of all MUMPS internal data for running BLR out-of-core - sum over all processors): 0 >>> linear system matrix = precond matrix: >>> Mat Object: 1 MPI processes >>> type: seqaij >>> rows=8, cols=8, bs=4 >>> total: nonzeros=64, allocated nonzeros=64 >>> total number of mallocs used during MatSetValues calls=0 >>> using I-node routines: found 2 nodes, limit used is 5 >>> linear system matrix = precond matrix: >>> Mat Object: 1 MPI processes >>> type: seqaij >>> rows=8, cols=8, bs=4 >>> total: nonzeros=64, allocated nonzeros=64 >>> total number of mallocs used during MatSetValues calls=0 >>> using I-node routines: found 2 nodes, limit used is 5 >>> Down solver (pre-smoother) on level 1 ------------------------------- >>> KSP Object: (Options_ProjectionL2_0mg_levels_1_) 1 MPI processes >>> type: chebyshev >>> eigenvalue estimates used: min = 0., max = 0. >>> eigenvalues estimate via gmres min 0., max 0. >>> eigenvalues estimated using gmres with translations [0. 0.1; 0. 1.1] >>> KSP Object: (Options_ProjectionL2_0mg_levels_1_esteig_) 1 MPI processes >>> type: gmres >>> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement >>> happy breakdown tolerance 1e-30 >>> maximum iterations=10, initial guess is zero >>> tolerances: relative=1e-12, absolute=1e-50, divergence=10000. >>> left preconditioning >>> using PRECONDITIONED norm type for convergence test >>> PC Object: (Options_ProjectionL2_0mg_levels_1_) 1 MPI processes >>> type: sor >>> type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. >>> linear system matrix = precond matrix: >>> Mat Object: (Options_ProjectionL2_0) 1 MPI processes >>> type: seqaij >>> rows=36, cols=36, bs=4 >>> total: nonzeros=656, allocated nonzeros=656 >>> total number of mallocs used during MatSetValues calls=0 >>> using I-node routines: found 9 nodes, limit used is 5 >>> estimating eigenvalues using noisy right hand side >>> maximum iterations=2, nonzero initial guess >>> tolerances: relative=1e-05, absolute=1e-50, divergence=10000. >>> left preconditioning >>> using NONE norm type for convergence test >>> PC Object: (Options_ProjectionL2_0mg_levels_1_) 1 MPI processes >>> type: sor >>> type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. >>> linear system matrix = precond matrix: >>> Mat Object: (Options_ProjectionL2_0) 1 MPI processes >>> type: seqaij >>> rows=36, cols=36, bs=4 >>> total: nonzeros=656, allocated nonzeros=656 >>> total number of mallocs used during MatSetValues calls=0 >>> using I-node routines: found 9 nodes, limit used is 5 >>> Up solver (post-smoother) same as down solver (pre-smoother) >>> linear system matrix = precond matrix: >>> Mat Object: (Options_ProjectionL2_0) 1 MPI processes >>> type: seqaij >>> rows=36, cols=36, bs=4 >>> total: nonzeros=656, allocated nonzeros=656 >>> total number of mallocs used during MatSetValues calls=0 >>> using I-node routines: found 9 nodes, limit used is 5 >>> >>> but I have this option left: >>> >>> Option left: name:-Options_ProjectionL2_0mg_coarse_sub_mat_mumps_icntl_24 value: 1 >>> >>> and as you can see above I end with: >>> >>> ICNTL(24) (detection of null pivot rows): 0 >>> >>> which is fatal in my case... >>> >>> Can you see where I did wrong? >>> >>> Thanks, >>> >>> Eric

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Lawrence Mitchell 19 Mar '21

19 Mar '21

> On 19 Mar 2021, at 03:51, Jed Brown <jed(a)jedbrown.org> wrote: > > It's a notable weakness of DMPlex that it does not apply such an ordering of dofs and I've complained to Matt about it many times over the years, but any blame rests solely with me for not carving out time to implement it here. I think there is a way to do this with plex (at least we do it), like so: DMPlexGetOrdering(dm, MATORDERINGRCM, NULL, &isperm); you might need to then invert this ordering. Now when you make a section you either say PetscSectionSetPermutation(section, isperm); before PetscSectionSetup(section); or, if you're using DMPlexCreateSection, pass in the isperm This gives RCM-based ordering for the dof numbering of the section. Cheers, Lawrence

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Pierre Jolivet 19 Mar '21

19 Mar '21

> On 19 Mar 2021, at 5:00 AM, Barry Smith <bsmith(a)petsc.dev> wrote: > > > Eric, > >> -Options_ProjectionL2_0mg_coarse_sub_mat_mumps_icntl_24 value: 1 > > If an option is skipped it is often due to the exact string name used with the option. I see your KSP option is Options_ProjectionL2_0mg_coarse_sub_mat_mumps_icntl_24 but then below I see > >> Coarse grid solver -- level ------------------------------- >> KSP Object: (Options_ProjectionL2_0mg_coarse_) 1 MPI processes >> type: preonly > > That is, it seems to be looking for an option without the sub. This is normal. For the coarsest level of multigrid it uses coarse otherwise it uses levels. Sub is used for block methods in PETSc such as block Jacobi methods but that doesn't seem to apply in your run with one process. It is possible since the run is on one process without a method such as block Jacobi the sub is just not relevant. There is something wrong with the option, IMHO, Eric is right in thinking that he should prepend sub_. The prefix is not being propagated properly, I’ll investigate. Here is a simple reproducer: $ mpirun -n 1 src/ksp/ksp/tests/ex60 -ksp_view -pc_type bjacobi // KO […] linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij […] $ mpirun -n 1 src/ksp/ksp/tests/ex60 -ksp_view -pc_type asm // OK […] linear system matrix = precond matrix: Mat Object: (sub_) 1 MPI processes type: seqaij […] $ mpirun -n 1 src/ksp/ksp/tests/ex60 -ksp_view -pc_type gasm // OK […] linear system matrix = precond matrix: Mat Object: (sub_) 1 MPI processes type: seqaij […] $ mpirun -n 4 src/ksp/ksp/tests/ex60 -ksp_view -pc_type bjacobi -pc_bjacobi_blocks 1 // OK […] linear system matrix = precond matrix: Mat Object: (sub_) 4 MPI processes type: mpiaij […] Eric, in the meantime, you can just put the MUMPS options in the global scope, i.e., -mat_mumps_icntl_24 1, but this will apply to all unprefixed MUMPS instances. Thanks, Pierre > You can run any code with your current options and with -help and then grep for particular options that you may wish to add. Or you can run with -ts/snes/ksp_view to see the option prefixes needed for each inner solve. > > I am not sure how to make the code bullet proof in your situation. ideally it would explain why your options don't work but I am not sure if that is possible. > > Barry > > > > > >> On Mar 18, 2021, at 8:46 PM, Eric Chamberland <Eric.Chamberland(a)giref.ulaval.ca> wrote: >> >> Hi again, >> >> ok, just saw that some matrices have lines of "0" in case of 3D hermite DOFs (ex: du/dz derivatives) when used into a 2D plane mesh... >> >> So, my last problem about hypre smoother is "normal". >> >> However, just to play with one of this matrix, I tried to do a "LU" with mumps icntl_24 option activated on the global system: fine it works. >> >> Then I tried to switche to GAMG with mumps for the coarse_sub level, but it seems my icntl_24 option is then ignored and I don't know why... >> >> See my KSP: >> >> KSP Object: (Options_ProjectionL2_0) 1 MPI processes >> type: bcgs >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-15, absolute=1e-15, divergence=1e+12 >> left preconditioning >> using PRECONDITIONED norm type for convergence test >> PC Object: (Options_ProjectionL2_0) 1 MPI processes >> type: gamg >> type is MULTIPLICATIVE, levels=2 cycles=v >> Cycles per PCApply=1 >> Using externally compute Galerkin coarse grid matrices >> GAMG specific options >> Threshold for dropping small values in graph on each level = >> Threshold scaling factor for each level not specified = 1. >> AGG specific options >> Symmetric graph false >> Number of levels to square graph 1 >> Number smoothing steps 1 >> Complexity: grid = 1.09756 >> Coarse grid solver -- level ------------------------------- >> KSP Object: (Options_ProjectionL2_0mg_coarse_) 1 MPI processes >> type: preonly >> maximum iterations=10000, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000. >> left preconditioning >> using NONE norm type for convergence test >> PC Object: (Options_ProjectionL2_0mg_coarse_) 1 MPI processes >> type: bjacobi >> number of blocks = 1 >> Local solver is the same for all blocks, as in the following KSP and PC objects on rank 0: >> KSP Object: (Options_ProjectionL2_0mg_coarse_sub_) 1 MPI processes >> type: preonly >> maximum iterations=1, initial guess is zero >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000. >> left preconditioning >> using NONE norm type for convergence test >> PC Object: (Options_ProjectionL2_0mg_coarse_sub_) 1 MPI processes >> type: lu >> out-of-place factorization >> tolerance for zero pivot 2.22045e-14 >> using diagonal shift on blocks to prevent zero pivot [INBLOCKS] >> matrix ordering: nd >> factor fill ratio given 0., needed 0. >> Factored matrix follows: >> Mat Object: 1 MPI processes >> type: mumps >> rows=8, cols=8 >> package used to perform factorization: mumps >> total: nonzeros=64, allocated nonzeros=64 >> MUMPS run parameters: >> SYM (matrix type): 0 >> PAR (host participation): 1 >> ICNTL(1) (output for error): 6 >> ICNTL(2) (output of diagnostic msg): 0 >> ICNTL(3) (output for global info): 0 >> ICNTL(4) (level of printing): 0 >> ICNTL(5) (input mat struct): 0 >> ICNTL(6) (matrix prescaling): 7 >> ICNTL(7) (sequential matrix ordering):7 >> ICNTL(8) (scaling strategy): 77 >> ICNTL(10) (max num of refinements): 0 >> ICNTL(11) (error analysis): 0 >> ICNTL(12) (efficiency control): 1 >> ICNTL(13) (sequential factorization of the root node): 0 >> ICNTL(14) (percentage of estimated workspace increase): 20 >> ICNTL(18) (input mat struct): 0 >> ICNTL(19) (Schur complement info): 0 >> ICNTL(20) (RHS sparse pattern): 0 >> ICNTL(21) (solution struct): 0 >> ICNTL(22) (in-core/out-of-core facility): 0 >> ICNTL(23) (max size of memory can be allocated locally):0 >> ICNTL(24) (detection of null pivot rows): 0 >> ICNTL(25) (computation of a null space basis): 0 >> ICNTL(26) (Schur options for RHS or solution): 0 >> ICNTL(27) (blocking size for multiple RHS): -32 >> ICNTL(28) (use parallel or sequential ordering): 1 >> ICNTL(29) (parallel ordering): 0 >> ICNTL(30) (user-specified set of entries in inv(A)): 0 >> ICNTL(31) (factors is discarded in the solve phase): 0 >> ICNTL(33) (compute determinant): 0 >> ICNTL(35) (activate BLR based factorization): 0 >> ICNTL(36) (choice of BLR factorization variant): 0 >> ICNTL(38) (estimated compression rate of LU factors): 333 >> CNTL(1) (relative pivoting threshold): 0.01 >> CNTL(2) (stopping criterion of refinement): 1.49012e-08 >> CNTL(3) (absolute pivoting threshold): 0. >> CNTL(4) (value of static pivoting): -1. >> CNTL(5) (fixation for null pivots): 0. >> CNTL(7) (dropping parameter for BLR): 0. >> RINFO(1) (local estimated flops for the elimination after analysis): >> [0] 308. >> RINFO(2) (local estimated flops for the assembly after factorization): >> [0] 0. >> RINFO(3) (local estimated flops for the elimination after factorization): >> [0] 0. >> INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): >> [0] 0 >> INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): >> [0] 0 >> INFO(23) (num of pivots eliminated on this processor after factorization): >> [0] 6 >> RINFOG(1) (global estimated flops for the elimination after analysis): 308. >> RINFOG(2) (global estimated flops for the assembly after factorization): 0. >> RINFOG(3) (global estimated flops for the elimination after factorization): 0. >> (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0.,0.)*(2^0) >> INFOG(3) (estimated real workspace for factors on all processors after analysis): 64 >> INFOG(4) (estimated integer workspace for factors on all processors after analysis): 35 >> INFOG(5) (estimated maximum front size in the complete tree): 8 >> INFOG(6) (number of nodes in the complete tree): 1 >> INFOG(7) (ordering option effectively use after analysis): 2 >> INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 100 >> INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 64 >> INFOG(10) (total integer space store the matrix factors after factorization): 35 >> INFOG(11) (order of largest frontal matrix after factorization): 8 >> INFOG(12) (number of off-diagonal pivots): 0 >> INFOG(13) (number of delayed pivots after factorization): 0 >> INFOG(14) (number of memory compress after factorization): 0 >> INFOG(15) (number of steps of iterative refinement after solution): 0 >> INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 0 >> INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 0 >> INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 0 >> INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 0 >> INFOG(20) (estimated number of entries in the factors): 64 >> INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 0 >> INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 0 >> INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 >> INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 >> INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 >> INFOG(28) (after factorization: number of null pivots encountered): 0 >> INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 0 >> INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 0, 0 >> INFOG(32) (after analysis: type of analysis done): 1 >> INFOG(33) (value used for ICNTL(8)): 7 >> INFOG(34) (exponent of the determinant if determinant is requested): 0 >> INFOG(35) (after factorization: number of entries taking into account BLR factor compression - sum over all processors): 0 >> INFOG(36) (after analysis: estimated size of all MUMPS internal data for running BLR in-core - value on the most memory consuming processor): 0 >> INFOG(37) (after analysis: estimated size of all MUMPS internal data for running BLR in-core - sum over all processors): 0 >> INFOG(38) (after analysis: estimated size of all MUMPS internal data for running BLR out-of-core - value on the most memory consuming processor): 0 >> INFOG(39) (after analysis: estimated size of all MUMPS internal data for running BLR out-of-core - sum over all processors): 0 >> linear system matrix = precond matrix: >> Mat Object: 1 MPI processes >> type: seqaij >> rows=8, cols=8, bs=4 >> total: nonzeros=64, allocated nonzeros=64 >> total number of mallocs used during MatSetValues calls=0 >> using I-node routines: found 2 nodes, limit used is 5 >> linear system matrix = precond matrix: >> Mat Object: 1 MPI processes >> type: seqaij >> rows=8, cols=8, bs=4 >> total: nonzeros=64, allocated nonzeros=64 >> total number of mallocs used during MatSetValues calls=0 >> using I-node routines: found 2 nodes, limit used is 5 >> Down solver (pre-smoother) on level 1 ------------------------------- >> KSP Object: (Options_ProjectionL2_0mg_levels_1_) 1 MPI processes >> type: chebyshev >> eigenvalue estimates used: min = 0., max = 0. >> eigenvalues estimate via gmres min 0., max 0. >> eigenvalues estimated using gmres with translations [0. 0.1; 0. 1.1] >> KSP Object: (Options_ProjectionL2_0mg_levels_1_esteig_) 1 MPI processes >> type: gmres >> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement >> happy breakdown tolerance 1e-30 >> maximum iterations=10, initial guess is zero >> tolerances: relative=1e-12, absolute=1e-50, divergence=10000. >> left preconditioning >> using PRECONDITIONED norm type for convergence test >> PC Object: (Options_ProjectionL2_0mg_levels_1_) 1 MPI processes >> type: sor >> type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. >> linear system matrix = precond matrix: >> Mat Object: (Options_ProjectionL2_0) 1 MPI processes >> type: seqaij >> rows=36, cols=36, bs=4 >> total: nonzeros=656, allocated nonzeros=656 >> total number of mallocs used during MatSetValues calls=0 >> using I-node routines: found 9 nodes, limit used is 5 >> estimating eigenvalues using noisy right hand side >> maximum iterations=2, nonzero initial guess >> tolerances: relative=1e-05, absolute=1e-50, divergence=10000. >> left preconditioning >> using NONE norm type for convergence test >> PC Object: (Options_ProjectionL2_0mg_levels_1_) 1 MPI processes >> type: sor >> type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. >> linear system matrix = precond matrix: >> Mat Object: (Options_ProjectionL2_0) 1 MPI processes >> type: seqaij >> rows=36, cols=36, bs=4 >> total: nonzeros=656, allocated nonzeros=656 >> total number of mallocs used during MatSetValues calls=0 >> using I-node routines: found 9 nodes, limit used is 5 >> Up solver (post-smoother) same as down solver (pre-smoother) >> linear system matrix = precond matrix: >> Mat Object: (Options_ProjectionL2_0) 1 MPI processes >> type: seqaij >> rows=36, cols=36, bs=4 >> total: nonzeros=656, allocated nonzeros=656 >> total number of mallocs used during MatSetValues calls=0 >> using I-node routines: found 9 nodes, limit used is 5 >> >> but I have this option left: >> >> Option left: name:-Options_ProjectionL2_0mg_coarse_sub_mat_mumps_icntl_24 value: 1 >> >> and as you can see above I end with: >> >> ICNTL(24) (detection of null pivot rows): 0 >> >> which is fatal in my case... >> >> Can you see where I did wrong? >> >> Thanks, >> >> Eric >> >> >

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Barry Smith 18 Mar '21

18 Mar '21

Eric, > -Options_ProjectionL2_0mg_coarse_sub_mat_mumps_icntl_24 value: 1 If an option is skipped it is often due to the exact string name used with the option. I see your KSP option is Options_ProjectionL2_0mg_coarse_sub_mat_mumps_icntl_24 but then below I see > Coarse grid solver -- level ------------------------------- > KSP Object: (Options_ProjectionL2_0mg_coarse_) 1 MPI processes > type: preonly That is, it seems to be looking for an option without the sub. This is normal. For the coarsest level of multigrid it uses coarse otherwise it uses levels. Sub is used for block methods in PETSc such as block Jacobi methods but that doesn't seem to apply in your run with one process. It is possible since the run is on one process without a method such as block Jacobi the sub is just not relevant. You can run any code with your current options and with -help and then grep for particular options that you may wish to add. Or you can run with -ts/snes/ksp_view to see the option prefixes needed for each inner solve. I am not sure how to make the code bullet proof in your situation. ideally it would explain why your options don't work but I am not sure if that is possible. Barry > On Mar 18, 2021, at 8:46 PM, Eric Chamberland <Eric.Chamberland(a)giref.ulaval.ca> wrote: > > Hi again, > > ok, just saw that some matrices have lines of "0" in case of 3D hermite DOFs (ex: du/dz derivatives) when used into a 2D plane mesh... > > So, my last problem about hypre smoother is "normal". > > However, just to play with one of this matrix, I tried to do a "LU" with mumps icntl_24 option activated on the global system: fine it works. > > Then I tried to switche to GAMG with mumps for the coarse_sub level, but it seems my icntl_24 option is then ignored and I don't know why... > > See my KSP: > > KSP Object: (Options_ProjectionL2_0) 1 MPI processes > type: bcgs > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-15, absolute=1e-15, divergence=1e+12 > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: (Options_ProjectionL2_0) 1 MPI processes > type: gamg > type is MULTIPLICATIVE, levels=2 cycles=v > Cycles per PCApply=1 > Using externally compute Galerkin coarse grid matrices > GAMG specific options > Threshold for dropping small values in graph on each level = > Threshold scaling factor for each level not specified = 1. > AGG specific options > Symmetric graph false > Number of levels to square graph 1 > Number smoothing steps 1 > Complexity: grid = 1.09756 > Coarse grid solver -- level ------------------------------- > KSP Object: (Options_ProjectionL2_0mg_coarse_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (Options_ProjectionL2_0mg_coarse_) 1 MPI processes > type: bjacobi > number of blocks = 1 > Local solver is the same for all blocks, as in the following KSP and PC objects on rank 0: > KSP Object: (Options_ProjectionL2_0mg_coarse_sub_) 1 MPI processes > type: preonly > maximum iterations=1, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (Options_ProjectionL2_0mg_coarse_sub_) 1 MPI processes > type: lu > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > using diagonal shift on blocks to prevent zero pivot [INBLOCKS] > matrix ordering: nd > factor fill ratio given 0., needed 0. > Factored matrix follows: > Mat Object: 1 MPI processes > type: mumps > rows=8, cols=8 > package used to perform factorization: mumps > total: nonzeros=64, allocated nonzeros=64 > MUMPS run parameters: > SYM (matrix type): 0 > PAR (host participation): 1 > ICNTL(1) (output for error): 6 > ICNTL(2) (output of diagnostic msg): 0 > ICNTL(3) (output for global info): 0 > ICNTL(4) (level of printing): 0 > ICNTL(5) (input mat struct): 0 > ICNTL(6) (matrix prescaling): 7 > ICNTL(7) (sequential matrix ordering):7 > ICNTL(8) (scaling strategy): 77 > ICNTL(10) (max num of refinements): 0 > ICNTL(11) (error analysis): 0 > ICNTL(12) (efficiency control): 1 > ICNTL(13) (sequential factorization of the root node): 0 > ICNTL(14) (percentage of estimated workspace increase): 20 > ICNTL(18) (input mat struct): 0 > ICNTL(19) (Schur complement info): 0 > ICNTL(20) (RHS sparse pattern): 0 > ICNTL(21) (solution struct): 0 > ICNTL(22) (in-core/out-of-core facility): 0 > ICNTL(23) (max size of memory can be allocated locally):0 > ICNTL(24) (detection of null pivot rows): 0 > ICNTL(25) (computation of a null space basis): 0 > ICNTL(26) (Schur options for RHS or solution): 0 > ICNTL(27) (blocking size for multiple RHS): -32 > ICNTL(28) (use parallel or sequential ordering): 1 > ICNTL(29) (parallel ordering): 0 > ICNTL(30) (user-specified set of entries in inv(A)): 0 > ICNTL(31) (factors is discarded in the solve phase): 0 > ICNTL(33) (compute determinant): 0 > ICNTL(35) (activate BLR based factorization): 0 > ICNTL(36) (choice of BLR factorization variant): 0 > ICNTL(38) (estimated compression rate of LU factors): 333 > CNTL(1) (relative pivoting threshold): 0.01 > CNTL(2) (stopping criterion of refinement): 1.49012e-08 > CNTL(3) (absolute pivoting threshold): 0. > CNTL(4) (value of static pivoting): -1. > CNTL(5) (fixation for null pivots): 0. > CNTL(7) (dropping parameter for BLR): 0. > RINFO(1) (local estimated flops for the elimination after analysis): > [0] 308. > RINFO(2) (local estimated flops for the assembly after factorization): > [0] 0. > RINFO(3) (local estimated flops for the elimination after factorization): > [0] 0. > INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): > [0] 0 > INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): > [0] 0 > INFO(23) (num of pivots eliminated on this processor after factorization): > [0] 6 > RINFOG(1) (global estimated flops for the elimination after analysis): 308. > RINFOG(2) (global estimated flops for the assembly after factorization): 0. > RINFOG(3) (global estimated flops for the elimination after factorization): 0. > (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0.,0.)*(2^0) > INFOG(3) (estimated real workspace for factors on all processors after analysis): 64 > INFOG(4) (estimated integer workspace for factors on all processors after analysis): 35 > INFOG(5) (estimated maximum front size in the complete tree): 8 > INFOG(6) (number of nodes in the complete tree): 1 > INFOG(7) (ordering option effectively use after analysis): 2 > INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 100 > INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 64 > INFOG(10) (total integer space store the matrix factors after factorization): 35 > INFOG(11) (order of largest frontal matrix after factorization): 8 > INFOG(12) (number of off-diagonal pivots): 0 > INFOG(13) (number of delayed pivots after factorization): 0 > INFOG(14) (number of memory compress after factorization): 0 > INFOG(15) (number of steps of iterative refinement after solution): 0 > INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 0 > INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 0 > INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 0 > INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 0 > INFOG(20) (estimated number of entries in the factors): 64 > INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 0 > INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 0 > INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 > INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 > INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 > INFOG(28) (after factorization: number of null pivots encountered): 0 > INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 0 > INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 0, 0 > INFOG(32) (after analysis: type of analysis done): 1 > INFOG(33) (value used for ICNTL(8)): 7 > INFOG(34) (exponent of the determinant if determinant is requested): 0 > INFOG(35) (after factorization: number of entries taking into account BLR factor compression - sum over all processors): 0 > INFOG(36) (after analysis: estimated size of all MUMPS internal data for running BLR in-core - value on the most memory consuming processor): 0 > INFOG(37) (after analysis: estimated size of all MUMPS internal data for running BLR in-core - sum over all processors): 0 > INFOG(38) (after analysis: estimated size of all MUMPS internal data for running BLR out-of-core - value on the most memory consuming processor): 0 > INFOG(39) (after analysis: estimated size of all MUMPS internal data for running BLR out-of-core - sum over all processors): 0 > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=8, cols=8, bs=4 > total: nonzeros=64, allocated nonzeros=64 > total number of mallocs used during MatSetValues calls=0 > using I-node routines: found 2 nodes, limit used is 5 > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=8, cols=8, bs=4 > total: nonzeros=64, allocated nonzeros=64 > total number of mallocs used during MatSetValues calls=0 > using I-node routines: found 2 nodes, limit used is 5 > Down solver (pre-smoother) on level 1 ------------------------------- > KSP Object: (Options_ProjectionL2_0mg_levels_1_) 1 MPI processes > type: chebyshev > eigenvalue estimates used: min = 0., max = 0. > eigenvalues estimate via gmres min 0., max 0. > eigenvalues estimated using gmres with translations [0. 0.1; 0. 1.1] > KSP Object: (Options_ProjectionL2_0mg_levels_1_esteig_) 1 MPI processes > type: gmres > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement > happy breakdown tolerance 1e-30 > maximum iterations=10, initial guess is zero > tolerances: relative=1e-12, absolute=1e-50, divergence=10000. > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: (Options_ProjectionL2_0mg_levels_1_) 1 MPI processes > type: sor > type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. > linear system matrix = precond matrix: > Mat Object: (Options_ProjectionL2_0) 1 MPI processes > type: seqaij > rows=36, cols=36, bs=4 > total: nonzeros=656, allocated nonzeros=656 > total number of mallocs used during MatSetValues calls=0 > using I-node routines: found 9 nodes, limit used is 5 > estimating eigenvalues using noisy right hand side > maximum iterations=2, nonzero initial guess > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (Options_ProjectionL2_0mg_levels_1_) 1 MPI processes > type: sor > type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. > linear system matrix = precond matrix: > Mat Object: (Options_ProjectionL2_0) 1 MPI processes > type: seqaij > rows=36, cols=36, bs=4 > total: nonzeros=656, allocated nonzeros=656 > total number of mallocs used during MatSetValues calls=0 > using I-node routines: found 9 nodes, limit used is 5 > Up solver (post-smoother) same as down solver (pre-smoother) > linear system matrix = precond matrix: > Mat Object: (Options_ProjectionL2_0) 1 MPI processes > type: seqaij > rows=36, cols=36, bs=4 > total: nonzeros=656, allocated nonzeros=656 > total number of mallocs used during MatSetValues calls=0 > using I-node routines: found 9 nodes, limit used is 5 > > but I have this option left: > > Option left: name:-Options_ProjectionL2_0mg_coarse_sub_mat_mumps_icntl_24 value: 1 > > and as you can see above I end with: > > ICNTL(24) (detection of null pivot rows): 0 > > which is fatal in my case... > > Can you see where I did wrong? > > Thanks, > > Eric > >

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Jed Brown 18 Mar '21

18 Mar '21

Note that this is specific to the node numbering, and that node numbering tends to produce poor results even for MatMult due to poor cache reuse of the vector. It's good practice after partitioning to use a locality-preserving ordering of dofs on a process (e.g., RCM if you use MatOrdering). This was shown in the PETSc-FUN3D papers circa 1999 and has been confirmed multiple times over the years by various members of this list (including me). I believe FEniCS and libMesh now do this by default (or at least have an option) and it was shown to perform better. It's a notable weakness of DMPlex that it does not apply such an ordering of dofs and I've complained to Matt about it many times over the years, but any blame rests solely with me for not carving out time to implement it here. Better SGS/SOR smoothing factors with simple OpenMP partitioning is an additional bonus, though I'm not a fan of using OpenMP in this way. Eric Chamberland <Eric.Chamberland(a)giref.ulaval.ca> writes: > Hi, > > For the knowledge of readers, I just read section 7.3 here: > > https://www.researchgate.net/publication/220411740_Multigrid_Smoothers_for_… > > And it is explained why multi-threading gives a poor result with the > Hybrid−SGS smoother... > > Eric > > > On 2021-03-15 2:50 p.m., Barry Smith wrote: >> >> I posted some information at the issue. >> >> IMHO it is likely a bug in one or more of hypre's smoothers that >> use OpenMP. We have never tested them before (and likely hypre has not >> tested all the combinations) and so would not have seen the bug. >> Hopefully they can just fix it. >> >> Barry >> >> I got the problem to occur with ex56 with 2 MPI ranks and 4 OpenMP >> threads, if I used less than 4 threads it did not generate an >> indefinite preconditioner. >> >> >>> On Mar 14, 2021, at 1:18 PM, Eric Chamberland >>> <Eric.Chamberland(a)giref.ulaval.ca >>> <mailto:[email protected]>> wrote: >>> >>> Done: >>> >>> https://github.com/hypre-space/hypre/issues/303 >>> >>> Maybe I will need some help about PETSc to answer their questions... >>> >>> Eric >>> >>> On 2021-03-14 3:44 a.m., Stefano Zampini wrote: >>>> Eric >>>> >>>> You should report these HYPRE issues upstream >>>> https://github.com/hypre-space/hypre/issues >>>> <https://github.com/hypre-space/hypre/issues> >>>> >>>> >>>>> On Mar 14, 2021, at 3:44 AM, Eric Chamberland >>>>> <Eric.Chamberland(a)giref.ulaval.ca >>>>> <mailto:[email protected]>> wrote: >>>>> >>>>> For us it clearly creates problems in real computations... >>>>> >>>>> I understand the need to have clean test for PETSc, but for me, it >>>>> reveals that hypre isn't usable with more than one thread for now... >>>>> >>>>> Another solution: force single-threaded configuration for hypre >>>>> until this is fixed? >>>>> >>>>> Eric >>>>> >>>>> On 2021-03-13 8:50 a.m., Pierre Jolivet wrote: >>>>>> -pc_hypre_boomeramg_relax_type_all Jacobi => >>>>>> Linear solve did not converge due to DIVERGED_INDEFINITE_PC >>>>>> iterations 3 >>>>>> -pc_hypre_boomeramg_relax_type_all l1scaled-Jacobi => >>>>>> OK, independently of the architecture it seems (Eric Docker image >>>>>> with 1 or 2 threads or my macOS), but contraction factor is higher >>>>>> Linear solve converged due to CONVERGED_RTOL iterations 8 >>>>>> Linear solve converged due to CONVERGED_RTOL iterations 24 >>>>>> Linear solve converged due to CONVERGED_RTOL iterations 26 >>>>>> v. currently >>>>>> Linear solve converged due to CONVERGED_RTOL iterations 7 >>>>>> Linear solve converged due to CONVERGED_RTOL iterations 9 >>>>>> Linear solve converged due to CONVERGED_RTOL iterations 10 >>>>>> >>>>>> Do we change this? Or should we force OMP_NUM_THREADS=1 for make test? >>>>>> >>>>>> Thanks, >>>>>> Pierre >>>>>> >>>>>>> On 13 Mar 2021, at 2:26 PM, Mark Adams <mfadams(a)lbl.gov >>>>>>> <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> Hypre uses a multiplicative smoother by default. It has a >>>>>>> chebyshev smoother. That with a Jacobi PC should be thread >>>>>>> invariant. >>>>>>> Mark >>>>>>> >>>>>>> On Sat, Mar 13, 2021 at 8:18 AM Pierre Jolivet <pierre(a)joliv.et >>>>>>> <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> >>>>>>>> On 13 Mar 2021, at 9:17 AM, Pierre Jolivet <pierre(a)joliv.et >>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>> >>>>>>>> Hello Eric, >>>>>>>> I’ve made an “interesting” discovery, so I’ll put back the >>>>>>>> list in c/c. >>>>>>>> It appears the following snippet of code which uses >>>>>>>> Allreduce() + lambda function + MPI_IN_PLACE is: >>>>>>>> - Valgrind-clean with MPICH; >>>>>>>> - Valgrind-clean with OpenMPI 4.0.5; >>>>>>>> - not Valgrind-clean with OpenMPI 4.1.0. >>>>>>>> I’m not sure who is to blame here, I’ll need to look at the >>>>>>>> MPI specification for what is required by the implementors >>>>>>>> and users in that case. >>>>>>>> >>>>>>>> In the meantime, I’ll do the following: >>>>>>>> - update config/BuildSystem/config/packages/OpenMPI.py to >>>>>>>> use OpenMPI 4.1.0, see if any other error appears; >>>>>>>> - provide a hotfix to bypass the segfaults; >>>>>>> >>>>>>> I can confirm that splitting the single Allreduce with my own >>>>>>> MPI_Op into two Allreduce with MAX and BAND fixes the >>>>>>> segfaults with OpenMPI (*). >>>>>>> >>>>>>>> - look at the hypre issue and whether they should be >>>>>>>> deferred to the hypre team. >>>>>>> >>>>>>> I don’t know if there is something wrong in hypre threading >>>>>>> or if it’s just a side effect of threading, but it seems that >>>>>>> the number of threads has a drastic effect on the quality of >>>>>>> the PC. >>>>>>> By default, it looks that there are two threads per process >>>>>>> with your Docker image. >>>>>>> If I force OMP_NUM_THREADS=1, then I get the same convergence >>>>>>> as in the output file. >>>>>>> >>>>>>> Thanks, >>>>>>> Pierre >>>>>>> >>>>>>> (*) https://gitlab.com/petsc/petsc/-/merge_requests/3712 >>>>>>> <https://gitlab.com/petsc/petsc/-/merge_requests/3712> >>>>>>> >>>>>>>> Thank you for the Docker files, they were really useful. >>>>>>>> If you want to avoid oversubscription failures, you can edit >>>>>>>> the file /opt/openmpi-4.1.0/etc/openmpi-default-hostfile and >>>>>>>> append the line: >>>>>>>> localhost slots=12 >>>>>>>> If you want to increase the timeout limit of PETSc test >>>>>>>> suite for each test, you can add the extra flag in your >>>>>>>> command line TIMEOUT=180 (default is 60, units are seconds). >>>>>>>> >>>>>>>> Thanks, I’ll ping you on GitLab when I’ve got something >>>>>>>> ready for you to try, >>>>>>>> Pierre >>>>>>>> >>>>>>>> <ompi.cxx> >>>>>>>> >>>>>>>>> On 12 Mar 2021, at 8:54 PM, Eric Chamberland >>>>>>>>> <Eric.Chamberland(a)giref.ulaval.ca >>>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>>> >>>>>>>>> Hi Pierre, >>>>>>>>> >>>>>>>>> I now have a docker container reproducing the problems here. >>>>>>>>> >>>>>>>>> Actually, if I look at >>>>>>>>> snes_tutorials-ex12_quad_singular_hpddm it fails like this: >>>>>>>>> >>>>>>>>> not ok snes_tutorials-ex12_quad_singular_hpddm # Error code: 59 >>>>>>>>> # Initial guess >>>>>>>>> # L_2 Error: 0.00803099 >>>>>>>>> # Initial Residual >>>>>>>>> # L_2 Residual: 1.09057 >>>>>>>>> # Au - b = Au + F(0) >>>>>>>>> # Linear L_2 Residual: 1.09057 >>>>>>>>> # [d470c54ce086:14127] Read -1, expected 4096, errno = 1 >>>>>>>>> # [d470c54ce086:14128] Read -1, expected 4096, errno = 1 >>>>>>>>> # [d470c54ce086:14129] Read -1, expected 4096, errno = 1 >>>>>>>>> # [3]PETSC ERROR: >>>>>>>>> ------------------------------------------------------------------------ >>>>>>>>> # [3]PETSC ERROR: Caught signal number 11 SEGV: >>>>>>>>> Segmentation Violation, probably memory access out of range >>>>>>>>> # [3]PETSC ERROR: Try option -start_in_debugger or >>>>>>>>> -on_error_attach_debugger >>>>>>>>> # [3]PETSC ERROR: or see >>>>>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>>>>>>>> <https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind> >>>>>>>>> # [3]PETSC ERROR: or try http://valgrind.org >>>>>>>>> <http://valgrind.org/> on GNU/linux and Apple Mac OS X to >>>>>>>>> find memory corruption errors >>>>>>>>> # [3]PETSC ERROR: likely location of problem given in stack >>>>>>>>> below >>>>>>>>> # [3]PETSC ERROR: --------------------- Stack Frames >>>>>>>>> ------------------------------------ >>>>>>>>> # [3]PETSC ERROR: Note: The EXACT line numbers in the stack >>>>>>>>> are not available, >>>>>>>>> # [3]PETSC ERROR: INSTEAD the line number of the start of >>>>>>>>> the function >>>>>>>>> # [3]PETSC ERROR: is given. >>>>>>>>> # [3]PETSC ERROR: [3] buildTwo line 987 >>>>>>>>> /opt/petsc-main/include/HPDDM_schwarz.hpp >>>>>>>>> # [3]PETSC ERROR: [3] next line 1130 >>>>>>>>> /opt/petsc-main/include/HPDDM_schwarz.hpp >>>>>>>>> # [3]PETSC ERROR: --------------------- Error Message >>>>>>>>> -------------------------------------------------------------- >>>>>>>>> # [3]PETSC ERROR: Signal received >>>>>>>>> # [3]PETSC ERROR: [0]PETSC ERROR: >>>>>>>>> ------------------------------------------------------------------------ >>>>>>>>> >>>>>>>>> also ex12_quad_hpddm_reuse_baij fails with a lot more "Read >>>>>>>>> -1, expected ..." which I don't know where they come from...? >>>>>>>>> >>>>>>>>> Hypre (like in diff-snes_tutorials-ex56_hypre) is also >>>>>>>>> having DIVERGED_INDEFINITE_PC failures... >>>>>>>>> >>>>>>>>> Please see the 3 attached docker files: >>>>>>>>> >>>>>>>>> 1) fedora_mkl_and_devtools : the DockerFile which install >>>>>>>>> fedore 33 with gnu compilers and MKL and everything to develop. >>>>>>>>> >>>>>>>>> 2) openmpi: the DockerFile to bluid OpenMPI >>>>>>>>> >>>>>>>>> 3) petsc: The las DockerFile that build/install and test PETSc >>>>>>>>> >>>>>>>>> I build the 3 like this: >>>>>>>>> >>>>>>>>> docker build -t fedora_mkl_and_devtools -f >>>>>>>>> fedora_mkl_and_devtools . >>>>>>>>> >>>>>>>>> docker build -t openmpi -f openmpi . >>>>>>>>> >>>>>>>>> docker build -t petsc -f petsc . >>>>>>>>> >>>>>>>>> Disclaimer: I am not a docker expert, so I may do things >>>>>>>>> that are not docker-stat-of-the-art but I am opened to >>>>>>>>> suggestions... ;) >>>>>>>>> >>>>>>>>> I have just ran it on my portable (long) which have not >>>>>>>>> enough cores, so many more tests failed (should force >>>>>>>>> --oversubscribe but don't know how to). I will relaunch on >>>>>>>>> my workstation in a few minutes. >>>>>>>>> >>>>>>>>> I will now test your branch! (sorry for the delay). >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Eric >>>>>>>>> >>>>>>>>> On 2021-03-11 9:03 a.m., Eric Chamberland wrote: >>>>>>>>>> >>>>>>>>>> Hi Pierre, >>>>>>>>>> >>>>>>>>>> ok, that's interesting! >>>>>>>>>> >>>>>>>>>> I will try to build a docker image until tomorrow and give >>>>>>>>>> you the exact recipe to reproduce the bugs. >>>>>>>>>> >>>>>>>>>> Eric >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 2021-03-11 2:46 a.m., Pierre Jolivet wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On 11 Mar 2021, at 6:16 AM, Barry Smith >>>>>>>>>>>> <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Eric, >>>>>>>>>>>> >>>>>>>>>>>> Sorry about not being more immediate. We still have >>>>>>>>>>>> this in our active email so you don't need to submit >>>>>>>>>>>> individual issues. We'll try to get to them as soon as >>>>>>>>>>>> we can. >>>>>>>>>>> >>>>>>>>>>> Indeed, I’m still trying to figure this out. >>>>>>>>>>> I realized that some of my configure flags were different >>>>>>>>>>> than yours, e.g., no --with-memalign. >>>>>>>>>>> I’ve also added SuperLU_DIST to my installation. >>>>>>>>>>> Still, I can’t reproduce any issue. >>>>>>>>>>> I will continue looking into this, it appears I’m seeing >>>>>>>>>>> some valgrind errors, but I don’t know if this is some >>>>>>>>>>> side effect of OpenMPI not being valgrind-clean (last >>>>>>>>>>> time I checked, there was no error with MPICH). >>>>>>>>>>> >>>>>>>>>>> Thank you for your patience, >>>>>>>>>>> Pierre >>>>>>>>>>> >>>>>>>>>>> /usr/bin/gmake -f gmakefile test test-fail=1 >>>>>>>>>>> Using MAKEFLAGS: test-fail=1 >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_baij.counts >>>>>>>>>>> ok snes_tutorials-ex12_quad_hpddm_reuse_baij >>>>>>>>>>> ok diff-snes_tutorials-ex12_quad_hpddm_reuse_baij >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist_2.counts >>>>>>>>>>> ok ksp_ksp_tests-ex33_superlu_dist_2 >>>>>>>>>>> ok diff-ksp_ksp_tests-ex33_superlu_dist_2 >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex49_superlu_dist.counts >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 >>>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 >>>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 >>>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 >>>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 >>>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 >>>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 >>>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 >>>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex50_tut_2.counts >>>>>>>>>>> ok ksp_ksp_tutorials-ex50_tut_2 >>>>>>>>>>> ok diff-ksp_ksp_tutorials-ex50_tut_2 >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist.counts >>>>>>>>>>> ok ksp_ksp_tests-ex33_superlu_dist >>>>>>>>>>> ok diff-ksp_ksp_tests-ex33_superlu_dist >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_hypre.counts >>>>>>>>>>> ok snes_tutorials-ex56_hypre >>>>>>>>>>> ok diff-snes_tutorials-ex56_hypre >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex56_2.counts >>>>>>>>>>> ok ksp_ksp_tutorials-ex56_2 >>>>>>>>>>> ok diff-ksp_ksp_tutorials-ex56_2 >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_elas.counts >>>>>>>>>>> ok snes_tutorials-ex17_3d_q3_trig_elas >>>>>>>>>>> ok diff-snes_tutorials-ex17_3d_q3_trig_elas >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij.counts >>>>>>>>>>> ok snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij >>>>>>>>>>> ok diff-snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist_3.counts >>>>>>>>>>> not ok ksp_ksp_tutorials-ex5_superlu_dist_3 # Error code: 1 >>>>>>>>>>> #srun: error: Unable to create step for job 1426755: More >>>>>>>>>>> processors requested than permitted >>>>>>>>>>> ok ksp_ksp_tutorials-ex5_superlu_dist_3 # SKIP Command >>>>>>>>>>> failed so no diff >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist.counts >>>>>>>>>>> ok ksp_ksp_tutorials-ex5f_superlu_dist # SKIP Fortran >>>>>>>>>>> required for this test >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_tri_parmetis_hpddm_baij.counts >>>>>>>>>>> ok snes_tutorials-ex12_tri_parmetis_hpddm_baij >>>>>>>>>>> ok diff-snes_tutorials-ex12_tri_parmetis_hpddm_baij >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_tut_3.counts >>>>>>>>>>> ok snes_tutorials-ex19_tut_3 >>>>>>>>>>> ok diff-snes_tutorials-ex19_tut_3 >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_vlap.counts >>>>>>>>>>> ok snes_tutorials-ex17_3d_q3_trig_vlap >>>>>>>>>>> ok diff-snes_tutorials-ex17_3d_q3_trig_vlap >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist_3.counts >>>>>>>>>>> ok ksp_ksp_tutorials-ex5f_superlu_dist_3 # SKIP Fortran >>>>>>>>>>> required for this test >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_superlu_dist.counts >>>>>>>>>>> ok snes_tutorials-ex19_superlu_dist >>>>>>>>>>> ok diff-snes_tutorials-ex19_superlu_dist >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre.counts >>>>>>>>>>> ok >>>>>>>>>>> snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre >>>>>>>>>>> ok >>>>>>>>>>> diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex49_hypre_nullspace.counts >>>>>>>>>>> ok ksp_ksp_tutorials-ex49_hypre_nullspace >>>>>>>>>>> ok diff-ksp_ksp_tutorials-ex49_hypre_nullspace >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_superlu_dist_2.counts >>>>>>>>>>> ok snes_tutorials-ex19_superlu_dist_2 >>>>>>>>>>> ok diff-snes_tutorials-ex19_superlu_dist_2 >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist_2.counts >>>>>>>>>>> not ok ksp_ksp_tutorials-ex5_superlu_dist_2 # Error code: 1 >>>>>>>>>>> #srun: error: Unable to create step for job 1426755: More >>>>>>>>>>> processors requested than permitted >>>>>>>>>>> ok ksp_ksp_tutorials-ex5_superlu_dist_2 # SKIP Command >>>>>>>>>>> failed so no diff >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre.counts >>>>>>>>>>> ok >>>>>>>>>>> snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre >>>>>>>>>>> ok >>>>>>>>>>> diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex64_1.counts >>>>>>>>>>> ok ksp_ksp_tutorials-ex64_1 >>>>>>>>>>> ok diff-ksp_ksp_tutorials-ex64_1 >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist.counts >>>>>>>>>>> not ok ksp_ksp_tutorials-ex5_superlu_dist # Error code: 1 >>>>>>>>>>> #srun: error: Unable to create step for job 1426755: More >>>>>>>>>>> processors requested than permitted >>>>>>>>>>> ok ksp_ksp_tutorials-ex5_superlu_dist # SKIP Command >>>>>>>>>>> failed so no diff >>>>>>>>>>> TEST >>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist_2.counts >>>>>>>>>>> ok ksp_ksp_tutorials-ex5f_superlu_dist_2 # SKIP Fortran >>>>>>>>>>> required for this test >>>>>>>>>>> >>>>>>>>>>>> Barry >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On Mar 10, 2021, at 11:03 PM, Eric Chamberland >>>>>>>>>>>>> <Eric.Chamberland(a)giref.ulaval.ca >>>>>>>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Barry, >>>>>>>>>>>>> >>>>>>>>>>>>> to get a some follow up on --with-openmp=1 failures, >>>>>>>>>>>>> shall I open gitlab issues for: >>>>>>>>>>>>> >>>>>>>>>>>>> a) all hypre failures giving DIVERGED_INDEFINITE_PC >>>>>>>>>>>>> >>>>>>>>>>>>> b) all superlu_dist failures giving different results >>>>>>>>>>>>> with initia and "Exceeded timeout limit of 60 s" >>>>>>>>>>>>> >>>>>>>>>>>>> c) hpddm failures "free(): invalid next size (fast)" >>>>>>>>>>>>> and "Segmentation Violation" >>>>>>>>>>>>> >>>>>>>>>>>>> d) all tao's "Exceeded timeout limit of 60 s" >>>>>>>>>>>>> >>>>>>>>>>>>> I don't see how I could do all these debugging by myself... >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> >>>>>>>>>>>>> Eric >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>>>> Professionnel de recherche >>>>>>>>>> GIREF/Université Laval >>>>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>>> -- >>>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>>> Professionnel de recherche >>>>>>>>> GIREF/Université Laval >>>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>>> <fedora_mkl_and_devtools.txt><openmpi.txt><petsc.txt> >>>>>>>> >>>>>>> >>>>>> >>>>> -- >>>>> Eric Chamberland, ing., M. Ing >>>>> Professionnel de recherche >>>>> GIREF/Université Laval >>>>> (418) 656-2131 poste 41 22 42 >>>> >>> -- >>> Eric Chamberland, ing., M. Ing >>> Professionnel de recherche >>> GIREF/Université Laval >>> (418) 656-2131 poste 41 22 42 >> > -- > Eric Chamberland, ing., M. Ing > Professionnel de recherche > GIREF/Université Laval > (418) 656-2131 poste 41 22 42

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Eric Chamberland 18 Mar '21

18 Mar '21

Hi, For the knowledge of readers, I just read section 7.3 here: https://www.researchgate.net/publication/220411740_Multigrid_Smoothers_for_… And it is explained why multi-threading gives a poor result with the Hybrid−SGS smoother... Eric On 2021-03-15 2:50 p.m., Barry Smith wrote: > > I posted some information at the issue. > > IMHO it is likely a bug in one or more of hypre's smoothers that > use OpenMP. We have never tested them before (and likely hypre has not > tested all the combinations) and so would not have seen the bug. > Hopefully they can just fix it. > > Barry > > I got the problem to occur with ex56 with 2 MPI ranks and 4 OpenMP > threads, if I used less than 4 threads it did not generate an > indefinite preconditioner. > > >> On Mar 14, 2021, at 1:18 PM, Eric Chamberland >> <Eric.Chamberland(a)giref.ulaval.ca >> <mailto:[email protected]>> wrote: >> >> Done: >> >> https://github.com/hypre-space/hypre/issues/303 >> >> Maybe I will need some help about PETSc to answer their questions... >> >> Eric >> >> On 2021-03-14 3:44 a.m., Stefano Zampini wrote: >>> Eric >>> >>> You should report these HYPRE issues upstream >>> https://github.com/hypre-space/hypre/issues >>> <https://github.com/hypre-space/hypre/issues> >>> >>> >>>> On Mar 14, 2021, at 3:44 AM, Eric Chamberland >>>> <Eric.Chamberland(a)giref.ulaval.ca >>>> <mailto:[email protected]>> wrote: >>>> >>>> For us it clearly creates problems in real computations... >>>> >>>> I understand the need to have clean test for PETSc, but for me, it >>>> reveals that hypre isn't usable with more than one thread for now... >>>> >>>> Another solution: force single-threaded configuration for hypre >>>> until this is fixed? >>>> >>>> Eric >>>> >>>> On 2021-03-13 8:50 a.m., Pierre Jolivet wrote: >>>>> -pc_hypre_boomeramg_relax_type_all Jacobi => >>>>> Linear solve did not converge due to DIVERGED_INDEFINITE_PC >>>>> iterations 3 >>>>> -pc_hypre_boomeramg_relax_type_all l1scaled-Jacobi => >>>>> OK, independently of the architecture it seems (Eric Docker image >>>>> with 1 or 2 threads or my macOS), but contraction factor is higher >>>>> Linear solve converged due to CONVERGED_RTOL iterations 8 >>>>> Linear solve converged due to CONVERGED_RTOL iterations 24 >>>>> Linear solve converged due to CONVERGED_RTOL iterations 26 >>>>> v. currently >>>>> Linear solve converged due to CONVERGED_RTOL iterations 7 >>>>> Linear solve converged due to CONVERGED_RTOL iterations 9 >>>>> Linear solve converged due to CONVERGED_RTOL iterations 10 >>>>> >>>>> Do we change this? Or should we force OMP_NUM_THREADS=1 for make test? >>>>> >>>>> Thanks, >>>>> Pierre >>>>> >>>>>> On 13 Mar 2021, at 2:26 PM, Mark Adams <mfadams(a)lbl.gov >>>>>> <mailto:[email protected]>> wrote: >>>>>> >>>>>> Hypre uses a multiplicative smoother by default. It has a >>>>>> chebyshev smoother. That with a Jacobi PC should be thread >>>>>> invariant. >>>>>> Mark >>>>>> >>>>>> On Sat, Mar 13, 2021 at 8:18 AM Pierre Jolivet <pierre(a)joliv.et >>>>>> <mailto:[email protected]>> wrote: >>>>>> >>>>>> >>>>>>> On 13 Mar 2021, at 9:17 AM, Pierre Jolivet <pierre(a)joliv.et >>>>>>> <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> Hello Eric, >>>>>>> I’ve made an “interesting” discovery, so I’ll put back the >>>>>>> list in c/c. >>>>>>> It appears the following snippet of code which uses >>>>>>> Allreduce() + lambda function + MPI_IN_PLACE is: >>>>>>> - Valgrind-clean with MPICH; >>>>>>> - Valgrind-clean with OpenMPI 4.0.5; >>>>>>> - not Valgrind-clean with OpenMPI 4.1.0. >>>>>>> I’m not sure who is to blame here, I’ll need to look at the >>>>>>> MPI specification for what is required by the implementors >>>>>>> and users in that case. >>>>>>> >>>>>>> In the meantime, I’ll do the following: >>>>>>> - update config/BuildSystem/config/packages/OpenMPI.py to >>>>>>> use OpenMPI 4.1.0, see if any other error appears; >>>>>>> - provide a hotfix to bypass the segfaults; >>>>>> >>>>>> I can confirm that splitting the single Allreduce with my own >>>>>> MPI_Op into two Allreduce with MAX and BAND fixes the >>>>>> segfaults with OpenMPI (*). >>>>>> >>>>>>> - look at the hypre issue and whether they should be >>>>>>> deferred to the hypre team. >>>>>> >>>>>> I don’t know if there is something wrong in hypre threading >>>>>> or if it’s just a side effect of threading, but it seems that >>>>>> the number of threads has a drastic effect on the quality of >>>>>> the PC. >>>>>> By default, it looks that there are two threads per process >>>>>> with your Docker image. >>>>>> If I force OMP_NUM_THREADS=1, then I get the same convergence >>>>>> as in the output file. >>>>>> >>>>>> Thanks, >>>>>> Pierre >>>>>> >>>>>> (*) https://gitlab.com/petsc/petsc/-/merge_requests/3712 >>>>>> <https://gitlab.com/petsc/petsc/-/merge_requests/3712> >>>>>> >>>>>>> Thank you for the Docker files, they were really useful. >>>>>>> If you want to avoid oversubscription failures, you can edit >>>>>>> the file /opt/openmpi-4.1.0/etc/openmpi-default-hostfile and >>>>>>> append the line: >>>>>>> localhost slots=12 >>>>>>> If you want to increase the timeout limit of PETSc test >>>>>>> suite for each test, you can add the extra flag in your >>>>>>> command line TIMEOUT=180 (default is 60, units are seconds). >>>>>>> >>>>>>> Thanks, I’ll ping you on GitLab when I’ve got something >>>>>>> ready for you to try, >>>>>>> Pierre >>>>>>> >>>>>>> <ompi.cxx> >>>>>>> >>>>>>>> On 12 Mar 2021, at 8:54 PM, Eric Chamberland >>>>>>>> <Eric.Chamberland(a)giref.ulaval.ca >>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>> >>>>>>>> Hi Pierre, >>>>>>>> >>>>>>>> I now have a docker container reproducing the problems here. >>>>>>>> >>>>>>>> Actually, if I look at >>>>>>>> snes_tutorials-ex12_quad_singular_hpddm it fails like this: >>>>>>>> >>>>>>>> not ok snes_tutorials-ex12_quad_singular_hpddm # Error code: 59 >>>>>>>> # Initial guess >>>>>>>> # L_2 Error: 0.00803099 >>>>>>>> # Initial Residual >>>>>>>> # L_2 Residual: 1.09057 >>>>>>>> # Au - b = Au + F(0) >>>>>>>> # Linear L_2 Residual: 1.09057 >>>>>>>> # [d470c54ce086:14127] Read -1, expected 4096, errno = 1 >>>>>>>> # [d470c54ce086:14128] Read -1, expected 4096, errno = 1 >>>>>>>> # [d470c54ce086:14129] Read -1, expected 4096, errno = 1 >>>>>>>> # [3]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> # [3]PETSC ERROR: Caught signal number 11 SEGV: >>>>>>>> Segmentation Violation, probably memory access out of range >>>>>>>> # [3]PETSC ERROR: Try option -start_in_debugger or >>>>>>>> -on_error_attach_debugger >>>>>>>> # [3]PETSC ERROR: or see >>>>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>>>>>>> <https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind> >>>>>>>> # [3]PETSC ERROR: or try http://valgrind.org >>>>>>>> <http://valgrind.org/> on GNU/linux and Apple Mac OS X to >>>>>>>> find memory corruption errors >>>>>>>> # [3]PETSC ERROR: likely location of problem given in stack >>>>>>>> below >>>>>>>> # [3]PETSC ERROR: --------------------- Stack Frames >>>>>>>> ------------------------------------ >>>>>>>> # [3]PETSC ERROR: Note: The EXACT line numbers in the stack >>>>>>>> are not available, >>>>>>>> # [3]PETSC ERROR: INSTEAD the line number of the start of >>>>>>>> the function >>>>>>>> # [3]PETSC ERROR: is given. >>>>>>>> # [3]PETSC ERROR: [3] buildTwo line 987 >>>>>>>> /opt/petsc-main/include/HPDDM_schwarz.hpp >>>>>>>> # [3]PETSC ERROR: [3] next line 1130 >>>>>>>> /opt/petsc-main/include/HPDDM_schwarz.hpp >>>>>>>> # [3]PETSC ERROR: --------------------- Error Message >>>>>>>> -------------------------------------------------------------- >>>>>>>> # [3]PETSC ERROR: Signal received >>>>>>>> # [3]PETSC ERROR: [0]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> >>>>>>>> also ex12_quad_hpddm_reuse_baij fails with a lot more "Read >>>>>>>> -1, expected ..." which I don't know where they come from...? >>>>>>>> >>>>>>>> Hypre (like in diff-snes_tutorials-ex56_hypre) is also >>>>>>>> having DIVERGED_INDEFINITE_PC failures... >>>>>>>> >>>>>>>> Please see the 3 attached docker files: >>>>>>>> >>>>>>>> 1) fedora_mkl_and_devtools : the DockerFile which install >>>>>>>> fedore 33 with gnu compilers and MKL and everything to develop. >>>>>>>> >>>>>>>> 2) openmpi: the DockerFile to bluid OpenMPI >>>>>>>> >>>>>>>> 3) petsc: The las DockerFile that build/install and test PETSc >>>>>>>> >>>>>>>> I build the 3 like this: >>>>>>>> >>>>>>>> docker build -t fedora_mkl_and_devtools -f >>>>>>>> fedora_mkl_and_devtools . >>>>>>>> >>>>>>>> docker build -t openmpi -f openmpi . >>>>>>>> >>>>>>>> docker build -t petsc -f petsc . >>>>>>>> >>>>>>>> Disclaimer: I am not a docker expert, so I may do things >>>>>>>> that are not docker-stat-of-the-art but I am opened to >>>>>>>> suggestions... ;) >>>>>>>> >>>>>>>> I have just ran it on my portable (long) which have not >>>>>>>> enough cores, so many more tests failed (should force >>>>>>>> --oversubscribe but don't know how to). I will relaunch on >>>>>>>> my workstation in a few minutes. >>>>>>>> >>>>>>>> I will now test your branch! (sorry for the delay). >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Eric >>>>>>>> >>>>>>>> On 2021-03-11 9:03 a.m., Eric Chamberland wrote: >>>>>>>>> >>>>>>>>> Hi Pierre, >>>>>>>>> >>>>>>>>> ok, that's interesting! >>>>>>>>> >>>>>>>>> I will try to build a docker image until tomorrow and give >>>>>>>>> you the exact recipe to reproduce the bugs. >>>>>>>>> >>>>>>>>> Eric >>>>>>>>> >>>>>>>>> >>>>>>>>> On 2021-03-11 2:46 a.m., Pierre Jolivet wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On 11 Mar 2021, at 6:16 AM, Barry Smith >>>>>>>>>>> <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Eric, >>>>>>>>>>> >>>>>>>>>>> Sorry about not being more immediate. We still have >>>>>>>>>>> this in our active email so you don't need to submit >>>>>>>>>>> individual issues. We'll try to get to them as soon as >>>>>>>>>>> we can. >>>>>>>>>> >>>>>>>>>> Indeed, I’m still trying to figure this out. >>>>>>>>>> I realized that some of my configure flags were different >>>>>>>>>> than yours, e.g., no --with-memalign. >>>>>>>>>> I’ve also added SuperLU_DIST to my installation. >>>>>>>>>> Still, I can’t reproduce any issue. >>>>>>>>>> I will continue looking into this, it appears I’m seeing >>>>>>>>>> some valgrind errors, but I don’t know if this is some >>>>>>>>>> side effect of OpenMPI not being valgrind-clean (last >>>>>>>>>> time I checked, there was no error with MPICH). >>>>>>>>>> >>>>>>>>>> Thank you for your patience, >>>>>>>>>> Pierre >>>>>>>>>> >>>>>>>>>> /usr/bin/gmake -f gmakefile test test-fail=1 >>>>>>>>>> Using MAKEFLAGS: test-fail=1 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_baij.counts >>>>>>>>>> ok snes_tutorials-ex12_quad_hpddm_reuse_baij >>>>>>>>>> ok diff-snes_tutorials-ex12_quad_hpddm_reuse_baij >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist_2.counts >>>>>>>>>> ok ksp_ksp_tests-ex33_superlu_dist_2 >>>>>>>>>> ok diff-ksp_ksp_tests-ex33_superlu_dist_2 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex49_superlu_dist.counts >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex50_tut_2.counts >>>>>>>>>> ok ksp_ksp_tutorials-ex50_tut_2 >>>>>>>>>> ok diff-ksp_ksp_tutorials-ex50_tut_2 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist.counts >>>>>>>>>> ok ksp_ksp_tests-ex33_superlu_dist >>>>>>>>>> ok diff-ksp_ksp_tests-ex33_superlu_dist >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_hypre.counts >>>>>>>>>> ok snes_tutorials-ex56_hypre >>>>>>>>>> ok diff-snes_tutorials-ex56_hypre >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex56_2.counts >>>>>>>>>> ok ksp_ksp_tutorials-ex56_2 >>>>>>>>>> ok diff-ksp_ksp_tutorials-ex56_2 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_elas.counts >>>>>>>>>> ok snes_tutorials-ex17_3d_q3_trig_elas >>>>>>>>>> ok diff-snes_tutorials-ex17_3d_q3_trig_elas >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij.counts >>>>>>>>>> ok snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij >>>>>>>>>> ok diff-snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist_3.counts >>>>>>>>>> not ok ksp_ksp_tutorials-ex5_superlu_dist_3 # Error code: 1 >>>>>>>>>> #srun: error: Unable to create step for job 1426755: More >>>>>>>>>> processors requested than permitted >>>>>>>>>> ok ksp_ksp_tutorials-ex5_superlu_dist_3 # SKIP Command >>>>>>>>>> failed so no diff >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist.counts >>>>>>>>>> ok ksp_ksp_tutorials-ex5f_superlu_dist # SKIP Fortran >>>>>>>>>> required for this test >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_tri_parmetis_hpddm_baij.counts >>>>>>>>>> ok snes_tutorials-ex12_tri_parmetis_hpddm_baij >>>>>>>>>> ok diff-snes_tutorials-ex12_tri_parmetis_hpddm_baij >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_tut_3.counts >>>>>>>>>> ok snes_tutorials-ex19_tut_3 >>>>>>>>>> ok diff-snes_tutorials-ex19_tut_3 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_vlap.counts >>>>>>>>>> ok snes_tutorials-ex17_3d_q3_trig_vlap >>>>>>>>>> ok diff-snes_tutorials-ex17_3d_q3_trig_vlap >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist_3.counts >>>>>>>>>> ok ksp_ksp_tutorials-ex5f_superlu_dist_3 # SKIP Fortran >>>>>>>>>> required for this test >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_superlu_dist.counts >>>>>>>>>> ok snes_tutorials-ex19_superlu_dist >>>>>>>>>> ok diff-snes_tutorials-ex19_superlu_dist >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre.counts >>>>>>>>>> ok >>>>>>>>>> snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre >>>>>>>>>> ok >>>>>>>>>> diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex49_hypre_nullspace.counts >>>>>>>>>> ok ksp_ksp_tutorials-ex49_hypre_nullspace >>>>>>>>>> ok diff-ksp_ksp_tutorials-ex49_hypre_nullspace >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_superlu_dist_2.counts >>>>>>>>>> ok snes_tutorials-ex19_superlu_dist_2 >>>>>>>>>> ok diff-snes_tutorials-ex19_superlu_dist_2 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist_2.counts >>>>>>>>>> not ok ksp_ksp_tutorials-ex5_superlu_dist_2 # Error code: 1 >>>>>>>>>> #srun: error: Unable to create step for job 1426755: More >>>>>>>>>> processors requested than permitted >>>>>>>>>> ok ksp_ksp_tutorials-ex5_superlu_dist_2 # SKIP Command >>>>>>>>>> failed so no diff >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre.counts >>>>>>>>>> ok >>>>>>>>>> snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre >>>>>>>>>> ok >>>>>>>>>> diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex64_1.counts >>>>>>>>>> ok ksp_ksp_tutorials-ex64_1 >>>>>>>>>> ok diff-ksp_ksp_tutorials-ex64_1 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist.counts >>>>>>>>>> not ok ksp_ksp_tutorials-ex5_superlu_dist # Error code: 1 >>>>>>>>>> #srun: error: Unable to create step for job 1426755: More >>>>>>>>>> processors requested than permitted >>>>>>>>>> ok ksp_ksp_tutorials-ex5_superlu_dist # SKIP Command >>>>>>>>>> failed so no diff >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist_2.counts >>>>>>>>>> ok ksp_ksp_tutorials-ex5f_superlu_dist_2 # SKIP Fortran >>>>>>>>>> required for this test >>>>>>>>>> >>>>>>>>>>> Barry >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On Mar 10, 2021, at 11:03 PM, Eric Chamberland >>>>>>>>>>>> <Eric.Chamberland(a)giref.ulaval.ca >>>>>>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Barry, >>>>>>>>>>>> >>>>>>>>>>>> to get a some follow up on --with-openmp=1 failures, >>>>>>>>>>>> shall I open gitlab issues for: >>>>>>>>>>>> >>>>>>>>>>>> a) all hypre failures giving DIVERGED_INDEFINITE_PC >>>>>>>>>>>> >>>>>>>>>>>> b) all superlu_dist failures giving different results >>>>>>>>>>>> with initia and "Exceeded timeout limit of 60 s" >>>>>>>>>>>> >>>>>>>>>>>> c) hpddm failures "free(): invalid next size (fast)" >>>>>>>>>>>> and "Segmentation Violation" >>>>>>>>>>>> >>>>>>>>>>>> d) all tao's "Exceeded timeout limit of 60 s" >>>>>>>>>>>> >>>>>>>>>>>> I don't see how I could do all these debugging by myself... >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Eric >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>>> Professionnel de recherche >>>>>>>>> GIREF/Université Laval >>>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>> -- >>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>> Professionnel de recherche >>>>>>>> GIREF/Université Laval >>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>> <fedora_mkl_and_devtools.txt><openmpi.txt><petsc.txt> >>>>>>> >>>>>> >>>>> >>>> -- >>>> Eric Chamberland, ing., M. Ing >>>> Professionnel de recherche >>>> GIREF/Université Laval >>>> (418) 656-2131 poste 41 22 42 >>> >> -- >> Eric Chamberland, ing., M. Ing >> Professionnel de recherche >> GIREF/Université Laval >> (418) 656-2131 poste 41 22 42 > -- Eric Chamberland, ing., M. Ing Professionnel de recherche GIREF/Université Laval (418) 656-2131 poste 41 22 42

1 0