March 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Eric Chamberland 22 Mar '21

22 Mar '21

I added some information here: https://github.com/xiaoyeli/superlu_dist/issues/69#issuecomment-804318719 Maybe someone can say more than I on what PETSc tries to do with the 2 mentioned tutorials that are timing out... Thanks, Eric On 2021-03-15 11:31 a.m., Eric Chamberland wrote: > > Reported timeout bugs to SuperLU_dist too: > > https://github.com/xiaoyeli/superlu_dist/issues/69 > > Eric > > > On 2021-03-14 2:18 p.m., Eric Chamberland wrote: >> >> Done: >> >> https://github.com/hypre-space/hypre/issues/303 >> >> Maybe I will need some help about PETSc to answer their questions... >> >> Eric >> >> On 2021-03-14 3:44 a.m., Stefano Zampini wrote: >>> Eric >>> >>> You should report these HYPRE issues upstream >>> https://github.com/hypre-space/hypre/issues >>> <https://github.com/hypre-space/hypre/issues> >>> >>> >>>> On Mar 14, 2021, at 3:44 AM, Eric Chamberland >>>> <Eric.Chamberland(a)giref.ulaval.ca >>>> <mailto:[email protected]>> wrote: >>>> >>>> For us it clearly creates problems in real computations... >>>> >>>> I understand the need to have clean test for PETSc, but for me, it >>>> reveals that hypre isn't usable with more than one thread for now... >>>> >>>> Another solution: force single-threaded configuration for hypre >>>> until this is fixed? >>>> >>>> Eric >>>> >>>> On 2021-03-13 8:50 a.m., Pierre Jolivet wrote: >>>>> -pc_hypre_boomeramg_relax_type_all Jacobi => >>>>> Linear solve did not converge due to DIVERGED_INDEFINITE_PC >>>>> iterations 3 >>>>> -pc_hypre_boomeramg_relax_type_all l1scaled-Jacobi => >>>>> OK, independently of the architecture it seems (Eric Docker image >>>>> with 1 or 2 threads or my macOS), but contraction factor is higher >>>>> Linear solve converged due to CONVERGED_RTOL iterations 8 >>>>> Linear solve converged due to CONVERGED_RTOL iterations 24 >>>>> Linear solve converged due to CONVERGED_RTOL iterations 26 >>>>> v. currently >>>>> Linear solve converged due to CONVERGED_RTOL iterations 7 >>>>> Linear solve converged due to CONVERGED_RTOL iterations 9 >>>>> Linear solve converged due to CONVERGED_RTOL iterations 10 >>>>> >>>>> Do we change this? Or should we force OMP_NUM_THREADS=1 for make test? >>>>> >>>>> Thanks, >>>>> Pierre >>>>> >>>>>> On 13 Mar 2021, at 2:26 PM, Mark Adams <mfadams(a)lbl.gov >>>>>> <mailto:[email protected]>> wrote: >>>>>> >>>>>> Hypre uses a multiplicative smoother by default. It has a >>>>>> chebyshev smoother. That with a Jacobi PC should be thread >>>>>> invariant. >>>>>> Mark >>>>>> >>>>>> On Sat, Mar 13, 2021 at 8:18 AM Pierre Jolivet <pierre(a)joliv.et >>>>>> <mailto:[email protected]>> wrote: >>>>>> >>>>>> >>>>>>> On 13 Mar 2021, at 9:17 AM, Pierre Jolivet <pierre(a)joliv.et >>>>>>> <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> Hello Eric, >>>>>>> I’ve made an “interesting” discovery, so I’ll put back the >>>>>>> list in c/c. >>>>>>> It appears the following snippet of code which uses >>>>>>> Allreduce() + lambda function + MPI_IN_PLACE is: >>>>>>> - Valgrind-clean with MPICH; >>>>>>> - Valgrind-clean with OpenMPI 4.0.5; >>>>>>> - not Valgrind-clean with OpenMPI 4.1.0. >>>>>>> I’m not sure who is to blame here, I’ll need to look at the >>>>>>> MPI specification for what is required by the implementors >>>>>>> and users in that case. >>>>>>> >>>>>>> In the meantime, I’ll do the following: >>>>>>> - update config/BuildSystem/config/packages/OpenMPI.py to >>>>>>> use OpenMPI 4.1.0, see if any other error appears; >>>>>>> - provide a hotfix to bypass the segfaults; >>>>>> >>>>>> I can confirm that splitting the single Allreduce with my own >>>>>> MPI_Op into two Allreduce with MAX and BAND fixes the >>>>>> segfaults with OpenMPI (*). >>>>>> >>>>>>> - look at the hypre issue and whether they should be >>>>>>> deferred to the hypre team. >>>>>> >>>>>> I don’t know if there is something wrong in hypre threading >>>>>> or if it’s just a side effect of threading, but it seems that >>>>>> the number of threads has a drastic effect on the quality of >>>>>> the PC. >>>>>> By default, it looks that there are two threads per process >>>>>> with your Docker image. >>>>>> If I force OMP_NUM_THREADS=1, then I get the same convergence >>>>>> as in the output file. >>>>>> >>>>>> Thanks, >>>>>> Pierre >>>>>> >>>>>> (*) https://gitlab.com/petsc/petsc/-/merge_requests/3712 >>>>>> <https://gitlab.com/petsc/petsc/-/merge_requests/3712> >>>>>> >>>>>>> Thank you for the Docker files, they were really useful. >>>>>>> If you want to avoid oversubscription failures, you can edit >>>>>>> the file /opt/openmpi-4.1.0/etc/openmpi-default-hostfile and >>>>>>> append the line: >>>>>>> localhost slots=12 >>>>>>> If you want to increase the timeout limit of PETSc test >>>>>>> suite for each test, you can add the extra flag in your >>>>>>> command line TIMEOUT=180 (default is 60, units are seconds). >>>>>>> >>>>>>> Thanks, I’ll ping you on GitLab when I’ve got something >>>>>>> ready for you to try, >>>>>>> Pierre >>>>>>> >>>>>>> <ompi.cxx> >>>>>>> >>>>>>>> On 12 Mar 2021, at 8:54 PM, Eric Chamberland >>>>>>>> <Eric.Chamberland(a)giref.ulaval.ca >>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>> >>>>>>>> Hi Pierre, >>>>>>>> >>>>>>>> I now have a docker container reproducing the problems here. >>>>>>>> >>>>>>>> Actually, if I look at >>>>>>>> snes_tutorials-ex12_quad_singular_hpddm it fails like this: >>>>>>>> >>>>>>>> not ok snes_tutorials-ex12_quad_singular_hpddm # Error code: 59 >>>>>>>> # Initial guess >>>>>>>> # L_2 Error: 0.00803099 >>>>>>>> # Initial Residual >>>>>>>> # L_2 Residual: 1.09057 >>>>>>>> # Au - b = Au + F(0) >>>>>>>> # Linear L_2 Residual: 1.09057 >>>>>>>> # [d470c54ce086:14127] Read -1, expected 4096, errno = 1 >>>>>>>> # [d470c54ce086:14128] Read -1, expected 4096, errno = 1 >>>>>>>> # [d470c54ce086:14129] Read -1, expected 4096, errno = 1 >>>>>>>> # [3]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> # [3]PETSC ERROR: Caught signal number 11 SEGV: >>>>>>>> Segmentation Violation, probably memory access out of range >>>>>>>> # [3]PETSC ERROR: Try option -start_in_debugger or >>>>>>>> -on_error_attach_debugger >>>>>>>> # [3]PETSC ERROR: or see >>>>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>>>>>>> <https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind> >>>>>>>> # [3]PETSC ERROR: or try http://valgrind.org >>>>>>>> <http://valgrind.org/> on GNU/linux and Apple Mac OS X to >>>>>>>> find memory corruption errors >>>>>>>> # [3]PETSC ERROR: likely location of problem given in >>>>>>>> stack below >>>>>>>> # [3]PETSC ERROR: --------------------- Stack Frames >>>>>>>> ------------------------------------ >>>>>>>> # [3]PETSC ERROR: Note: The EXACT line numbers in the >>>>>>>> stack are not available, >>>>>>>> # [3]PETSC ERROR: INSTEAD the line number of the >>>>>>>> start of the function >>>>>>>> # [3]PETSC ERROR: is given. >>>>>>>> # [3]PETSC ERROR: [3] buildTwo line 987 >>>>>>>> /opt/petsc-main/include/HPDDM_schwarz.hpp >>>>>>>> # [3]PETSC ERROR: [3] next line 1130 >>>>>>>> /opt/petsc-main/include/HPDDM_schwarz.hpp >>>>>>>> # [3]PETSC ERROR: --------------------- Error Message >>>>>>>> -------------------------------------------------------------- >>>>>>>> # [3]PETSC ERROR: Signal received >>>>>>>> # [3]PETSC ERROR: [0]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> >>>>>>>> also ex12_quad_hpddm_reuse_baij fails with a lot more "Read >>>>>>>> -1, expected ..." which I don't know where they come from...? >>>>>>>> >>>>>>>> Hypre (like in diff-snes_tutorials-ex56_hypre) is also >>>>>>>> having DIVERGED_INDEFINITE_PC failures... >>>>>>>> >>>>>>>> Please see the 3 attached docker files: >>>>>>>> >>>>>>>> 1) fedora_mkl_and_devtools : the DockerFile which install >>>>>>>> fedore 33 with gnu compilers and MKL and everything to develop. >>>>>>>> >>>>>>>> 2) openmpi: the DockerFile to bluid OpenMPI >>>>>>>> >>>>>>>> 3) petsc: The las DockerFile that build/install and test PETSc >>>>>>>> >>>>>>>> I build the 3 like this: >>>>>>>> >>>>>>>> docker build -t fedora_mkl_and_devtools -f >>>>>>>> fedora_mkl_and_devtools . >>>>>>>> >>>>>>>> docker build -t openmpi -f openmpi . >>>>>>>> >>>>>>>> docker build -t petsc -f petsc . >>>>>>>> >>>>>>>> Disclaimer: I am not a docker expert, so I may do things >>>>>>>> that are not docker-stat-of-the-art but I am opened to >>>>>>>> suggestions... ;) >>>>>>>> >>>>>>>> I have just ran it on my portable (long) which have not >>>>>>>> enough cores, so many more tests failed (should force >>>>>>>> --oversubscribe but don't know how to). I will relaunch on >>>>>>>> my workstation in a few minutes. >>>>>>>> >>>>>>>> I will now test your branch! (sorry for the delay). >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Eric >>>>>>>> >>>>>>>> On 2021-03-11 9:03 a.m., Eric Chamberland wrote: >>>>>>>>> >>>>>>>>> Hi Pierre, >>>>>>>>> >>>>>>>>> ok, that's interesting! >>>>>>>>> >>>>>>>>> I will try to build a docker image until tomorrow and give >>>>>>>>> you the exact recipe to reproduce the bugs. >>>>>>>>> >>>>>>>>> Eric >>>>>>>>> >>>>>>>>> >>>>>>>>> On 2021-03-11 2:46 a.m., Pierre Jolivet wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On 11 Mar 2021, at 6:16 AM, Barry Smith >>>>>>>>>>> <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Eric, >>>>>>>>>>> >>>>>>>>>>> Sorry about not being more immediate. We still have >>>>>>>>>>> this in our active email so you don't need to submit >>>>>>>>>>> individual issues. We'll try to get to them as soon as >>>>>>>>>>> we can. >>>>>>>>>> >>>>>>>>>> Indeed, I’m still trying to figure this out. >>>>>>>>>> I realized that some of my configure flags were different >>>>>>>>>> than yours, e.g., no --with-memalign. >>>>>>>>>> I’ve also added SuperLU_DIST to my installation. >>>>>>>>>> Still, I can’t reproduce any issue. >>>>>>>>>> I will continue looking into this, it appears I’m seeing >>>>>>>>>> some valgrind errors, but I don’t know if this is some >>>>>>>>>> side effect of OpenMPI not being valgrind-clean (last >>>>>>>>>> time I checked, there was no error with MPICH). >>>>>>>>>> >>>>>>>>>> Thank you for your patience, >>>>>>>>>> Pierre >>>>>>>>>> >>>>>>>>>> /usr/bin/gmake -f gmakefile test test-fail=1 >>>>>>>>>> Using MAKEFLAGS: test-fail=1 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_baij.counts >>>>>>>>>> ok snes_tutorials-ex12_quad_hpddm_reuse_baij >>>>>>>>>> ok diff-snes_tutorials-ex12_quad_hpddm_reuse_baij >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist_2.counts >>>>>>>>>> ok ksp_ksp_tests-ex33_superlu_dist_2 >>>>>>>>>> ok diff-ksp_ksp_tests-ex33_superlu_dist_2 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex49_superlu_dist.counts >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex50_tut_2.counts >>>>>>>>>> ok ksp_ksp_tutorials-ex50_tut_2 >>>>>>>>>> ok diff-ksp_ksp_tutorials-ex50_tut_2 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist.counts >>>>>>>>>> ok ksp_ksp_tests-ex33_superlu_dist >>>>>>>>>> ok diff-ksp_ksp_tests-ex33_superlu_dist >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_hypre.counts >>>>>>>>>> ok snes_tutorials-ex56_hypre >>>>>>>>>> ok diff-snes_tutorials-ex56_hypre >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex56_2.counts >>>>>>>>>> ok ksp_ksp_tutorials-ex56_2 >>>>>>>>>> ok diff-ksp_ksp_tutorials-ex56_2 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_elas.counts >>>>>>>>>> ok snes_tutorials-ex17_3d_q3_trig_elas >>>>>>>>>> ok diff-snes_tutorials-ex17_3d_q3_trig_elas >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij.counts >>>>>>>>>> ok snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij >>>>>>>>>> ok diff-snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist_3.counts >>>>>>>>>> not ok ksp_ksp_tutorials-ex5_superlu_dist_3 # Error code: 1 >>>>>>>>>> #srun: error: Unable to create step for job 1426755: More >>>>>>>>>> processors requested than permitted >>>>>>>>>> ok ksp_ksp_tutorials-ex5_superlu_dist_3 # SKIP Command >>>>>>>>>> failed so no diff >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist.counts >>>>>>>>>> ok ksp_ksp_tutorials-ex5f_superlu_dist # SKIP Fortran >>>>>>>>>> required for this test >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_tri_parmetis_hpddm_baij.counts >>>>>>>>>> ok snes_tutorials-ex12_tri_parmetis_hpddm_baij >>>>>>>>>> ok diff-snes_tutorials-ex12_tri_parmetis_hpddm_baij >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_tut_3.counts >>>>>>>>>> ok snes_tutorials-ex19_tut_3 >>>>>>>>>> ok diff-snes_tutorials-ex19_tut_3 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_vlap.counts >>>>>>>>>> ok snes_tutorials-ex17_3d_q3_trig_vlap >>>>>>>>>> ok diff-snes_tutorials-ex17_3d_q3_trig_vlap >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist_3.counts >>>>>>>>>> ok ksp_ksp_tutorials-ex5f_superlu_dist_3 # SKIP Fortran >>>>>>>>>> required for this test >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_superlu_dist.counts >>>>>>>>>> ok snes_tutorials-ex19_superlu_dist >>>>>>>>>> ok diff-snes_tutorials-ex19_superlu_dist >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre.counts >>>>>>>>>> ok >>>>>>>>>> snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre >>>>>>>>>> ok >>>>>>>>>> diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex49_hypre_nullspace.counts >>>>>>>>>> ok ksp_ksp_tutorials-ex49_hypre_nullspace >>>>>>>>>> ok diff-ksp_ksp_tutorials-ex49_hypre_nullspace >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_superlu_dist_2.counts >>>>>>>>>> ok snes_tutorials-ex19_superlu_dist_2 >>>>>>>>>> ok diff-snes_tutorials-ex19_superlu_dist_2 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist_2.counts >>>>>>>>>> not ok ksp_ksp_tutorials-ex5_superlu_dist_2 # Error code: 1 >>>>>>>>>> #srun: error: Unable to create step for job 1426755: More >>>>>>>>>> processors requested than permitted >>>>>>>>>> ok ksp_ksp_tutorials-ex5_superlu_dist_2 # SKIP Command >>>>>>>>>> failed so no diff >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre.counts >>>>>>>>>> ok >>>>>>>>>> snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre >>>>>>>>>> ok >>>>>>>>>> diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex64_1.counts >>>>>>>>>> ok ksp_ksp_tutorials-ex64_1 >>>>>>>>>> ok diff-ksp_ksp_tutorials-ex64_1 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist.counts >>>>>>>>>> not ok ksp_ksp_tutorials-ex5_superlu_dist # Error code: 1 >>>>>>>>>> #srun: error: Unable to create step for job 1426755: More >>>>>>>>>> processors requested than permitted >>>>>>>>>> ok ksp_ksp_tutorials-ex5_superlu_dist # SKIP Command >>>>>>>>>> failed so no diff >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist_2.counts >>>>>>>>>> ok ksp_ksp_tutorials-ex5f_superlu_dist_2 # SKIP Fortran >>>>>>>>>> required for this test >>>>>>>>>> >>>>>>>>>>> Barry >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On Mar 10, 2021, at 11:03 PM, Eric Chamberland >>>>>>>>>>>> <Eric.Chamberland(a)giref.ulaval.ca >>>>>>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Barry, >>>>>>>>>>>> >>>>>>>>>>>> to get a some follow up on --with-openmp=1 failures, >>>>>>>>>>>> shall I open gitlab issues for: >>>>>>>>>>>> >>>>>>>>>>>> a) all hypre failures giving DIVERGED_INDEFINITE_PC >>>>>>>>>>>> >>>>>>>>>>>> b) all superlu_dist failures giving different results >>>>>>>>>>>> with initia and "Exceeded timeout limit of 60 s" >>>>>>>>>>>> >>>>>>>>>>>> c) hpddm failures "free(): invalid next size (fast)" >>>>>>>>>>>> and "Segmentation Violation" >>>>>>>>>>>> >>>>>>>>>>>> d) all tao's "Exceeded timeout limit of 60 s" >>>>>>>>>>>> >>>>>>>>>>>> I don't see how I could do all these debugging by myself... >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Eric >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>>> Professionnel de recherche >>>>>>>>> GIREF/Université Laval >>>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>> -- >>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>> Professionnel de recherche >>>>>>>> GIREF/Université Laval >>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>> <fedora_mkl_and_devtools.txt><openmpi.txt><petsc.txt> >>>>>>> >>>>>> >>>>> >>>> -- >>>> Eric Chamberland, ing., M. Ing >>>> Professionnel de recherche >>>> GIREF/Université Laval >>>> (418) 656-2131 poste 41 22 42 >>> >> -- >> Eric Chamberland, ing., M. Ing >> Professionnel de recherche >> GIREF/Université Laval >> (418) 656-2131 poste 41 22 42 > -- > Eric Chamberland, ing., M. Ing > Professionnel de recherche > GIREF/Université Laval > (418) 656-2131 poste 41 22 42 -- Eric Chamberland, ing., M. Ing Professionnel de recherche GIREF/Université Laval (418) 656-2131 poste 41 22 42

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Jed Brown 19 Mar '21

19 Mar '21

Matthew Knepley <knepley(a)gmail.com> writes: >> PetscSectionSetUp currently calculates offsets by walking the points in >> order. I'd like to be able to walk them in a different ordering, perhaps >> specified via an IS permutation, which could be computed using >> MATORDERINGRCM or directly via BFS of a closure (saves building a Mat). >> > > You can do this using > > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PetscSection/P… > > This is what Lawrence is doing. He permutes the points so that interior > points come first, then points that are not > shared but touch shared points, then shared points. Cool, this sounds easy enough now. What user interface would you like to use to have DMPlex apply an ordering when constructing a section after DMAddField()? DMPlexSetDofOrderingType(DM, MatOrderingType) To be set before DMSetUp with "natural" as default and "rcm" (or a new "bfs") taking a fast path that just walks the closure instead of building a matrix?

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Matthew Knepley 19 Mar '21

19 Mar '21

On Fri, Mar 19, 2021 at 2:55 PM Jed Brown <jed(a)jedbrown.org> wrote: > Matthew Knepley <knepley(a)gmail.com> writes: > > > On Fri, Mar 19, 2021 at 2:21 PM Jed Brown <jed(a)jedbrown.org> wrote: > > > >> Matthew Knepley <knepley(a)gmail.com> writes: > >> > >> >> Notice how the permutations are contained within the vertices {0, > ..., > >> 8}, > >> >> edges {9, ..., 24}, and cells {25, ..., 32}. I would like to get rid > of > >> >> that restriction, but you've said it would have significant non-local > >> >> consequences so I haven't tried. > >> >> > >> > > >> > Yes, that is not a good idea, even in theory. It wrecks the nice > >> contiguity > >> > we use for topological operations. > >> > > >> > As Lawrence points out, this need not affect your smoother because dof > >> > permutations can break this stratification. It has been that way for a > >> > decade. > >> > >> I only want to break dof stratification, not point stratification. The > >> points produced by the stratified RCM are okay (at least as Lawrence > >> described it; not sure how it's done in DMPlexGetOrdering, but can't be > >> anything hard). > >> > >> What code needs to be written to get unstratified RCM-like dofs? > Lawrence > >> evidently has this code in Firedrake, but I want it in DMPlex, tested > and > >> probably done by default because most users would benefit from it. I > could > >> swear you said it would break some other assumptions and thus not > >> everything would work, but perhaps we weren't talking about the same > thing. > >> > > > > It is possible that I misunderstood what you were asking for. What > Section > > requires for everything to work is that dofs follow the point ordering. > > I think what you mean is that the dofs for any point are contiguous, not > that the dofs for point p come before the dofs for point p+1. > Yes. > > The point ordering can be anything we want since we can use a > permutation from the mesh point ordering. However, some stuff depends on > all dofs associated with a point being contiguous. Not everything does, > since I wrote something for LibMesh which allows them to order in a field > major way, instead of point major, but it is probably not possible to > individually permute unknowns. Usually, this is not what we want anyway I > would assume. > > I don't want to break up the dofs for a point, just an unstratified > locality-preserving ordering of dofs (not points). > Good. > PetscSectionSetUp currently calculates offsets by walking the points in > order. I'd like to be able to walk them in a different ordering, perhaps > specified via an IS permutation, which could be computed using > MATORDERINGRCM or directly via BFS of a closure (saves building a Mat). > You can do this using https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PetscSection/P… This is what Lawrence is doing. He permutes the points so that interior points come first, then points that are not shared but touch shared points, then shared points. Thanks, Matt > 1. Would this break anything you can think of? > 2. How would you like it implemented? > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Jed Brown 19 Mar '21

19 Mar '21

Matthew Knepley <knepley(a)gmail.com> writes: > On Fri, Mar 19, 2021 at 2:21 PM Jed Brown <jed(a)jedbrown.org> wrote: > >> Matthew Knepley <knepley(a)gmail.com> writes: >> >> >> Notice how the permutations are contained within the vertices {0, ..., >> 8}, >> >> edges {9, ..., 24}, and cells {25, ..., 32}. I would like to get rid of >> >> that restriction, but you've said it would have significant non-local >> >> consequences so I haven't tried. >> >> >> > >> > Yes, that is not a good idea, even in theory. It wrecks the nice >> contiguity >> > we use for topological operations. >> > >> > As Lawrence points out, this need not affect your smoother because dof >> > permutations can break this stratification. It has been that way for a >> > decade. >> >> I only want to break dof stratification, not point stratification. The >> points produced by the stratified RCM are okay (at least as Lawrence >> described it; not sure how it's done in DMPlexGetOrdering, but can't be >> anything hard). >> >> What code needs to be written to get unstratified RCM-like dofs? Lawrence >> evidently has this code in Firedrake, but I want it in DMPlex, tested and >> probably done by default because most users would benefit from it. I could >> swear you said it would break some other assumptions and thus not >> everything would work, but perhaps we weren't talking about the same thing. >> > > It is possible that I misunderstood what you were asking for. What Section > requires for everything to work is that dofs follow the point ordering. I think what you mean is that the dofs for any point are contiguous, not that the dofs for point p come before the dofs for point p+1. > The point ordering can be anything we want since we can use a permutation from the mesh point ordering. However, some stuff depends on all dofs associated with a point being contiguous. Not everything does, since I wrote something for LibMesh which allows them to order in a field major way, instead of point major, but it is probably not possible to individually permute unknowns. Usually, this is not what we want anyway I would assume. I don't want to break up the dofs for a point, just an unstratified locality-preserving ordering of dofs (not points). PetscSectionSetUp currently calculates offsets by walking the points in order. I'd like to be able to walk them in a different ordering, perhaps specified via an IS permutation, which could be computed using MATORDERINGRCM or directly via BFS of a closure (saves building a Mat). 1. Would this break anything you can think of? 2. How would you like it implemented?

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Matthew Knepley 19 Mar '21

19 Mar '21

On Fri, Mar 19, 2021 at 2:21 PM Jed Brown <jed(a)jedbrown.org> wrote: > Matthew Knepley <knepley(a)gmail.com> writes: > > >> Notice how the permutations are contained within the vertices {0, ..., > 8}, > >> edges {9, ..., 24}, and cells {25, ..., 32}. I would like to get rid of > >> that restriction, but you've said it would have significant non-local > >> consequences so I haven't tried. > >> > > > > Yes, that is not a good idea, even in theory. It wrecks the nice > contiguity > > we use for topological operations. > > > > As Lawrence points out, this need not affect your smoother because dof > > permutations can break this stratification. It has been that way for a > > decade. > > I only want to break dof stratification, not point stratification. The > points produced by the stratified RCM are okay (at least as Lawrence > described it; not sure how it's done in DMPlexGetOrdering, but can't be > anything hard). > > What code needs to be written to get unstratified RCM-like dofs? Lawrence > evidently has this code in Firedrake, but I want it in DMPlex, tested and > probably done by default because most users would benefit from it. I could > swear you said it would break some other assumptions and thus not > everything would work, but perhaps we weren't talking about the same thing. > It is possible that I misunderstood what you were asking for. What Section requires for everything to work is that dofs follow the point ordering. The point ordering can be anything we want since we can use a permutation from the mesh point ordering. However, some stuff depends on all dofs associated with a point being contiguous. Not everything does, since I wrote something for LibMesh which allows them to order in a field major way, instead of point major, but it is probably not possible to individually permute unknowns. Usually, this is not what we want anyway I would assume. Thanks, Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Jed Brown 19 Mar '21

19 Mar '21

Matthew Knepley <knepley(a)gmail.com> writes: >> Notice how the permutations are contained within the vertices {0, ..., 8}, >> edges {9, ..., 24}, and cells {25, ..., 32}. I would like to get rid of >> that restriction, but you've said it would have significant non-local >> consequences so I haven't tried. >> > > Yes, that is not a good idea, even in theory. It wrecks the nice contiguity > we use for topological operations. > > As Lawrence points out, this need not affect your smoother because dof > permutations can break this stratification. It has been that way for a > decade. I only want to break dof stratification, not point stratification. The points produced by the stratified RCM are okay (at least as Lawrence described it; not sure how it's done in DMPlexGetOrdering, but can't be anything hard). What code needs to be written to get unstratified RCM-like dofs? Lawrence evidently has this code in Firedrake, but I want it in DMPlex, tested and probably done by default because most users would benefit from it. I could swear you said it would break some other assumptions and thus not everything would work, but perhaps we weren't talking about the same thing.

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Matthew Knepley 19 Mar '21

19 Mar '21

On Fri, Mar 19, 2021 at 11:52 AM Lawrence Mitchell <wence(a)gmx.li> wrote: > > On 19 Mar 2021, at 15:20, Jed Brown <jed(a)jedbrown.org> wrote: > > > >> I guess the only issue is now one needs to always remember to traverse > cells in the reordered_cells order (although if you call DMPlexPermute that > is automatic?). > > > > Can DMPlexPermute handle a non-stratified permutation? > > I think the answer here is still no (as per your previous statement). The > way we work around this in Firedrake is as I described above. Effectively > we always traverse the cells in order specified by the RCM reordering. For > traversal of facets and other entities we then build a traversal order once > by traversing the cells and greedily ordering the facets/edges/etc... > > I think you could get this behaviour from dmplexpermute if one wrote it so > that it permutes the cells with RCM and then compatibly permutes > lower-dimensional entities. This would maintain stratification, but I think > keep good cache locality for all entity traversals. But I do not understand > the details of what dmplexpermute is doing at the moment. > I thought Plex already did that. Maybe I am wrong, it was a long time ago. Matt > Cheers, > > Lawrence -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Matthew Knepley 19 Mar '21

19 Mar '21

On Fri, Mar 19, 2021 at 10:21 AM Jed Brown <jed(a)jedbrown.org> wrote: > Matthew Knepley <knepley(a)gmail.com> writes: > > > On Fri, Mar 19, 2021 at 7:30 AM Lawrence Mitchell <wence(a)gmx.li> wrote: > > > >> > On 19 Mar 2021, at 03:51, Jed Brown <jed(a)jedbrown.org> wrote: > >> > > >> > It's a notable weakness of DMPlex that it does not apply such an > >> ordering of dofs and I've complained to Matt about it many times over > the > >> years, but any blame rests solely with me for not carving out time to > >> implement it here. > >> > >> I think there is a way to do this with plex (at least we do it), like > so: > >> > >> DMPlexGetOrdering(dm, MATORDERINGRCM, NULL, &isperm); > >> > >> you might need to then invert this ordering. > >> > >> Now when you make a section you either say > >> > >> PetscSectionSetPermutation(section, isperm); > >> > > > > You can do that if you only want to permute the dofs, but usually I want > to > > reorder everything so that any Section comes out right > > > > DMPlexPermute() > > Last we talked, and all that's tested, this can only permute within > strata, not across strata. That would incur the "basically point Jacobi" > problem for naive OpenMP smoothers highlighted in Allison's paper and (what > I care about more) the memory locality impact is similar to moving from > interlaced fields to segregated fields. > > diff --git i/src/dm/impls/plex/tests/ex10.c > w/src/dm/impls/plex/tests/ex10.c > index eed9aa878e..cfe35dc1a6 100644 > --- i/src/dm/impls/plex/tests/ex10.c > +++ w/src/dm/impls/plex/tests/ex10.c > @@ -80,6 +80,7 @@ PetscErrorCode TestReordering(DM dm, AppCtx *user) > > PetscFunctionBegin; > ierr = DMPlexGetOrdering(dm, order, NULL, &perm);CHKERRQ(ierr); > + ierr = ISView(perm, NULL);CHKERRQ(ierr); > ierr = DMPlexPermute(dm, perm, &pdm);CHKERRQ(ierr); > ierr = PetscObjectSetOptionsPrefix((PetscObject) pdm, > "perm_");CHKERRQ(ierr); > ierr = DMSetFromOptions(pdm);CHKERRQ(ierr); > > > > $ $PETSC_ARCH/tests/dm/impls/plex/tests/ex10 -dim 2 -num_dof 1,1,1 > -perm_dm_view > IS Object: 1 MPI processes > type: general > Number of indices in set 33 > 0 0 > 1 2 > 2 1 > 3 5 > 4 3 > 5 7 > 6 6 > 7 4 > 8 8 > 9 9 > 10 13 > 11 10 > 12 11 > 13 15 > 14 12 > 15 14 > 16 16 > 17 17 > 18 18 > 19 19 > 20 22 > 21 23 > 22 21 > 23 20 > 24 24 > 25 28 > 26 29 > 27 25 > 28 30 > 29 31 > 30 32 > 31 26 > 32 27 > DM Object: (perm_) 1 MPI processes > type: plex > DM_0x564eba4a45d0_1 in 2 dimensions: > 0-cells: 9 > 1-cells: 16 > 2-cells: 8 > Labels: > celltype: 3 strata with value/size (0 (9), 3 (8), 1 (16)) > depth: 3 strata with value/size (0 (9), 1 (16), 2 (8)) > marker: 1 strata with value/size (1 (16)) > Face Sets: 1 strata with value/size (1 (8)) > Field Field_0: > adjacency FEM > Ordering method rcm reduced bandwidth from 51 to 51 > > > Notice how the permutations are contained within the vertices {0, ..., 8}, > edges {9, ..., 24}, and cells {25, ..., 32}. I would like to get rid of > that restriction, but you've said it would have significant non-local > consequences so I haven't tried. > Yes, that is not a good idea, even in theory. It wrecks the nice contiguity we use for topological operations. As Lawrence points out, this need not affect your smoother because dof permutations can break this stratification. It has been that way for a decade. Thanks, Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Lawrence Mitchell 19 Mar '21

19 Mar '21

> On 19 Mar 2021, at 15:20, Jed Brown <jed(a)jedbrown.org> wrote: > >> I guess the only issue is now one needs to always remember to traverse cells in the reordered_cells order (although if you call DMPlexPermute that is automatic?). > > Can DMPlexPermute handle a non-stratified permutation? I think the answer here is still no (as per your previous statement). The way we work around this in Firedrake is as I described above. Effectively we always traverse the cells in order specified by the RCM reordering. For traversal of facets and other entities we then build a traversal order once by traversing the cells and greedily ordering the facets/edges/etc... I think you could get this behaviour from dmplexpermute if one wrote it so that it permutes the cells with RCM and then compatibly permutes lower-dimensional entities. This would maintain stratification, but I think keep good cache locality for all entity traversals. But I do not understand the details of what dmplexpermute is doing at the moment. Cheers, Lawrence

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Jed Brown 19 Mar '21

19 Mar '21

Lawrence Mitchell <wence(a)gmx.li> writes: >> On 19 Mar 2021, at 14:21, Jed Brown <jed(a)jedbrown.org> wrote: >> >> Notice how the permutations are contained within the vertices {0, ..., 8}, edges {9, ..., 24}, and cells {25, ..., 32}. I would like to get rid of that restriction, but you've said it would have significant non-local consequences so I haven't tried. > > What we do, and I'm not sure if this is exactly what you want, is the following. > > Use DMPlexGetOrdering to compute permutatations of points with RCM. As you say, this is stratified. > > Then, we traverse cells in this RCM ordering and compute a permutation of the full plex chart greedily. > > Pseudo-code: > > permutation = empty(pstart - pend) > i = 0 > for cell in reordered_cells: > points = plex.getclosure(cell) > for point in points: > if not seen(point): > permutation[i] = point > i += 1 > > Now when we create a section, we say > > PetscSectionSetPermutation(sec, permutation) > > And then when the section offsets are calculated, the points are visited and numbered in the order provided by the permutation. > > This seems to work, here are the sparsity patterns for a Q3 mass matrix on an unstructured quad mesh (from gmsh). One with RCM ordering applied, the other without). This result looks good. > I guess the only issue is now one needs to always remember to traverse cells in the reordered_cells order (although if you call DMPlexPermute that is automatic?). Can DMPlexPermute handle a non-stratified permutation?

1 0