March 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Mark Adams 13 Mar '21

13 Mar '21

Hypre uses a multiplicative smoother by default. It has a chebyshev smoother. That with a Jacobi PC should be thread invariant. Mark On Sat, Mar 13, 2021 at 8:18 AM Pierre Jolivet <pierre(a)joliv.et> wrote: > > On 13 Mar 2021, at 9:17 AM, Pierre Jolivet <pierre(a)joliv.et> wrote: > > Hello Eric, > I’ve made an “interesting” discovery, so I’ll put back the list in c/c. > It appears the following snippet of code which uses Allreduce() + lambda > function + MPI_IN_PLACE is: > - Valgrind-clean with MPICH; > - Valgrind-clean with OpenMPI 4.0.5; > - not Valgrind-clean with OpenMPI 4.1.0. > I’m not sure who is to blame here, I’ll need to look at the MPI > specification for what is required by the implementors and users in that > case. > > In the meantime, I’ll do the following: > - update config/BuildSystem/config/packages/OpenMPI.py to use OpenMPI > 4.1.0, see if any other error appears; > - provide a hotfix to bypass the segfaults; > > > I can confirm that splitting the single Allreduce with my own MPI_Op into > two Allreduce with MAX and BAND fixes the segfaults with OpenMPI (*). > > - look at the hypre issue and whether they should be deferred to the hypre > team. > > > I don’t know if there is something wrong in hypre threading or if it’s > just a side effect of threading, but it seems that the number of threads > has a drastic effect on the quality of the PC. > By default, it looks that there are two threads per process with your > Docker image. > If I force OMP_NUM_THREADS=1, then I get the same convergence as in the > output file. > > Thanks, > Pierre > > (*) https://gitlab.com/petsc/petsc/-/merge_requests/3712 > > Thank you for the Docker files, they were really useful. > If you want to avoid oversubscription failures, you can edit the file > /opt/openmpi-4.1.0/etc/openmpi-default-hostfile and append the line: > localhost slots=12 > If you want to increase the timeout limit of PETSc test suite for each > test, you can add the extra flag in your command line TIMEOUT=180 (default > is 60, units are seconds). > > Thanks, I’ll ping you on GitLab when I’ve got something ready for you to > try, > Pierre > > <ompi.cxx> > > On 12 Mar 2021, at 8:54 PM, Eric Chamberland < > Eric.Chamberland(a)giref.ulaval.ca> wrote: > > Hi Pierre, > > I now have a docker container reproducing the problems here. > > Actually, if I look at snes_tutorials-ex12_quad_singular_hpddm it fails > like this: > > not ok snes_tutorials-ex12_quad_singular_hpddm # Error code: 59 > # Initial guess > # L_2 Error: 0.00803099 > # Initial Residual > # L_2 Residual: 1.09057 > # Au - b = Au + F(0) > # Linear L_2 Residual: 1.09057 > # [d470c54ce086:14127] Read -1, expected 4096, errno = 1 > # [d470c54ce086:14128] Read -1, expected 4096, errno = 1 > # [d470c54ce086:14129] Read -1, expected 4096, errno = 1 > # [3]PETSC ERROR: > ------------------------------------------------------------------------ > # [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation > Violation, probably memory access out of range > # [3]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > # [3]PETSC ERROR: or see > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > # [3]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple > Mac OS X to find memory corruption errors > # [3]PETSC ERROR: likely location of problem given in stack below > # [3]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > # [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > # [3]PETSC ERROR: INSTEAD the line number of the start of the > function > # [3]PETSC ERROR: is given. > # [3]PETSC ERROR: [3] buildTwo line 987 > /opt/petsc-main/include/HPDDM_schwarz.hpp > # [3]PETSC ERROR: [3] next line 1130 > /opt/petsc-main/include/HPDDM_schwarz.hpp > # [3]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > # [3]PETSC ERROR: Signal received > # [3]PETSC ERROR: [0]PETSC ERROR: > ------------------------------------------------------------------------ > > also ex12_quad_hpddm_reuse_baij fails with a lot more "Read -1, expected > ..." which I don't know where they come from...? > > Hypre (like in diff-snes_tutorials-ex56_hypre) is also having > DIVERGED_INDEFINITE_PC failures... > > Please see the 3 attached docker files: > > 1) fedora_mkl_and_devtools : the DockerFile which install fedore 33 with > gnu compilers and MKL and everything to develop. > > 2) openmpi: the DockerFile to bluid OpenMPI > > 3) petsc: The las DockerFile that build/install and test PETSc > > I build the 3 like this: > > docker build -t fedora_mkl_and_devtools -f fedora_mkl_and_devtools . > > docker build -t openmpi -f openmpi . > > docker build -t petsc -f petsc . > > Disclaimer: I am not a docker expert, so I may do things that are not > docker-stat-of-the-art but I am opened to suggestions... ;) > > I have just ran it on my portable (long) which have not enough cores, so > many more tests failed (should force --oversubscribe but don't know how > to). I will relaunch on my workstation in a few minutes. > > I will now test your branch! (sorry for the delay). > > Thanks, > > Eric > On 2021-03-11 9:03 a.m., Eric Chamberland wrote: > > Hi Pierre, > > ok, that's interesting! > > I will try to build a docker image until tomorrow and give you the exact > recipe to reproduce the bugs. > > Eric > > > On 2021-03-11 2:46 a.m., Pierre Jolivet wrote: > > > > On 11 Mar 2021, at 6:16 AM, Barry Smith <bsmith(a)petsc.dev> wrote: > > > Eric, > > Sorry about not being more immediate. We still have this in our active > email so you don't need to submit individual issues. We'll try to get to > them as soon as we can. > > > Indeed, I’m still trying to figure this out. > I realized that some of my configure flags were different than yours, > e.g., no --with-memalign. > I’ve also added SuperLU_DIST to my installation. > Still, I can’t reproduce any issue. > I will continue looking into this, it appears I’m seeing some valgrind > errors, but I don’t know if this is some side effect of OpenMPI not being > valgrind-clean (last time I checked, there was no error with MPICH). > > Thank you for your patience, > Pierre > > /usr/bin/gmake -f gmakefile test test-fail=1 > Using MAKEFLAGS: test-fail=1 > TEST > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_baij.counts > ok snes_tutorials-ex12_quad_hpddm_reuse_baij > ok diff-snes_tutorials-ex12_quad_hpddm_reuse_baij > TEST > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist_2.counts > ok ksp_ksp_tests-ex33_superlu_dist_2 > ok diff-ksp_ksp_tests-ex33_superlu_dist_2 > TEST > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex49_superlu_dist.counts > ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 > ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 > ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 > ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 > ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 > ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 > ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 > ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 > ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 > ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 > ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 > ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 > ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 > ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 > ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 > ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 > TEST > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex50_tut_2.counts > ok ksp_ksp_tutorials-ex50_tut_2 > ok diff-ksp_ksp_tutorials-ex50_tut_2 > TEST > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist.counts > ok ksp_ksp_tests-ex33_superlu_dist > ok diff-ksp_ksp_tests-ex33_superlu_dist > TEST > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_hypre.counts > ok snes_tutorials-ex56_hypre > ok diff-snes_tutorials-ex56_hypre > TEST > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex56_2.counts > ok ksp_ksp_tutorials-ex56_2 > ok diff-ksp_ksp_tutorials-ex56_2 > TEST > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_elas.counts > ok snes_tutorials-ex17_3d_q3_trig_elas > ok diff-snes_tutorials-ex17_3d_q3_trig_elas > TEST > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij.counts > ok snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij > ok diff-snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij > TEST > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist_3.counts > not ok ksp_ksp_tutorials-ex5_superlu_dist_3 # Error code: 1 > # srun: error: Unable to create step for job 1426755: More processors > requested than permitted > ok ksp_ksp_tutorials-ex5_superlu_dist_3 # SKIP Command failed so no diff > TEST > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist.counts > ok ksp_ksp_tutorials-ex5f_superlu_dist # SKIP Fortran required for this > test > TEST > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_tri_parmetis_hpddm_baij.counts > ok snes_tutorials-ex12_tri_parmetis_hpddm_baij > ok diff-snes_tutorials-ex12_tri_parmetis_hpddm_baij > TEST > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_tut_3.counts > ok snes_tutorials-ex19_tut_3 > ok diff-snes_tutorials-ex19_tut_3 > TEST > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_vlap.counts > ok snes_tutorials-ex17_3d_q3_trig_vlap > ok diff-snes_tutorials-ex17_3d_q3_trig_vlap > TEST > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist_3.counts > ok ksp_ksp_tutorials-ex5f_superlu_dist_3 # SKIP Fortran required for this > test > TEST > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_superlu_dist.counts > ok snes_tutorials-ex19_superlu_dist > ok diff-snes_tutorials-ex19_superlu_dist > TEST > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre.counts > ok snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre > ok diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre > TEST > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex49_hypre_nullspace.counts > ok ksp_ksp_tutorials-ex49_hypre_nullspace > ok diff-ksp_ksp_tutorials-ex49_hypre_nullspace > TEST > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_superlu_dist_2.counts > ok snes_tutorials-ex19_superlu_dist_2 > ok diff-snes_tutorials-ex19_superlu_dist_2 > TEST > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist_2.counts > not ok ksp_ksp_tutorials-ex5_superlu_dist_2 # Error code: 1 > # srun: error: Unable to create step for job 1426755: More processors > requested than permitted > ok ksp_ksp_tutorials-ex5_superlu_dist_2 # SKIP Command failed so no diff > TEST > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre.counts > ok snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre > ok diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre > TEST > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex64_1.counts > ok ksp_ksp_tutorials-ex64_1 > ok diff-ksp_ksp_tutorials-ex64_1 > TEST > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist.counts > not ok ksp_ksp_tutorials-ex5_superlu_dist # Error code: 1 > # srun: error: Unable to create step for job 1426755: More processors > requested than permitted > ok ksp_ksp_tutorials-ex5_superlu_dist # SKIP Command failed so no diff > TEST > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist_2.counts > ok ksp_ksp_tutorials-ex5f_superlu_dist_2 # SKIP Fortran required for this > test > > Barry > > > On Mar 10, 2021, at 11:03 PM, Eric Chamberland < > Eric.Chamberland(a)giref.ulaval.ca> wrote: > > Barry, > > to get a some follow up on --with-openmp=1 failures, shall I open gitlab > issues for: > > a) all hypre failures giving DIVERGED_INDEFINITE_PC > > b) all superlu_dist failures giving different results with initia and > "Exceeded timeout limit of 60 s" > > c) hpddm failures "free(): invalid next size (fast)" and "Segmentation > Violation" > > d) all tao's "Exceeded timeout limit of 60 s" > > I don't see how I could do all these debugging by myself... > > Thanks, > > Eric > > > > -- > Eric Chamberland, ing., M. Ing > Professionnel de recherche > GIREF/Université Laval > (418) 656-2131 poste 41 22 42 > > -- > Eric Chamberland, ing., M. Ing > Professionnel de recherche > GIREF/Université Laval > (418) 656-2131 poste 41 22 42 > > <fedora_mkl_and_devtools.txt><openmpi.txt><petsc.txt> > > > >

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Pierre Jolivet 13 Mar '21

13 Mar '21

> On 13 Mar 2021, at 9:17 AM, Pierre Jolivet <pierre(a)joliv.et> wrote: > > Hello Eric, > I’ve made an “interesting” discovery, so I’ll put back the list in c/c. > It appears the following snippet of code which uses Allreduce() + lambda function + MPI_IN_PLACE is: > - Valgrind-clean with MPICH; > - Valgrind-clean with OpenMPI 4.0.5; > - not Valgrind-clean with OpenMPI 4.1.0. > I’m not sure who is to blame here, I’ll need to look at the MPI specification for what is required by the implementors and users in that case. > > In the meantime, I’ll do the following: > - update config/BuildSystem/config/packages/OpenMPI.py to use OpenMPI 4.1.0, see if any other error appears; > - provide a hotfix to bypass the segfaults; I can confirm that splitting the single Allreduce with my own MPI_Op into two Allreduce with MAX and BAND fixes the segfaults with OpenMPI (*). > - look at the hypre issue and whether they should be deferred to the hypre team. I don’t know if there is something wrong in hypre threading or if it’s just a side effect of threading, but it seems that the number of threads has a drastic effect on the quality of the PC. By default, it looks that there are two threads per process with your Docker image. If I force OMP_NUM_THREADS=1, then I get the same convergence as in the output file. Thanks, Pierre (*) https://gitlab.com/petsc/petsc/-/merge_requests/3712 <https://gitlab.com/petsc/petsc/-/merge_requests/3712> > Thank you for the Docker files, they were really useful. > If you want to avoid oversubscription failures, you can edit the file /opt/openmpi-4.1.0/etc/openmpi-default-hostfile and append the line: > localhost slots=12 > If you want to increase the timeout limit of PETSc test suite for each test, you can add the extra flag in your command line TIMEOUT=180 (default is 60, units are seconds). > > Thanks, I’ll ping you on GitLab when I’ve got something ready for you to try, > Pierre > > <ompi.cxx> > >> On 12 Mar 2021, at 8:54 PM, Eric Chamberland <Eric.Chamberland(a)giref.ulaval.ca <mailto:[email protected]>> wrote: >> >> Hi Pierre, >> >> I now have a docker container reproducing the problems here. >> >> Actually, if I look at snes_tutorials-ex12_quad_singular_hpddm it fails like this: >> >> not ok snes_tutorials-ex12_quad_singular_hpddm # Error code: 59 >> # Initial guess >> # L_2 Error: 0.00803099 >> # Initial Residual >> # L_2 Residual: 1.09057 >> # Au - b = Au + F(0) >> # Linear L_2 Residual: 1.09057 >> # [d470c54ce086:14127] Read -1, expected 4096, errno = 1 >> # [d470c54ce086:14128] Read -1, expected 4096, errno = 1 >> # [d470c54ce086:14129] Read -1, expected 4096, errno = 1 >> # [3]PETSC ERROR: ------------------------------------------------------------------------ >> # [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >> # [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> # [3]PETSC ERROR: or see https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind <https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind> >> # [3]PETSC ERROR: or try http://valgrind.org <http://valgrind.org/> on GNU/linux and Apple Mac OS X to find memory corruption errors >> # [3]PETSC ERROR: likely location of problem given in stack below >> # [3]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >> # [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >> # [3]PETSC ERROR: INSTEAD the line number of the start of the function >> # [3]PETSC ERROR: is given. >> # [3]PETSC ERROR: [3] buildTwo line 987 /opt/petsc-main/include/HPDDM_schwarz.hpp >> # [3]PETSC ERROR: [3] next line 1130 /opt/petsc-main/include/HPDDM_schwarz.hpp >> # [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> # [3]PETSC ERROR: Signal received >> # [3]PETSC ERROR: [0]PETSC ERROR: ------------------------------------------------------------------------ >> >> also ex12_quad_hpddm_reuse_baij fails with a lot more "Read -1, expected ..." which I don't know where they come from...? >> >> Hypre (like in diff-snes_tutorials-ex56_hypre) is also having DIVERGED_INDEFINITE_PC failures... >> >> Please see the 3 attached docker files: >> >> 1) fedora_mkl_and_devtools : the DockerFile which install fedore 33 with gnu compilers and MKL and everything to develop. >> >> 2) openmpi: the DockerFile to bluid OpenMPI >> >> 3) petsc: The las DockerFile that build/install and test PETSc >> >> I build the 3 like this: >> >> docker build -t fedora_mkl_and_devtools -f fedora_mkl_and_devtools . >> >> docker build -t openmpi -f openmpi . >> >> docker build -t petsc -f petsc . >> >> Disclaimer: I am not a docker expert, so I may do things that are not docker-stat-of-the-art but I am opened to suggestions... ;) >> >> I have just ran it on my portable (long) which have not enough cores, so many more tests failed (should force --oversubscribe but don't know how to). I will relaunch on my workstation in a few minutes. >> >> I will now test your branch! (sorry for the delay). >> >> Thanks, >> >> Eric >> >> On 2021-03-11 9:03 a.m., Eric Chamberland wrote: >>> Hi Pierre, >>> >>> ok, that's interesting! >>> >>> I will try to build a docker image until tomorrow and give you the exact recipe to reproduce the bugs. >>> >>> Eric >>> >>> >>> >>> On 2021-03-11 2:46 a.m., Pierre Jolivet wrote: >>>> >>>> >>>>> On 11 Mar 2021, at 6:16 AM, Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >>>>> >>>>> >>>>> Eric, >>>>> >>>>> Sorry about not being more immediate. We still have this in our active email so you don't need to submit individual issues. We'll try to get to them as soon as we can. >>>> >>>> Indeed, I’m still trying to figure this out. >>>> I realized that some of my configure flags were different than yours, e.g., no --with-memalign. >>>> I’ve also added SuperLU_DIST to my installation. >>>> Still, I can’t reproduce any issue. >>>> I will continue looking into this, it appears I’m seeing some valgrind errors, but I don’t know if this is some side effect of OpenMPI not being valgrind-clean (last time I checked, there was no error with MPICH). >>>> >>>> Thank you for your patience, >>>> Pierre >>>> >>>> /usr/bin/gmake -f gmakefile test test-fail=1 >>>> Using MAKEFLAGS: test-fail=1 >>>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_baij.counts >>>> ok snes_tutorials-ex12_quad_hpddm_reuse_baij >>>> ok diff-snes_tutorials-ex12_quad_hpddm_reuse_baij >>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist_2.counts >>>> ok ksp_ksp_tests-ex33_superlu_dist_2 >>>> ok diff-ksp_ksp_tests-ex33_superlu_dist_2 >>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex49_superlu_dist.counts >>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 >>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 >>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 >>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 >>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 >>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 >>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 >>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 >>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 >>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 >>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 >>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 >>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 >>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 >>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 >>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 >>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex50_tut_2.counts >>>> ok ksp_ksp_tutorials-ex50_tut_2 >>>> ok diff-ksp_ksp_tutorials-ex50_tut_2 >>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist.counts >>>> ok ksp_ksp_tests-ex33_superlu_dist >>>> ok diff-ksp_ksp_tests-ex33_superlu_dist >>>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_hypre.counts >>>> ok snes_tutorials-ex56_hypre >>>> ok diff-snes_tutorials-ex56_hypre >>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex56_2.counts >>>> ok ksp_ksp_tutorials-ex56_2 >>>> ok diff-ksp_ksp_tutorials-ex56_2 >>>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_elas.counts >>>> ok snes_tutorials-ex17_3d_q3_trig_elas >>>> ok diff-snes_tutorials-ex17_3d_q3_trig_elas >>>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij.counts >>>> ok snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij >>>> ok diff-snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij >>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist_3.counts >>>> not ok ksp_ksp_tutorials-ex5_superlu_dist_3 # Error code: 1 >>>> # srun: error: Unable to create step for job 1426755: More processors requested than permitted >>>> ok ksp_ksp_tutorials-ex5_superlu_dist_3 # SKIP Command failed so no diff >>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist.counts >>>> ok ksp_ksp_tutorials-ex5f_superlu_dist # SKIP Fortran required for this test >>>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_tri_parmetis_hpddm_baij.counts >>>> ok snes_tutorials-ex12_tri_parmetis_hpddm_baij >>>> ok diff-snes_tutorials-ex12_tri_parmetis_hpddm_baij >>>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_tut_3.counts >>>> ok snes_tutorials-ex19_tut_3 >>>> ok diff-snes_tutorials-ex19_tut_3 >>>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_vlap.counts >>>> ok snes_tutorials-ex17_3d_q3_trig_vlap >>>> ok diff-snes_tutorials-ex17_3d_q3_trig_vlap >>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist_3.counts >>>> ok ksp_ksp_tutorials-ex5f_superlu_dist_3 # SKIP Fortran required for this test >>>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_superlu_dist.counts >>>> ok snes_tutorials-ex19_superlu_dist >>>> ok diff-snes_tutorials-ex19_superlu_dist >>>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre.counts >>>> ok snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre >>>> ok diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre >>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex49_hypre_nullspace.counts >>>> ok ksp_ksp_tutorials-ex49_hypre_nullspace >>>> ok diff-ksp_ksp_tutorials-ex49_hypre_nullspace >>>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_superlu_dist_2.counts >>>> ok snes_tutorials-ex19_superlu_dist_2 >>>> ok diff-snes_tutorials-ex19_superlu_dist_2 >>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist_2.counts >>>> not ok ksp_ksp_tutorials-ex5_superlu_dist_2 # Error code: 1 >>>> # srun: error: Unable to create step for job 1426755: More processors requested than permitted >>>> ok ksp_ksp_tutorials-ex5_superlu_dist_2 # SKIP Command failed so no diff >>>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre.counts >>>> ok snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre >>>> ok diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre >>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex64_1.counts >>>> ok ksp_ksp_tutorials-ex64_1 >>>> ok diff-ksp_ksp_tutorials-ex64_1 >>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist.counts >>>> not ok ksp_ksp_tutorials-ex5_superlu_dist # Error code: 1 >>>> # srun: error: Unable to create step for job 1426755: More processors requested than permitted >>>> ok ksp_ksp_tutorials-ex5_superlu_dist # SKIP Command failed so no diff >>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist_2.counts >>>> ok ksp_ksp_tutorials-ex5f_superlu_dist_2 # SKIP Fortran required for this test >>>> >>>>> Barry >>>>> >>>>> >>>>>> On Mar 10, 2021, at 11:03 PM, Eric Chamberland <Eric.Chamberland(a)giref.ulaval.ca <mailto:[email protected]>> wrote: >>>>>> >>>>>> Barry, >>>>>> >>>>>> to get a some follow up on --with-openmp=1 failures, shall I open gitlab issues for: >>>>>> >>>>>> a) all hypre failures giving DIVERGED_INDEFINITE_PC >>>>>> >>>>>> b) all superlu_dist failures giving different results with initia and "Exceeded timeout limit of 60 s" >>>>>> >>>>>> c) hpddm failures "free(): invalid next size (fast)" and "Segmentation Violation" >>>>>> >>>>>> d) all tao's "Exceeded timeout limit of 60 s" >>>>>> >>>>>> I don't see how I could do all these debugging by myself... >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Eric >>>>>> >>>>>> >>>>> >>>> >>> -- >>> Eric Chamberland, ing., M. Ing >>> Professionnel de recherche >>> GIREF/Université Laval >>> (418) 656-2131 poste 41 22 42 >> -- >> Eric Chamberland, ing., M. Ing >> Professionnel de recherche >> GIREF/Université Laval >> (418) 656-2131 poste 41 22 42 >> <fedora_mkl_and_devtools.txt><openmpi.txt><petsc.txt> >

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Eric Chamberland 13 Mar '21

13 Mar '21

Hi Pierre! On 2021-03-13 3:17 a.m., Pierre Jolivet wrote: > Hello Eric, > I’ve made an “interesting” discovery, so I’ll put back the list in c/c. > It appears the following snippet of code which uses Allreduce() + > lambda function + MPI_IN_PLACE is: > - Valgrind-clean with MPICH; > - Valgrind-clean with OpenMPI 4.0.5; > - not Valgrind-clean with OpenMPI 4.1.0. > I’m not sure who is to blame here, I’ll need to look at the MPI > specification for what is required by the implementors and users in > that case. > > In the meantime, I’ll do the following: > - update config/BuildSystem/config/packages/OpenMPI.py to use OpenMPI > 4.1.0, see if any other error appears; ok, I think it is a good idea since 4.1 is the "stable" version... > - provide a hotfix to bypass the segfaults; If OpenMPI guys fix the issue, maybe it will be included in the 4.1.1 release (4.1.1rc1 is still open for modifications I think)... An idea: I our code, we "block" compilations/usages of buggy MPIs versions... > - look at the hypre issue and whether they should be deferred to the > hypre team. Oh thanks for this! :) > > Thank you for the Docker files, they were really useful. > If you want to avoid oversubscription failures, you can edit the file > /opt/openmpi-4.1.0/etc/openmpi-default-hostfile and append the line: > localhost slots=12 > If you want to increase the timeout limit of PETSc test suite for each > test, you can add the extra flag in your command line TIMEOUT=180 > (default is 60, units are seconds). Noted, I will add this to my scripts and the dockerfiles too... > > Thanks, I’ll ping you on GitLab when I’ve got something ready for you > to try, Thank you for all your work! Eric > Pierre > -- Eric Chamberland, ing., M. Ing Professionnel de recherche GIREF/Université Laval (418) 656-2131 poste 41 22 42

1 0

Re: [petsc-dev] Argonne GPU Virtual Hackathon - Accepted
by Patrick Sanan 13 Mar '21

13 Mar '21

Another thing perhaps of interest is the stencil-based GPU matrix assembly functionality that Mark introduced. > Am 13.03.2021 um 07:59 schrieb Stefano Zampini <stefano.zampini(a)gmail.com>: > > The COO assembly is entirely based on thrust primitives, I don’t have much experience to say we will get a serious speedup by writing our own kernels, but it is definitely worth a try if we will end up adopting COO as entry point for GPU irregular assembly. > Jed, you mentioned BDDC deluxe, what do you mean by that? Porting setup/application of deluxe scaling onto GPU? > > Timings are not so bad for me joining the hackaton. > >> On Mar 13, 2021, at 8:17 AM, Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >> >> >> >>> On Mar 12, 2021, at 10:49 PM, Jed Brown <jed(a)jedbrown.org <mailto:[email protected]>> wrote: >>> >>> Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> writes: >>> >>>>> On Mar 12, 2021, at 6:58 PM, Jed Brown <jed(a)jedbrown.org <mailto:[email protected]>> wrote: >>>>> >>>>> Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> writes: >>>>> >>>>>> I think we should start porting the PetscFE infrastructure, numerical integrations, vector and matrix assembly to GPUs soon. It is dog slow on CPUs and should be able to deliver higher performance on GPUs. >>>>> >>>>> IMO, this comes via interfaces to libCEED, not rolling yet another way to invoke quadrature routines on GPUs. >>>> >>>> I am not talking about matrix-free stuff, that definitely belongs in libCEED, no reason to rewrite. >>>> >>>> But does libCEED also support the traditional finite element construction process where the matrices are built explicitly? Or does it provide some of the code, integration points, integration formula etc. that could be shared and used as a starting point? If it includes all of these "traditional" things then we should definitely get it all hooked into PetscFE/DMPLEX and go to town. (But yes not so much need for the GPU hackathon since it is wiring more than GPU code). The way I have always heard about libCEED was as a matrix-free engine, so I may have miss understood. It is definitely not my intention to start a project that reproduces functionality that we can just use. >>> >>> MFEM wants this too and it's in a draft libCEED PR right now. My intent is to ensure it's compatible with Stefano's split-phase COO assembly. >> >> Cool, would this be something that, in combination with perhaps some libCEED folk, could be incorporated in the Hackathon? Anyone can join our group Hackathon group, they don't have to have any financial connection with "PETSc". >> >>> >>>> We do need solid support for traditional finite element assembly on GPUs, matrix-free finite elements alone is not enough. >>> >>> Agreed, and while libCEED could be further optimized for lowest order, even naive assembly will be faster than what's in DMPlex. >

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Pierre Jolivet 13 Mar '21

13 Mar '21

Hello Eric, I’ve made an “interesting” discovery, so I’ll put back the list in c/c. It appears the following snippet of code which uses Allreduce() + lambda function + MPI_IN_PLACE is: - Valgrind-clean with MPICH; - Valgrind-clean with OpenMPI 4.0.5; - not Valgrind-clean with OpenMPI 4.1.0. I’m not sure who is to blame here, I’ll need to look at the MPI specification for what is required by the implementors and users in that case. In the meantime, I’ll do the following: - update config/BuildSystem/config/packages/OpenMPI.py to use OpenMPI 4.1.0, see if any other error appears; - provide a hotfix to bypass the segfaults; - look at the hypre issue and whether they should be deferred to the hypre team. Thank you for the Docker files, they were really useful. If you want to avoid oversubscription failures, you can edit the file /opt/openmpi-4.1.0/etc/openmpi-default-hostfile and append the line: localhost slots=12 If you want to increase the timeout limit of PETSc test suite for each test, you can add the extra flag in your command line TIMEOUT=180 (default is 60, units are seconds). Thanks, I’ll ping you on GitLab when I’ve got something ready for you to try, Pierre > On 12 Mar 2021, at 8:54 PM, Eric Chamberland <Eric.Chamberland(a)giref.ulaval.ca> wrote: > > Hi Pierre, > > I now have a docker container reproducing the problems here. > > Actually, if I look at snes_tutorials-ex12_quad_singular_hpddm it fails like this: > > not ok snes_tutorials-ex12_quad_singular_hpddm # Error code: 59 > # Initial guess > # L_2 Error: 0.00803099 > # Initial Residual > # L_2 Residual: 1.09057 > # Au - b = Au + F(0) > # Linear L_2 Residual: 1.09057 > # [d470c54ce086:14127] Read -1, expected 4096, errno = 1 > # [d470c54ce086:14128] Read -1, expected 4096, errno = 1 > # [d470c54ce086:14129] Read -1, expected 4096, errno = 1 > # [3]PETSC ERROR: ------------------------------------------------------------------------ > # [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > # [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > # [3]PETSC ERROR: or see https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind <https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind> > # [3]PETSC ERROR: or try http://valgrind.org <http://valgrind.org/> on GNU/linux and Apple Mac OS X to find memory corruption errors > # [3]PETSC ERROR: likely location of problem given in stack below > # [3]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > # [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > # [3]PETSC ERROR: INSTEAD the line number of the start of the function > # [3]PETSC ERROR: is given. > # [3]PETSC ERROR: [3] buildTwo line 987 /opt/petsc-main/include/HPDDM_schwarz.hpp > # [3]PETSC ERROR: [3] next line 1130 /opt/petsc-main/include/HPDDM_schwarz.hpp > # [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > # [3]PETSC ERROR: Signal received > # [3]PETSC ERROR: [0]PETSC ERROR: ------------------------------------------------------------------------ > > also ex12_quad_hpddm_reuse_baij fails with a lot more "Read -1, expected ..." which I don't know where they come from...? > > Hypre (like in diff-snes_tutorials-ex56_hypre) is also having DIVERGED_INDEFINITE_PC failures... > > Please see the 3 attached docker files: > > 1) fedora_mkl_and_devtools : the DockerFile which install fedore 33 with gnu compilers and MKL and everything to develop. > > 2) openmpi: the DockerFile to bluid OpenMPI > > 3) petsc: The las DockerFile that build/install and test PETSc > > I build the 3 like this: > > docker build -t fedora_mkl_and_devtools -f fedora_mkl_and_devtools . > > docker build -t openmpi -f openmpi . > > docker build -t petsc -f petsc . > > Disclaimer: I am not a docker expert, so I may do things that are not docker-stat-of-the-art but I am opened to suggestions... ;) > > I have just ran it on my portable (long) which have not enough cores, so many more tests failed (should force --oversubscribe but don't know how to). I will relaunch on my workstation in a few minutes. > > I will now test your branch! (sorry for the delay). > > Thanks, > > Eric > > On 2021-03-11 9:03 a.m., Eric Chamberland wrote: >> Hi Pierre, >> >> ok, that's interesting! >> >> I will try to build a docker image until tomorrow and give you the exact recipe to reproduce the bugs. >> >> Eric >> >> >> >> On 2021-03-11 2:46 a.m., Pierre Jolivet wrote: >>> >>> >>>> On 11 Mar 2021, at 6:16 AM, Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >>>> >>>> >>>> Eric, >>>> >>>> Sorry about not being more immediate. We still have this in our active email so you don't need to submit individual issues. We'll try to get to them as soon as we can. >>> >>> Indeed, I’m still trying to figure this out. >>> I realized that some of my configure flags were different than yours, e.g., no --with-memalign. >>> I’ve also added SuperLU_DIST to my installation. >>> Still, I can’t reproduce any issue. >>> I will continue looking into this, it appears I’m seeing some valgrind errors, but I don’t know if this is some side effect of OpenMPI not being valgrind-clean (last time I checked, there was no error with MPICH). >>> >>> Thank you for your patience, >>> Pierre >>> >>> /usr/bin/gmake -f gmakefile test test-fail=1 >>> Using MAKEFLAGS: test-fail=1 >>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_baij.counts >>> ok snes_tutorials-ex12_quad_hpddm_reuse_baij >>> ok diff-snes_tutorials-ex12_quad_hpddm_reuse_baij >>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist_2.counts >>> ok ksp_ksp_tests-ex33_superlu_dist_2 >>> ok diff-ksp_ksp_tests-ex33_superlu_dist_2 >>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex49_superlu_dist.counts >>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 >>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 >>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 >>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 >>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 >>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 >>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 >>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 >>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 >>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 >>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 >>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 >>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 >>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 >>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 >>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 >>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex50_tut_2.counts >>> ok ksp_ksp_tutorials-ex50_tut_2 >>> ok diff-ksp_ksp_tutorials-ex50_tut_2 >>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist.counts >>> ok ksp_ksp_tests-ex33_superlu_dist >>> ok diff-ksp_ksp_tests-ex33_superlu_dist >>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_hypre.counts >>> ok snes_tutorials-ex56_hypre >>> ok diff-snes_tutorials-ex56_hypre >>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex56_2.counts >>> ok ksp_ksp_tutorials-ex56_2 >>> ok diff-ksp_ksp_tutorials-ex56_2 >>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_elas.counts >>> ok snes_tutorials-ex17_3d_q3_trig_elas >>> ok diff-snes_tutorials-ex17_3d_q3_trig_elas >>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij.counts >>> ok snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij >>> ok diff-snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij >>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist_3.counts >>> not ok ksp_ksp_tutorials-ex5_superlu_dist_3 # Error code: 1 >>> # srun: error: Unable to create step for job 1426755: More processors requested than permitted >>> ok ksp_ksp_tutorials-ex5_superlu_dist_3 # SKIP Command failed so no diff >>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist.counts >>> ok ksp_ksp_tutorials-ex5f_superlu_dist # SKIP Fortran required for this test >>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_tri_parmetis_hpddm_baij.counts >>> ok snes_tutorials-ex12_tri_parmetis_hpddm_baij >>> ok diff-snes_tutorials-ex12_tri_parmetis_hpddm_baij >>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_tut_3.counts >>> ok snes_tutorials-ex19_tut_3 >>> ok diff-snes_tutorials-ex19_tut_3 >>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_vlap.counts >>> ok snes_tutorials-ex17_3d_q3_trig_vlap >>> ok diff-snes_tutorials-ex17_3d_q3_trig_vlap >>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist_3.counts >>> ok ksp_ksp_tutorials-ex5f_superlu_dist_3 # SKIP Fortran required for this test >>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_superlu_dist.counts >>> ok snes_tutorials-ex19_superlu_dist >>> ok diff-snes_tutorials-ex19_superlu_dist >>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre.counts >>> ok snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre >>> ok diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre >>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex49_hypre_nullspace.counts >>> ok ksp_ksp_tutorials-ex49_hypre_nullspace >>> ok diff-ksp_ksp_tutorials-ex49_hypre_nullspace >>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_superlu_dist_2.counts >>> ok snes_tutorials-ex19_superlu_dist_2 >>> ok diff-snes_tutorials-ex19_superlu_dist_2 >>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist_2.counts >>> not ok ksp_ksp_tutorials-ex5_superlu_dist_2 # Error code: 1 >>> # srun: error: Unable to create step for job 1426755: More processors requested than permitted >>> ok ksp_ksp_tutorials-ex5_superlu_dist_2 # SKIP Command failed so no diff >>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre.counts >>> ok snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre >>> ok diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre >>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex64_1.counts >>> ok ksp_ksp_tutorials-ex64_1 >>> ok diff-ksp_ksp_tutorials-ex64_1 >>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist.counts >>> not ok ksp_ksp_tutorials-ex5_superlu_dist # Error code: 1 >>> # srun: error: Unable to create step for job 1426755: More processors requested than permitted >>> ok ksp_ksp_tutorials-ex5_superlu_dist # SKIP Command failed so no diff >>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist_2.counts >>> ok ksp_ksp_tutorials-ex5f_superlu_dist_2 # SKIP Fortran required for this test >>> >>>> Barry >>>> >>>> >>>>> On Mar 10, 2021, at 11:03 PM, Eric Chamberland <Eric.Chamberland(a)giref.ulaval.ca <mailto:[email protected]>> wrote: >>>>> >>>>> Barry, >>>>> >>>>> to get a some follow up on --with-openmp=1 failures, shall I open gitlab issues for: >>>>> >>>>> a) all hypre failures giving DIVERGED_INDEFINITE_PC >>>>> >>>>> b) all superlu_dist failures giving different results with initia and "Exceeded timeout limit of 60 s" >>>>> >>>>> c) hpddm failures "free(): invalid next size (fast)" and "Segmentation Violation" >>>>> >>>>> d) all tao's "Exceeded timeout limit of 60 s" >>>>> >>>>> I don't see how I could do all these debugging by myself... >>>>> >>>>> Thanks, >>>>> >>>>> Eric >>>>> >>>>> >>>> >>> >> -- >> Eric Chamberland, ing., M. Ing >> Professionnel de recherche >> GIREF/Université Laval >> (418) 656-2131 poste 41 22 42 > -- > Eric Chamberland, ing., M. Ing > Professionnel de recherche > GIREF/Université Laval > (418) 656-2131 poste 41 22 42 > <fedora_mkl_and_devtools.txt><openmpi.txt><petsc.txt>

1 0

Re: [petsc-dev] Argonne GPU Virtual Hackathon - Accepted
by Stefano Zampini 13 Mar '21

13 Mar '21

The COO assembly is entirely based on thrust primitives, I don’t have much experience to say we will get a serious speedup by writing our own kernels, but it is definitely worth a try if we will end up adopting COO as entry point for GPU irregular assembly. Jed, you mentioned BDDC deluxe, what do you mean by that? Porting setup/application of deluxe scaling onto GPU? Timings are not so bad for me joining the hackaton. > On Mar 13, 2021, at 8:17 AM, Barry Smith <bsmith(a)petsc.dev> wrote: > > > >> On Mar 12, 2021, at 10:49 PM, Jed Brown <jed(a)jedbrown.org> wrote: >> >> Barry Smith <bsmith(a)petsc.dev> writes: >> >>>> On Mar 12, 2021, at 6:58 PM, Jed Brown <jed(a)jedbrown.org> wrote: >>>> >>>> Barry Smith <bsmith(a)petsc.dev> writes: >>>> >>>>> I think we should start porting the PetscFE infrastructure, numerical integrations, vector and matrix assembly to GPUs soon. It is dog slow on CPUs and should be able to deliver higher performance on GPUs. >>>> >>>> IMO, this comes via interfaces to libCEED, not rolling yet another way to invoke quadrature routines on GPUs. >>> >>> I am not talking about matrix-free stuff, that definitely belongs in libCEED, no reason to rewrite. >>> >>> But does libCEED also support the traditional finite element construction process where the matrices are built explicitly? Or does it provide some of the code, integration points, integration formula etc. that could be shared and used as a starting point? If it includes all of these "traditional" things then we should definitely get it all hooked into PetscFE/DMPLEX and go to town. (But yes not so much need for the GPU hackathon since it is wiring more than GPU code). The way I have always heard about libCEED was as a matrix-free engine, so I may have miss understood. It is definitely not my intention to start a project that reproduces functionality that we can just use. >> >> MFEM wants this too and it's in a draft libCEED PR right now. My intent is to ensure it's compatible with Stefano's split-phase COO assembly. > > Cool, would this be something that, in combination with perhaps some libCEED folk, could be incorporated in the Hackathon? Anyone can join our group Hackathon group, they don't have to have any financial connection with "PETSc". > >> >>> We do need solid support for traditional finite element assembly on GPUs, matrix-free finite elements alone is not enough. >> >> Agreed, and while libCEED could be further optimized for lowest order, even naive assembly will be faster than what's in DMPlex.

1 0

Re: [petsc-dev] Argonne GPU Virtual Hackathon - Accepted
by Jeremy Luke Thompson 13 Mar '21

13 Mar '21

I would potentially be interested in joining in on a libCEED + PETSc integration effort for the hackathon. On Fri, Mar 12, 2021, 10:17 PM Barry Smith <bsmith(a)petsc.dev> wrote: > > > > On Mar 12, 2021, at 10:49 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > > > Barry Smith <bsmith(a)petsc.dev> writes: > > > >>> On Mar 12, 2021, at 6:58 PM, Jed Brown <jed(a)jedbrown.org> wrote: > >>> > >>> Barry Smith <bsmith(a)petsc.dev> writes: > >>> > >>>> I think we should start porting the PetscFE infrastructure, > numerical integrations, vector and matrix assembly to GPUs soon. It is dog > slow on CPUs and should be able to deliver higher performance on GPUs. > >>> > >>> IMO, this comes via interfaces to libCEED, not rolling yet another way > to invoke quadrature routines on GPUs. > >> > >> I am not talking about matrix-free stuff, that definitely belongs in > libCEED, no reason to rewrite. > >> > >> But does libCEED also support the traditional finite element > construction process where the matrices are built explicitly? Or does it > provide some of the code, integration points, integration formula etc. that > could be shared and used as a starting point? If it includes all of these > "traditional" things then we should definitely get it all hooked into > PetscFE/DMPLEX and go to town. (But yes not so much need for the GPU > hackathon since it is wiring more than GPU code). The way I have always > heard about libCEED was as a matrix-free engine, so I may have miss > understood. It is definitely not my intention to start a project that > reproduces functionality that we can just use. > > > > MFEM wants this too and it's in a draft libCEED PR right now. My intent > is to ensure it's compatible with Stefano's split-phase COO assembly. > > Cool, would this be something that, in combination with perhaps some > libCEED folk, could be incorporated in the Hackathon? Anyone can join our > group Hackathon group, they don't have to have any financial connection > with "PETSc". > > > > >> We do need solid support for traditional finite element assembly on > GPUs, matrix-free finite elements alone is not enough. > > > > Agreed, and while libCEED could be further optimized for lowest order, > even naive assembly will be faster than what's in DMPlex. > >

1 0

Re: [petsc-dev] Argonne GPU Virtual Hackathon - Accepted
by Barry Smith 12 Mar '21

12 Mar '21

> On Mar 12, 2021, at 10:49 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > Barry Smith <bsmith(a)petsc.dev> writes: > >>> On Mar 12, 2021, at 6:58 PM, Jed Brown <jed(a)jedbrown.org> wrote: >>> >>> Barry Smith <bsmith(a)petsc.dev> writes: >>> >>>> I think we should start porting the PetscFE infrastructure, numerical integrations, vector and matrix assembly to GPUs soon. It is dog slow on CPUs and should be able to deliver higher performance on GPUs. >>> >>> IMO, this comes via interfaces to libCEED, not rolling yet another way to invoke quadrature routines on GPUs. >> >> I am not talking about matrix-free stuff, that definitely belongs in libCEED, no reason to rewrite. >> >> But does libCEED also support the traditional finite element construction process where the matrices are built explicitly? Or does it provide some of the code, integration points, integration formula etc. that could be shared and used as a starting point? If it includes all of these "traditional" things then we should definitely get it all hooked into PetscFE/DMPLEX and go to town. (But yes not so much need for the GPU hackathon since it is wiring more than GPU code). The way I have always heard about libCEED was as a matrix-free engine, so I may have miss understood. It is definitely not my intention to start a project that reproduces functionality that we can just use. > > MFEM wants this too and it's in a draft libCEED PR right now. My intent is to ensure it's compatible with Stefano's split-phase COO assembly. Cool, would this be something that, in combination with perhaps some libCEED folk, could be incorporated in the Hackathon? Anyone can join our group Hackathon group, they don't have to have any financial connection with "PETSc". > >> We do need solid support for traditional finite element assembly on GPUs, matrix-free finite elements alone is not enough. > > Agreed, and while libCEED could be further optimized for lowest order, even naive assembly will be faster than what's in DMPlex.

1 0

Re: [petsc-dev] Argonne GPU Virtual Hackathon - Accepted
by Jed Brown 12 Mar '21

12 Mar '21

Barry Smith <bsmith(a)petsc.dev> writes: >> On Mar 12, 2021, at 6:58 PM, Jed Brown <jed(a)jedbrown.org> wrote: >> >> Barry Smith <bsmith(a)petsc.dev> writes: >> >>> I think we should start porting the PetscFE infrastructure, numerical integrations, vector and matrix assembly to GPUs soon. It is dog slow on CPUs and should be able to deliver higher performance on GPUs. >> >> IMO, this comes via interfaces to libCEED, not rolling yet another way to invoke quadrature routines on GPUs. > > I am not talking about matrix-free stuff, that definitely belongs in libCEED, no reason to rewrite. > > But does libCEED also support the traditional finite element construction process where the matrices are built explicitly? Or does it provide some of the code, integration points, integration formula etc. that could be shared and used as a starting point? If it includes all of these "traditional" things then we should definitely get it all hooked into PetscFE/DMPLEX and go to town. (But yes not so much need for the GPU hackathon since it is wiring more than GPU code). The way I have always heard about libCEED was as a matrix-free engine, so I may have miss understood. It is definitely not my intention to start a project that reproduces functionality that we can just use. MFEM wants this too and it's in a draft libCEED PR right now. My intent is to ensure it's compatible with Stefano's split-phase COO assembly. > We do need solid support for traditional finite element assembly on GPUs, matrix-free finite elements alone is not enough. Agreed, and while libCEED could be further optimized for lowest order, even naive assembly will be faster than what's in DMPlex.

1 0

Re: [petsc-dev] Argonne GPU Virtual Hackathon - Accepted
by Barry Smith 12 Mar '21

12 Mar '21

> On Mar 12, 2021, at 6:58 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > Barry Smith <bsmith(a)petsc.dev> writes: > >> I think we should start porting the PetscFE infrastructure, numerical integrations, vector and matrix assembly to GPUs soon. It is dog slow on CPUs and should be able to deliver higher performance on GPUs. > > IMO, this comes via interfaces to libCEED, not rolling yet another way to invoke quadrature routines on GPUs. I am not talking about matrix-free stuff, that definitely belongs in libCEED, no reason to rewrite. But does libCEED also support the traditional finite element construction process where the matrices are built explicitly? Or does it provide some of the code, integration points, integration formula etc. that could be shared and used as a starting point? If it includes all of these "traditional" things then we should definitely get it all hooked into PetscFE/DMPLEX and go to town. (But yes not so much need for the GPU hackathon since it is wiring more than GPU code). The way I have always heard about libCEED was as a matrix-free engine, so I may have miss understood. It is definitely not my intention to start a project that reproduces functionality that we can just use. We do need solid support for traditional finite element assembly on GPUs, matrix-free finite elements alone is not enough. > > DMPlex setup and distribution could conceivably be ported to GPUs, but it would be a monumental task and that stuff is usually done once and intermediate structures don't necessarily fit in device memory even when they fit comfortably in DRAM. Way too big of a task for one hackathon. I am not talking about the setup and distribution, I don't see that as a priority.

1 0