March 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] MatTransposeMatMult() bug
by Matthew Knepley 17 Mar '21

17 Mar '21

On Wed, Mar 17, 2021 at 3:27 PM Zhang, Hong via petsc-dev < petsc-dev(a)mcs.anl.gov> wrote: > Pierre, > Do you mean a possible bug in C=AtB MatTransposeMatMult()? > Can you provide a stand-alone test without hpddm that reproduces this > error? > Hong, you should be able to just configure with --download-hpddm and then run that ex76 test. Thanks, Matt > Hong > ------------------------------ > *From:* petsc-dev <petsc-dev-bounces(a)mcs.anl.gov> on behalf of Pierre > Jolivet <pierre(a)joliv.et> > *Sent:* Wednesday, March 17, 2021 4:31 AM > *To:* For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov > > > *Subject:* [petsc-dev] MatTransposeMatMult() bug > > Hello, > While trying out Stefano’s PCApplyMat_MG() code (*), we stumbled upon > weird numerical errors when reusing a Mat for both MatProduct_AB and > MatProduct_AtB. > This reminded me that there has been a long-standing issue > with MatTransposeMatMult(), see > https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/pc/impls/hpddm/hpddm.cx…, > that I never looked into. > I’ve now been trying to figure this out, because this has side effects in > multiple places (PCMG and PCHPDDM at least), and thus could impact > user-code as well? > With this commit: > https://gitlab.com/petsc/petsc/-/commit/03d8bd538039defc2fcc3e37d523735c4aa… > + > $ mpirun -n 4 src/ksp/ksp/tutorials/ex76 -ksp_converged_reason -pc_type > hpddm -pc_hpddm_levels_1_eps_nev 20 -ksp_type preonly -mat_type aij > -load_dir ${DATAFILESPATH}/matrices/hpddm/GENEO -rhs 2 > -pc_hpddm_coarse_correction balanced -C_input_mattransposematmult > -D_output_mattransposematmult > I’m seeing that C is nonzero, but D is full of zeros. > Mat Object: 4 MPI processes > type: mpidense > 5.7098316584361917e-08 1.0159399260517841e-07 > 1.5812349976211856e-07 2.0688121715350138e-07 > 2.4887556933361981e-08 4.8111092300772958e-08 > 1.4606298643602107e-07 1.7213611729839211e-07 > […] > Mat Object: 4 MPI processes > type: mpidense > 0.0000000000000000e+00 0.0000000000000000e+00 > 0.0000000000000000e+00 0.0000000000000000e+00 > 0.0000000000000000e+00 0.0000000000000000e+00 > 0.0000000000000000e+00 0.0000000000000000e+00 > […] > > If one switches to a MatType which has no MatProduct_AtB implementation > with B of type MPIDense (reminder: in that case, the product is computed > column-by-column), e.g., -mat_type sbaij, one gets the expected result. > Mat Object: 4 MPI processes > type: mpidense > 7.2003197398135299e-01 9.5191869895699011e-01 > 6.1793966541680234e-02 9.3884397585488877e-01 > 1.0022337823233585e-02 2.4653068080134588e-01 > 1.4463931936094099e-01 8.6111517670701687e-01 > > Is there a bug somewhere with the MatAIJ implementation, or am I doing > something which is not allowed by the MatProduct() machinery? > > Thanks, > Pierre > > (*) https://gitlab.com/petsc/petsc/-/merge_requests/3717 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] MatTransposeMatMult() bug
by Pierre Jolivet 17 Mar '21

17 Mar '21

Thanks Hong, this has been fixed by Stefano there: https://gitlab.com/petsc/petsc/-/commit/38faa66a3f98cdf432182095b5ca9cf1fc3… <https://gitlab.com/petsc/petsc/-/commit/38faa66a3f98cdf432182095b5ca9cf1fc3…> Pierre > On 17 Mar 2021, at 8:27 PM, Zhang, Hong <hzhang(a)mcs.anl.gov> wrote: > > Pierre, > Do you mean a possible bug in C=AtB MatTransposeMatMult()? > Can you provide a stand-alone test without hpddm that reproduces this error? > Hong > From: petsc-dev <petsc-dev-bounces(a)mcs.anl.gov> on behalf of Pierre Jolivet <pierre(a)joliv.et> > Sent: Wednesday, March 17, 2021 4:31 AM > To: For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov> > Subject: [petsc-dev] MatTransposeMatMult() bug > > Hello, > While trying out Stefano’s PCApplyMat_MG() code (*), we stumbled upon weird numerical errors when reusing a Mat for both MatProduct_AB and MatProduct_AtB. > This reminded me that there has been a long-standing issue with MatTransposeMatMult(), see https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/pc/impls/hpddm/hpddm.cx… <https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/pc/impls/hpddm/hpddm.cx…>, that I never looked into. > I’ve now been trying to figure this out, because this has side effects in multiple places (PCMG and PCHPDDM at least), and thus could impact user-code as well? > With this commit: https://gitlab.com/petsc/petsc/-/commit/03d8bd538039defc2fcc3e37d523735c4aa… <https://gitlab.com/petsc/petsc/-/commit/03d8bd538039defc2fcc3e37d523735c4aa…> > + > $ mpirun -n 4 src/ksp/ksp/tutorials/ex76 -ksp_converged_reason -pc_type hpddm -pc_hpddm_levels_1_eps_nev 20 -ksp_type preonly -mat_type aij -load_dir ${DATAFILESPATH}/matrices/hpddm/GENEO -rhs 2 -pc_hpddm_coarse_correction balanced -C_input_mattransposematmult -D_output_mattransposematmult > I’m seeing that C is nonzero, but D is full of zeros. > Mat Object: 4 MPI processes > type: mpidense > 5.7098316584361917e-08 1.0159399260517841e-07 > 1.5812349976211856e-07 2.0688121715350138e-07 > 2.4887556933361981e-08 4.8111092300772958e-08 > 1.4606298643602107e-07 1.7213611729839211e-07 > […] > Mat Object: 4 MPI processes > type: mpidense > 0.0000000000000000e+00 0.0000000000000000e+00 > 0.0000000000000000e+00 0.0000000000000000e+00 > 0.0000000000000000e+00 0.0000000000000000e+00 > 0.0000000000000000e+00 0.0000000000000000e+00 > […] > > If one switches to a MatType which has no MatProduct_AtB implementation with B of type MPIDense (reminder: in that case, the product is computed column-by-column), e.g., -mat_type sbaij, one gets the expected result. > Mat Object: 4 MPI processes > type: mpidense > 7.2003197398135299e-01 9.5191869895699011e-01 > 6.1793966541680234e-02 9.3884397585488877e-01 > 1.0022337823233585e-02 2.4653068080134588e-01 > 1.4463931936094099e-01 8.6111517670701687e-01 > > Is there a bug somewhere with the MatAIJ implementation, or am I doing something which is not allowed by the MatProduct() machinery? > > Thanks, > Pierre > > (*) https://gitlab.com/petsc/petsc/-/merge_requests/3717 <https://gitlab.com/petsc/petsc/-/merge_requests/3717>

1 0

Re: [petsc-dev] MatTransposeMatMult() bug
by Zhang, Hong 17 Mar '21

17 Mar '21

Pierre, Do you mean a possible bug in C=AtB MatTransposeMatMult()? Can you provide a stand-alone test without hpddm that reproduces this error? Hong ________________________________ From: petsc-dev <petsc-dev-bounces(a)mcs.anl.gov> on behalf of Pierre Jolivet <pierre(a)joliv.et> Sent: Wednesday, March 17, 2021 4:31 AM To: For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov> Subject: [petsc-dev] MatTransposeMatMult() bug Hello, While trying out Stefano’s PCApplyMat_MG() code (*), we stumbled upon weird numerical errors when reusing a Mat for both MatProduct_AB and MatProduct_AtB. This reminded me that there has been a long-standing issue with MatTransposeMatMult(), see https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/pc/impls/hpddm/hpddm.cx…, that I never looked into. I’ve now been trying to figure this out, because this has side effects in multiple places (PCMG and PCHPDDM at least), and thus could impact user-code as well? With this commit: https://gitlab.com/petsc/petsc/-/commit/03d8bd538039defc2fcc3e37d523735c4aa… + $ mpirun -n 4 src/ksp/ksp/tutorials/ex76 -ksp_converged_reason -pc_type hpddm -pc_hpddm_levels_1_eps_nev 20 -ksp_type preonly -mat_type aij -load_dir ${DATAFILESPATH}/matrices/hpddm/GENEO -rhs 2 -pc_hpddm_coarse_correction balanced -C_input_mattransposematmult -D_output_mattransposematmult I’m seeing that C is nonzero, but D is full of zeros. Mat Object: 4 MPI processes type: mpidense 5.7098316584361917e-08 1.0159399260517841e-07 1.5812349976211856e-07 2.0688121715350138e-07 2.4887556933361981e-08 4.8111092300772958e-08 1.4606298643602107e-07 1.7213611729839211e-07 […] Mat Object: 4 MPI processes type: mpidense 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 […] If one switches to a MatType which has no MatProduct_AtB implementation with B of type MPIDense (reminder: in that case, the product is computed column-by-column), e.g., -mat_type sbaij, one gets the expected result. Mat Object: 4 MPI processes type: mpidense 7.2003197398135299e-01 9.5191869895699011e-01 6.1793966541680234e-02 9.3884397585488877e-01 1.0022337823233585e-02 2.4653068080134588e-01 1.4463931936094099e-01 8.6111517670701687e-01 Is there a bug somewhere with the MatAIJ implementation, or am I doing something which is not allowed by the MatProduct() machinery? Thanks, Pierre (*) https://gitlab.com/petsc/petsc/-/merge_requests/3717

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Eric Chamberland 17 Mar '21

17 Mar '21

Thanks Barry, Just to report: I tried to switch to the proposed smoother by default in our code: -pc_hypre_boomeramg_relax_type_all l1scaled-SOR/Jacobi However, I have some failures, even if I compiled without --with-openmp=1. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Error in external library [0]PETSC ERROR: Error in jac->setup(): error code 12 [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.14.5, Mar 03, 2021 [0]PETSC ERROR: /home/mefpp_ericc/GIREF/bin/Test.EstimationGradientHessien.dev on a named rohan by ericc Wed Mar 17 11:05:23 2021 [0]PETSC ERROR: Configure options --prefix=/opt/petsc-3.14.5_debug_openmpi-4.1.0 --with-mpi-compilers=1 --with-mpi-dir=/opt/openmpi-4.1.0 --with-cxx-dialect=C++14 --with-make-np=12 --with-shared-libraries=1 --with-debugging=yes --with-memalign=64 --with-visibility=0 --with-64-bit-indices=0 --download-ml=yes --download-mumps=yes --download-superlu=yes --download-hpddm=yes --download-slepc=yes --download-superlu_dist=yes --download-parmetis=yes --download-ptscotch=yes --download-metis=yes --download-strumpack=yes --download-suitesparse=yes --download-hypre=yes --with-blaslapack-dir=/opt/intel/oneapi/mkl/2021.1.1/env/../lib/intel64 --with-mkl_pardiso-dir=/opt/intel/oneapi/mkl/2021.1.1/env/.. --with-mkl_cpardiso-dir=/opt/intel/oneapi/mkl/2021.1.1/env/.. --with-scalapack=1 --with-scalapack-include=/opt/intel/oneapi/mkl/2021.1.1/env/../include --with-scalapack-lib="-L/opt/intel/oneapi/mkl/2021.1.1/env/../lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64" [0]PETSC ERROR: #1 PCSetUp_HYPRE() line 408 in /tmp/petsc-3.14.5-debug/src/ksp/pc/impls/hypre/hypre.c [0]PETSC ERROR: #2 PCSetUp() line 1009 in /tmp/petsc-3.14.5-debug/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #3 KSPSetUp() line 406 in /tmp/petsc-3.14.5-debug/src/ksp/ksp/interface/itfunc.c But it seems to happen only on some cases, actually hermitian elements which have a lots of DOF per vertices... It seems to work well otherwise, with some results differences I still have to analyse... Do you think this might be a PETSc bug? Does the error code is from PETSc or Hypre? (if from hypre, I suggest to say "Hypre error code: 12" instead...) Thanks, Eric On 2021-03-15 2:50 p.m., Barry Smith wrote: > > I posted some information at the issue. > > IMHO it is likely a bug in one or more of hypre's smoothers that > use OpenMP. We have never tested them before (and likely hypre has not > tested all the combinations) and so would not have seen the bug. > Hopefully they can just fix it. > > Barry > > I got the problem to occur with ex56 with 2 MPI ranks and 4 OpenMP > threads, if I used less than 4 threads it did not generate an > indefinite preconditioner. > > >> On Mar 14, 2021, at 1:18 PM, Eric Chamberland >> <Eric.Chamberland(a)giref.ulaval.ca >> <mailto:[email protected]>> wrote: >> >> Done: >> >> https://github.com/hypre-space/hypre/issues/303 >> >> Maybe I will need some help about PETSc to answer their questions... >> >> Eric >> >> On 2021-03-14 3:44 a.m., Stefano Zampini wrote: >>> Eric >>> >>> You should report these HYPRE issues upstream >>> https://github.com/hypre-space/hypre/issues >>> <https://github.com/hypre-space/hypre/issues> >>> >>> >>>> On Mar 14, 2021, at 3:44 AM, Eric Chamberland >>>> <Eric.Chamberland(a)giref.ulaval.ca >>>> <mailto:[email protected]>> wrote: >>>> >>>> For us it clearly creates problems in real computations... >>>> >>>> I understand the need to have clean test for PETSc, but for me, it >>>> reveals that hypre isn't usable with more than one thread for now... >>>> >>>> Another solution: force single-threaded configuration for hypre >>>> until this is fixed? >>>> >>>> Eric >>>> >>>> On 2021-03-13 8:50 a.m., Pierre Jolivet wrote: >>>>> -pc_hypre_boomeramg_relax_type_all Jacobi => >>>>> Linear solve did not converge due to DIVERGED_INDEFINITE_PC >>>>> iterations 3 >>>>> -pc_hypre_boomeramg_relax_type_all l1scaled-Jacobi => >>>>> OK, independently of the architecture it seems (Eric Docker image >>>>> with 1 or 2 threads or my macOS), but contraction factor is higher >>>>> Linear solve converged due to CONVERGED_RTOL iterations 8 >>>>> Linear solve converged due to CONVERGED_RTOL iterations 24 >>>>> Linear solve converged due to CONVERGED_RTOL iterations 26 >>>>> v. currently >>>>> Linear solve converged due to CONVERGED_RTOL iterations 7 >>>>> Linear solve converged due to CONVERGED_RTOL iterations 9 >>>>> Linear solve converged due to CONVERGED_RTOL iterations 10 >>>>> >>>>> Do we change this? Or should we force OMP_NUM_THREADS=1 for make test? >>>>> >>>>> Thanks, >>>>> Pierre >>>>> >>>>>> On 13 Mar 2021, at 2:26 PM, Mark Adams <mfadams(a)lbl.gov >>>>>> <mailto:[email protected]>> wrote: >>>>>> >>>>>> Hypre uses a multiplicative smoother by default. It has a >>>>>> chebyshev smoother. That with a Jacobi PC should be thread >>>>>> invariant. >>>>>> Mark >>>>>> >>>>>> On Sat, Mar 13, 2021 at 8:18 AM Pierre Jolivet <pierre(a)joliv.et >>>>>> <mailto:[email protected]>> wrote: >>>>>> >>>>>> >>>>>>> On 13 Mar 2021, at 9:17 AM, Pierre Jolivet <pierre(a)joliv.et >>>>>>> <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> Hello Eric, >>>>>>> I’ve made an “interesting” discovery, so I’ll put back the >>>>>>> list in c/c. >>>>>>> It appears the following snippet of code which uses >>>>>>> Allreduce() + lambda function + MPI_IN_PLACE is: >>>>>>> - Valgrind-clean with MPICH; >>>>>>> - Valgrind-clean with OpenMPI 4.0.5; >>>>>>> - not Valgrind-clean with OpenMPI 4.1.0. >>>>>>> I’m not sure who is to blame here, I’ll need to look at the >>>>>>> MPI specification for what is required by the implementors >>>>>>> and users in that case. >>>>>>> >>>>>>> In the meantime, I’ll do the following: >>>>>>> - update config/BuildSystem/config/packages/OpenMPI.py to >>>>>>> use OpenMPI 4.1.0, see if any other error appears; >>>>>>> - provide a hotfix to bypass the segfaults; >>>>>> >>>>>> I can confirm that splitting the single Allreduce with my own >>>>>> MPI_Op into two Allreduce with MAX and BAND fixes the >>>>>> segfaults with OpenMPI (*). >>>>>> >>>>>>> - look at the hypre issue and whether they should be >>>>>>> deferred to the hypre team. >>>>>> >>>>>> I don’t know if there is something wrong in hypre threading >>>>>> or if it’s just a side effect of threading, but it seems that >>>>>> the number of threads has a drastic effect on the quality of >>>>>> the PC. >>>>>> By default, it looks that there are two threads per process >>>>>> with your Docker image. >>>>>> If I force OMP_NUM_THREADS=1, then I get the same convergence >>>>>> as in the output file. >>>>>> >>>>>> Thanks, >>>>>> Pierre >>>>>> >>>>>> (*) https://gitlab.com/petsc/petsc/-/merge_requests/3712 >>>>>> <https://gitlab.com/petsc/petsc/-/merge_requests/3712> >>>>>> >>>>>>> Thank you for the Docker files, they were really useful. >>>>>>> If you want to avoid oversubscription failures, you can edit >>>>>>> the file /opt/openmpi-4.1.0/etc/openmpi-default-hostfile and >>>>>>> append the line: >>>>>>> localhost slots=12 >>>>>>> If you want to increase the timeout limit of PETSc test >>>>>>> suite for each test, you can add the extra flag in your >>>>>>> command line TIMEOUT=180 (default is 60, units are seconds). >>>>>>> >>>>>>> Thanks, I’ll ping you on GitLab when I’ve got something >>>>>>> ready for you to try, >>>>>>> Pierre >>>>>>> >>>>>>> <ompi.cxx> >>>>>>> >>>>>>>> On 12 Mar 2021, at 8:54 PM, Eric Chamberland >>>>>>>> <Eric.Chamberland(a)giref.ulaval.ca >>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>> >>>>>>>> Hi Pierre, >>>>>>>> >>>>>>>> I now have a docker container reproducing the problems here. >>>>>>>> >>>>>>>> Actually, if I look at >>>>>>>> snes_tutorials-ex12_quad_singular_hpddm it fails like this: >>>>>>>> >>>>>>>> not ok snes_tutorials-ex12_quad_singular_hpddm # Error code: 59 >>>>>>>> # Initial guess >>>>>>>> # L_2 Error: 0.00803099 >>>>>>>> # Initial Residual >>>>>>>> # L_2 Residual: 1.09057 >>>>>>>> # Au - b = Au + F(0) >>>>>>>> # Linear L_2 Residual: 1.09057 >>>>>>>> # [d470c54ce086:14127] Read -1, expected 4096, errno = 1 >>>>>>>> # [d470c54ce086:14128] Read -1, expected 4096, errno = 1 >>>>>>>> # [d470c54ce086:14129] Read -1, expected 4096, errno = 1 >>>>>>>> # [3]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> # [3]PETSC ERROR: Caught signal number 11 SEGV: >>>>>>>> Segmentation Violation, probably memory access out of range >>>>>>>> # [3]PETSC ERROR: Try option -start_in_debugger or >>>>>>>> -on_error_attach_debugger >>>>>>>> # [3]PETSC ERROR: or see >>>>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>>>>>>> <https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind> >>>>>>>> # [3]PETSC ERROR: or try http://valgrind.org >>>>>>>> <http://valgrind.org/> on GNU/linux and Apple Mac OS X to >>>>>>>> find memory corruption errors >>>>>>>> # [3]PETSC ERROR: likely location of problem given in stack >>>>>>>> below >>>>>>>> # [3]PETSC ERROR: --------------------- Stack Frames >>>>>>>> ------------------------------------ >>>>>>>> # [3]PETSC ERROR: Note: The EXACT line numbers in the stack >>>>>>>> are not available, >>>>>>>> # [3]PETSC ERROR: INSTEAD the line number of the start of >>>>>>>> the function >>>>>>>> # [3]PETSC ERROR: is given. >>>>>>>> # [3]PETSC ERROR: [3] buildTwo line 987 >>>>>>>> /opt/petsc-main/include/HPDDM_schwarz.hpp >>>>>>>> # [3]PETSC ERROR: [3] next line 1130 >>>>>>>> /opt/petsc-main/include/HPDDM_schwarz.hpp >>>>>>>> # [3]PETSC ERROR: --------------------- Error Message >>>>>>>> -------------------------------------------------------------- >>>>>>>> # [3]PETSC ERROR: Signal received >>>>>>>> # [3]PETSC ERROR: [0]PETSC ERROR: >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> >>>>>>>> also ex12_quad_hpddm_reuse_baij fails with a lot more "Read >>>>>>>> -1, expected ..." which I don't know where they come from...? >>>>>>>> >>>>>>>> Hypre (like in diff-snes_tutorials-ex56_hypre) is also >>>>>>>> having DIVERGED_INDEFINITE_PC failures... >>>>>>>> >>>>>>>> Please see the 3 attached docker files: >>>>>>>> >>>>>>>> 1) fedora_mkl_and_devtools : the DockerFile which install >>>>>>>> fedore 33 with gnu compilers and MKL and everything to develop. >>>>>>>> >>>>>>>> 2) openmpi: the DockerFile to bluid OpenMPI >>>>>>>> >>>>>>>> 3) petsc: The las DockerFile that build/install and test PETSc >>>>>>>> >>>>>>>> I build the 3 like this: >>>>>>>> >>>>>>>> docker build -t fedora_mkl_and_devtools -f >>>>>>>> fedora_mkl_and_devtools . >>>>>>>> >>>>>>>> docker build -t openmpi -f openmpi . >>>>>>>> >>>>>>>> docker build -t petsc -f petsc . >>>>>>>> >>>>>>>> Disclaimer: I am not a docker expert, so I may do things >>>>>>>> that are not docker-stat-of-the-art but I am opened to >>>>>>>> suggestions... ;) >>>>>>>> >>>>>>>> I have just ran it on my portable (long) which have not >>>>>>>> enough cores, so many more tests failed (should force >>>>>>>> --oversubscribe but don't know how to). I will relaunch on >>>>>>>> my workstation in a few minutes. >>>>>>>> >>>>>>>> I will now test your branch! (sorry for the delay). >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Eric >>>>>>>> >>>>>>>> On 2021-03-11 9:03 a.m., Eric Chamberland wrote: >>>>>>>>> >>>>>>>>> Hi Pierre, >>>>>>>>> >>>>>>>>> ok, that's interesting! >>>>>>>>> >>>>>>>>> I will try to build a docker image until tomorrow and give >>>>>>>>> you the exact recipe to reproduce the bugs. >>>>>>>>> >>>>>>>>> Eric >>>>>>>>> >>>>>>>>> >>>>>>>>> On 2021-03-11 2:46 a.m., Pierre Jolivet wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On 11 Mar 2021, at 6:16 AM, Barry Smith >>>>>>>>>>> <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Eric, >>>>>>>>>>> >>>>>>>>>>> Sorry about not being more immediate. We still have >>>>>>>>>>> this in our active email so you don't need to submit >>>>>>>>>>> individual issues. We'll try to get to them as soon as >>>>>>>>>>> we can. >>>>>>>>>> >>>>>>>>>> Indeed, I’m still trying to figure this out. >>>>>>>>>> I realized that some of my configure flags were different >>>>>>>>>> than yours, e.g., no --with-memalign. >>>>>>>>>> I’ve also added SuperLU_DIST to my installation. >>>>>>>>>> Still, I can’t reproduce any issue. >>>>>>>>>> I will continue looking into this, it appears I’m seeing >>>>>>>>>> some valgrind errors, but I don’t know if this is some >>>>>>>>>> side effect of OpenMPI not being valgrind-clean (last >>>>>>>>>> time I checked, there was no error with MPICH). >>>>>>>>>> >>>>>>>>>> Thank you for your patience, >>>>>>>>>> Pierre >>>>>>>>>> >>>>>>>>>> /usr/bin/gmake -f gmakefile test test-fail=1 >>>>>>>>>> Using MAKEFLAGS: test-fail=1 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_baij.counts >>>>>>>>>> ok snes_tutorials-ex12_quad_hpddm_reuse_baij >>>>>>>>>> ok diff-snes_tutorials-ex12_quad_hpddm_reuse_baij >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist_2.counts >>>>>>>>>> ok ksp_ksp_tests-ex33_superlu_dist_2 >>>>>>>>>> ok diff-ksp_ksp_tests-ex33_superlu_dist_2 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex49_superlu_dist.counts >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 >>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 >>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex50_tut_2.counts >>>>>>>>>> ok ksp_ksp_tutorials-ex50_tut_2 >>>>>>>>>> ok diff-ksp_ksp_tutorials-ex50_tut_2 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist.counts >>>>>>>>>> ok ksp_ksp_tests-ex33_superlu_dist >>>>>>>>>> ok diff-ksp_ksp_tests-ex33_superlu_dist >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_hypre.counts >>>>>>>>>> ok snes_tutorials-ex56_hypre >>>>>>>>>> ok diff-snes_tutorials-ex56_hypre >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex56_2.counts >>>>>>>>>> ok ksp_ksp_tutorials-ex56_2 >>>>>>>>>> ok diff-ksp_ksp_tutorials-ex56_2 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_elas.counts >>>>>>>>>> ok snes_tutorials-ex17_3d_q3_trig_elas >>>>>>>>>> ok diff-snes_tutorials-ex17_3d_q3_trig_elas >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij.counts >>>>>>>>>> ok snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij >>>>>>>>>> ok diff-snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist_3.counts >>>>>>>>>> not ok ksp_ksp_tutorials-ex5_superlu_dist_3 # Error code: 1 >>>>>>>>>> #srun: error: Unable to create step for job 1426755: More >>>>>>>>>> processors requested than permitted >>>>>>>>>> ok ksp_ksp_tutorials-ex5_superlu_dist_3 # SKIP Command >>>>>>>>>> failed so no diff >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist.counts >>>>>>>>>> ok ksp_ksp_tutorials-ex5f_superlu_dist # SKIP Fortran >>>>>>>>>> required for this test >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_tri_parmetis_hpddm_baij.counts >>>>>>>>>> ok snes_tutorials-ex12_tri_parmetis_hpddm_baij >>>>>>>>>> ok diff-snes_tutorials-ex12_tri_parmetis_hpddm_baij >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_tut_3.counts >>>>>>>>>> ok snes_tutorials-ex19_tut_3 >>>>>>>>>> ok diff-snes_tutorials-ex19_tut_3 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_vlap.counts >>>>>>>>>> ok snes_tutorials-ex17_3d_q3_trig_vlap >>>>>>>>>> ok diff-snes_tutorials-ex17_3d_q3_trig_vlap >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist_3.counts >>>>>>>>>> ok ksp_ksp_tutorials-ex5f_superlu_dist_3 # SKIP Fortran >>>>>>>>>> required for this test >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_superlu_dist.counts >>>>>>>>>> ok snes_tutorials-ex19_superlu_dist >>>>>>>>>> ok diff-snes_tutorials-ex19_superlu_dist >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre.counts >>>>>>>>>> ok >>>>>>>>>> snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre >>>>>>>>>> ok >>>>>>>>>> diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex49_hypre_nullspace.counts >>>>>>>>>> ok ksp_ksp_tutorials-ex49_hypre_nullspace >>>>>>>>>> ok diff-ksp_ksp_tutorials-ex49_hypre_nullspace >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_superlu_dist_2.counts >>>>>>>>>> ok snes_tutorials-ex19_superlu_dist_2 >>>>>>>>>> ok diff-snes_tutorials-ex19_superlu_dist_2 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist_2.counts >>>>>>>>>> not ok ksp_ksp_tutorials-ex5_superlu_dist_2 # Error code: 1 >>>>>>>>>> #srun: error: Unable to create step for job 1426755: More >>>>>>>>>> processors requested than permitted >>>>>>>>>> ok ksp_ksp_tutorials-ex5_superlu_dist_2 # SKIP Command >>>>>>>>>> failed so no diff >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre.counts >>>>>>>>>> ok >>>>>>>>>> snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre >>>>>>>>>> ok >>>>>>>>>> diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex64_1.counts >>>>>>>>>> ok ksp_ksp_tutorials-ex64_1 >>>>>>>>>> ok diff-ksp_ksp_tutorials-ex64_1 >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist.counts >>>>>>>>>> not ok ksp_ksp_tutorials-ex5_superlu_dist # Error code: 1 >>>>>>>>>> #srun: error: Unable to create step for job 1426755: More >>>>>>>>>> processors requested than permitted >>>>>>>>>> ok ksp_ksp_tutorials-ex5_superlu_dist # SKIP Command >>>>>>>>>> failed so no diff >>>>>>>>>> TEST >>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist_2.counts >>>>>>>>>> ok ksp_ksp_tutorials-ex5f_superlu_dist_2 # SKIP Fortran >>>>>>>>>> required for this test >>>>>>>>>> >>>>>>>>>>> Barry >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On Mar 10, 2021, at 11:03 PM, Eric Chamberland >>>>>>>>>>>> <Eric.Chamberland(a)giref.ulaval.ca >>>>>>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Barry, >>>>>>>>>>>> >>>>>>>>>>>> to get a some follow up on --with-openmp=1 failures, >>>>>>>>>>>> shall I open gitlab issues for: >>>>>>>>>>>> >>>>>>>>>>>> a) all hypre failures giving DIVERGED_INDEFINITE_PC >>>>>>>>>>>> >>>>>>>>>>>> b) all superlu_dist failures giving different results >>>>>>>>>>>> with initia and "Exceeded timeout limit of 60 s" >>>>>>>>>>>> >>>>>>>>>>>> c) hpddm failures "free(): invalid next size (fast)" >>>>>>>>>>>> and "Segmentation Violation" >>>>>>>>>>>> >>>>>>>>>>>> d) all tao's "Exceeded timeout limit of 60 s" >>>>>>>>>>>> >>>>>>>>>>>> I don't see how I could do all these debugging by myself... >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Eric >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>>> Professionnel de recherche >>>>>>>>> GIREF/Université Laval >>>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>> -- >>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>> Professionnel de recherche >>>>>>>> GIREF/Université Laval >>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>> <fedora_mkl_and_devtools.txt><openmpi.txt><petsc.txt> >>>>>>> >>>>>> >>>>> >>>> -- >>>> Eric Chamberland, ing., M. Ing >>>> Professionnel de recherche >>>> GIREF/Université Laval >>>> (418) 656-2131 poste 41 22 42 >>> >> -- >> Eric Chamberland, ing., M. Ing >> Professionnel de recherche >> GIREF/Université Laval >> (418) 656-2131 poste 41 22 42 > -- Eric Chamberland, ing., M. Ing Professionnel de recherche GIREF/Université Laval (418) 656-2131 poste 41 22 42

1 0

Re: [petsc-dev] Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines
by Satish Balay 17 Mar '21

17 Mar '21

On Wed, 17 Mar 2021, Blaise A Bourdin wrote: > > > On Mar 16, 2021, at 11:17 PM, Barry Smith <bsmith(a)petsc.dev<mailto:[email protected]>> wrote: > > > So it actually does not load everything at the call to use but waits until the subroutine definitions and then only loads what you ask to import? > > Sorry, this is odd, with C include or python import it immediately loads up everything as soon as it sees the include or import, there is no way later to say "wo hoarsy I didn't really mean you should get everything I asked, for please only load up a subset". > > > But then Fortran was always weird, when I saw presentations of the Fortran standards committee members I always felt like they read every third page of some computer science book but never realized they missed many of the pages. > > Which proves that they know their target audience ;) > > >From the standpoint of somebody who also read every third page of a computer science book, fortran modules sounded like a good idea on the paper (I basically think of them as auto-generated header files), but they are miserable in practice, and in particular a major block to cross compiler compatibility since the format of .mod file was left to the discretion of the compiler writers. > Not just (a)cross compiler - different versions of gfortran are incompatible with each other's .mod format.. Satish > Blaise > > -- > A.K. & Shirley Barton Professor of Mathematics > Adjunct Professor of Mechanical Engineering > Adjunct of the Center for Computation & Technology > Louisiana State University, Lockett Hall Room 344, Baton Rouge, LA 70803, USA > Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 Web http://www.math.lsu.edu/~bourdin > >

1 0

Re: [petsc-dev] Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines
by Blaise A Bourdin 17 Mar '21

17 Mar '21

On Mar 16, 2021, at 11:17 PM, Barry Smith <bsmith(a)petsc.dev<mailto:[email protected]>> wrote: So it actually does not load everything at the call to use but waits until the subroutine definitions and then only loads what you ask to import? Sorry, this is odd, with C include or python import it immediately loads up everything as soon as it sees the include or import, there is no way later to say "wo hoarsy I didn't really mean you should get everything I asked, for please only load up a subset". But then Fortran was always weird, when I saw presentations of the Fortran standards committee members I always felt like they read every third page of some computer science book but never realized they missed many of the pages. Which proves that they know their target audience ;) >From the standpoint of somebody who also read every third page of a computer science book, fortran modules sounded like a good idea on the paper (I basically think of them as auto-generated header files), but they are miserable in practice, and in particular a major block to cross compiler compatibility since the format of .mod file was left to the discretion of the compiler writers. Blaise -- A.K. & Shirley Barton Professor of Mathematics Adjunct Professor of Mechanical Engineering Adjunct of the Center for Computation & Technology Louisiana State University, Lockett Hall Room 344, Baton Rouge, LA 70803, USA Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 Web http://www.math.lsu.edu/~bourdin

1 0

MatTransposeMatMult() bug
by Pierre Jolivet 17 Mar '21

17 Mar '21

Hello, While trying out Stefano’s PCApplyMat_MG() code (*), we stumbled upon weird numerical errors when reusing a Mat for both MatProduct_AB and MatProduct_AtB. This reminded me that there has been a long-standing issue with MatTransposeMatMult(), see https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/pc/impls/hpddm/hpddm.cx… <https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/pc/impls/hpddm/hpddm.cx…>, that I never looked into. I’ve now been trying to figure this out, because this has side effects in multiple places (PCMG and PCHPDDM at least), and thus could impact user-code as well? With this commit: https://gitlab.com/petsc/petsc/-/commit/03d8bd538039defc2fcc3e37d523735c4aa… <https://gitlab.com/petsc/petsc/-/commit/03d8bd538039defc2fcc3e37d523735c4aa…> + $ mpirun -n 4 src/ksp/ksp/tutorials/ex76 -ksp_converged_reason -pc_type hpddm -pc_hpddm_levels_1_eps_nev 20 -ksp_type preonly -mat_type aij -load_dir ${DATAFILESPATH}/matrices/hpddm/GENEO -rhs 2 -pc_hpddm_coarse_correction balanced -C_input_mattransposematmult -D_output_mattransposematmult I’m seeing that C is nonzero, but D is full of zeros. Mat Object: 4 MPI processes type: mpidense 5.7098316584361917e-08 1.0159399260517841e-07 1.5812349976211856e-07 2.0688121715350138e-07 2.4887556933361981e-08 4.8111092300772958e-08 1.4606298643602107e-07 1.7213611729839211e-07 […] Mat Object: 4 MPI processes type: mpidense 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 […] If one switches to a MatType which has no MatProduct_AtB implementation with B of type MPIDense (reminder: in that case, the product is computed column-by-column), e.g., -mat_type sbaij, one gets the expected result. Mat Object: 4 MPI processes type: mpidense 7.2003197398135299e-01 9.5191869895699011e-01 6.1793966541680234e-02 9.3884397585488877e-01 1.0022337823233585e-02 2.4653068080134588e-01 1.4463931936094099e-01 8.6111517670701687e-01 Is there a bug somewhere with the MatAIJ implementation, or am I doing something which is not allowed by the MatProduct() machinery? Thanks, Pierre (*) https://gitlab.com/petsc/petsc/-/merge_requests/3717 <https://gitlab.com/petsc/petsc/-/merge_requests/3717>

1 0

Re: [petsc-dev] Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines
by Barry Smith 16 Mar '21

16 Mar '21

So it actually does not load everything at the call to use but waits until the subroutine definitions and then only loads what you ask to import? Sorry, this is odd, with C include or python import it immediately loads up everything as soon as it sees the include or import, there is no way later to say "wo hoarsy I didn't really mean you should get everything I asked, for please only load up a subset". But then Fortran was always weird, when I saw presentations of the Fortran standards committee members I always felt like they read every third page of some computer science book but never realized they missed many of the pages. But, ok, if it works, it works. Barry > On Mar 16, 2021, at 6:54 PM, Jacob Faibussowitsch <jacob.fai(a)gmail.com> wrote: > >> I am puzzled by the import tXXX, how it can work? How does it know what module file to look in for the import? With use xxx the xxx gets translated directly to a file name. But with import tXXX that is impossible. > > Its similar to nested namespace semantics, you “use" the super-namespace at the module level then import the objects at interface of that module. i.e.: > > Module foo > Use bar <—— module contains bop > Interface > Subroutine baz(x) > Import bop <—— hoists bop out of bar defined in super-scope > Bop x > End subroutine > End interface > End module > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > Cell: (312) 694-3391 > >> On Mar 16, 2021, at 18:25, Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >> >> >> >> >>> On Mar 16, 2021, at 5:03 PM, Satish Balay <balay(a)mcs.anl.gov <mailto:[email protected]>> wrote: >>> >>> On Tue, 16 Mar 2021, Barry Smith wrote: >>> >>>> >>>> Jacob, >>>> >>>> You very well may be right, splitting may not be needed or relevant. I only advocated it since for Mat it did fix a VM gfortran memory issue. How closely the IBM compiler and gfortran issues are I do not know. >>>> >>>> My MR Mat split is read to to go into main. >>>> >>>> https://gitlab.com/petsc/petsc/-/merge_requests/3715 <https://gitlab.com/petsc/petsc/-/merge_requests/3715> <https://gitlab.com/petsc/petsc/-/merge_requests/3715 <https://gitlab.com/petsc/petsc/-/merge_requests/3715>> The MPI one is still needing some babysitting, hopefully it can pass the CI and get into main soon. >>>> >>>> Both should be in main before starting to muck with changing the use stuff for the IBM compiler. >>>> >>>> Satish, does your Vec file split pass the CI? If so I think we should also get it into main before messing with the use for the IBM compiler. >>> >>> Barry - if the switch from 'use' to 'import' is sufficient [for VM builds] - should we still split the sources? Right now it changes only the auto-fortran stubs, fixing custom stubs should give further improvements. >> >> If the import solves the VM gfortran problem then I see no reason to split the source (it is a little easier to maintain as one file). >> >> So maybe, for now use the import and later if the VM gfortran problem continues we split the source. >> >> Barry >> >> I am puzzled by the import tXXX, how it can work? How does it know what module file to look in for the import? With use xxx the xxx gets translated directly to a file name. But with import tXXX that is impossible. >> >> >> >>> >>> If spliting sources is useful - I'll look into pushing the Vec split into this MR. >>> >>> BTW: Matt - can you test balay/reorg-f90-for-xlf branch against 'gfortran VM'? >>> >>> Satish >>> >>>> >>>> >>>> Who has access to the wonky IBM compiler? Is there any way to have general access so it can be added to the CI and we can maintain code that works with it? Or does someone need to fix things one at a time, first Vec, then Mat, then .... >>>> >>>> >>>> Barry >>>> >>>> >>>>> On Mar 16, 2021, at 2:14 PM, Satish Balay via petsc-dev <petsc-dev(a)mcs.anl.gov <mailto:[email protected]>> wrote: >>>>> >>>>> On Tue, 16 Mar 2021, Matthew Knepley wrote: >>>>> >>>>>> On Tue, Mar 16, 2021 at 2:42 PM Jacob Faibussowitsch <jacob.fai(a)gmail.com <mailto:[email protected]>> >>>>>> wrote: >>>>>> >>>>>>> To me the best solution is to first split the files when possible (it is >>>>>>> often not or just too painful) and then adding imports if needed. >>>>>>> >>>>>>> >>>>>>> I’m not sure I agree. Bear in mind my familiarity with fortran is very >>>>>>> limited, but to me the “use” statement is similar to #include <header> in C >>>>>>> or using namespace xxx in cpp. >>>>>>> >>>>>>> Our fortran interface is not optimal code atm, it’s as if we put a >>>>>>> "#include <petscvec.h>” before every Vec function in the C source >>>>>>> (assuming include guards are not used). Sure you can fix this by splitting >>>>>>> the vec src files in two, but at the end of the day that’s a bandaid >>>>>>> solution. I think I can get the import stuff to work reasonably well (a lot >>>>>>> of the infrastructure to capture types is already present in the current >>>>>>> generatefortranstubs). >>>>>>> >>>>>> >>>>>> If you can get this to work, we should definitely do it. Was there a >>>>>> question about getting it to work? >>>>> >>>>> Likely these additional changes should be done starting with Barry's split-up of sources. >>>>> >>>>> Satish >>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Best regards, >>>>>>> >>>>>>> Jacob Faibussowitsch >>>>>>> (Jacob Fai - booss - oh - vitch) >>>>>>> Cell: (312) 694-3391 >>>>>>> >>>>>>> On Mar 16, 2021, at 12:56, Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> >>>>>>> Jacob, >>>>>>> >>>>>>> I split the mat definitions in the MR >>>>>>> https://gitlab.com/petsc/petsc/-/merge_requests/3696 <https://gitlab.com/petsc/petsc/-/merge_requests/3696> and this reduced >>>>>>> memory requirements enough to get builds through on some VM that failed >>>>>>> previously ran out of memory (with gfortran). >>>>>>> >>>>>>> The petsc*mod.F90 files are all handwritten so it is ok to manually >>>>>>> change them with imports if that helps the IBM compiler. To me the best >>>>>>> solution is to first split the files when possible (it is often not or just >>>>>>> too painful) and then adding imports if needed. >>>>>>> >>>>>>> Note also the MR >>>>>>> https://gitlab.com/petsc/petsc/-/merge_requests/3715/diffs <https://gitlab.com/petsc/petsc/-/merge_requests/3715/diffs> which may help >>>>>>> a great deal with the IBM compiler or not. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mar 16, 2021, at 11:47 AM, Jacob Faibussowitsch <jacob.fai(a)gmail.com <mailto:[email protected]>> >>>>>>> wrote: >>>>>>> >>>>>>> Ok something I have gotten to work, but only doing things by hand in >>>>>>> petscvecmod.F90: >>>>>>> >>>>>>> diff --git a/src/vec/f90-mod/petscvecmod.F90 >>>>>>> b/src/vec/f90-mod/petscvecmod.F90 >>>>>>> index 0c447156b9..ef3e2e2e55 100644 >>>>>>> --- a/src/vec/f90-mod/petscvecmod.F90 >>>>>>> +++ b/src/vec/f90-mod/petscvecmod.F90 >>>>>>> module petscisdef >>>>>>> use petscisdefdummy >>>>>>> interface operator(.ne.) >>>>>>> function isnotequal(A,B) >>>>>>> - use petscisdefdummy >>>>>>> + import tIS >>>>>>> logical isnotequal >>>>>>> >>>>>>> type(tIS), intent(in) :: A,B >>>>>>> end function isnotequal >>>>>>> >>>>>>> This works for everything wrapped in a module which contains an interface >>>>>>> block. For dangling functions the following works: >>>>>>> >>>>>>> function isnotequal(A,B) >>>>>>> - use petscisdefdummy >>>>>>> + use petscisdef, only: tIs >>>>>>> logical isnotequal >>>>>>> type(tIS), intent(in) :: A,B >>>>>>> isnotequal = (A%v .ne. B%v) >>>>>>> end function isnotequal >>>>>>> >>>>>>> Do our fortran stub generators have any notion of types? Specifically >>>>>>> types that originate from petsc? >>>>>>> >>>>>>> Best regards, >>>>>>> >>>>>>> Jacob Faibussowitsch >>>>>>> (Jacob Fai - booss - oh - vitch) >>>>>>> Cell: (312) 694-3391 >>>>>>> >>>>>>> On Mar 16, 2021, at 11:11, Satish Balay <balay(a)mcs.anl.gov <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> On Tue, 16 Mar 2021, Jacob Faibussowitsch wrote: >>>>>>> >>>>>>> My [partial] change is in branch balay/reorg-f90-for-xlf >>>>>>> >>>>>>> >>>>>>> Satish is this branch pushed? I’d like to send it to the ibm folks to see >>>>>>> if it works for them as well. >>>>>>> >>>>>>> >>>>>>> Sorry - pushed now. From what I remember - it didn't work. >>>>>>> >>>>>>> Satish >>>>>>> >>>>>>> >>>>>>> They also added this extra follow up: >>>>>>> >>>>>>> The change we did in the source files is to replace all the "use pet*" >>>>>>> statements in all the Interface blocks with IMPORT statement. >>>>>>> >>>>>>> The nature of this workaround is to reduce the number of symbols that the >>>>>>> compiler have to create, which exceeded the limitation and caused ICE. >>>>>>> >>>>>>> Every USE statement in an interface block opens up the module symbol file >>>>>>> and reads all the symbols from it and creates an entity for each symbol in >>>>>>> compiler. This is unnecessary when the module already has the same USE >>>>>>> statement in the module scope. Instead, users can use IMPORT statement to >>>>>>> make the module symbols accessible inside interface face blocks. >>>>>>> >>>>>>> With the change, the compiler would only read the module symbol file once >>>>>>> and create one set of symbols where the old code reads the module symbol >>>>>>> files as many times as the number of the USE statements in Interface blocks >>>>>>> and create that many sets of duplicate symbols. Replacing those USE >>>>>>> statements with IMPORT statements also shortens the compile time >>>>>>> significantly. >>>>>>> >>>>>>> Best regards, >>>>>>> >>>>>>> Jacob Faibussowitsch >>>>>>> (Jacob Fai - booss - oh - vitch) >>>>>>> Cell: (312) 694-3391 >>>>>>> >>>>>>> On Mar 3, 2021, at 13:43, Satish Balay via petsc-dev < >>>>>>> petsc-dev(a)mcs.anl.gov <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> The only change I can get working (i.e avoid compile errors) is to split >>>>>>> petscvecmod.F90 into 2 source files - but I don't know if this will help >>>>>>> with xlf.. >>>>>>> >>>>>>> My [partial] change is in branch balay/reorg-f90-for-xlf >>>>>>> >>>>>>> Satish >>>>>>> >>>>>>> On Wed, 3 Mar 2021, Satish Balay via petsc-dev wrote: >>>>>>> >>>>>>> On Wed, 3 Mar 2021, Satish Balay via petsc-dev wrote: >>>>>>> >>>>>>> Sure - once any change works locally [for gcc and xlf] >>>>>>> >>>>>>> When I try - I get a bunch of errors.. [yet to digest them.] >>>>>>> >>>>>>> >>>>>>> Can you please give the following source code workaround a try? >>>>>>> Since there is already "use petscvecdefdummy" at the module scope, one >>>>>>> workaround might be to remove the unnecessary "use petscvecdefdummy" in >>>>>>> vecnotequal and vecequals >>>>>>> and all similar procedures. >>>>>>> >>>>>>> For example, the test case has: >>>>>>> module petscvecdef >>>>>>> use petscvecdefdummy >>>>>>> ... >>>>>>> function vecnotequal(A,B) >>>>>>> use petscvecdefdummy >>>>>>> logical vecnotequal >>>>>>> type(tVec), intent(in) :: A,B >>>>>>> vecnotequal = (A%v .ne. B%v) >>>>>>> end function >>>>>>> >>>>>>> >>>>>>> >>>>>>> Ok - try this suggestion: >>>>>>> >>>>>>> diff --git a/src/vec/f90-mod/petscvecmod.F90 >>>>>>> b/src/vec/f90-mod/petscvecmod.F90 >>>>>>> index 0c447156b9..81968c7ca1 100644 >>>>>>> --- a/src/vec/f90-mod/petscvecmod.F90 >>>>>>> +++ b/src/vec/f90-mod/petscvecmod.F90 >>>>>>> @@ -77,7 +77,6 @@ >>>>>>> use petscvecdefdummy >>>>>>> interface operator(.ne.) >>>>>>> function vecnotequal(A,B) >>>>>>> - use petscvecdefdummy >>>>>>> logical vecnotequal >>>>>>> type(tVec), intent(in) :: A,B >>>>>>> end function >>>>>>> >>>>>>> >>>>>>> >>>>>>> FC arch-linux-c-debug/obj/vec/f90-mod/petscvecmod.o >>>>>>> /home/balay/petsc/src/vec/f90-mod/petscvecmod.F90:81:22: >>>>>>> >>>>>>> 81 | type(tVec), intent(in) :: A,B >>>>>>> | 1 >>>>>>> Error: Derived type ‘tvec’ at (1) is being used before it is defined >>>>>>> >>>>>>> /home/balay/petsc/include/../src/vec/f90-mod/ftn-auto-interfaces/petscis.h90:2:10: >>>>>>> >>>>>>> 2 | use petscvecdef >>>>>>> | 1 >>>>>>> Fatal Error: Cannot open module file ‘petscvecdef.mod’ for reading at (1): >>>>>>> No such file or directory >>>>>>> <<<<<< >>>>>>> >>>>>>> Satish >

1 0

Re: [petsc-dev] Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines
by Jacob Faibussowitsch 16 Mar '21

16 Mar '21

> I am puzzled by the import tXXX, how it can work? How does it know what module file to look in for the import? With use xxx the xxx gets translated directly to a file name. But with import tXXX that is impossible. Its similar to nested namespace semantics, you “use" the super-namespace at the module level then import the objects at interface of that module. i.e.: Module foo Use bar <—— module contains bop Interface Subroutine baz(x) Import bop <—— hoists bop out of bar defined in super-scope Bop x End subroutine End interface End module Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) Cell: (312) 694-3391 > On Mar 16, 2021, at 18:25, Barry Smith <bsmith(a)petsc.dev> wrote: > > > > >> On Mar 16, 2021, at 5:03 PM, Satish Balay <balay(a)mcs.anl.gov> wrote: >> >> On Tue, 16 Mar 2021, Barry Smith wrote: >> >>> >>> Jacob, >>> >>> You very well may be right, splitting may not be needed or relevant. I only advocated it since for Mat it did fix a VM gfortran memory issue. How closely the IBM compiler and gfortran issues are I do not know. >>> >>> My MR Mat split is read to to go into main. >>> >>> https://gitlab.com/petsc/petsc/-/merge_requests/3715 <https://gitlab.com/petsc/petsc/-/merge_requests/3715> The MPI one is still needing some babysitting, hopefully it can pass the CI and get into main soon. >>> >>> Both should be in main before starting to muck with changing the use stuff for the IBM compiler. >>> >>> Satish, does your Vec file split pass the CI? If so I think we should also get it into main before messing with the use for the IBM compiler. >> >> Barry - if the switch from 'use' to 'import' is sufficient [for VM builds] - should we still split the sources? Right now it changes only the auto-fortran stubs, fixing custom stubs should give further improvements. > > If the import solves the VM gfortran problem then I see no reason to split the source (it is a little easier to maintain as one file). > > So maybe, for now use the import and later if the VM gfortran problem continues we split the source. > > Barry > > I am puzzled by the import tXXX, how it can work? How does it know what module file to look in for the import? With use xxx the xxx gets translated directly to a file name. But with import tXXX that is impossible. > > > >> >> If spliting sources is useful - I'll look into pushing the Vec split into this MR. >> >> BTW: Matt - can you test balay/reorg-f90-for-xlf branch against 'gfortran VM'? >> >> Satish >> >>> >>> >>> Who has access to the wonky IBM compiler? Is there any way to have general access so it can be added to the CI and we can maintain code that works with it? Or does someone need to fix things one at a time, first Vec, then Mat, then .... >>> >>> >>> Barry >>> >>> >>>> On Mar 16, 2021, at 2:14 PM, Satish Balay via petsc-dev <petsc-dev(a)mcs.anl.gov> wrote: >>>> >>>> On Tue, 16 Mar 2021, Matthew Knepley wrote: >>>> >>>>> On Tue, Mar 16, 2021 at 2:42 PM Jacob Faibussowitsch <jacob.fai(a)gmail.com> >>>>> wrote: >>>>> >>>>>> To me the best solution is to first split the files when possible (it is >>>>>> often not or just too painful) and then adding imports if needed. >>>>>> >>>>>> >>>>>> I’m not sure I agree. Bear in mind my familiarity with fortran is very >>>>>> limited, but to me the “use” statement is similar to #include <header> in C >>>>>> or using namespace xxx in cpp. >>>>>> >>>>>> Our fortran interface is not optimal code atm, it’s as if we put a >>>>>> "#include <petscvec.h>” before every Vec function in the C source >>>>>> (assuming include guards are not used). Sure you can fix this by splitting >>>>>> the vec src files in two, but at the end of the day that’s a bandaid >>>>>> solution. I think I can get the import stuff to work reasonably well (a lot >>>>>> of the infrastructure to capture types is already present in the current >>>>>> generatefortranstubs). >>>>>> >>>>> >>>>> If you can get this to work, we should definitely do it. Was there a >>>>> question about getting it to work? >>>> >>>> Likely these additional changes should be done starting with Barry's split-up of sources. >>>> >>>> Satish >>>> >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Best regards, >>>>>> >>>>>> Jacob Faibussowitsch >>>>>> (Jacob Fai - booss - oh - vitch) >>>>>> Cell: (312) 694-3391 >>>>>> >>>>>> On Mar 16, 2021, at 12:56, Barry Smith <bsmith(a)petsc.dev> wrote: >>>>>> >>>>>> >>>>>> Jacob, >>>>>> >>>>>> I split the mat definitions in the MR >>>>>> https://gitlab.com/petsc/petsc/-/merge_requests/3696 and this reduced >>>>>> memory requirements enough to get builds through on some VM that failed >>>>>> previously ran out of memory (with gfortran). >>>>>> >>>>>> The petsc*mod.F90 files are all handwritten so it is ok to manually >>>>>> change them with imports if that helps the IBM compiler. To me the best >>>>>> solution is to first split the files when possible (it is often not or just >>>>>> too painful) and then adding imports if needed. >>>>>> >>>>>> Note also the MR >>>>>> https://gitlab.com/petsc/petsc/-/merge_requests/3715/diffs which may help >>>>>> a great deal with the IBM compiler or not. >>>>>> >>>>>> Thanks >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>> >>>>>> On Mar 16, 2021, at 11:47 AM, Jacob Faibussowitsch <jacob.fai(a)gmail.com> >>>>>> wrote: >>>>>> >>>>>> Ok something I have gotten to work, but only doing things by hand in >>>>>> petscvecmod.F90: >>>>>> >>>>>> diff --git a/src/vec/f90-mod/petscvecmod.F90 >>>>>> b/src/vec/f90-mod/petscvecmod.F90 >>>>>> index 0c447156b9..ef3e2e2e55 100644 >>>>>> --- a/src/vec/f90-mod/petscvecmod.F90 >>>>>> +++ b/src/vec/f90-mod/petscvecmod.F90 >>>>>> module petscisdef >>>>>> use petscisdefdummy >>>>>> interface operator(.ne.) >>>>>> function isnotequal(A,B) >>>>>> - use petscisdefdummy >>>>>> + import tIS >>>>>> logical isnotequal >>>>>> >>>>>> type(tIS), intent(in) :: A,B >>>>>> end function isnotequal >>>>>> >>>>>> This works for everything wrapped in a module which contains an interface >>>>>> block. For dangling functions the following works: >>>>>> >>>>>> function isnotequal(A,B) >>>>>> - use petscisdefdummy >>>>>> + use petscisdef, only: tIs >>>>>> logical isnotequal >>>>>> type(tIS), intent(in) :: A,B >>>>>> isnotequal = (A%v .ne. B%v) >>>>>> end function isnotequal >>>>>> >>>>>> Do our fortran stub generators have any notion of types? Specifically >>>>>> types that originate from petsc? >>>>>> >>>>>> Best regards, >>>>>> >>>>>> Jacob Faibussowitsch >>>>>> (Jacob Fai - booss - oh - vitch) >>>>>> Cell: (312) 694-3391 >>>>>> >>>>>> On Mar 16, 2021, at 11:11, Satish Balay <balay(a)mcs.anl.gov> wrote: >>>>>> >>>>>> On Tue, 16 Mar 2021, Jacob Faibussowitsch wrote: >>>>>> >>>>>> My [partial] change is in branch balay/reorg-f90-for-xlf >>>>>> >>>>>> >>>>>> Satish is this branch pushed? I’d like to send it to the ibm folks to see >>>>>> if it works for them as well. >>>>>> >>>>>> >>>>>> Sorry - pushed now. From what I remember - it didn't work. >>>>>> >>>>>> Satish >>>>>> >>>>>> >>>>>> They also added this extra follow up: >>>>>> >>>>>> The change we did in the source files is to replace all the "use pet*" >>>>>> statements in all the Interface blocks with IMPORT statement. >>>>>> >>>>>> The nature of this workaround is to reduce the number of symbols that the >>>>>> compiler have to create, which exceeded the limitation and caused ICE. >>>>>> >>>>>> Every USE statement in an interface block opens up the module symbol file >>>>>> and reads all the symbols from it and creates an entity for each symbol in >>>>>> compiler. This is unnecessary when the module already has the same USE >>>>>> statement in the module scope. Instead, users can use IMPORT statement to >>>>>> make the module symbols accessible inside interface face blocks. >>>>>> >>>>>> With the change, the compiler would only read the module symbol file once >>>>>> and create one set of symbols where the old code reads the module symbol >>>>>> files as many times as the number of the USE statements in Interface blocks >>>>>> and create that many sets of duplicate symbols. Replacing those USE >>>>>> statements with IMPORT statements also shortens the compile time >>>>>> significantly. >>>>>> >>>>>> Best regards, >>>>>> >>>>>> Jacob Faibussowitsch >>>>>> (Jacob Fai - booss - oh - vitch) >>>>>> Cell: (312) 694-3391 >>>>>> >>>>>> On Mar 3, 2021, at 13:43, Satish Balay via petsc-dev < >>>>>> petsc-dev(a)mcs.anl.gov> wrote: >>>>>> >>>>>> The only change I can get working (i.e avoid compile errors) is to split >>>>>> petscvecmod.F90 into 2 source files - but I don't know if this will help >>>>>> with xlf.. >>>>>> >>>>>> My [partial] change is in branch balay/reorg-f90-for-xlf >>>>>> >>>>>> Satish >>>>>> >>>>>> On Wed, 3 Mar 2021, Satish Balay via petsc-dev wrote: >>>>>> >>>>>> On Wed, 3 Mar 2021, Satish Balay via petsc-dev wrote: >>>>>> >>>>>> Sure - once any change works locally [for gcc and xlf] >>>>>> >>>>>> When I try - I get a bunch of errors.. [yet to digest them.] >>>>>> >>>>>> >>>>>> Can you please give the following source code workaround a try? >>>>>> Since there is already "use petscvecdefdummy" at the module scope, one >>>>>> workaround might be to remove the unnecessary "use petscvecdefdummy" in >>>>>> vecnotequal and vecequals >>>>>> and all similar procedures. >>>>>> >>>>>> For example, the test case has: >>>>>> module petscvecdef >>>>>> use petscvecdefdummy >>>>>> ... >>>>>> function vecnotequal(A,B) >>>>>> use petscvecdefdummy >>>>>> logical vecnotequal >>>>>> type(tVec), intent(in) :: A,B >>>>>> vecnotequal = (A%v .ne. B%v) >>>>>> end function >>>>>> >>>>>> >>>>>> >>>>>> Ok - try this suggestion: >>>>>> >>>>>> diff --git a/src/vec/f90-mod/petscvecmod.F90 >>>>>> b/src/vec/f90-mod/petscvecmod.F90 >>>>>> index 0c447156b9..81968c7ca1 100644 >>>>>> --- a/src/vec/f90-mod/petscvecmod.F90 >>>>>> +++ b/src/vec/f90-mod/petscvecmod.F90 >>>>>> @@ -77,7 +77,6 @@ >>>>>> use petscvecdefdummy >>>>>> interface operator(.ne.) >>>>>> function vecnotequal(A,B) >>>>>> - use petscvecdefdummy >>>>>> logical vecnotequal >>>>>> type(tVec), intent(in) :: A,B >>>>>> end function >>>>>> >>>>>> >>>>>> >>>>>> FC arch-linux-c-debug/obj/vec/f90-mod/petscvecmod.o >>>>>> /home/balay/petsc/src/vec/f90-mod/petscvecmod.F90:81:22: >>>>>> >>>>>> 81 | type(tVec), intent(in) :: A,B >>>>>> | 1 >>>>>> Error: Derived type ‘tvec’ at (1) is being used before it is defined >>>>>> >>>>>> /home/balay/petsc/include/../src/vec/f90-mod/ftn-auto-interfaces/petscis.h90:2:10: >>>>>> >>>>>> 2 | use petscvecdef >>>>>> | 1 >>>>>> Fatal Error: Cannot open module file ‘petscvecdef.mod’ for reading at (1): >>>>>> No such file or directory >>>>>> <<<<<< >>>>>> >>>>>> Satish

1 0

Re: [petsc-dev] Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines
by Barry Smith 16 Mar '21

16 Mar '21

> On Mar 16, 2021, at 5:03 PM, Satish Balay <balay(a)mcs.anl.gov> wrote: > > On Tue, 16 Mar 2021, Barry Smith wrote: > >> >> Jacob, >> >> You very well may be right, splitting may not be needed or relevant. I only advocated it since for Mat it did fix a VM gfortran memory issue. How closely the IBM compiler and gfortran issues are I do not know. >> >> My MR Mat split is read to to go into main. >> >> https://gitlab.com/petsc/petsc/-/merge_requests/3715 <https://gitlab.com/petsc/petsc/-/merge_requests/3715> The MPI one is still needing some babysitting, hopefully it can pass the CI and get into main soon. >> >> Both should be in main before starting to muck with changing the use stuff for the IBM compiler. >> >> Satish, does your Vec file split pass the CI? If so I think we should also get it into main before messing with the use for the IBM compiler. > > Barry - if the switch from 'use' to 'import' is sufficient [for VM builds] - should we still split the sources? Right now it changes only the auto-fortran stubs, fixing custom stubs should give further improvements. If the import solves the VM gfortran problem then I see no reason to split the source (it is a little easier to maintain as one file). So maybe, for now use the import and later if the VM gfortran problem continues we split the source. Barry I am puzzled by the import tXXX, how it can work? How does it know what module file to look in for the import? With use xxx the xxx gets translated directly to a file name. But with import tXXX that is impossible. > > If spliting sources is useful - I'll look into pushing the Vec split into this MR. > > BTW: Matt - can you test balay/reorg-f90-for-xlf branch against 'gfortran VM'? > > Satish > >> >> >> Who has access to the wonky IBM compiler? Is there any way to have general access so it can be added to the CI and we can maintain code that works with it? Or does someone need to fix things one at a time, first Vec, then Mat, then .... >> >> >> Barry >> >> >>> On Mar 16, 2021, at 2:14 PM, Satish Balay via petsc-dev <petsc-dev(a)mcs.anl.gov> wrote: >>> >>> On Tue, 16 Mar 2021, Matthew Knepley wrote: >>> >>>> On Tue, Mar 16, 2021 at 2:42 PM Jacob Faibussowitsch <jacob.fai(a)gmail.com> >>>> wrote: >>>> >>>>> To me the best solution is to first split the files when possible (it is >>>>> often not or just too painful) and then adding imports if needed. >>>>> >>>>> >>>>> I’m not sure I agree. Bear in mind my familiarity with fortran is very >>>>> limited, but to me the “use” statement is similar to #include <header> in C >>>>> or using namespace xxx in cpp. >>>>> >>>>> Our fortran interface is not optimal code atm, it’s as if we put a >>>>> "#include <petscvec.h>” before every Vec function in the C source >>>>> (assuming include guards are not used). Sure you can fix this by splitting >>>>> the vec src files in two, but at the end of the day that’s a bandaid >>>>> solution. I think I can get the import stuff to work reasonably well (a lot >>>>> of the infrastructure to capture types is already present in the current >>>>> generatefortranstubs). >>>>> >>>> >>>> If you can get this to work, we should definitely do it. Was there a >>>> question about getting it to work? >>> >>> Likely these additional changes should be done starting with Barry's split-up of sources. >>> >>> Satish >>> >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Best regards, >>>>> >>>>> Jacob Faibussowitsch >>>>> (Jacob Fai - booss - oh - vitch) >>>>> Cell: (312) 694-3391 >>>>> >>>>> On Mar 16, 2021, at 12:56, Barry Smith <bsmith(a)petsc.dev> wrote: >>>>> >>>>> >>>>> Jacob, >>>>> >>>>> I split the mat definitions in the MR >>>>> https://gitlab.com/petsc/petsc/-/merge_requests/3696 and this reduced >>>>> memory requirements enough to get builds through on some VM that failed >>>>> previously ran out of memory (with gfortran). >>>>> >>>>> The petsc*mod.F90 files are all handwritten so it is ok to manually >>>>> change them with imports if that helps the IBM compiler. To me the best >>>>> solution is to first split the files when possible (it is often not or just >>>>> too painful) and then adding imports if needed. >>>>> >>>>> Note also the MR >>>>> https://gitlab.com/petsc/petsc/-/merge_requests/3715/diffs which may help >>>>> a great deal with the IBM compiler or not. >>>>> >>>>> Thanks >>>>> >>>>> Barry >>>>> >>>>> >>>>> >>>>> On Mar 16, 2021, at 11:47 AM, Jacob Faibussowitsch <jacob.fai(a)gmail.com> >>>>> wrote: >>>>> >>>>> Ok something I have gotten to work, but only doing things by hand in >>>>> petscvecmod.F90: >>>>> >>>>> diff --git a/src/vec/f90-mod/petscvecmod.F90 >>>>> b/src/vec/f90-mod/petscvecmod.F90 >>>>> index 0c447156b9..ef3e2e2e55 100644 >>>>> --- a/src/vec/f90-mod/petscvecmod.F90 >>>>> +++ b/src/vec/f90-mod/petscvecmod.F90 >>>>> module petscisdef >>>>> use petscisdefdummy >>>>> interface operator(.ne.) >>>>> function isnotequal(A,B) >>>>> - use petscisdefdummy >>>>> + import tIS >>>>> logical isnotequal >>>>> >>>>> type(tIS), intent(in) :: A,B >>>>> end function isnotequal >>>>> >>>>> This works for everything wrapped in a module which contains an interface >>>>> block. For dangling functions the following works: >>>>> >>>>> function isnotequal(A,B) >>>>> - use petscisdefdummy >>>>> + use petscisdef, only: tIs >>>>> logical isnotequal >>>>> type(tIS), intent(in) :: A,B >>>>> isnotequal = (A%v .ne. B%v) >>>>> end function isnotequal >>>>> >>>>> Do our fortran stub generators have any notion of types? Specifically >>>>> types that originate from petsc? >>>>> >>>>> Best regards, >>>>> >>>>> Jacob Faibussowitsch >>>>> (Jacob Fai - booss - oh - vitch) >>>>> Cell: (312) 694-3391 >>>>> >>>>> On Mar 16, 2021, at 11:11, Satish Balay <balay(a)mcs.anl.gov> wrote: >>>>> >>>>> On Tue, 16 Mar 2021, Jacob Faibussowitsch wrote: >>>>> >>>>> My [partial] change is in branch balay/reorg-f90-for-xlf >>>>> >>>>> >>>>> Satish is this branch pushed? I’d like to send it to the ibm folks to see >>>>> if it works for them as well. >>>>> >>>>> >>>>> Sorry - pushed now. From what I remember - it didn't work. >>>>> >>>>> Satish >>>>> >>>>> >>>>> They also added this extra follow up: >>>>> >>>>> The change we did in the source files is to replace all the "use pet*" >>>>> statements in all the Interface blocks with IMPORT statement. >>>>> >>>>> The nature of this workaround is to reduce the number of symbols that the >>>>> compiler have to create, which exceeded the limitation and caused ICE. >>>>> >>>>> Every USE statement in an interface block opens up the module symbol file >>>>> and reads all the symbols from it and creates an entity for each symbol in >>>>> compiler. This is unnecessary when the module already has the same USE >>>>> statement in the module scope. Instead, users can use IMPORT statement to >>>>> make the module symbols accessible inside interface face blocks. >>>>> >>>>> With the change, the compiler would only read the module symbol file once >>>>> and create one set of symbols where the old code reads the module symbol >>>>> files as many times as the number of the USE statements in Interface blocks >>>>> and create that many sets of duplicate symbols. Replacing those USE >>>>> statements with IMPORT statements also shortens the compile time >>>>> significantly. >>>>> >>>>> Best regards, >>>>> >>>>> Jacob Faibussowitsch >>>>> (Jacob Fai - booss - oh - vitch) >>>>> Cell: (312) 694-3391 >>>>> >>>>> On Mar 3, 2021, at 13:43, Satish Balay via petsc-dev < >>>>> petsc-dev(a)mcs.anl.gov> wrote: >>>>> >>>>> The only change I can get working (i.e avoid compile errors) is to split >>>>> petscvecmod.F90 into 2 source files - but I don't know if this will help >>>>> with xlf.. >>>>> >>>>> My [partial] change is in branch balay/reorg-f90-for-xlf >>>>> >>>>> Satish >>>>> >>>>> On Wed, 3 Mar 2021, Satish Balay via petsc-dev wrote: >>>>> >>>>> On Wed, 3 Mar 2021, Satish Balay via petsc-dev wrote: >>>>> >>>>> Sure - once any change works locally [for gcc and xlf] >>>>> >>>>> When I try - I get a bunch of errors.. [yet to digest them.] >>>>> >>>>> >>>>> Can you please give the following source code workaround a try? >>>>> Since there is already "use petscvecdefdummy" at the module scope, one >>>>> workaround might be to remove the unnecessary "use petscvecdefdummy" in >>>>> vecnotequal and vecequals >>>>> and all similar procedures. >>>>> >>>>> For example, the test case has: >>>>> module petscvecdef >>>>> use petscvecdefdummy >>>>> ... >>>>> function vecnotequal(A,B) >>>>> use petscvecdefdummy >>>>> logical vecnotequal >>>>> type(tVec), intent(in) :: A,B >>>>> vecnotequal = (A%v .ne. B%v) >>>>> end function >>>>> >>>>> >>>>> >>>>> Ok - try this suggestion: >>>>> >>>>> diff --git a/src/vec/f90-mod/petscvecmod.F90 >>>>> b/src/vec/f90-mod/petscvecmod.F90 >>>>> index 0c447156b9..81968c7ca1 100644 >>>>> --- a/src/vec/f90-mod/petscvecmod.F90 >>>>> +++ b/src/vec/f90-mod/petscvecmod.F90 >>>>> @@ -77,7 +77,6 @@ >>>>> use petscvecdefdummy >>>>> interface operator(.ne.) >>>>> function vecnotequal(A,B) >>>>> - use petscvecdefdummy >>>>> logical vecnotequal >>>>> type(tVec), intent(in) :: A,B >>>>> end function >>>>> >>>>> >>>>> >>>>> FC arch-linux-c-debug/obj/vec/f90-mod/petscvecmod.o >>>>> /home/balay/petsc/src/vec/f90-mod/petscvecmod.F90:81:22: >>>>> >>>>> 81 | type(tVec), intent(in) :: A,B >>>>> | 1 >>>>> Error: Derived type ‘tvec’ at (1) is being used before it is defined >>>>> >>>>> /home/balay/petsc/include/../src/vec/f90-mod/ftn-auto-interfaces/petscis.h90:2:10: >>>>> >>>>> 2 | use petscvecdef >>>>> | 1 >>>>> Fatal Error: Cannot open module file ‘petscvecdef.mod’ for reading at (1): >>>>> No such file or directory >>>>> <<<<<< >>>>> >>>>> Satish >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >> >>

1 0