petsc-dev

Download

[email protected]

March 2021

20 participants
252 discussions

Re: [petsc-dev] Fwd: Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines
by Satish Balay 03 Mar '21

03 Mar '21

On Wed, 3 Mar 2021, Jacob Faibussowitsch wrote: > Hello All, > > I discovered a compiler bug in the IBM xl fortran compiler a few weeks ago that would crash the compiler when compiling petsc fortran interfaces. The TL;DR of it is that the xl compiler creates a function dictionary for every function imported in fortran modules, and since petsc fortran interfaces seem to import entire packages writ-large this exceeds the number of dictionary entries (2**21): > > > The reason for the Internal Compiler Error is because we can't grow an interal dictionary anymore (ie we hit a 2**21 limit). > > The file contains many module procedures and interfaces that use the same helper module. As a result, we are importing the dictionary entries for that module repeatedly reaching > > the limit. > > > > Can you please give the following source code workaround a try? > > Since there is already "use petscvecdefdummy" at the module scope, one workaround might be to remove the unnecessary "use petscvecdefdummy" in vecnotequal and vecequals > > and all similar procedures. > > > > For example, the test case has: > > module petscvecdef > > use petscvecdefdummy > > ... > > function vecnotequal(A,B) > > use petscvecdefdummy > > logical vecnotequal > > type(tVec), intent(in) :: A,B > > vecnotequal = (A%v .ne. B%v) > > end function > > function vecequals(A,B) > > use petscvecdefdummy > > logical vecequals > > type(tVec), intent(in) :: A,B > > vecequals = (A%v .eq. B%v) > > end function > > ... > > end module > > Another workaround would be to put the procedure definitions from this large module into several submodules. Each submodule would be able to accommodate a dictionary with 2**21 entries. > > > > > > Please let us know if one of the above workarounds resolve the issue. > > > The proposed fix from IBM would be to pull “use moduleXXX” out of subroutines or to have our auto-fortran interfaces detect which symbols to include from the respective modules and only include those in the subroutines. I’m not familiar at all with how the interfaces are generated so I don’t even know if this is possible. I'm not sure what would happen if these 'use' statements are removed [whats required and what can be removed?] The relevant code that adds this is in lib/petsc/bin/maint/generatefortranstubs.py fd.write(' use petsc'+mansec+'def\n') Satish > > IBM provided the following additional explanation and example. Can the process used to generate these routines and functions determine the specific symbols required and then use the only keyword or import statement to include them? > > > > When factoring out use statements out of module procedures, you can just delete them. But you can't completely remove them from interface blocks. Instead, you can limit them either by using use <module>, only: <symbol> or import <symbol> . if the hundreds of use statements in the program are factored out / limited in this way, that should reduce the dictionary size sufficiently for the program to compile. > > > > For example > > Interface > > Subroutine VecRestoreArrayReadF90(v,array,ierr) > > use petscvecdef > > real(kind=selected_real_kind(10)), pointer :: array(:) > > integer(kind=selected_int_kind(5)) ierr > > type(tVec) v > > End Subroutine > > End Interface > > > > imports all symbols from petscvecdef into the dictionary even though we only need tVec . So we can either: > > > > Interface > > Subroutine VecRestoreArrayReadF90(v,array,ierr) > > use petscvecdef, only: tVec > > implicit none > > real(kind=selected_real_kind(10)), pointer :: array(:) > > integer(kind=selected_int_kind(5)) ierr > > type(tVec) v > > End Subroutine > > End Interface > > > > or if use petscvecdef is used in the outer scope, we can: > > Interface > > Subroutine VecRestoreArrayReadF90(v,array,ierr) > > import tVec > > implicit none > > real(kind=selected_real_kind(10)), pointer :: array(:) > > integer(kind=selected_int_kind(5)) ierr > > type(tVec) v > > End Subroutine > > End Interface > > (The two methods (use, only vs import) are equivalent in terms of impact to the dictionary.) > > > > Is this compiler ~feature~ something that we intend to work around? Thoughts? > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > Cell: (312) 694-3391 > > > Begin forwarded message: > > > > From: "Roy Musselman" <roymuss(a)us.ibm.com> > > Subject: Re: Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines > > Date: March 3, 2021 at 08:23:17 CST > > To: Jacob Faibussowitsch <faibuss2(a)illinois.edu> > > Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov> > > > > Hi Jacob, > > I tried the first suggestion and commented out the use statements called within the functions. However, I hit the following error complaining about specific symbol dependencies provided by the library. > > > > .../src/vec/f90-mod/petscvecmod.F90", line 107.37: 1514-084 (S) Identifier a is being declared with type name tvec which has not been defined in a derived type definition. > > > > IBM provided the following additional explanation and example. Can the process used to generate these routines and functions determine the specific symbols required and then use the only keyword or import statement to include them? > > > > When factoring out use statements out of module procedures, you can just delete them. But you can't completely remove them from interface blocks. Instead, you can limit them either by using use <module>, only: <symbol> or import <symbol> . if the hundreds of use statements in the program are factored out / limited in this way, that should reduce the dictionary size sufficiently for the program to compile. > > > > For example > > Interface > > Subroutine VecRestoreArrayReadF90(v,array,ierr) > > use petscvecdef > > real(kind=selected_real_kind(10)), pointer :: array(:) > > integer(kind=selected_int_kind(5)) ierr > > type(tVec) v > > End Subroutine > > End Interface > > > > imports all symbols from petscvecdef into the dictionary even though we only need tVec . So we can either: > > > > Interface > > Subroutine VecRestoreArrayReadF90(v,array,ierr) > > use petscvecdef, only: tVec > > implicit none > > real(kind=selected_real_kind(10)), pointer :: array(:) > > integer(kind=selected_int_kind(5)) ierr > > type(tVec) v > > End Subroutine > > End Interface > > > > or if use petscvecdef is used in the outer scope, we can: > > Interface > > Subroutine VecRestoreArrayReadF90(v,array,ierr) > > import tVec > > implicit none > > real(kind=selected_real_kind(10)), pointer :: array(:) > > integer(kind=selected_int_kind(5)) ierr > > type(tVec) v > > End Subroutine > > End Interface > > (The two methods (use, only vs import) are equivalent in terms of impact to the dictionary.) > > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Roy Musselman > > IBM HPC Application Analyst at Lawrence Livermore National Lab > > email: roymuss(a)us.ibm.com <mailto:[email protected]> > > LLNL office: 925-422-6033 > > Cell: 507-358-8895, Home: 507-281-9565 > > > > Roy Musselman---02/24/2021 07:08:45 PM---Hi Jacob, I opened the ticket with IBM: case TS005062693 and and the local LLNL Sierra Jira Ticket > > > > From: Roy Musselman/Rochester/Contr/IBM > > To: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> > > Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> > > Date: 02/24/2021 07:08 PM > > Subject: Re: [EXTERNAL] Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines > > > > > > > > Hi Jacob, > > I opened the ticket with IBM: case TS005062693 and and the local LLNL Sierra Jira Ticket at > > https://lc.llnl.gov/jira/projects/SIERRA/issues/SIERRA-111?filter=allissues <https://urldefense.com/v3/__https://lc.llnl.gov/jira/projects/SIERRA/issues…> > > > > Today IBM provided the response below. I don't know when I'll have time to try it on the reproducer I gave IBM. Perhaps early next week. Can you review this and see if it helps? > > > > The reason for the Internal Compiler Error is because we can't grow an interal dictionary anymore (ie we hit a 2**21 limit). > > The file contains many module procedures and interfaces that use the same helper module. As a result, we are importing the dictionary entries for that module repeatedly reaching > > the limit. > > > > Can you please give the following source code workaround a try? > > Since there is already "use petscvecdefdummy" at the module scope, one workaround might be to remove the unnecessary "use petscvecdefdummy" in vecnotequal and vecequals > > and all similar procedures. > > > > For example, the test case has: > > module petscvecdef > > use petscvecdefdummy > > ... > > function vecnotequal(A,B) > > use petscvecdefdummy > > logical vecnotequal > > type(tVec), intent(in) :: A,B > > vecnotequal = (A%v .ne. B%v) > > end function > > function vecequals(A,B) > > use petscvecdefdummy > > logical vecequals > > type(tVec), intent(in) :: A,B > > vecequals = (A%v .eq. B%v) > > end function > > ... > > end module > > Another workaround would be to put the procedure definitions from this large module into several submodules. Each submodule would be able to accommodate a dictionary with 2**21 entries. > > > > > > Please let us know if one of the above workarounds resolve the issue. > > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Roy Musselman > > IBM HPC Application Analyst at Lawrence Livermore National Lab > > email: roymuss(a)us.ibm.com <mailto:[email protected]> > > LLNL office: 925-422-6033 > > Cell: 507-358-8895, Home: 507-281-9565 > > > > > > Roy Musselman---02/21/2021 09:42:55 PM---Hi Jacob, After some more experimentation, I think I may have found what is triggering the ICE. It > > > > From: Roy Musselman/Rochester/Contr/IBM > > To: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> > > Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> > > Date: 02/21/2021 09:42 PM > > Subject: Re: [EXTERNAL] Re: xlf90_r Internal Compiler Error > > > > > > Hi Jacob, > > > > After some more experimentation, I think I may have found what is triggering the ICE. It doesn't appear to be related to the subroutine name length. I think the compiler may be hitting an internal limit of the number of subroutines within a module. There are 358 subroutines contained in the expanded petscmatmod.F90. Removing 4 subroutines will allow the compile to complete successfully, so the limit must be 354 subroutines. Is it possible for you to bust up petscmatmod into multiple modules? I'll package up the reproducer and pass it on to the compiler development team. > > > > I've asked for user feedback a couple years ago, when the IBM Power9 CORAL-1 Sierra systems were deployed, but received minimal responses. DOE is now working with Cray (aka HPE) developing the environment for the CORAL-2 system (El Capitan). I'll pass your request to the LLNL person I know that is dealing with math libraries for CORAL-2. > > > > We use the spack tool to download and build petsc and its specified dependencies. I switched between the PETSC versions by changing the PETSCDIR variable in the script I shared with you. I've attached a tar ball containing the scripts used to build PETSc via spack. > > > > [attachment "bld-petsc-spack.tgz" deleted by Roy Musselman/Rochester/Contr/IBM] > > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Roy Musselman > > IBM HPC Application Analyst at Lawrence Livermore National Lab > > email: roymuss(a)us.ibm.com <mailto:[email protected]> > > LLNL office: 925-422-6033 > > Cell: 507-358-8895, Home: 507-281-9565 > > > > > > Jacob Faibussowitsch ---02/21/2021 12:24:11 PM---Hi Roy, > I'm not sure which projects at LLNL are using PETSc or if they chose to build their own ve > > > > From: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> > > To: Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> > > Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> > > Date: 02/21/2021 12:24 PM > > Subject: [EXTERNAL] Re: xlf90_r Internal Compiler Error > > > > > > > > Hi Roy, I'm not sure which projects at LLNL are using PETSc or if they chose to build their own version. Entirely unrelated to our problem, but is it possible to find this out? It would be great if yes, but also completely fine if not. PETSc > > Hi Roy, > > I'm not sure which projects at LLNL are using PETSc or if they chose to build their own version. > > Entirely unrelated to our problem, but is it possible to find this out? It would be great if yes, but also completely fine if not. PETSc is potentially undergoing a rather transformative rewrite over the next few years and we’d like to gather current usage data to get a better idea of where PETSc fits into our users workflows. But we aren’t sure how to gather this data (we don’t particularly want to scrape and silently send it off without users consent/knowledge) absent user questionnaires and HPC usage statistics. > > If you are interested, I can share with you the spack recipes I use to build petsc with hdf5, hypre, and suplerlu-dist. > > Yes that would be quite useful. I can let it percolate through our dev channels for any other recommendations etc. > > 3.14.0 and 3.14.1 > > > > "../roymuss/spack-stage-petsc-3.14.0-on3lboy4slkz65tsjttgfmwghzky54jj/spack-src/src/vec/f90-mod/petscvecmod.F90", line 9.13: 1514-219 (S) Unable to access module symbol file for module petscisdefdummy. Check path and file permissions of file. Use association not done for this module. > > 1501-511 Compilation failed for file petscvecmod.F90. > > How exactly did you switch between versions? PETSc has 2 types of fortran bindings, “ftn-custom” and “ftn-auto” (technically 3 including the F90 files, but those simply call either of the two preceding ones), a copy of which you will find in every src directory. As the names imply ftn-auto is auto generated while ftn-custom is hand-written. > > > > This also means that the ftn-auto files are __not__ tracked by git, so a simple git checkout [new-tag] may not properly dispose of the old auto-generated files (very rare, but IIRC we made a major enough change to the fortran bindings within the last year to warrant having to "make deletefortranstubs" before rebuilding). > > Adding the option -qlanglvl=2003std or -qlanglvl=2008std produces a bunch of other warning messages, but it still encounters the ICE. So, I'm uncertain if the subroutine name length is the root of the problem. > > Our current compiler flag selection philosophy is to require a minimum but choose the maximum available reasonable flag for the compiler (I.e. we require C99, but very often you will find that your code is compiled with C11 or C17 if they are available). It is therefore odd that configure did not use the same methodology for fortran compilers. I will relay this on our side. > > Is it possible for you to use subroutines that are less than 32 characters and see if that works four you? Have you used other fortran 90 compilers and do any of them complain of this? > > Of all of the small quirks fortran has this is probably the most esoteric one I’ve come across… I’ve attached a list of all the F90 compilers, and their flags which we use in CI/CD (all of which is run multiple times daily and __must__ pass). I got them all via grep, so there may be some duplicates here or there. As for using shorter names, this is also something we can look at, but since none of the other compilers have had issues with this I’m not sure this is the change to make. > > Are there any unusual or questionable language constructs used in any of the functions mentioned above that may possibly challenge the compiler? > > Not that I am aware of, but again I will ask around our dev channels and see if anything comes to mind. > > > > > > Best regards, > > > > Jacob Faibussowitsch > > (Jacob Fai - booss - oh - vitch) > > Cell: (312) 694-3391[attachment "compilerList" deleted by Roy Musselman/Rochester/Contr/IBM] > > On Feb 20, 2021, at 22:05, Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> wrote: > > Hi Jacob, > > Thanks for letting me know that you are a PETSc developer and that you are testing it on the LLNL lassen system. I've used the spack build tool to build and deploy a few versions on the systems. I'm not sure which projects at LLNL are using PETSc or if they chose to build their own version. I did however provide a single precision version upon request that was integrated with MVAPICH2-MPI instead of the IBM-provided Spectrum-MPI. Here's what's available on the systems today. > > > > > ml avail petsc > > ----------------------------------------------------- /usr/tcetmp/modulefiles/Core ----------------------------------------------------- > > petsc/default petsc/3.10.2 petsc/3.11.3 petsc/3.13.0 (D) > > petsc/3.13.1-mvapich2-2020.01.09-xl-2020.03.18.single > > > > If you are interested, I can share with you the spack recipes I use to build petsc with hdf5, hypre, and suplerlu-dist. > > > > After several attempts I was able to reproduce the Internal Compiler Errro (ICE) that you are seeing using version 3.14.4. I've whittled it down to the petscmatmod.F90 file and it's specific dependencies. > > The following script is what I'm using. Note that in the 2nd set of compiles, the -E option is used to expand all included source files and headers and encapsulating it into a single large source file. This can be used to help isolate the source of the problem. > > > > #!/bin/bash > > > > PETSCDIR="../roymuss/spack-stage-petsc-3.14.4-eh5arny7l3cqjlltlfpjp6f4jofbnmz6/spack-src" > > OPTIONS=" -qmoddir=moddir -I$PETSCDIR/arch-linux-c-opt/include -I$PETSCDIR/include" > > mkdir -p moddir > > > > set -x > > > > # Compile original source files including dependencies > > if [ 0 = 1 ]; then > > mpif90 -c -g $OPTIONS $PETSCDIR/src/sys/f90-mod/petscsysmod.F90 -o petscsysmod.o > > mpif90 -c -g $OPTIONS $PETSCDIR/src/vec/f90-mod/petscvecmod.F90 -o petscvecmod.o > > mpif90 -c -g $OPTIONS $PETSCDIR/src/mat/f90-mod/petscmatmod.F90 -o petscmatmod.o > > fi > > > > # Use -E option to expand source into full source files > > if [ 0 = 1 ]; then > > mpif90 -c -g -E $OPTIONS $PETSCDIR/src/sys/f90-mod/petscsysmod.F90 -o full_petscsysmod.F90 > > mpif90 -c -g -E $OPTIONS $PETSCDIR/src/vec/f90-mod/petscvecmod.F90 -o full_petscvecmod.F90 > > mpif90 -c -g -E $OPTIONS $PETSCDIR/src/mat/f90-mod/petscmatmod.F90 -o full_petscmatmod.F90 > > fi > > > > # Compile from full source files > > if [ 1 = 1 ]; then > > mpif90 -c -g -Imoddir -qmoddir=moddir full_petscsysmod.F90 -o full_petscsysmod.o > > mpif90 -c -g -Imoddir -qmoddir=moddir full_petscvecmod.F90 -o full_petscvecmod.o > > mpif90 -V -c -g -Imoddir -qmoddir=moddir full_petscmatmod.F90 -o full_petscmatmod.o > > fi > > > > <eof> > > > > Petsc 3.13.6 it the most recent version that did not fail. I tried all subsequent versions and got the folowing results: > > > > 3.14.0 and 3.14.1 > > > > "../roymuss/spack-stage-petsc-3.14.0-on3lboy4slkz65tsjttgfmwghzky54jj/spack-src/src/vec/f90-mod/petscvecmod.F90", line 9.13: 1514-219 (S) Unable to access module symbol file for module petscisdefdummy. Check path and file permissions of file. Use association not done for this module. > > 1501-511 Compilation failed for file petscvecmod.F90. > > > > 3.14.2, 3.14.3, and 3.14.4 > > > > . . . > > ** matnullspaceequals === End of Compilation 8 === > > *** Error in `/usr/tce/packages/xl/xl-2020.11.12/xlf/16.1.1/exe/xlfentry': free(): invalid pointer: 0x0000200001740018 *** > > > > Examining the tail end of petscmatmod.F90 > > > > > > 80 function matnullspaceequals(A,B) > > 81 use petscmatdefdummy > > 82 logical matnullspaceequals > > 83 type(tMatNullSpace), intent(in) :: A,B > > 84 matnullspaceequals = (A%v .eq. B%v) > > 85 end function > > 86 > > 87 #if defined(_WIN32) && defined(PETSC_USE_SHARED_LIBRARIES) > > 88 !DEC$ ATTRIBUTES DLLEXPORT::matnotequal > > 89 !DEC$ ATTRIBUTES DLLEXPORT::matequals > > 90 !DEC$ ATTRIBUTES DLLEXPORT::matfdcoloringnotequal > > 91 !DEC$ ATTRIBUTES DLLEXPORT::matfdcoloringequals > > 92 !DEC$ ATTRIBUTES DLLEXPORT::matnullspacenotequal > > 93 !DEC$ ATTRIBUTES DLLEXPORT::matnullspaceequals > > 94 #endif > > 95 module petscmat > > 96 use petscmatdef > > 97 use petscvec > > 98 #include <../src/mat/f90-mod/petscmat.h90> > > 99 interface > > 100 #include <../src/mat/f90-mod/ftn-auto-interfaces/petscmat.h90> > > 101 end interface > > 102 end module > > 103 > > > > Compiling the matnullspaceequals function was successful just before hitting the error. The error goes away when removing either or both of the #include lines 98 and 100. Both #include statements are required to produce the error. The 3.13.6 and 3.14.4 version of the file identified in the first #include at line 98 are identical. The file identified in line 100 is different between 3.13.6 and 3.14.4. > > Just looking at the list of subroutines contained within each version, the following are the differences. > > > > Old subroutines available in 3.13.6 but removed from 4.14.4 > > subroutine MatFreeIntermediateDataStructures(a,z) > > > > New subroutines available in 4.14.4 but not contained in 3.13.6 > > subroutine MatDenseReplaceArray(a,b,z) > > subroutine MatIsShell(a,b,z) > > subroutine MatRARtMultEqual(a,b,c,d,e,z) > > subroutine MatScaLAPACKGetBlockSizes(a,b,c,z) > > subroutine MatScaLAPACKSetBlockSizes(a,b,c,z) > > subroutine MatSeqAIJCUSPARSESetGenerateTranspose(a,b,z) > > subroutine MatSeqAIJSetTotalPreallocation(a,b,z) > > subroutine MatSetLayouts(a,b,c,z) > > > > Methodically removing the new subroutines did not provide a consistent result. But I did notice the extra long subroutine name MatSeqAIJCUSPARSESetGenerateTranspose had 37 characters. > > A little research found: In Fortran 90/95 the maximum length was 31 characters, in Fortran 2003 it is now 63 characters. I found the following subroutines with greater than 31 characters > > > > subroutine MatCreateMPIMatConcatenateSeqMat > > subroutine MatFactorFactorizeSchurComplement > > subroutine MatMPIAdjCreateNonemptySubcommMat > > subroutine MatSeqAIJCUSPARSESetGenerateTranspose > > subroutine MatMPIAIJSetUseScalableIncreaseOverlap > > subroutine MatFactorSolveSchurComplementTranspose > > > > I individually ifdef'd them out of the source file and was able to compile the files successfully without encountering the ICE. > > > > I'm not exactly sure what the maximum subroutine name length that the XLF compiler allows, but if it is only 31, it would be useful if the compiler detected this and issue a message instead of the ICE. > > Adding the option -qlanglvl=2003std or -qlanglvl=2008std produces a bunch of other warning messages, but it still encounters the ICE. So, I'm uncertain if the subroutine name length is the root of the problem. > > > > Is it possible for you to use subroutines that are less than 32 characters and see if that works four you? Have you used other fortran 90 compilers and do any of them complain of this? > > Are there any unusual or questionable language constructs used in any of the functions mentioned above that may possibly challenge the compiler? > > > > I'll package this up and send it to the IBM XL compiler development team for their examination and comment. > > > > Best Regards, > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Roy Musselman > > IBM HPC Application Analyst at Lawrence Livermore National Lab > > email: roymuss(a)us.ibm.com <mailto:[email protected]> > > LLNL office: 925-422-6033 > > Cell: 507-358-8895, Home: 507-281-9565 > > > > <graycol.gif>Jacob Faibussowitsch ---02/18/2021 02:17:05 PM---> The most recently built version available on the CORAL systems is 3.13.0. (ml load petsc/3.13.0) W > > > > From: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> > > To: Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> > > Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> > > Date: 02/18/2021 02:17 PM > > Subject: [EXTERNAL] Re: xlf90_r Internal Compiler Error > > > > > > > > > > > > The most recently built version available on the CORAL systems... > > This Message Is From an External Sender > > This message came from outside your organization. > > The most recently built version available on the CORAL systems is 3.13.0. (ml load petsc/3.13.0) Will that work for you? > > I am building petsc from source as part of development work on petsc itself so modules are unfortunately not useful here. > > The files you sent me do not contain all the dependencies (other mod files) required to reproduce the error. > > I'll attempt to build version 3.14.4 from scratch and recreate the failing symptom you are observing. > > Yes, petsc uses an automated system to generate the fortran files from C which goes about 20 rabbit holes deeper than I was willing to dig. Let me know if you run into trouble configuring and building petsc, I can point you in the right direction. I’ve attached a “reconfigure” script with this email, it contains all of the arguments I used to configure petsc successfully on Lassen. If you place it into your $PETSC_DIR (i.e. the folder titled “petsc” and that contains a “configure” file) and run: > > > > $ python3 ./reconfigure-arch-linux-c-debug.py > > > > It should work. If not, you will have to > > > > $ ./configure —all-the-args —in-the-reconfigure —file > > > > Best regards, > > > > Jacob Faibussowitsch > > (Jacob Fai - booss - oh - vitch) > > Cell: (312) 694-3391[attachment "reconfigure-arch-linux-c-debug.py" deleted by Roy Musselman/Rochester/Contr/IBM] > > On Feb 18, 2021, at 15:07, Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> wrote: > > Hi Jacob, > > > > The source file appears to come from the PETSc 3.14.4 library. The most recently built version available on the CORAL systems is 3.13.0. (ml load petsc/3.13.0) Will that work for you? > > The files you sent me do not contain all the dependencies (other mod files) required to reproduce the error. > > I'll attempt to build version 3.14.4 from scratch and recreate the failing symptom you are observing. > > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Roy Musselman > > IBM HPC Application Analyst at Lawrence Livermore National Lab > > email: roymuss(a)us.ibm.com <mailto:[email protected]> > > LLNL office: 925-422-6033 > > Cell: 507-358-8895, Home: 507-281-9565 > > > > <graycol.gif>Roy Musselman---02/18/2021 11:18:20 AM---I'll take a look. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Roy Musselman > > > > From: Roy Musselman/Rochester/Contr/IBM > > To: LC Hotline <lc-hotline(a)llnl.gov <mailto:[email protected]>> > > Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> > > Date: 02/18/2021 11:18 AM > > Subject: Re: [EXTERNAL] FW: xlf90_r Internal Compiler Error > > > > > > > > > > > > I'll take a look. > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Roy Musselman > > IBM HPC Application Analyst at Lawrence Livermore National Lab > > email: roymuss(a)us.ibm.com <mailto:[email protected]> > > LLNL office: 925-422-6033 > > Cell: 507-358-8895, Home: 507-281-9565 > > > > > > <graycol.gif>LC Hotline ---02/18/2021 11:03:55 AM---Hi John, Roy, Can you help this user with the problem that he is seeing when he tries to build with > > > > From: LC Hotline <lc-hotline(a)llnl.gov <mailto:[email protected]>> > > To: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>>, Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> > > Date: 02/18/2021 11:03 AM > > Subject: [EXTERNAL] FW: xlf90_r Internal Compiler Error > > > > > > > > Hi John, Roy, Can you help this user with the problem that he is... > > This Message Is From an External Sender > > This message came from outside your organization. > > Hi John, Roy, > > > > Can you help this user with the problem that he is seeing when he tries to build with xlf90 on Lassen? > > > > Thanks, > > Ryan > > -- > > LC Hotline > > > > From: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> > > Date: Wednesday, February 17, 2021 at 5:27 PM > > To: LC Hotline <lc-hotline(a)llnl.gov <mailto:[email protected]>> > > Subject: xlf90_r Internal Compiler Error > > > > Hello LC Support, > > > > While compiling my application on Lassen I seem have run afoul of the xlf90 mpi compiler wrapper with the following error: > > > > *** Error in `/usr/tce/packages/xl/xl-2020.11.12/xlf/16.1.1/exe/xlfentry': free(): invalid pointer: 0x0000200001740018 *** > > > > I’m fairly certain this isn’t my fault as this is code that compiles regularly on extensive CI/CD under various other compilers and machines, but you can never rule it out. I have included a verbose full log of my make run (which includes a comprehensive rundown of the environment) as well as a separate file containing the error message and stack trace from the compiler. Additionally I have also included the file which I believe is causing the error. Let me know if there is anything else I should send. > > > > P.S. My list of loaded modules: > > > > Currently Loaded Modules: > > 1) StdEnv (S) 4) cuda/11.1.1 7) valgrind/3.16.1 > > 2) clang/ibm-11.0.0 5) python/3.8.2 8) lapack/3.9.0-xl-2020.11.12 > > 3) spectrum-mpi/rolling-release 6) cmake/3.18.0 9) hip/3.0.0 > > > > Best regards, > > > > Jacob Faibussowitsch > > (Jacob Fai - booss - oh - vitch) > > Cell: (312) 694-3391[attachment "errorReport.zip" deleted by Roy Musselman/Rochester/Contr/IBM] > >

1 0

Re: [petsc-dev] Fwd: Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines
by Satish Balay 03 Mar '21

03 Mar '21

On Wed, 3 Mar 2021, Jacob Faibussowitsch wrote: > Hello All, > > I discovered a compiler bug in the IBM xl fortran compiler a few weeks ago that would crash the compiler when compiling petsc fortran interfaces. The TL;DR of it is that the xl compiler creates a function dictionary for every function imported in fortran modules, and since petsc fortran interfaces seem to import entire packages writ-large this exceeds the number of dictionary entries (2**21): > > > The reason for the Internal Compiler Error is because we can't grow an interal dictionary anymore (ie we hit a 2**21 limit). > > The file contains many module procedures and interfaces that use the same helper module. As a result, we are importing the dictionary entries for that module repeatedly reaching > > the limit. > > > > Can you please give the following source code workaround a try? > > Since there is already "use petscvecdefdummy" at the module scope, one workaround might be to remove the unnecessary "use petscvecdefdummy" in vecnotequal and vecequals > > and all similar procedures. This sounds reasonable - but the change might be tedious [to make without breaking some required dependency]. Perhaps it will also help gfortran RAM requirements.. Satish > > > > For example, the test case has: > > module petscvecdef > > use petscvecdefdummy > > ... > > function vecnotequal(A,B) > > use petscvecdefdummy > > logical vecnotequal > > type(tVec), intent(in) :: A,B > > vecnotequal = (A%v .ne. B%v) > > end function > > function vecequals(A,B) > > use petscvecdefdummy > > logical vecequals > > type(tVec), intent(in) :: A,B > > vecequals = (A%v .eq. B%v) > > end function > > ... > > end module > > Another workaround would be to put the procedure definitions from this large module into several submodules. Each submodule would be able to accommodate a dictionary with 2**21 entries. > > > > > > Please let us know if one of the above workarounds resolve the issue. > > > The proposed fix from IBM would be to pull “use moduleXXX” out of subroutines or to have our auto-fortran interfaces detect which symbols to include from the respective modules and only include those in the subroutines. I’m not familiar at all with how the interfaces are generated so I don’t even know if this is possible. > > IBM provided the following additional explanation and example. Can the process used to generate these routines and functions determine the specific symbols required and then use the only keyword or import statement to include them? > > > > When factoring out use statements out of module procedures, you can just delete them. But you can't completely remove them from interface blocks. Instead, you can limit them either by using use <module>, only: <symbol> or import <symbol> . if the hundreds of use statements in the program are factored out / limited in this way, that should reduce the dictionary size sufficiently for the program to compile. > > > > For example > > Interface > > Subroutine VecRestoreArrayReadF90(v,array,ierr) > > use petscvecdef > > real(kind=selected_real_kind(10)), pointer :: array(:) > > integer(kind=selected_int_kind(5)) ierr > > type(tVec) v > > End Subroutine > > End Interface > > > > imports all symbols from petscvecdef into the dictionary even though we only need tVec . So we can either: > > > > Interface > > Subroutine VecRestoreArrayReadF90(v,array,ierr) > > use petscvecdef, only: tVec > > implicit none > > real(kind=selected_real_kind(10)), pointer :: array(:) > > integer(kind=selected_int_kind(5)) ierr > > type(tVec) v > > End Subroutine > > End Interface > > > > or if use petscvecdef is used in the outer scope, we can: > > Interface > > Subroutine VecRestoreArrayReadF90(v,array,ierr) > > import tVec > > implicit none > > real(kind=selected_real_kind(10)), pointer :: array(:) > > integer(kind=selected_int_kind(5)) ierr > > type(tVec) v > > End Subroutine > > End Interface > > (The two methods (use, only vs import) are equivalent in terms of impact to the dictionary.) > > > > Is this compiler ~feature~ something that we intend to work around? Thoughts? > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > Cell: (312) 694-3391 > > > Begin forwarded message: > > > > From: "Roy Musselman" <roymuss(a)us.ibm.com> > > Subject: Re: Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines > > Date: March 3, 2021 at 08:23:17 CST > > To: Jacob Faibussowitsch <faibuss2(a)illinois.edu> > > Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov> > > > > Hi Jacob, > > I tried the first suggestion and commented out the use statements called within the functions. However, I hit the following error complaining about specific symbol dependencies provided by the library. > > > > .../src/vec/f90-mod/petscvecmod.F90", line 107.37: 1514-084 (S) Identifier a is being declared with type name tvec which has not been defined in a derived type definition. > > > > IBM provided the following additional explanation and example. Can the process used to generate these routines and functions determine the specific symbols required and then use the only keyword or import statement to include them? > > > > When factoring out use statements out of module procedures, you can just delete them. But you can't completely remove them from interface blocks. Instead, you can limit them either by using use <module>, only: <symbol> or import <symbol> . if the hundreds of use statements in the program are factored out / limited in this way, that should reduce the dictionary size sufficiently for the program to compile. > > > > For example > > Interface > > Subroutine VecRestoreArrayReadF90(v,array,ierr) > > use petscvecdef > > real(kind=selected_real_kind(10)), pointer :: array(:) > > integer(kind=selected_int_kind(5)) ierr > > type(tVec) v > > End Subroutine > > End Interface > > > > imports all symbols from petscvecdef into the dictionary even though we only need tVec . So we can either: > > > > Interface > > Subroutine VecRestoreArrayReadF90(v,array,ierr) > > use petscvecdef, only: tVec > > implicit none > > real(kind=selected_real_kind(10)), pointer :: array(:) > > integer(kind=selected_int_kind(5)) ierr > > type(tVec) v > > End Subroutine > > End Interface > > > > or if use petscvecdef is used in the outer scope, we can: > > Interface > > Subroutine VecRestoreArrayReadF90(v,array,ierr) > > import tVec > > implicit none > > real(kind=selected_real_kind(10)), pointer :: array(:) > > integer(kind=selected_int_kind(5)) ierr > > type(tVec) v > > End Subroutine > > End Interface > > (The two methods (use, only vs import) are equivalent in terms of impact to the dictionary.) > > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Roy Musselman > > IBM HPC Application Analyst at Lawrence Livermore National Lab > > email: roymuss(a)us.ibm.com <mailto:[email protected]> > > LLNL office: 925-422-6033 > > Cell: 507-358-8895, Home: 507-281-9565 > > > > Roy Musselman---02/24/2021 07:08:45 PM---Hi Jacob, I opened the ticket with IBM: case TS005062693 and and the local LLNL Sierra Jira Ticket > > > > From: Roy Musselman/Rochester/Contr/IBM > > To: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> > > Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> > > Date: 02/24/2021 07:08 PM > > Subject: Re: [EXTERNAL] Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines > > > > > > > > Hi Jacob, > > I opened the ticket with IBM: case TS005062693 and and the local LLNL Sierra Jira Ticket at > > https://lc.llnl.gov/jira/projects/SIERRA/issues/SIERRA-111?filter=allissues <https://urldefense.com/v3/__https://lc.llnl.gov/jira/projects/SIERRA/issues…> > > > > Today IBM provided the response below. I don't know when I'll have time to try it on the reproducer I gave IBM. Perhaps early next week. Can you review this and see if it helps? > > > > The reason for the Internal Compiler Error is because we can't grow an interal dictionary anymore (ie we hit a 2**21 limit). > > The file contains many module procedures and interfaces that use the same helper module. As a result, we are importing the dictionary entries for that module repeatedly reaching > > the limit. > > > > Can you please give the following source code workaround a try? > > Since there is already "use petscvecdefdummy" at the module scope, one workaround might be to remove the unnecessary "use petscvecdefdummy" in vecnotequal and vecequals > > and all similar procedures. > > > > For example, the test case has: > > module petscvecdef > > use petscvecdefdummy > > ... > > function vecnotequal(A,B) > > use petscvecdefdummy > > logical vecnotequal > > type(tVec), intent(in) :: A,B > > vecnotequal = (A%v .ne. B%v) > > end function > > function vecequals(A,B) > > use petscvecdefdummy > > logical vecequals > > type(tVec), intent(in) :: A,B > > vecequals = (A%v .eq. B%v) > > end function > > ... > > end module > > Another workaround would be to put the procedure definitions from this large module into several submodules. Each submodule would be able to accommodate a dictionary with 2**21 entries. > > > > > > Please let us know if one of the above workarounds resolve the issue. > > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Roy Musselman > > IBM HPC Application Analyst at Lawrence Livermore National Lab > > email: roymuss(a)us.ibm.com <mailto:[email protected]> > > LLNL office: 925-422-6033 > > Cell: 507-358-8895, Home: 507-281-9565 > > > > > > Roy Musselman---02/21/2021 09:42:55 PM---Hi Jacob, After some more experimentation, I think I may have found what is triggering the ICE. It > > > > From: Roy Musselman/Rochester/Contr/IBM > > To: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> > > Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> > > Date: 02/21/2021 09:42 PM > > Subject: Re: [EXTERNAL] Re: xlf90_r Internal Compiler Error > > > > > > Hi Jacob, > > > > After some more experimentation, I think I may have found what is triggering the ICE. It doesn't appear to be related to the subroutine name length. I think the compiler may be hitting an internal limit of the number of subroutines within a module. There are 358 subroutines contained in the expanded petscmatmod.F90. Removing 4 subroutines will allow the compile to complete successfully, so the limit must be 354 subroutines. Is it possible for you to bust up petscmatmod into multiple modules? I'll package up the reproducer and pass it on to the compiler development team. > > > > I've asked for user feedback a couple years ago, when the IBM Power9 CORAL-1 Sierra systems were deployed, but received minimal responses. DOE is now working with Cray (aka HPE) developing the environment for the CORAL-2 system (El Capitan). I'll pass your request to the LLNL person I know that is dealing with math libraries for CORAL-2. > > > > We use the spack tool to download and build petsc and its specified dependencies. I switched between the PETSC versions by changing the PETSCDIR variable in the script I shared with you. I've attached a tar ball containing the scripts used to build PETSc via spack. > > > > [attachment "bld-petsc-spack.tgz" deleted by Roy Musselman/Rochester/Contr/IBM] > > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Roy Musselman > > IBM HPC Application Analyst at Lawrence Livermore National Lab > > email: roymuss(a)us.ibm.com <mailto:[email protected]> > > LLNL office: 925-422-6033 > > Cell: 507-358-8895, Home: 507-281-9565 > > > > > > Jacob Faibussowitsch ---02/21/2021 12:24:11 PM---Hi Roy, > I'm not sure which projects at LLNL are using PETSc or if they chose to build their own ve > > > > From: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> > > To: Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> > > Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> > > Date: 02/21/2021 12:24 PM > > Subject: [EXTERNAL] Re: xlf90_r Internal Compiler Error > > > > > > > > Hi Roy, I'm not sure which projects at LLNL are using PETSc or if they chose to build their own version. Entirely unrelated to our problem, but is it possible to find this out? It would be great if yes, but also completely fine if not. PETSc > > Hi Roy, > > I'm not sure which projects at LLNL are using PETSc or if they chose to build their own version. > > Entirely unrelated to our problem, but is it possible to find this out? It would be great if yes, but also completely fine if not. PETSc is potentially undergoing a rather transformative rewrite over the next few years and we’d like to gather current usage data to get a better idea of where PETSc fits into our users workflows. But we aren’t sure how to gather this data (we don’t particularly want to scrape and silently send it off without users consent/knowledge) absent user questionnaires and HPC usage statistics. > > If you are interested, I can share with you the spack recipes I use to build petsc with hdf5, hypre, and suplerlu-dist. > > Yes that would be quite useful. I can let it percolate through our dev channels for any other recommendations etc. > > 3.14.0 and 3.14.1 > > > > "../roymuss/spack-stage-petsc-3.14.0-on3lboy4slkz65tsjttgfmwghzky54jj/spack-src/src/vec/f90-mod/petscvecmod.F90", line 9.13: 1514-219 (S) Unable to access module symbol file for module petscisdefdummy. Check path and file permissions of file. Use association not done for this module. > > 1501-511 Compilation failed for file petscvecmod.F90. > > How exactly did you switch between versions? PETSc has 2 types of fortran bindings, “ftn-custom” and “ftn-auto” (technically 3 including the F90 files, but those simply call either of the two preceding ones), a copy of which you will find in every src directory. As the names imply ftn-auto is auto generated while ftn-custom is hand-written. > > > > This also means that the ftn-auto files are __not__ tracked by git, so a simple git checkout [new-tag] may not properly dispose of the old auto-generated files (very rare, but IIRC we made a major enough change to the fortran bindings within the last year to warrant having to "make deletefortranstubs" before rebuilding). > > Adding the option -qlanglvl=2003std or -qlanglvl=2008std produces a bunch of other warning messages, but it still encounters the ICE. So, I'm uncertain if the subroutine name length is the root of the problem. > > Our current compiler flag selection philosophy is to require a minimum but choose the maximum available reasonable flag for the compiler (I.e. we require C99, but very often you will find that your code is compiled with C11 or C17 if they are available). It is therefore odd that configure did not use the same methodology for fortran compilers. I will relay this on our side. > > Is it possible for you to use subroutines that are less than 32 characters and see if that works four you? Have you used other fortran 90 compilers and do any of them complain of this? > > Of all of the small quirks fortran has this is probably the most esoteric one I’ve come across… I’ve attached a list of all the F90 compilers, and their flags which we use in CI/CD (all of which is run multiple times daily and __must__ pass). I got them all via grep, so there may be some duplicates here or there. As for using shorter names, this is also something we can look at, but since none of the other compilers have had issues with this I’m not sure this is the change to make. > > Are there any unusual or questionable language constructs used in any of the functions mentioned above that may possibly challenge the compiler? > > Not that I am aware of, but again I will ask around our dev channels and see if anything comes to mind. > > > > > > Best regards, > > > > Jacob Faibussowitsch > > (Jacob Fai - booss - oh - vitch) > > Cell: (312) 694-3391[attachment "compilerList" deleted by Roy Musselman/Rochester/Contr/IBM] > > On Feb 20, 2021, at 22:05, Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> wrote: > > Hi Jacob, > > Thanks for letting me know that you are a PETSc developer and that you are testing it on the LLNL lassen system. I've used the spack build tool to build and deploy a few versions on the systems. I'm not sure which projects at LLNL are using PETSc or if they chose to build their own version. I did however provide a single precision version upon request that was integrated with MVAPICH2-MPI instead of the IBM-provided Spectrum-MPI. Here's what's available on the systems today. > > > > > ml avail petsc > > ----------------------------------------------------- /usr/tcetmp/modulefiles/Core ----------------------------------------------------- > > petsc/default petsc/3.10.2 petsc/3.11.3 petsc/3.13.0 (D) > > petsc/3.13.1-mvapich2-2020.01.09-xl-2020.03.18.single > > > > If you are interested, I can share with you the spack recipes I use to build petsc with hdf5, hypre, and suplerlu-dist. > > > > After several attempts I was able to reproduce the Internal Compiler Errro (ICE) that you are seeing using version 3.14.4. I've whittled it down to the petscmatmod.F90 file and it's specific dependencies. > > The following script is what I'm using. Note that in the 2nd set of compiles, the -E option is used to expand all included source files and headers and encapsulating it into a single large source file. This can be used to help isolate the source of the problem. > > > > #!/bin/bash > > > > PETSCDIR="../roymuss/spack-stage-petsc-3.14.4-eh5arny7l3cqjlltlfpjp6f4jofbnmz6/spack-src" > > OPTIONS=" -qmoddir=moddir -I$PETSCDIR/arch-linux-c-opt/include -I$PETSCDIR/include" > > mkdir -p moddir > > > > set -x > > > > # Compile original source files including dependencies > > if [ 0 = 1 ]; then > > mpif90 -c -g $OPTIONS $PETSCDIR/src/sys/f90-mod/petscsysmod.F90 -o petscsysmod.o > > mpif90 -c -g $OPTIONS $PETSCDIR/src/vec/f90-mod/petscvecmod.F90 -o petscvecmod.o > > mpif90 -c -g $OPTIONS $PETSCDIR/src/mat/f90-mod/petscmatmod.F90 -o petscmatmod.o > > fi > > > > # Use -E option to expand source into full source files > > if [ 0 = 1 ]; then > > mpif90 -c -g -E $OPTIONS $PETSCDIR/src/sys/f90-mod/petscsysmod.F90 -o full_petscsysmod.F90 > > mpif90 -c -g -E $OPTIONS $PETSCDIR/src/vec/f90-mod/petscvecmod.F90 -o full_petscvecmod.F90 > > mpif90 -c -g -E $OPTIONS $PETSCDIR/src/mat/f90-mod/petscmatmod.F90 -o full_petscmatmod.F90 > > fi > > > > # Compile from full source files > > if [ 1 = 1 ]; then > > mpif90 -c -g -Imoddir -qmoddir=moddir full_petscsysmod.F90 -o full_petscsysmod.o > > mpif90 -c -g -Imoddir -qmoddir=moddir full_petscvecmod.F90 -o full_petscvecmod.o > > mpif90 -V -c -g -Imoddir -qmoddir=moddir full_petscmatmod.F90 -o full_petscmatmod.o > > fi > > > > <eof> > > > > Petsc 3.13.6 it the most recent version that did not fail. I tried all subsequent versions and got the folowing results: > > > > 3.14.0 and 3.14.1 > > > > "../roymuss/spack-stage-petsc-3.14.0-on3lboy4slkz65tsjttgfmwghzky54jj/spack-src/src/vec/f90-mod/petscvecmod.F90", line 9.13: 1514-219 (S) Unable to access module symbol file for module petscisdefdummy. Check path and file permissions of file. Use association not done for this module. > > 1501-511 Compilation failed for file petscvecmod.F90. > > > > 3.14.2, 3.14.3, and 3.14.4 > > > > . . . > > ** matnullspaceequals === End of Compilation 8 === > > *** Error in `/usr/tce/packages/xl/xl-2020.11.12/xlf/16.1.1/exe/xlfentry': free(): invalid pointer: 0x0000200001740018 *** > > > > Examining the tail end of petscmatmod.F90 > > > > > > 80 function matnullspaceequals(A,B) > > 81 use petscmatdefdummy > > 82 logical matnullspaceequals > > 83 type(tMatNullSpace), intent(in) :: A,B > > 84 matnullspaceequals = (A%v .eq. B%v) > > 85 end function > > 86 > > 87 #if defined(_WIN32) && defined(PETSC_USE_SHARED_LIBRARIES) > > 88 !DEC$ ATTRIBUTES DLLEXPORT::matnotequal > > 89 !DEC$ ATTRIBUTES DLLEXPORT::matequals > > 90 !DEC$ ATTRIBUTES DLLEXPORT::matfdcoloringnotequal > > 91 !DEC$ ATTRIBUTES DLLEXPORT::matfdcoloringequals > > 92 !DEC$ ATTRIBUTES DLLEXPORT::matnullspacenotequal > > 93 !DEC$ ATTRIBUTES DLLEXPORT::matnullspaceequals > > 94 #endif > > 95 module petscmat > > 96 use petscmatdef > > 97 use petscvec > > 98 #include <../src/mat/f90-mod/petscmat.h90> > > 99 interface > > 100 #include <../src/mat/f90-mod/ftn-auto-interfaces/petscmat.h90> > > 101 end interface > > 102 end module > > 103 > > > > Compiling the matnullspaceequals function was successful just before hitting the error. The error goes away when removing either or both of the #include lines 98 and 100. Both #include statements are required to produce the error. The 3.13.6 and 3.14.4 version of the file identified in the first #include at line 98 are identical. The file identified in line 100 is different between 3.13.6 and 3.14.4. > > Just looking at the list of subroutines contained within each version, the following are the differences. > > > > Old subroutines available in 3.13.6 but removed from 4.14.4 > > subroutine MatFreeIntermediateDataStructures(a,z) > > > > New subroutines available in 4.14.4 but not contained in 3.13.6 > > subroutine MatDenseReplaceArray(a,b,z) > > subroutine MatIsShell(a,b,z) > > subroutine MatRARtMultEqual(a,b,c,d,e,z) > > subroutine MatScaLAPACKGetBlockSizes(a,b,c,z) > > subroutine MatScaLAPACKSetBlockSizes(a,b,c,z) > > subroutine MatSeqAIJCUSPARSESetGenerateTranspose(a,b,z) > > subroutine MatSeqAIJSetTotalPreallocation(a,b,z) > > subroutine MatSetLayouts(a,b,c,z) > > > > Methodically removing the new subroutines did not provide a consistent result. But I did notice the extra long subroutine name MatSeqAIJCUSPARSESetGenerateTranspose had 37 characters. > > A little research found: In Fortran 90/95 the maximum length was 31 characters, in Fortran 2003 it is now 63 characters. I found the following subroutines with greater than 31 characters > > > > subroutine MatCreateMPIMatConcatenateSeqMat > > subroutine MatFactorFactorizeSchurComplement > > subroutine MatMPIAdjCreateNonemptySubcommMat > > subroutine MatSeqAIJCUSPARSESetGenerateTranspose > > subroutine MatMPIAIJSetUseScalableIncreaseOverlap > > subroutine MatFactorSolveSchurComplementTranspose > > > > I individually ifdef'd them out of the source file and was able to compile the files successfully without encountering the ICE. > > > > I'm not exactly sure what the maximum subroutine name length that the XLF compiler allows, but if it is only 31, it would be useful if the compiler detected this and issue a message instead of the ICE. > > Adding the option -qlanglvl=2003std or -qlanglvl=2008std produces a bunch of other warning messages, but it still encounters the ICE. So, I'm uncertain if the subroutine name length is the root of the problem. > > > > Is it possible for you to use subroutines that are less than 32 characters and see if that works four you? Have you used other fortran 90 compilers and do any of them complain of this? > > Are there any unusual or questionable language constructs used in any of the functions mentioned above that may possibly challenge the compiler? > > > > I'll package this up and send it to the IBM XL compiler development team for their examination and comment. > > > > Best Regards, > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Roy Musselman > > IBM HPC Application Analyst at Lawrence Livermore National Lab > > email: roymuss(a)us.ibm.com <mailto:[email protected]> > > LLNL office: 925-422-6033 > > Cell: 507-358-8895, Home: 507-281-9565 > > > > <graycol.gif>Jacob Faibussowitsch ---02/18/2021 02:17:05 PM---> The most recently built version available on the CORAL systems is 3.13.0. (ml load petsc/3.13.0) W > > > > From: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> > > To: Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> > > Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> > > Date: 02/18/2021 02:17 PM > > Subject: [EXTERNAL] Re: xlf90_r Internal Compiler Error > > > > > > > > > > > > The most recently built version available on the CORAL systems... > > This Message Is From an External Sender > > This message came from outside your organization. > > The most recently built version available on the CORAL systems is 3.13.0. (ml load petsc/3.13.0) Will that work for you? > > I am building petsc from source as part of development work on petsc itself so modules are unfortunately not useful here. > > The files you sent me do not contain all the dependencies (other mod files) required to reproduce the error. > > I'll attempt to build version 3.14.4 from scratch and recreate the failing symptom you are observing. > > Yes, petsc uses an automated system to generate the fortran files from C which goes about 20 rabbit holes deeper than I was willing to dig. Let me know if you run into trouble configuring and building petsc, I can point you in the right direction. I’ve attached a “reconfigure” script with this email, it contains all of the arguments I used to configure petsc successfully on Lassen. If you place it into your $PETSC_DIR (i.e. the folder titled “petsc” and that contains a “configure” file) and run: > > > > $ python3 ./reconfigure-arch-linux-c-debug.py > > > > It should work. If not, you will have to > > > > $ ./configure —all-the-args —in-the-reconfigure —file > > > > Best regards, > > > > Jacob Faibussowitsch > > (Jacob Fai - booss - oh - vitch) > > Cell: (312) 694-3391[attachment "reconfigure-arch-linux-c-debug.py" deleted by Roy Musselman/Rochester/Contr/IBM] > > On Feb 18, 2021, at 15:07, Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> wrote: > > Hi Jacob, > > > > The source file appears to come from the PETSc 3.14.4 library. The most recently built version available on the CORAL systems is 3.13.0. (ml load petsc/3.13.0) Will that work for you? > > The files you sent me do not contain all the dependencies (other mod files) required to reproduce the error. > > I'll attempt to build version 3.14.4 from scratch and recreate the failing symptom you are observing. > > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Roy Musselman > > IBM HPC Application Analyst at Lawrence Livermore National Lab > > email: roymuss(a)us.ibm.com <mailto:[email protected]> > > LLNL office: 925-422-6033 > > Cell: 507-358-8895, Home: 507-281-9565 > > > > <graycol.gif>Roy Musselman---02/18/2021 11:18:20 AM---I'll take a look. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Roy Musselman > > > > From: Roy Musselman/Rochester/Contr/IBM > > To: LC Hotline <lc-hotline(a)llnl.gov <mailto:[email protected]>> > > Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> > > Date: 02/18/2021 11:18 AM > > Subject: Re: [EXTERNAL] FW: xlf90_r Internal Compiler Error > > > > > > > > > > > > I'll take a look. > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Roy Musselman > > IBM HPC Application Analyst at Lawrence Livermore National Lab > > email: roymuss(a)us.ibm.com <mailto:[email protected]> > > LLNL office: 925-422-6033 > > Cell: 507-358-8895, Home: 507-281-9565 > > > > > > <graycol.gif>LC Hotline ---02/18/2021 11:03:55 AM---Hi John, Roy, Can you help this user with the problem that he is seeing when he tries to build with > > > > From: LC Hotline <lc-hotline(a)llnl.gov <mailto:[email protected]>> > > To: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>>, Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> > > Date: 02/18/2021 11:03 AM > > Subject: [EXTERNAL] FW: xlf90_r Internal Compiler Error > > > > > > > > Hi John, Roy, Can you help this user with the problem that he is... > > This Message Is From an External Sender > > This message came from outside your organization. > > Hi John, Roy, > > > > Can you help this user with the problem that he is seeing when he tries to build with xlf90 on Lassen? > > > > Thanks, > > Ryan > > -- > > LC Hotline > > > > From: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> > > Date: Wednesday, February 17, 2021 at 5:27 PM > > To: LC Hotline <lc-hotline(a)llnl.gov <mailto:[email protected]>> > > Subject: xlf90_r Internal Compiler Error > > > > Hello LC Support, > > > > While compiling my application on Lassen I seem have run afoul of the xlf90 mpi compiler wrapper with the following error: > > > > *** Error in `/usr/tce/packages/xl/xl-2020.11.12/xlf/16.1.1/exe/xlfentry': free(): invalid pointer: 0x0000200001740018 *** > > > > I’m fairly certain this isn’t my fault as this is code that compiles regularly on extensive CI/CD under various other compilers and machines, but you can never rule it out. I have included a verbose full log of my make run (which includes a comprehensive rundown of the environment) as well as a separate file containing the error message and stack trace from the compiler. Additionally I have also included the file which I believe is causing the error. Let me know if there is anything else I should send. > > > > P.S. My list of loaded modules: > > > > Currently Loaded Modules: > > 1) StdEnv (S) 4) cuda/11.1.1 7) valgrind/3.16.1 > > 2) clang/ibm-11.0.0 5) python/3.8.2 8) lapack/3.9.0-xl-2020.11.12 > > 3) spectrum-mpi/rolling-release 6) cmake/3.18.0 9) hip/3.0.0 > > > > Best regards, > > > > Jacob Faibussowitsch > > (Jacob Fai - booss - oh - vitch) > > Cell: (312) 694-3391[attachment "errorReport.zip" deleted by Roy Musselman/Rochester/Contr/IBM] > >

1 0

Re: [petsc-dev] Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines
by Jed Brown 03 Mar '21

03 Mar '21

I wonder if gfortran has a similar "bug" but larger capacity that would explain why it's so much more expensive to compile the Fortran interface files than to compile all of PETSc. Barry Smith <bsmith(a)petsc.dev> writes: > PETSc stacks the Fortran modules in the same way it stacks the C include files. So the TAO module includes all the Fortran modules below it etc. It would be nearly impossible to disentangle the bits and pieces without introducing a more painful user experience. For example use PCTypes, use PCFunctions, use KSPTypes, .... impossible to use and impossible to maintain. > > This is a completely artificial bug of IBM's own making in their compiler that we should not have to work around. > > Barry > > >> On Mar 3, 2021, at 12:10 PM, Jacob Faibussowitsch <jacob.fai(a)gmail.com> wrote: >> >> Hello All, >> >> I discovered a compiler bug in the IBM xl fortran compiler a few weeks ago that would crash the compiler when compiling petsc fortran interfaces. The TL;DR of it is that the xl compiler creates a function dictionary for every function imported in fortran modules, and since petsc fortran interfaces seem to import entire packages writ-large this exceeds the number of dictionary entries (2**21): >> >>> The reason for the Internal Compiler Error is because we can't grow an interal dictionary anymore (ie we hit a 2**21 limit). >>> The file contains many module procedures and interfaces that use the same helper module. As a result, we are importing the dictionary entries for that module repeatedly reaching >>> the limit. >>> >>> Can you please give the following source code workaround a try? >>> Since there is already "use petscvecdefdummy" at the module scope, one workaround might be to remove the unnecessary "use petscvecdefdummy" in vecnotequal and vecequals >>> and all similar procedures. >>> >>> For example, the test case has: >>> module petscvecdef >>> use petscvecdefdummy >>> ... >>> function vecnotequal(A,B) >>> use petscvecdefdummy >>> logical vecnotequal >>> type(tVec), intent(in) :: A,B >>> vecnotequal = (A%v .ne. B%v) >>> end function >>> function vecequals(A,B) >>> use petscvecdefdummy >>> logical vecequals >>> type(tVec), intent(in) :: A,B >>> vecequals = (A%v .eq. B%v) >>> end function >>> ... >>> end module >>> Another workaround would be to put the procedure definitions from this large module into several submodules. Each submodule would be able to accommodate a dictionary with 2**21 entries. >>> >>> >>> Please let us know if one of the above workarounds resolve the issue. >> >> >> The proposed fix from IBM would be to pull “use moduleXXX” out of subroutines or to have our auto-fortran interfaces detect which symbols to include from the respective modules and only include those in the subroutines. I’m not familiar at all with how the interfaces are generated so I don’t even know if this is possible. >>> IBM provided the following additional explanation and example. Can the process used to generate these routines and functions determine the specific symbols required and then use the only keyword or import statement to include them? >>> >>> When factoring out use statements out of module procedures, you can just delete them. But you can't completely remove them from interface blocks. Instead, you can limit them either by using use <module>, only: <symbol> or import <symbol> . if the hundreds of use statements in the program are factored out / limited in this way, that should reduce the dictionary size sufficiently for the program to compile. >>> >>> For example >>> Interface >>> Subroutine VecRestoreArrayReadF90(v,array,ierr) >>> use petscvecdef >>> real(kind=selected_real_kind(10)), pointer :: array(:) >>> integer(kind=selected_int_kind(5)) ierr >>> type(tVec) v >>> End Subroutine >>> End Interface >>> >>> imports all symbols from petscvecdef into the dictionary even though we only need tVec . So we can either: >>> >>> Interface >>> Subroutine VecRestoreArrayReadF90(v,array,ierr) >>> use petscvecdef, only: tVec >>> implicit none >>> real(kind=selected_real_kind(10)), pointer :: array(:) >>> integer(kind=selected_int_kind(5)) ierr >>> type(tVec) v >>> End Subroutine >>> End Interface >>> >>> or if use petscvecdef is used in the outer scope, we can: >>> Interface >>> Subroutine VecRestoreArrayReadF90(v,array,ierr) >>> import tVec >>> implicit none >>> real(kind=selected_real_kind(10)), pointer :: array(:) >>> integer(kind=selected_int_kind(5)) ierr >>> type(tVec) v >>> End Subroutine >>> End Interface >>> (The two methods (use, only vs import) are equivalent in terms of impact to the dictionary.) >>> >> >> Is this compiler ~feature~ something that we intend to work around? Thoughts? >> >> Best regards, >> >> Jacob Faibussowitsch >> (Jacob Fai - booss - oh - vitch) >> Cell: (312) 694-3391 >> >>> Begin forwarded message: >>> >>> From: "Roy Musselman" <roymuss(a)us.ibm.com <mailto:[email protected]>> >>> Subject: Re: Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines >>> Date: March 3, 2021 at 08:23:17 CST >>> To: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> >>> Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> >>> >>> Hi Jacob, >>> I tried the first suggestion and commented out the use statements called within the functions. However, I hit the following error complaining about specific symbol dependencies provided by the library. >>> >>> .../src/vec/f90-mod/petscvecmod.F90", line 107.37: 1514-084 (S) Identifier a is being declared with type name tvec which has not been defined in a derived type definition. >>> >>> IBM provided the following additional explanation and example. Can the process used to generate these routines and functions determine the specific symbols required and then use the only keyword or import statement to include them? >>> >>> When factoring out use statements out of module procedures, you can just delete them. But you can't completely remove them from interface blocks. Instead, you can limit them either by using use <module>, only: <symbol> or import <symbol> . if the hundreds of use statements in the program are factored out / limited in this way, that should reduce the dictionary size sufficiently for the program to compile. >>> >>> For example >>> Interface >>> Subroutine VecRestoreArrayReadF90(v,array,ierr) >>> use petscvecdef >>> real(kind=selected_real_kind(10)), pointer :: array(:) >>> integer(kind=selected_int_kind(5)) ierr >>> type(tVec) v >>> End Subroutine >>> End Interface >>> >>> imports all symbols from petscvecdef into the dictionary even though we only need tVec . So we can either: >>> >>> Interface >>> Subroutine VecRestoreArrayReadF90(v,array,ierr) >>> use petscvecdef, only: tVec >>> implicit none >>> real(kind=selected_real_kind(10)), pointer :: array(:) >>> integer(kind=selected_int_kind(5)) ierr >>> type(tVec) v >>> End Subroutine >>> End Interface >>> >>> or if use petscvecdef is used in the outer scope, we can: >>> Interface >>> Subroutine VecRestoreArrayReadF90(v,array,ierr) >>> import tVec >>> implicit none >>> real(kind=selected_real_kind(10)), pointer :: array(:) >>> integer(kind=selected_int_kind(5)) ierr >>> type(tVec) v >>> End Subroutine >>> End Interface >>> (The two methods (use, only vs import) are equivalent in terms of impact to the dictionary.) >>> >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> Roy Musselman >>> IBM HPC Application Analyst at Lawrence Livermore National Lab >>> email: roymuss(a)us.ibm.com <mailto:[email protected]> >>> LLNL office: 925-422-6033 >>> Cell: 507-358-8895, Home: 507-281-9565 >>> >>> <graycol.gif>Roy Musselman---02/24/2021 07:08:45 PM---Hi Jacob, I opened the ticket with IBM: case TS005062693 and and the local LLNL Sierra Jira Ticket >>> >>> From: Roy Musselman/Rochester/Contr/IBM >>> To: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> >>> Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> >>> Date: 02/24/2021 07:08 PM >>> Subject: Re: [EXTERNAL] Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines >>> >>> >>> >>> Hi Jacob, >>> I opened the ticket with IBM: case TS005062693 and and the local LLNL Sierra Jira Ticket at >>> https://lc.llnl.gov/jira/projects/SIERRA/issues/SIERRA-111?filter=allissues <https://urldefense.com/v3/__https://lc.llnl.gov/jira/projects/SIERRA/issues…> >>> >>> Today IBM provided the response below. I don't know when I'll have time to try it on the reproducer I gave IBM. Perhaps early next week. Can you review this and see if it helps? >>> >>> The reason for the Internal Compiler Error is because we can't grow an interal dictionary anymore (ie we hit a 2**21 limit). >>> The file contains many module procedures and interfaces that use the same helper module. As a result, we are importing the dictionary entries for that module repeatedly reaching >>> the limit. >>> >>> Can you please give the following source code workaround a try? >>> Since there is already "use petscvecdefdummy" at the module scope, one workaround might be to remove the unnecessary "use petscvecdefdummy" in vecnotequal and vecequals >>> and all similar procedures. >>> >>> For example, the test case has: >>> module petscvecdef >>> use petscvecdefdummy >>> ... >>> function vecnotequal(A,B) >>> use petscvecdefdummy >>> logical vecnotequal >>> type(tVec), intent(in) :: A,B >>> vecnotequal = (A%v .ne. B%v) >>> end function >>> function vecequals(A,B) >>> use petscvecdefdummy >>> logical vecequals >>> type(tVec), intent(in) :: A,B >>> vecequals = (A%v .eq. B%v) >>> end function >>> ... >>> end module >>> Another workaround would be to put the procedure definitions from this large module into several submodules. Each submodule would be able to accommodate a dictionary with 2**21 entries. >>> >>> >>> Please let us know if one of the above workarounds resolve the issue. >>> >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> Roy Musselman >>> IBM HPC Application Analyst at Lawrence Livermore National Lab >>> email: roymuss(a)us.ibm.com <mailto:[email protected]> >>> LLNL office: 925-422-6033 >>> Cell: 507-358-8895, Home: 507-281-9565 >>> >>> >>> <graycol.gif>Roy Musselman---02/21/2021 09:42:55 PM---Hi Jacob, After some more experimentation, I think I may have found what is triggering the ICE. It >>> >>> From: Roy Musselman/Rochester/Contr/IBM >>> To: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> >>> Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> >>> Date: 02/21/2021 09:42 PM >>> Subject: Re: [EXTERNAL] Re: xlf90_r Internal Compiler Error >>> >>> >>> Hi Jacob, >>> >>> After some more experimentation, I think I may have found what is triggering the ICE. It doesn't appear to be related to the subroutine name length. I think the compiler may be hitting an internal limit of the number of subroutines within a module. There are 358 subroutines contained in the expanded petscmatmod.F90. Removing 4 subroutines will allow the compile to complete successfully, so the limit must be 354 subroutines. Is it possible for you to bust up petscmatmod into multiple modules? I'll package up the reproducer and pass it on to the compiler development team. >>> >>> I've asked for user feedback a couple years ago, when the IBM Power9 CORAL-1 Sierra systems were deployed, but received minimal responses. DOE is now working with Cray (aka HPE) developing the environment for the CORAL-2 system (El Capitan). I'll pass your request to the LLNL person I know that is dealing with math libraries for CORAL-2. >>> >>> We use the spack tool to download and build petsc and its specified dependencies. I switched between the PETSC versions by changing the PETSCDIR variable in the script I shared with you. I've attached a tar ball containing the scripts used to build PETSc via spack. >>> >>> [attachment "bld-petsc-spack.tgz" deleted by Roy Musselman/Rochester/Contr/IBM] >>> >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> Roy Musselman >>> IBM HPC Application Analyst at Lawrence Livermore National Lab >>> email: roymuss(a)us.ibm.com <mailto:[email protected]> >>> LLNL office: 925-422-6033 >>> Cell: 507-358-8895, Home: 507-281-9565 >>> >>> >>> <graycol.gif>Jacob Faibussowitsch ---02/21/2021 12:24:11 PM---Hi Roy, > I'm not sure which projects at LLNL are using PETSc or if they chose to build their own ve >>> >>> From: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> >>> To: Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> >>> Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> >>> Date: 02/21/2021 12:24 PM >>> Subject: [EXTERNAL] Re: xlf90_r Internal Compiler Error >>> >>> >>> >>> Hi Roy, I'm not sure which projects at LLNL are using PETSc or if they chose to build their own version. Entirely unrelated to our problem, but is it possible to find this out? It would be great if yes, but also completely fine if not. PETSc >>> Hi Roy, >>> I'm not sure which projects at LLNL are using PETSc or if they chose to build their own version. >>> Entirely unrelated to our problem, but is it possible to find this out? It would be great if yes, but also completely fine if not. PETSc is potentially undergoing a rather transformative rewrite over the next few years and we’d like to gather current usage data to get a better idea of where PETSc fits into our users workflows. But we aren’t sure how to gather this data (we don’t particularly want to scrape and silently send it off without users consent/knowledge) absent user questionnaires and HPC usage statistics. >>> If you are interested, I can share with you the spack recipes I use to build petsc with hdf5, hypre, and suplerlu-dist. >>> Yes that would be quite useful. I can let it percolate through our dev channels for any other recommendations etc. >>> 3.14.0 and 3.14.1 >>> >>> "../roymuss/spack-stage-petsc-3.14.0-on3lboy4slkz65tsjttgfmwghzky54jj/spack-src/src/vec/f90-mod/petscvecmod.F90", line 9.13: 1514-219 (S) Unable to access module symbol file for module petscisdefdummy. Check path and file permissions of file. Use association not done for this module. >>> 1501-511 Compilation failed for file petscvecmod.F90. >>> How exactly did you switch between versions? PETSc has 2 types of fortran bindings, “ftn-custom” and “ftn-auto” (technically 3 including the F90 files, but those simply call either of the two preceding ones), a copy of which you will find in every src directory. As the names imply ftn-auto is auto generated while ftn-custom is hand-written. >>> >>> This also means that the ftn-auto files are __not__ tracked by git, so a simple git checkout [new-tag] may not properly dispose of the old auto-generated files (very rare, but IIRC we made a major enough change to the fortran bindings within the last year to warrant having to "make deletefortranstubs" before rebuilding). >>> Adding the option -qlanglvl=2003std or -qlanglvl=2008std produces a bunch of other warning messages, but it still encounters the ICE. So, I'm uncertain if the subroutine name length is the root of the problem. >>> Our current compiler flag selection philosophy is to require a minimum but choose the maximum available reasonable flag for the compiler (I.e. we require C99, but very often you will find that your code is compiled with C11 or C17 if they are available). It is therefore odd that configure did not use the same methodology for fortran compilers. I will relay this on our side. >>> Is it possible for you to use subroutines that are less than 32 characters and see if that works four you? Have you used other fortran 90 compilers and do any of them complain of this? >>> Of all of the small quirks fortran has this is probably the most esoteric one I’ve come across… I’ve attached a list of all the F90 compilers, and their flags which we use in CI/CD (all of which is run multiple times daily and __must__ pass). I got them all via grep, so there may be some duplicates here or there. As for using shorter names, this is also something we can look at, but since none of the other compilers have had issues with this I’m not sure this is the change to make. >>> Are there any unusual or questionable language constructs used in any of the functions mentioned above that may possibly challenge the compiler? >>> Not that I am aware of, but again I will ask around our dev channels and see if anything comes to mind. >>> >>> >>> Best regards, >>> >>> Jacob Faibussowitsch >>> (Jacob Fai - booss - oh - vitch) >>> Cell: (312) 694-3391[attachment "compilerList" deleted by Roy Musselman/Rochester/Contr/IBM] >>> On Feb 20, 2021, at 22:05, Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> wrote: >>> Hi Jacob, >>> Thanks for letting me know that you are a PETSc developer and that you are testing it on the LLNL lassen system. I've used the spack build tool to build and deploy a few versions on the systems. I'm not sure which projects at LLNL are using PETSc or if they chose to build their own version. I did however provide a single precision version upon request that was integrated with MVAPICH2-MPI instead of the IBM-provided Spectrum-MPI. Here's what's available on the systems today. >>> >>> > ml avail petsc >>> ----------------------------------------------------- /usr/tcetmp/modulefiles/Core ----------------------------------------------------- >>> petsc/default petsc/3.10.2 petsc/3.11.3 petsc/3.13.0 (D) >>> petsc/3.13.1-mvapich2-2020.01.09-xl-2020.03.18.single >>> >>> If you are interested, I can share with you the spack recipes I use to build petsc with hdf5, hypre, and suplerlu-dist. >>> >>> After several attempts I was able to reproduce the Internal Compiler Errro (ICE) that you are seeing using version 3.14.4. I've whittled it down to the petscmatmod.F90 file and it's specific dependencies. >>> The following script is what I'm using. Note that in the 2nd set of compiles, the -E option is used to expand all included source files and headers and encapsulating it into a single large source file. This can be used to help isolate the source of the problem. >>> >>> #!/bin/bash >>> >>> PETSCDIR="../roymuss/spack-stage-petsc-3.14.4-eh5arny7l3cqjlltlfpjp6f4jofbnmz6/spack-src" >>> OPTIONS=" -qmoddir=moddir -I$PETSCDIR/arch-linux-c-opt/include -I$PETSCDIR/include" >>> mkdir -p moddir >>> >>> set -x >>> >>> # Compile original source files including dependencies >>> if [ 0 = 1 ]; then >>> mpif90 -c -g $OPTIONS $PETSCDIR/src/sys/f90-mod/petscsysmod.F90 -o petscsysmod.o >>> mpif90 -c -g $OPTIONS $PETSCDIR/src/vec/f90-mod/petscvecmod.F90 -o petscvecmod.o >>> mpif90 -c -g $OPTIONS $PETSCDIR/src/mat/f90-mod/petscmatmod.F90 -o petscmatmod.o >>> fi >>> >>> # Use -E option to expand source into full source files >>> if [ 0 = 1 ]; then >>> mpif90 -c -g -E $OPTIONS $PETSCDIR/src/sys/f90-mod/petscsysmod.F90 -o full_petscsysmod.F90 >>> mpif90 -c -g -E $OPTIONS $PETSCDIR/src/vec/f90-mod/petscvecmod.F90 -o full_petscvecmod.F90 >>> mpif90 -c -g -E $OPTIONS $PETSCDIR/src/mat/f90-mod/petscmatmod.F90 -o full_petscmatmod.F90 >>> fi >>> >>> # Compile from full source files >>> if [ 1 = 1 ]; then >>> mpif90 -c -g -Imoddir -qmoddir=moddir full_petscsysmod.F90 -o full_petscsysmod.o >>> mpif90 -c -g -Imoddir -qmoddir=moddir full_petscvecmod.F90 -o full_petscvecmod.o >>> mpif90 -V -c -g -Imoddir -qmoddir=moddir full_petscmatmod.F90 -o full_petscmatmod.o >>> fi >>> >>> <eof> >>> >>> Petsc 3.13.6 it the most recent version that did not fail. I tried all subsequent versions and got the folowing results: >>> >>> 3.14.0 and 3.14.1 >>> >>> "../roymuss/spack-stage-petsc-3.14.0-on3lboy4slkz65tsjttgfmwghzky54jj/spack-src/src/vec/f90-mod/petscvecmod.F90", line 9.13: 1514-219 (S) Unable to access module symbol file for module petscisdefdummy. Check path and file permissions of file. Use association not done for this module. >>> 1501-511 Compilation failed for file petscvecmod.F90. >>> >>> 3.14.2, 3.14.3, and 3.14.4 >>> >>> . . . >>> ** matnullspaceequals === End of Compilation 8 === >>> *** Error in `/usr/tce/packages/xl/xl-2020.11.12/xlf/16.1.1/exe/xlfentry': free(): invalid pointer: 0x0000200001740018 *** >>> >>> Examining the tail end of petscmatmod.F90 >>> >>> >>> 80 function matnullspaceequals(A,B) >>> 81 use petscmatdefdummy >>> 82 logical matnullspaceequals >>> 83 type(tMatNullSpace), intent(in) :: A,B >>> 84 matnullspaceequals = (A%v .eq. B%v) >>> 85 end function >>> 86 >>> 87 #if defined(_WIN32) && defined(PETSC_USE_SHARED_LIBRARIES) >>> 88 !DEC$ ATTRIBUTES DLLEXPORT::matnotequal >>> 89 !DEC$ ATTRIBUTES DLLEXPORT::matequals >>> 90 !DEC$ ATTRIBUTES DLLEXPORT::matfdcoloringnotequal >>> 91 !DEC$ ATTRIBUTES DLLEXPORT::matfdcoloringequals >>> 92 !DEC$ ATTRIBUTES DLLEXPORT::matnullspacenotequal >>> 93 !DEC$ ATTRIBUTES DLLEXPORT::matnullspaceequals >>> 94 #endif >>> 95 module petscmat >>> 96 use petscmatdef >>> 97 use petscvec >>> 98 #include <../src/mat/f90-mod/petscmat.h90> >>> 99 interface >>> 100 #include <../src/mat/f90-mod/ftn-auto-interfaces/petscmat.h90> >>> 101 end interface >>> 102 end module >>> 103 >>> >>> Compiling the matnullspaceequals function was successful just before hitting the error. The error goes away when removing either or both of the #include lines 98 and 100. Both #include statements are required to produce the error. The 3.13.6 and 3.14.4 version of the file identified in the first #include at line 98 are identical. The file identified in line 100 is different between 3.13.6 and 3.14.4. >>> Just looking at the list of subroutines contained within each version, the following are the differences. >>> >>> Old subroutines available in 3.13.6 but removed from 4.14.4 >>> subroutine MatFreeIntermediateDataStructures(a,z) >>> >>> New subroutines available in 4.14.4 but not contained in 3.13.6 >>> subroutine MatDenseReplaceArray(a,b,z) >>> subroutine MatIsShell(a,b,z) >>> subroutine MatRARtMultEqual(a,b,c,d,e,z) >>> subroutine MatScaLAPACKGetBlockSizes(a,b,c,z) >>> subroutine MatScaLAPACKSetBlockSizes(a,b,c,z) >>> subroutine MatSeqAIJCUSPARSESetGenerateTranspose(a,b,z) >>> subroutine MatSeqAIJSetTotalPreallocation(a,b,z) >>> subroutine MatSetLayouts(a,b,c,z) >>> >>> Methodically removing the new subroutines did not provide a consistent result. But I did notice the extra long subroutine name MatSeqAIJCUSPARSESetGenerateTranspose had 37 characters. >>> A little research found: In Fortran 90/95 the maximum length was 31 characters, in Fortran 2003 it is now 63 characters. I found the following subroutines with greater than 31 characters >>> >>> subroutine MatCreateMPIMatConcatenateSeqMat >>> subroutine MatFactorFactorizeSchurComplement >>> subroutine MatMPIAdjCreateNonemptySubcommMat >>> subroutine MatSeqAIJCUSPARSESetGenerateTranspose >>> subroutine MatMPIAIJSetUseScalableIncreaseOverlap >>> subroutine MatFactorSolveSchurComplementTranspose >>> >>> I individually ifdef'd them out of the source file and was able to compile the files successfully without encountering the ICE. >>> >>> I'm not exactly sure what the maximum subroutine name length that the XLF compiler allows, but if it is only 31, it would be useful if the compiler detected this and issue a message instead of the ICE. >>> Adding the option -qlanglvl=2003std or -qlanglvl=2008std produces a bunch of other warning messages, but it still encounters the ICE. So, I'm uncertain if the subroutine name length is the root of the problem. >>> >>> Is it possible for you to use subroutines that are less than 32 characters and see if that works four you? Have you used other fortran 90 compilers and do any of them complain of this? >>> Are there any unusual or questionable language constructs used in any of the functions mentioned above that may possibly challenge the compiler? >>> >>> I'll package this up and send it to the IBM XL compiler development team for their examination and comment. >>> >>> Best Regards, >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> Roy Musselman >>> IBM HPC Application Analyst at Lawrence Livermore National Lab >>> email: roymuss(a)us.ibm.com <mailto:[email protected]> >>> LLNL office: 925-422-6033 >>> Cell: 507-358-8895, Home: 507-281-9565 >>> >>> <graycol.gif>Jacob Faibussowitsch ---02/18/2021 02:17:05 PM---> The most recently built version available on the CORAL systems is 3.13.0. (ml load petsc/3.13.0) W >>> >>> From: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> >>> To: Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> >>> Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> >>> Date: 02/18/2021 02:17 PM >>> Subject: [EXTERNAL] Re: xlf90_r Internal Compiler Error >>> >>> >>> >>> >>> >>> The most recently built version available on the CORAL systems... >>> This Message Is From an External Sender >>> This message came from outside your organization. >>> The most recently built version available on the CORAL systems is 3.13.0. (ml load petsc/3.13.0) Will that work for you? >>> I am building petsc from source as part of development work on petsc itself so modules are unfortunately not useful here. >>> The files you sent me do not contain all the dependencies (other mod files) required to reproduce the error. >>> I'll attempt to build version 3.14.4 from scratch and recreate the failing symptom you are observing. >>> Yes, petsc uses an automated system to generate the fortran files from C which goes about 20 rabbit holes deeper than I was willing to dig. Let me know if you run into trouble configuring and building petsc, I can point you in the right direction. I’ve attached a “reconfigure” script with this email, it contains all of the arguments I used to configure petsc successfully on Lassen. If you place it into your $PETSC_DIR (i.e. the folder titled “petsc” and that contains a “configure” file) and run: >>> >>> $ python3 ./reconfigure-arch-linux-c-debug.py >>> >>> It should work. If not, you will have to >>> >>> $ ./configure —all-the-args —in-the-reconfigure —file >>> >>> Best regards, >>> >>> Jacob Faibussowitsch >>> (Jacob Fai - booss - oh - vitch) >>> Cell: (312) 694-3391[attachment "reconfigure-arch-linux-c-debug.py" deleted by Roy Musselman/Rochester/Contr/IBM] >>> On Feb 18, 2021, at 15:07, Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> wrote: >>> Hi Jacob, >>> >>> The source file appears to come from the PETSc 3.14.4 library. The most recently built version available on the CORAL systems is 3.13.0. (ml load petsc/3.13.0) Will that work for you? >>> The files you sent me do not contain all the dependencies (other mod files) required to reproduce the error. >>> I'll attempt to build version 3.14.4 from scratch and recreate the failing symptom you are observing. >>> >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> Roy Musselman >>> IBM HPC Application Analyst at Lawrence Livermore National Lab >>> email: roymuss(a)us.ibm.com <mailto:[email protected]> >>> LLNL office: 925-422-6033 >>> Cell: 507-358-8895, Home: 507-281-9565 >>> >>> <graycol.gif>Roy Musselman---02/18/2021 11:18:20 AM---I'll take a look. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Roy Musselman >>> >>> From: Roy Musselman/Rochester/Contr/IBM >>> To: LC Hotline <lc-hotline(a)llnl.gov <mailto:[email protected]>> >>> Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> >>> Date: 02/18/2021 11:18 AM >>> Subject: Re: [EXTERNAL] FW: xlf90_r Internal Compiler Error >>> >>> >>> >>> >>> >>> I'll take a look. >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> Roy Musselman >>> IBM HPC Application Analyst at Lawrence Livermore National Lab >>> email: roymuss(a)us.ibm.com <mailto:[email protected]> >>> LLNL office: 925-422-6033 >>> Cell: 507-358-8895, Home: 507-281-9565 >>> >>> >>> <graycol.gif>LC Hotline ---02/18/2021 11:03:55 AM---Hi John, Roy, Can you help this user with the problem that he is seeing when he tries to build with >>> >>> From: LC Hotline <lc-hotline(a)llnl.gov <mailto:[email protected]>> >>> To: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>>, Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> >>> Date: 02/18/2021 11:03 AM >>> Subject: [EXTERNAL] FW: xlf90_r Internal Compiler Error >>> >>> >>> >>> Hi John, Roy, Can you help this user with the problem that he is... >>> This Message Is From an External Sender >>> This message came from outside your organization. >>> Hi John, Roy, >>> >>> Can you help this user with the problem that he is seeing when he tries to build with xlf90 on Lassen? >>> >>> Thanks, >>> Ryan >>> -- >>> LC Hotline >>> >>> From: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> >>> Date: Wednesday, February 17, 2021 at 5:27 PM >>> To: LC Hotline <lc-hotline(a)llnl.gov <mailto:[email protected]>> >>> Subject: xlf90_r Internal Compiler Error >>> >>> Hello LC Support, >>> >>> While compiling my application on Lassen I seem have run afoul of the xlf90 mpi compiler wrapper with the following error: >>> >>> *** Error in `/usr/tce/packages/xl/xl-2020.11.12/xlf/16.1.1/exe/xlfentry': free(): invalid pointer: 0x0000200001740018 *** >>> >>> I’m fairly certain this isn’t my fault as this is code that compiles regularly on extensive CI/CD under various other compilers and machines, but you can never rule it out. I have included a verbose full log of my make run (which includes a comprehensive rundown of the environment) as well as a separate file containing the error message and stack trace from the compiler. Additionally I have also included the file which I believe is causing the error. Let me know if there is anything else I should send. >>> >>> P.S. My list of loaded modules: >>> >>> Currently Loaded Modules: >>> 1) StdEnv (S) 4) cuda/11.1.1 7) valgrind/3.16.1 >>> 2) clang/ibm-11.0.0 5) python/3.8.2 8) lapack/3.9.0-xl-2020.11.12 >>> 3) spectrum-mpi/rolling-release 6) cmake/3.18.0 9) hip/3.0.0 >>> >>> Best regards, >>> >>> Jacob Faibussowitsch >>> (Jacob Fai - booss - oh - vitch) >>> Cell: (312) 694-3391[attachment "errorReport.zip" deleted by Roy Musselman/Rochester/Contr/IBM] >>

1 0

Re: [petsc-dev] Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines
by Barry Smith 03 Mar '21

03 Mar '21

PETSc stacks the Fortran modules in the same way it stacks the C include files. So the TAO module includes all the Fortran modules below it etc. It would be nearly impossible to disentangle the bits and pieces without introducing a more painful user experience. For example use PCTypes, use PCFunctions, use KSPTypes, .... impossible to use and impossible to maintain. This is a completely artificial bug of IBM's own making in their compiler that we should not have to work around. Barry > On Mar 3, 2021, at 12:10 PM, Jacob Faibussowitsch <jacob.fai(a)gmail.com> wrote: > > Hello All, > > I discovered a compiler bug in the IBM xl fortran compiler a few weeks ago that would crash the compiler when compiling petsc fortran interfaces. The TL;DR of it is that the xl compiler creates a function dictionary for every function imported in fortran modules, and since petsc fortran interfaces seem to import entire packages writ-large this exceeds the number of dictionary entries (2**21): > >> The reason for the Internal Compiler Error is because we can't grow an interal dictionary anymore (ie we hit a 2**21 limit). >> The file contains many module procedures and interfaces that use the same helper module. As a result, we are importing the dictionary entries for that module repeatedly reaching >> the limit. >> >> Can you please give the following source code workaround a try? >> Since there is already "use petscvecdefdummy" at the module scope, one workaround might be to remove the unnecessary "use petscvecdefdummy" in vecnotequal and vecequals >> and all similar procedures. >> >> For example, the test case has: >> module petscvecdef >> use petscvecdefdummy >> ... >> function vecnotequal(A,B) >> use petscvecdefdummy >> logical vecnotequal >> type(tVec), intent(in) :: A,B >> vecnotequal = (A%v .ne. B%v) >> end function >> function vecequals(A,B) >> use petscvecdefdummy >> logical vecequals >> type(tVec), intent(in) :: A,B >> vecequals = (A%v .eq. B%v) >> end function >> ... >> end module >> Another workaround would be to put the procedure definitions from this large module into several submodules. Each submodule would be able to accommodate a dictionary with 2**21 entries. >> >> >> Please let us know if one of the above workarounds resolve the issue. > > > The proposed fix from IBM would be to pull “use moduleXXX” out of subroutines or to have our auto-fortran interfaces detect which symbols to include from the respective modules and only include those in the subroutines. I’m not familiar at all with how the interfaces are generated so I don’t even know if this is possible. >> IBM provided the following additional explanation and example. Can the process used to generate these routines and functions determine the specific symbols required and then use the only keyword or import statement to include them? >> >> When factoring out use statements out of module procedures, you can just delete them. But you can't completely remove them from interface blocks. Instead, you can limit them either by using use <module>, only: <symbol> or import <symbol> . if the hundreds of use statements in the program are factored out / limited in this way, that should reduce the dictionary size sufficiently for the program to compile. >> >> For example >> Interface >> Subroutine VecRestoreArrayReadF90(v,array,ierr) >> use petscvecdef >> real(kind=selected_real_kind(10)), pointer :: array(:) >> integer(kind=selected_int_kind(5)) ierr >> type(tVec) v >> End Subroutine >> End Interface >> >> imports all symbols from petscvecdef into the dictionary even though we only need tVec . So we can either: >> >> Interface >> Subroutine VecRestoreArrayReadF90(v,array,ierr) >> use petscvecdef, only: tVec >> implicit none >> real(kind=selected_real_kind(10)), pointer :: array(:) >> integer(kind=selected_int_kind(5)) ierr >> type(tVec) v >> End Subroutine >> End Interface >> >> or if use petscvecdef is used in the outer scope, we can: >> Interface >> Subroutine VecRestoreArrayReadF90(v,array,ierr) >> import tVec >> implicit none >> real(kind=selected_real_kind(10)), pointer :: array(:) >> integer(kind=selected_int_kind(5)) ierr >> type(tVec) v >> End Subroutine >> End Interface >> (The two methods (use, only vs import) are equivalent in terms of impact to the dictionary.) >> > > Is this compiler ~feature~ something that we intend to work around? Thoughts? > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > Cell: (312) 694-3391 > >> Begin forwarded message: >> >> From: "Roy Musselman" <roymuss(a)us.ibm.com <mailto:[email protected]>> >> Subject: Re: Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines >> Date: March 3, 2021 at 08:23:17 CST >> To: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> >> Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> >> >> Hi Jacob, >> I tried the first suggestion and commented out the use statements called within the functions. However, I hit the following error complaining about specific symbol dependencies provided by the library. >> >> .../src/vec/f90-mod/petscvecmod.F90", line 107.37: 1514-084 (S) Identifier a is being declared with type name tvec which has not been defined in a derived type definition. >> >> IBM provided the following additional explanation and example. Can the process used to generate these routines and functions determine the specific symbols required and then use the only keyword or import statement to include them? >> >> When factoring out use statements out of module procedures, you can just delete them. But you can't completely remove them from interface blocks. Instead, you can limit them either by using use <module>, only: <symbol> or import <symbol> . if the hundreds of use statements in the program are factored out / limited in this way, that should reduce the dictionary size sufficiently for the program to compile. >> >> For example >> Interface >> Subroutine VecRestoreArrayReadF90(v,array,ierr) >> use petscvecdef >> real(kind=selected_real_kind(10)), pointer :: array(:) >> integer(kind=selected_int_kind(5)) ierr >> type(tVec) v >> End Subroutine >> End Interface >> >> imports all symbols from petscvecdef into the dictionary even though we only need tVec . So we can either: >> >> Interface >> Subroutine VecRestoreArrayReadF90(v,array,ierr) >> use petscvecdef, only: tVec >> implicit none >> real(kind=selected_real_kind(10)), pointer :: array(:) >> integer(kind=selected_int_kind(5)) ierr >> type(tVec) v >> End Subroutine >> End Interface >> >> or if use petscvecdef is used in the outer scope, we can: >> Interface >> Subroutine VecRestoreArrayReadF90(v,array,ierr) >> import tVec >> implicit none >> real(kind=selected_real_kind(10)), pointer :: array(:) >> integer(kind=selected_int_kind(5)) ierr >> type(tVec) v >> End Subroutine >> End Interface >> (The two methods (use, only vs import) are equivalent in terms of impact to the dictionary.) >> >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> Roy Musselman >> IBM HPC Application Analyst at Lawrence Livermore National Lab >> email: roymuss(a)us.ibm.com <mailto:[email protected]> >> LLNL office: 925-422-6033 >> Cell: 507-358-8895, Home: 507-281-9565 >> >> <graycol.gif>Roy Musselman---02/24/2021 07:08:45 PM---Hi Jacob, I opened the ticket with IBM: case TS005062693 and and the local LLNL Sierra Jira Ticket >> >> From: Roy Musselman/Rochester/Contr/IBM >> To: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> >> Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> >> Date: 02/24/2021 07:08 PM >> Subject: Re: [EXTERNAL] Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines >> >> >> >> Hi Jacob, >> I opened the ticket with IBM: case TS005062693 and and the local LLNL Sierra Jira Ticket at >> https://lc.llnl.gov/jira/projects/SIERRA/issues/SIERRA-111?filter=allissues <https://urldefense.com/v3/__https://lc.llnl.gov/jira/projects/SIERRA/issues…> >> >> Today IBM provided the response below. I don't know when I'll have time to try it on the reproducer I gave IBM. Perhaps early next week. Can you review this and see if it helps? >> >> The reason for the Internal Compiler Error is because we can't grow an interal dictionary anymore (ie we hit a 2**21 limit). >> The file contains many module procedures and interfaces that use the same helper module. As a result, we are importing the dictionary entries for that module repeatedly reaching >> the limit. >> >> Can you please give the following source code workaround a try? >> Since there is already "use petscvecdefdummy" at the module scope, one workaround might be to remove the unnecessary "use petscvecdefdummy" in vecnotequal and vecequals >> and all similar procedures. >> >> For example, the test case has: >> module petscvecdef >> use petscvecdefdummy >> ... >> function vecnotequal(A,B) >> use petscvecdefdummy >> logical vecnotequal >> type(tVec), intent(in) :: A,B >> vecnotequal = (A%v .ne. B%v) >> end function >> function vecequals(A,B) >> use petscvecdefdummy >> logical vecequals >> type(tVec), intent(in) :: A,B >> vecequals = (A%v .eq. B%v) >> end function >> ... >> end module >> Another workaround would be to put the procedure definitions from this large module into several submodules. Each submodule would be able to accommodate a dictionary with 2**21 entries. >> >> >> Please let us know if one of the above workarounds resolve the issue. >> >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> Roy Musselman >> IBM HPC Application Analyst at Lawrence Livermore National Lab >> email: roymuss(a)us.ibm.com <mailto:[email protected]> >> LLNL office: 925-422-6033 >> Cell: 507-358-8895, Home: 507-281-9565 >> >> >> <graycol.gif>Roy Musselman---02/21/2021 09:42:55 PM---Hi Jacob, After some more experimentation, I think I may have found what is triggering the ICE. It >> >> From: Roy Musselman/Rochester/Contr/IBM >> To: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> >> Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> >> Date: 02/21/2021 09:42 PM >> Subject: Re: [EXTERNAL] Re: xlf90_r Internal Compiler Error >> >> >> Hi Jacob, >> >> After some more experimentation, I think I may have found what is triggering the ICE. It doesn't appear to be related to the subroutine name length. I think the compiler may be hitting an internal limit of the number of subroutines within a module. There are 358 subroutines contained in the expanded petscmatmod.F90. Removing 4 subroutines will allow the compile to complete successfully, so the limit must be 354 subroutines. Is it possible for you to bust up petscmatmod into multiple modules? I'll package up the reproducer and pass it on to the compiler development team. >> >> I've asked for user feedback a couple years ago, when the IBM Power9 CORAL-1 Sierra systems were deployed, but received minimal responses. DOE is now working with Cray (aka HPE) developing the environment for the CORAL-2 system (El Capitan). I'll pass your request to the LLNL person I know that is dealing with math libraries for CORAL-2. >> >> We use the spack tool to download and build petsc and its specified dependencies. I switched between the PETSC versions by changing the PETSCDIR variable in the script I shared with you. I've attached a tar ball containing the scripts used to build PETSc via spack. >> >> [attachment "bld-petsc-spack.tgz" deleted by Roy Musselman/Rochester/Contr/IBM] >> >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> Roy Musselman >> IBM HPC Application Analyst at Lawrence Livermore National Lab >> email: roymuss(a)us.ibm.com <mailto:[email protected]> >> LLNL office: 925-422-6033 >> Cell: 507-358-8895, Home: 507-281-9565 >> >> >> <graycol.gif>Jacob Faibussowitsch ---02/21/2021 12:24:11 PM---Hi Roy, > I'm not sure which projects at LLNL are using PETSc or if they chose to build their own ve >> >> From: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> >> To: Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> >> Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> >> Date: 02/21/2021 12:24 PM >> Subject: [EXTERNAL] Re: xlf90_r Internal Compiler Error >> >> >> >> Hi Roy, I'm not sure which projects at LLNL are using PETSc or if they chose to build their own version. Entirely unrelated to our problem, but is it possible to find this out? It would be great if yes, but also completely fine if not. PETSc >> Hi Roy, >> I'm not sure which projects at LLNL are using PETSc or if they chose to build their own version. >> Entirely unrelated to our problem, but is it possible to find this out? It would be great if yes, but also completely fine if not. PETSc is potentially undergoing a rather transformative rewrite over the next few years and we’d like to gather current usage data to get a better idea of where PETSc fits into our users workflows. But we aren’t sure how to gather this data (we don’t particularly want to scrape and silently send it off without users consent/knowledge) absent user questionnaires and HPC usage statistics. >> If you are interested, I can share with you the spack recipes I use to build petsc with hdf5, hypre, and suplerlu-dist. >> Yes that would be quite useful. I can let it percolate through our dev channels for any other recommendations etc. >> 3.14.0 and 3.14.1 >> >> "../roymuss/spack-stage-petsc-3.14.0-on3lboy4slkz65tsjttgfmwghzky54jj/spack-src/src/vec/f90-mod/petscvecmod.F90", line 9.13: 1514-219 (S) Unable to access module symbol file for module petscisdefdummy. Check path and file permissions of file. Use association not done for this module. >> 1501-511 Compilation failed for file petscvecmod.F90. >> How exactly did you switch between versions? PETSc has 2 types of fortran bindings, “ftn-custom” and “ftn-auto” (technically 3 including the F90 files, but those simply call either of the two preceding ones), a copy of which you will find in every src directory. As the names imply ftn-auto is auto generated while ftn-custom is hand-written. >> >> This also means that the ftn-auto files are __not__ tracked by git, so a simple git checkout [new-tag] may not properly dispose of the old auto-generated files (very rare, but IIRC we made a major enough change to the fortran bindings within the last year to warrant having to "make deletefortranstubs" before rebuilding). >> Adding the option -qlanglvl=2003std or -qlanglvl=2008std produces a bunch of other warning messages, but it still encounters the ICE. So, I'm uncertain if the subroutine name length is the root of the problem. >> Our current compiler flag selection philosophy is to require a minimum but choose the maximum available reasonable flag for the compiler (I.e. we require C99, but very often you will find that your code is compiled with C11 or C17 if they are available). It is therefore odd that configure did not use the same methodology for fortran compilers. I will relay this on our side. >> Is it possible for you to use subroutines that are less than 32 characters and see if that works four you? Have you used other fortran 90 compilers and do any of them complain of this? >> Of all of the small quirks fortran has this is probably the most esoteric one I’ve come across… I’ve attached a list of all the F90 compilers, and their flags which we use in CI/CD (all of which is run multiple times daily and __must__ pass). I got them all via grep, so there may be some duplicates here or there. As for using shorter names, this is also something we can look at, but since none of the other compilers have had issues with this I’m not sure this is the change to make. >> Are there any unusual or questionable language constructs used in any of the functions mentioned above that may possibly challenge the compiler? >> Not that I am aware of, but again I will ask around our dev channels and see if anything comes to mind. >> >> >> Best regards, >> >> Jacob Faibussowitsch >> (Jacob Fai - booss - oh - vitch) >> Cell: (312) 694-3391[attachment "compilerList" deleted by Roy Musselman/Rochester/Contr/IBM] >> On Feb 20, 2021, at 22:05, Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> wrote: >> Hi Jacob, >> Thanks for letting me know that you are a PETSc developer and that you are testing it on the LLNL lassen system. I've used the spack build tool to build and deploy a few versions on the systems. I'm not sure which projects at LLNL are using PETSc or if they chose to build their own version. I did however provide a single precision version upon request that was integrated with MVAPICH2-MPI instead of the IBM-provided Spectrum-MPI. Here's what's available on the systems today. >> >> > ml avail petsc >> ----------------------------------------------------- /usr/tcetmp/modulefiles/Core ----------------------------------------------------- >> petsc/default petsc/3.10.2 petsc/3.11.3 petsc/3.13.0 (D) >> petsc/3.13.1-mvapich2-2020.01.09-xl-2020.03.18.single >> >> If you are interested, I can share with you the spack recipes I use to build petsc with hdf5, hypre, and suplerlu-dist. >> >> After several attempts I was able to reproduce the Internal Compiler Errro (ICE) that you are seeing using version 3.14.4. I've whittled it down to the petscmatmod.F90 file and it's specific dependencies. >> The following script is what I'm using. Note that in the 2nd set of compiles, the -E option is used to expand all included source files and headers and encapsulating it into a single large source file. This can be used to help isolate the source of the problem. >> >> #!/bin/bash >> >> PETSCDIR="../roymuss/spack-stage-petsc-3.14.4-eh5arny7l3cqjlltlfpjp6f4jofbnmz6/spack-src" >> OPTIONS=" -qmoddir=moddir -I$PETSCDIR/arch-linux-c-opt/include -I$PETSCDIR/include" >> mkdir -p moddir >> >> set -x >> >> # Compile original source files including dependencies >> if [ 0 = 1 ]; then >> mpif90 -c -g $OPTIONS $PETSCDIR/src/sys/f90-mod/petscsysmod.F90 -o petscsysmod.o >> mpif90 -c -g $OPTIONS $PETSCDIR/src/vec/f90-mod/petscvecmod.F90 -o petscvecmod.o >> mpif90 -c -g $OPTIONS $PETSCDIR/src/mat/f90-mod/petscmatmod.F90 -o petscmatmod.o >> fi >> >> # Use -E option to expand source into full source files >> if [ 0 = 1 ]; then >> mpif90 -c -g -E $OPTIONS $PETSCDIR/src/sys/f90-mod/petscsysmod.F90 -o full_petscsysmod.F90 >> mpif90 -c -g -E $OPTIONS $PETSCDIR/src/vec/f90-mod/petscvecmod.F90 -o full_petscvecmod.F90 >> mpif90 -c -g -E $OPTIONS $PETSCDIR/src/mat/f90-mod/petscmatmod.F90 -o full_petscmatmod.F90 >> fi >> >> # Compile from full source files >> if [ 1 = 1 ]; then >> mpif90 -c -g -Imoddir -qmoddir=moddir full_petscsysmod.F90 -o full_petscsysmod.o >> mpif90 -c -g -Imoddir -qmoddir=moddir full_petscvecmod.F90 -o full_petscvecmod.o >> mpif90 -V -c -g -Imoddir -qmoddir=moddir full_petscmatmod.F90 -o full_petscmatmod.o >> fi >> >> <eof> >> >> Petsc 3.13.6 it the most recent version that did not fail. I tried all subsequent versions and got the folowing results: >> >> 3.14.0 and 3.14.1 >> >> "../roymuss/spack-stage-petsc-3.14.0-on3lboy4slkz65tsjttgfmwghzky54jj/spack-src/src/vec/f90-mod/petscvecmod.F90", line 9.13: 1514-219 (S) Unable to access module symbol file for module petscisdefdummy. Check path and file permissions of file. Use association not done for this module. >> 1501-511 Compilation failed for file petscvecmod.F90. >> >> 3.14.2, 3.14.3, and 3.14.4 >> >> . . . >> ** matnullspaceequals === End of Compilation 8 === >> *** Error in `/usr/tce/packages/xl/xl-2020.11.12/xlf/16.1.1/exe/xlfentry': free(): invalid pointer: 0x0000200001740018 *** >> >> Examining the tail end of petscmatmod.F90 >> >> >> 80 function matnullspaceequals(A,B) >> 81 use petscmatdefdummy >> 82 logical matnullspaceequals >> 83 type(tMatNullSpace), intent(in) :: A,B >> 84 matnullspaceequals = (A%v .eq. B%v) >> 85 end function >> 86 >> 87 #if defined(_WIN32) && defined(PETSC_USE_SHARED_LIBRARIES) >> 88 !DEC$ ATTRIBUTES DLLEXPORT::matnotequal >> 89 !DEC$ ATTRIBUTES DLLEXPORT::matequals >> 90 !DEC$ ATTRIBUTES DLLEXPORT::matfdcoloringnotequal >> 91 !DEC$ ATTRIBUTES DLLEXPORT::matfdcoloringequals >> 92 !DEC$ ATTRIBUTES DLLEXPORT::matnullspacenotequal >> 93 !DEC$ ATTRIBUTES DLLEXPORT::matnullspaceequals >> 94 #endif >> 95 module petscmat >> 96 use petscmatdef >> 97 use petscvec >> 98 #include <../src/mat/f90-mod/petscmat.h90> >> 99 interface >> 100 #include <../src/mat/f90-mod/ftn-auto-interfaces/petscmat.h90> >> 101 end interface >> 102 end module >> 103 >> >> Compiling the matnullspaceequals function was successful just before hitting the error. The error goes away when removing either or both of the #include lines 98 and 100. Both #include statements are required to produce the error. The 3.13.6 and 3.14.4 version of the file identified in the first #include at line 98 are identical. The file identified in line 100 is different between 3.13.6 and 3.14.4. >> Just looking at the list of subroutines contained within each version, the following are the differences. >> >> Old subroutines available in 3.13.6 but removed from 4.14.4 >> subroutine MatFreeIntermediateDataStructures(a,z) >> >> New subroutines available in 4.14.4 but not contained in 3.13.6 >> subroutine MatDenseReplaceArray(a,b,z) >> subroutine MatIsShell(a,b,z) >> subroutine MatRARtMultEqual(a,b,c,d,e,z) >> subroutine MatScaLAPACKGetBlockSizes(a,b,c,z) >> subroutine MatScaLAPACKSetBlockSizes(a,b,c,z) >> subroutine MatSeqAIJCUSPARSESetGenerateTranspose(a,b,z) >> subroutine MatSeqAIJSetTotalPreallocation(a,b,z) >> subroutine MatSetLayouts(a,b,c,z) >> >> Methodically removing the new subroutines did not provide a consistent result. But I did notice the extra long subroutine name MatSeqAIJCUSPARSESetGenerateTranspose had 37 characters. >> A little research found: In Fortran 90/95 the maximum length was 31 characters, in Fortran 2003 it is now 63 characters. I found the following subroutines with greater than 31 characters >> >> subroutine MatCreateMPIMatConcatenateSeqMat >> subroutine MatFactorFactorizeSchurComplement >> subroutine MatMPIAdjCreateNonemptySubcommMat >> subroutine MatSeqAIJCUSPARSESetGenerateTranspose >> subroutine MatMPIAIJSetUseScalableIncreaseOverlap >> subroutine MatFactorSolveSchurComplementTranspose >> >> I individually ifdef'd them out of the source file and was able to compile the files successfully without encountering the ICE. >> >> I'm not exactly sure what the maximum subroutine name length that the XLF compiler allows, but if it is only 31, it would be useful if the compiler detected this and issue a message instead of the ICE. >> Adding the option -qlanglvl=2003std or -qlanglvl=2008std produces a bunch of other warning messages, but it still encounters the ICE. So, I'm uncertain if the subroutine name length is the root of the problem. >> >> Is it possible for you to use subroutines that are less than 32 characters and see if that works four you? Have you used other fortran 90 compilers and do any of them complain of this? >> Are there any unusual or questionable language constructs used in any of the functions mentioned above that may possibly challenge the compiler? >> >> I'll package this up and send it to the IBM XL compiler development team for their examination and comment. >> >> Best Regards, >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> Roy Musselman >> IBM HPC Application Analyst at Lawrence Livermore National Lab >> email: roymuss(a)us.ibm.com <mailto:[email protected]> >> LLNL office: 925-422-6033 >> Cell: 507-358-8895, Home: 507-281-9565 >> >> <graycol.gif>Jacob Faibussowitsch ---02/18/2021 02:17:05 PM---> The most recently built version available on the CORAL systems is 3.13.0. (ml load petsc/3.13.0) W >> >> From: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> >> To: Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> >> Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> >> Date: 02/18/2021 02:17 PM >> Subject: [EXTERNAL] Re: xlf90_r Internal Compiler Error >> >> >> >> >> >> The most recently built version available on the CORAL systems... >> This Message Is From an External Sender >> This message came from outside your organization. >> The most recently built version available on the CORAL systems is 3.13.0. (ml load petsc/3.13.0) Will that work for you? >> I am building petsc from source as part of development work on petsc itself so modules are unfortunately not useful here. >> The files you sent me do not contain all the dependencies (other mod files) required to reproduce the error. >> I'll attempt to build version 3.14.4 from scratch and recreate the failing symptom you are observing. >> Yes, petsc uses an automated system to generate the fortran files from C which goes about 20 rabbit holes deeper than I was willing to dig. Let me know if you run into trouble configuring and building petsc, I can point you in the right direction. I’ve attached a “reconfigure” script with this email, it contains all of the arguments I used to configure petsc successfully on Lassen. If you place it into your $PETSC_DIR (i.e. the folder titled “petsc” and that contains a “configure” file) and run: >> >> $ python3 ./reconfigure-arch-linux-c-debug.py >> >> It should work. If not, you will have to >> >> $ ./configure —all-the-args —in-the-reconfigure —file >> >> Best regards, >> >> Jacob Faibussowitsch >> (Jacob Fai - booss - oh - vitch) >> Cell: (312) 694-3391[attachment "reconfigure-arch-linux-c-debug.py" deleted by Roy Musselman/Rochester/Contr/IBM] >> On Feb 18, 2021, at 15:07, Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> wrote: >> Hi Jacob, >> >> The source file appears to come from the PETSc 3.14.4 library. The most recently built version available on the CORAL systems is 3.13.0. (ml load petsc/3.13.0) Will that work for you? >> The files you sent me do not contain all the dependencies (other mod files) required to reproduce the error. >> I'll attempt to build version 3.14.4 from scratch and recreate the failing symptom you are observing. >> >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> Roy Musselman >> IBM HPC Application Analyst at Lawrence Livermore National Lab >> email: roymuss(a)us.ibm.com <mailto:[email protected]> >> LLNL office: 925-422-6033 >> Cell: 507-358-8895, Home: 507-281-9565 >> >> <graycol.gif>Roy Musselman---02/18/2021 11:18:20 AM---I'll take a look. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Roy Musselman >> >> From: Roy Musselman/Rochester/Contr/IBM >> To: LC Hotline <lc-hotline(a)llnl.gov <mailto:[email protected]>> >> Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> >> Date: 02/18/2021 11:18 AM >> Subject: Re: [EXTERNAL] FW: xlf90_r Internal Compiler Error >> >> >> >> >> >> I'll take a look. >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> Roy Musselman >> IBM HPC Application Analyst at Lawrence Livermore National Lab >> email: roymuss(a)us.ibm.com <mailto:[email protected]> >> LLNL office: 925-422-6033 >> Cell: 507-358-8895, Home: 507-281-9565 >> >> >> <graycol.gif>LC Hotline ---02/18/2021 11:03:55 AM---Hi John, Roy, Can you help this user with the problem that he is seeing when he tries to build with >> >> From: LC Hotline <lc-hotline(a)llnl.gov <mailto:[email protected]>> >> To: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>>, Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> >> Date: 02/18/2021 11:03 AM >> Subject: [EXTERNAL] FW: xlf90_r Internal Compiler Error >> >> >> >> Hi John, Roy, Can you help this user with the problem that he is... >> This Message Is From an External Sender >> This message came from outside your organization. >> Hi John, Roy, >> >> Can you help this user with the problem that he is seeing when he tries to build with xlf90 on Lassen? >> >> Thanks, >> Ryan >> -- >> LC Hotline >> >> From: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> >> Date: Wednesday, February 17, 2021 at 5:27 PM >> To: LC Hotline <lc-hotline(a)llnl.gov <mailto:[email protected]>> >> Subject: xlf90_r Internal Compiler Error >> >> Hello LC Support, >> >> While compiling my application on Lassen I seem have run afoul of the xlf90 mpi compiler wrapper with the following error: >> >> *** Error in `/usr/tce/packages/xl/xl-2020.11.12/xlf/16.1.1/exe/xlfentry': free(): invalid pointer: 0x0000200001740018 *** >> >> I’m fairly certain this isn’t my fault as this is code that compiles regularly on extensive CI/CD under various other compilers and machines, but you can never rule it out. I have included a verbose full log of my make run (which includes a comprehensive rundown of the environment) as well as a separate file containing the error message and stack trace from the compiler. Additionally I have also included the file which I believe is causing the error. Let me know if there is anything else I should send. >> >> P.S. My list of loaded modules: >> >> Currently Loaded Modules: >> 1) StdEnv (S) 4) cuda/11.1.1 7) valgrind/3.16.1 >> 2) clang/ibm-11.0.0 5) python/3.8.2 8) lapack/3.9.0-xl-2020.11.12 >> 3) spectrum-mpi/rolling-release 6) cmake/3.18.0 9) hip/3.0.0 >> >> Best regards, >> >> Jacob Faibussowitsch >> (Jacob Fai - booss - oh - vitch) >> Cell: (312) 694-3391[attachment "errorReport.zip" deleted by Roy Musselman/Rochester/Contr/IBM] >

1 0

Fwd: Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines
by Jacob Faibussowitsch 03 Mar '21

03 Mar '21

Hello All, I discovered a compiler bug in the IBM xl fortran compiler a few weeks ago that would crash the compiler when compiling petsc fortran interfaces. The TL;DR of it is that the xl compiler creates a function dictionary for every function imported in fortran modules, and since petsc fortran interfaces seem to import entire packages writ-large this exceeds the number of dictionary entries (2**21): > The reason for the Internal Compiler Error is because we can't grow an interal dictionary anymore (ie we hit a 2**21 limit). > The file contains many module procedures and interfaces that use the same helper module. As a result, we are importing the dictionary entries for that module repeatedly reaching > the limit. > > Can you please give the following source code workaround a try? > Since there is already "use petscvecdefdummy" at the module scope, one workaround might be to remove the unnecessary "use petscvecdefdummy" in vecnotequal and vecequals > and all similar procedures. > > For example, the test case has: > module petscvecdef > use petscvecdefdummy > ... > function vecnotequal(A,B) > use petscvecdefdummy > logical vecnotequal > type(tVec), intent(in) :: A,B > vecnotequal = (A%v .ne. B%v) > end function > function vecequals(A,B) > use petscvecdefdummy > logical vecequals > type(tVec), intent(in) :: A,B > vecequals = (A%v .eq. B%v) > end function > ... > end module > Another workaround would be to put the procedure definitions from this large module into several submodules. Each submodule would be able to accommodate a dictionary with 2**21 entries. > > > Please let us know if one of the above workarounds resolve the issue. The proposed fix from IBM would be to pull “use moduleXXX” out of subroutines or to have our auto-fortran interfaces detect which symbols to include from the respective modules and only include those in the subroutines. I’m not familiar at all with how the interfaces are generated so I don’t even know if this is possible. > IBM provided the following additional explanation and example. Can the process used to generate these routines and functions determine the specific symbols required and then use the only keyword or import statement to include them? > > When factoring out use statements out of module procedures, you can just delete them. But you can't completely remove them from interface blocks. Instead, you can limit them either by using use <module>, only: <symbol> or import <symbol> . if the hundreds of use statements in the program are factored out / limited in this way, that should reduce the dictionary size sufficiently for the program to compile. > > For example > Interface > Subroutine VecRestoreArrayReadF90(v,array,ierr) > use petscvecdef > real(kind=selected_real_kind(10)), pointer :: array(:) > integer(kind=selected_int_kind(5)) ierr > type(tVec) v > End Subroutine > End Interface > > imports all symbols from petscvecdef into the dictionary even though we only need tVec . So we can either: > > Interface > Subroutine VecRestoreArrayReadF90(v,array,ierr) > use petscvecdef, only: tVec > implicit none > real(kind=selected_real_kind(10)), pointer :: array(:) > integer(kind=selected_int_kind(5)) ierr > type(tVec) v > End Subroutine > End Interface > > or if use petscvecdef is used in the outer scope, we can: > Interface > Subroutine VecRestoreArrayReadF90(v,array,ierr) > import tVec > implicit none > real(kind=selected_real_kind(10)), pointer :: array(:) > integer(kind=selected_int_kind(5)) ierr > type(tVec) v > End Subroutine > End Interface > (The two methods (use, only vs import) are equivalent in terms of impact to the dictionary.) > Is this compiler ~feature~ something that we intend to work around? Thoughts? Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) Cell: (312) 694-3391 > Begin forwarded message: > > From: "Roy Musselman" <roymuss(a)us.ibm.com> > Subject: Re: Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines > Date: March 3, 2021 at 08:23:17 CST > To: Jacob Faibussowitsch <faibuss2(a)illinois.edu> > Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov> > > Hi Jacob, > I tried the first suggestion and commented out the use statements called within the functions. However, I hit the following error complaining about specific symbol dependencies provided by the library. > > .../src/vec/f90-mod/petscvecmod.F90", line 107.37: 1514-084 (S) Identifier a is being declared with type name tvec which has not been defined in a derived type definition. > > IBM provided the following additional explanation and example. Can the process used to generate these routines and functions determine the specific symbols required and then use the only keyword or import statement to include them? > > When factoring out use statements out of module procedures, you can just delete them. But you can't completely remove them from interface blocks. Instead, you can limit them either by using use <module>, only: <symbol> or import <symbol> . if the hundreds of use statements in the program are factored out / limited in this way, that should reduce the dictionary size sufficiently for the program to compile. > > For example > Interface > Subroutine VecRestoreArrayReadF90(v,array,ierr) > use petscvecdef > real(kind=selected_real_kind(10)), pointer :: array(:) > integer(kind=selected_int_kind(5)) ierr > type(tVec) v > End Subroutine > End Interface > > imports all symbols from petscvecdef into the dictionary even though we only need tVec . So we can either: > > Interface > Subroutine VecRestoreArrayReadF90(v,array,ierr) > use petscvecdef, only: tVec > implicit none > real(kind=selected_real_kind(10)), pointer :: array(:) > integer(kind=selected_int_kind(5)) ierr > type(tVec) v > End Subroutine > End Interface > > or if use petscvecdef is used in the outer scope, we can: > Interface > Subroutine VecRestoreArrayReadF90(v,array,ierr) > import tVec > implicit none > real(kind=selected_real_kind(10)), pointer :: array(:) > integer(kind=selected_int_kind(5)) ierr > type(tVec) v > End Subroutine > End Interface > (The two methods (use, only vs import) are equivalent in terms of impact to the dictionary.) > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Roy Musselman > IBM HPC Application Analyst at Lawrence Livermore National Lab > email: roymuss(a)us.ibm.com <mailto:[email protected]> > LLNL office: 925-422-6033 > Cell: 507-358-8895, Home: 507-281-9565 > > Roy Musselman---02/24/2021 07:08:45 PM---Hi Jacob, I opened the ticket with IBM: case TS005062693 and and the local LLNL Sierra Jira Ticket > > From: Roy Musselman/Rochester/Contr/IBM > To: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> > Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> > Date: 02/24/2021 07:08 PM > Subject: Re: [EXTERNAL] Case TS005062693 - XLF: ICE in xlfentry compiling a module with 358 subroutines > > > > Hi Jacob, > I opened the ticket with IBM: case TS005062693 and and the local LLNL Sierra Jira Ticket at > https://lc.llnl.gov/jira/projects/SIERRA/issues/SIERRA-111?filter=allissues <https://urldefense.com/v3/__https://lc.llnl.gov/jira/projects/SIERRA/issues…> > > Today IBM provided the response below. I don't know when I'll have time to try it on the reproducer I gave IBM. Perhaps early next week. Can you review this and see if it helps? > > The reason for the Internal Compiler Error is because we can't grow an interal dictionary anymore (ie we hit a 2**21 limit). > The file contains many module procedures and interfaces that use the same helper module. As a result, we are importing the dictionary entries for that module repeatedly reaching > the limit. > > Can you please give the following source code workaround a try? > Since there is already "use petscvecdefdummy" at the module scope, one workaround might be to remove the unnecessary "use petscvecdefdummy" in vecnotequal and vecequals > and all similar procedures. > > For example, the test case has: > module petscvecdef > use petscvecdefdummy > ... > function vecnotequal(A,B) > use petscvecdefdummy > logical vecnotequal > type(tVec), intent(in) :: A,B > vecnotequal = (A%v .ne. B%v) > end function > function vecequals(A,B) > use petscvecdefdummy > logical vecequals > type(tVec), intent(in) :: A,B > vecequals = (A%v .eq. B%v) > end function > ... > end module > Another workaround would be to put the procedure definitions from this large module into several submodules. Each submodule would be able to accommodate a dictionary with 2**21 entries. > > > Please let us know if one of the above workarounds resolve the issue. > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Roy Musselman > IBM HPC Application Analyst at Lawrence Livermore National Lab > email: roymuss(a)us.ibm.com <mailto:[email protected]> > LLNL office: 925-422-6033 > Cell: 507-358-8895, Home: 507-281-9565 > > > Roy Musselman---02/21/2021 09:42:55 PM---Hi Jacob, After some more experimentation, I think I may have found what is triggering the ICE. It > > From: Roy Musselman/Rochester/Contr/IBM > To: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> > Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> > Date: 02/21/2021 09:42 PM > Subject: Re: [EXTERNAL] Re: xlf90_r Internal Compiler Error > > > Hi Jacob, > > After some more experimentation, I think I may have found what is triggering the ICE. It doesn't appear to be related to the subroutine name length. I think the compiler may be hitting an internal limit of the number of subroutines within a module. There are 358 subroutines contained in the expanded petscmatmod.F90. Removing 4 subroutines will allow the compile to complete successfully, so the limit must be 354 subroutines. Is it possible for you to bust up petscmatmod into multiple modules? I'll package up the reproducer and pass it on to the compiler development team. > > I've asked for user feedback a couple years ago, when the IBM Power9 CORAL-1 Sierra systems were deployed, but received minimal responses. DOE is now working with Cray (aka HPE) developing the environment for the CORAL-2 system (El Capitan). I'll pass your request to the LLNL person I know that is dealing with math libraries for CORAL-2. > > We use the spack tool to download and build petsc and its specified dependencies. I switched between the PETSC versions by changing the PETSCDIR variable in the script I shared with you. I've attached a tar ball containing the scripts used to build PETSc via spack. > > [attachment "bld-petsc-spack.tgz" deleted by Roy Musselman/Rochester/Contr/IBM] > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Roy Musselman > IBM HPC Application Analyst at Lawrence Livermore National Lab > email: roymuss(a)us.ibm.com <mailto:[email protected]> > LLNL office: 925-422-6033 > Cell: 507-358-8895, Home: 507-281-9565 > > > Jacob Faibussowitsch ---02/21/2021 12:24:11 PM---Hi Roy, > I'm not sure which projects at LLNL are using PETSc or if they chose to build their own ve > > From: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> > To: Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> > Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> > Date: 02/21/2021 12:24 PM > Subject: [EXTERNAL] Re: xlf90_r Internal Compiler Error > > > > Hi Roy, I'm not sure which projects at LLNL are using PETSc or if they chose to build their own version. Entirely unrelated to our problem, but is it possible to find this out? It would be great if yes, but also completely fine if not. PETSc > Hi Roy, > I'm not sure which projects at LLNL are using PETSc or if they chose to build their own version. > Entirely unrelated to our problem, but is it possible to find this out? It would be great if yes, but also completely fine if not. PETSc is potentially undergoing a rather transformative rewrite over the next few years and we’d like to gather current usage data to get a better idea of where PETSc fits into our users workflows. But we aren’t sure how to gather this data (we don’t particularly want to scrape and silently send it off without users consent/knowledge) absent user questionnaires and HPC usage statistics. > If you are interested, I can share with you the spack recipes I use to build petsc with hdf5, hypre, and suplerlu-dist. > Yes that would be quite useful. I can let it percolate through our dev channels for any other recommendations etc. > 3.14.0 and 3.14.1 > > "../roymuss/spack-stage-petsc-3.14.0-on3lboy4slkz65tsjttgfmwghzky54jj/spack-src/src/vec/f90-mod/petscvecmod.F90", line 9.13: 1514-219 (S) Unable to access module symbol file for module petscisdefdummy. Check path and file permissions of file. Use association not done for this module. > 1501-511 Compilation failed for file petscvecmod.F90. > How exactly did you switch between versions? PETSc has 2 types of fortran bindings, “ftn-custom” and “ftn-auto” (technically 3 including the F90 files, but those simply call either of the two preceding ones), a copy of which you will find in every src directory. As the names imply ftn-auto is auto generated while ftn-custom is hand-written. > > This also means that the ftn-auto files are __not__ tracked by git, so a simple git checkout [new-tag] may not properly dispose of the old auto-generated files (very rare, but IIRC we made a major enough change to the fortran bindings within the last year to warrant having to "make deletefortranstubs" before rebuilding). > Adding the option -qlanglvl=2003std or -qlanglvl=2008std produces a bunch of other warning messages, but it still encounters the ICE. So, I'm uncertain if the subroutine name length is the root of the problem. > Our current compiler flag selection philosophy is to require a minimum but choose the maximum available reasonable flag for the compiler (I.e. we require C99, but very often you will find that your code is compiled with C11 or C17 if they are available). It is therefore odd that configure did not use the same methodology for fortran compilers. I will relay this on our side. > Is it possible for you to use subroutines that are less than 32 characters and see if that works four you? Have you used other fortran 90 compilers and do any of them complain of this? > Of all of the small quirks fortran has this is probably the most esoteric one I’ve come across… I’ve attached a list of all the F90 compilers, and their flags which we use in CI/CD (all of which is run multiple times daily and __must__ pass). I got them all via grep, so there may be some duplicates here or there. As for using shorter names, this is also something we can look at, but since none of the other compilers have had issues with this I’m not sure this is the change to make. > Are there any unusual or questionable language constructs used in any of the functions mentioned above that may possibly challenge the compiler? > Not that I am aware of, but again I will ask around our dev channels and see if anything comes to mind. > > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > Cell: (312) 694-3391[attachment "compilerList" deleted by Roy Musselman/Rochester/Contr/IBM] > On Feb 20, 2021, at 22:05, Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> wrote: > Hi Jacob, > Thanks for letting me know that you are a PETSc developer and that you are testing it on the LLNL lassen system. I've used the spack build tool to build and deploy a few versions on the systems. I'm not sure which projects at LLNL are using PETSc or if they chose to build their own version. I did however provide a single precision version upon request that was integrated with MVAPICH2-MPI instead of the IBM-provided Spectrum-MPI. Here's what's available on the systems today. > > > ml avail petsc > ----------------------------------------------------- /usr/tcetmp/modulefiles/Core ----------------------------------------------------- > petsc/default petsc/3.10.2 petsc/3.11.3 petsc/3.13.0 (D) > petsc/3.13.1-mvapich2-2020.01.09-xl-2020.03.18.single > > If you are interested, I can share with you the spack recipes I use to build petsc with hdf5, hypre, and suplerlu-dist. > > After several attempts I was able to reproduce the Internal Compiler Errro (ICE) that you are seeing using version 3.14.4. I've whittled it down to the petscmatmod.F90 file and it's specific dependencies. > The following script is what I'm using. Note that in the 2nd set of compiles, the -E option is used to expand all included source files and headers and encapsulating it into a single large source file. This can be used to help isolate the source of the problem. > > #!/bin/bash > > PETSCDIR="../roymuss/spack-stage-petsc-3.14.4-eh5arny7l3cqjlltlfpjp6f4jofbnmz6/spack-src" > OPTIONS=" -qmoddir=moddir -I$PETSCDIR/arch-linux-c-opt/include -I$PETSCDIR/include" > mkdir -p moddir > > set -x > > # Compile original source files including dependencies > if [ 0 = 1 ]; then > mpif90 -c -g $OPTIONS $PETSCDIR/src/sys/f90-mod/petscsysmod.F90 -o petscsysmod.o > mpif90 -c -g $OPTIONS $PETSCDIR/src/vec/f90-mod/petscvecmod.F90 -o petscvecmod.o > mpif90 -c -g $OPTIONS $PETSCDIR/src/mat/f90-mod/petscmatmod.F90 -o petscmatmod.o > fi > > # Use -E option to expand source into full source files > if [ 0 = 1 ]; then > mpif90 -c -g -E $OPTIONS $PETSCDIR/src/sys/f90-mod/petscsysmod.F90 -o full_petscsysmod.F90 > mpif90 -c -g -E $OPTIONS $PETSCDIR/src/vec/f90-mod/petscvecmod.F90 -o full_petscvecmod.F90 > mpif90 -c -g -E $OPTIONS $PETSCDIR/src/mat/f90-mod/petscmatmod.F90 -o full_petscmatmod.F90 > fi > > # Compile from full source files > if [ 1 = 1 ]; then > mpif90 -c -g -Imoddir -qmoddir=moddir full_petscsysmod.F90 -o full_petscsysmod.o > mpif90 -c -g -Imoddir -qmoddir=moddir full_petscvecmod.F90 -o full_petscvecmod.o > mpif90 -V -c -g -Imoddir -qmoddir=moddir full_petscmatmod.F90 -o full_petscmatmod.o > fi > > <eof> > > Petsc 3.13.6 it the most recent version that did not fail. I tried all subsequent versions and got the folowing results: > > 3.14.0 and 3.14.1 > > "../roymuss/spack-stage-petsc-3.14.0-on3lboy4slkz65tsjttgfmwghzky54jj/spack-src/src/vec/f90-mod/petscvecmod.F90", line 9.13: 1514-219 (S) Unable to access module symbol file for module petscisdefdummy. Check path and file permissions of file. Use association not done for this module. > 1501-511 Compilation failed for file petscvecmod.F90. > > 3.14.2, 3.14.3, and 3.14.4 > > . . . > ** matnullspaceequals === End of Compilation 8 === > *** Error in `/usr/tce/packages/xl/xl-2020.11.12/xlf/16.1.1/exe/xlfentry': free(): invalid pointer: 0x0000200001740018 *** > > Examining the tail end of petscmatmod.F90 > > > 80 function matnullspaceequals(A,B) > 81 use petscmatdefdummy > 82 logical matnullspaceequals > 83 type(tMatNullSpace), intent(in) :: A,B > 84 matnullspaceequals = (A%v .eq. B%v) > 85 end function > 86 > 87 #if defined(_WIN32) && defined(PETSC_USE_SHARED_LIBRARIES) > 88 !DEC$ ATTRIBUTES DLLEXPORT::matnotequal > 89 !DEC$ ATTRIBUTES DLLEXPORT::matequals > 90 !DEC$ ATTRIBUTES DLLEXPORT::matfdcoloringnotequal > 91 !DEC$ ATTRIBUTES DLLEXPORT::matfdcoloringequals > 92 !DEC$ ATTRIBUTES DLLEXPORT::matnullspacenotequal > 93 !DEC$ ATTRIBUTES DLLEXPORT::matnullspaceequals > 94 #endif > 95 module petscmat > 96 use petscmatdef > 97 use petscvec > 98 #include <../src/mat/f90-mod/petscmat.h90> > 99 interface > 100 #include <../src/mat/f90-mod/ftn-auto-interfaces/petscmat.h90> > 101 end interface > 102 end module > 103 > > Compiling the matnullspaceequals function was successful just before hitting the error. The error goes away when removing either or both of the #include lines 98 and 100. Both #include statements are required to produce the error. The 3.13.6 and 3.14.4 version of the file identified in the first #include at line 98 are identical. The file identified in line 100 is different between 3.13.6 and 3.14.4. > Just looking at the list of subroutines contained within each version, the following are the differences. > > Old subroutines available in 3.13.6 but removed from 4.14.4 > subroutine MatFreeIntermediateDataStructures(a,z) > > New subroutines available in 4.14.4 but not contained in 3.13.6 > subroutine MatDenseReplaceArray(a,b,z) > subroutine MatIsShell(a,b,z) > subroutine MatRARtMultEqual(a,b,c,d,e,z) > subroutine MatScaLAPACKGetBlockSizes(a,b,c,z) > subroutine MatScaLAPACKSetBlockSizes(a,b,c,z) > subroutine MatSeqAIJCUSPARSESetGenerateTranspose(a,b,z) > subroutine MatSeqAIJSetTotalPreallocation(a,b,z) > subroutine MatSetLayouts(a,b,c,z) > > Methodically removing the new subroutines did not provide a consistent result. But I did notice the extra long subroutine name MatSeqAIJCUSPARSESetGenerateTranspose had 37 characters. > A little research found: In Fortran 90/95 the maximum length was 31 characters, in Fortran 2003 it is now 63 characters. I found the following subroutines with greater than 31 characters > > subroutine MatCreateMPIMatConcatenateSeqMat > subroutine MatFactorFactorizeSchurComplement > subroutine MatMPIAdjCreateNonemptySubcommMat > subroutine MatSeqAIJCUSPARSESetGenerateTranspose > subroutine MatMPIAIJSetUseScalableIncreaseOverlap > subroutine MatFactorSolveSchurComplementTranspose > > I individually ifdef'd them out of the source file and was able to compile the files successfully without encountering the ICE. > > I'm not exactly sure what the maximum subroutine name length that the XLF compiler allows, but if it is only 31, it would be useful if the compiler detected this and issue a message instead of the ICE. > Adding the option -qlanglvl=2003std or -qlanglvl=2008std produces a bunch of other warning messages, but it still encounters the ICE. So, I'm uncertain if the subroutine name length is the root of the problem. > > Is it possible for you to use subroutines that are less than 32 characters and see if that works four you? Have you used other fortran 90 compilers and do any of them complain of this? > Are there any unusual or questionable language constructs used in any of the functions mentioned above that may possibly challenge the compiler? > > I'll package this up and send it to the IBM XL compiler development team for their examination and comment. > > Best Regards, > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Roy Musselman > IBM HPC Application Analyst at Lawrence Livermore National Lab > email: roymuss(a)us.ibm.com <mailto:[email protected]> > LLNL office: 925-422-6033 > Cell: 507-358-8895, Home: 507-281-9565 > > <graycol.gif>Jacob Faibussowitsch ---02/18/2021 02:17:05 PM---> The most recently built version available on the CORAL systems is 3.13.0. (ml load petsc/3.13.0) W > > From: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> > To: Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> > Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> > Date: 02/18/2021 02:17 PM > Subject: [EXTERNAL] Re: xlf90_r Internal Compiler Error > > > > > > The most recently built version available on the CORAL systems... > This Message Is From an External Sender > This message came from outside your organization. > The most recently built version available on the CORAL systems is 3.13.0. (ml load petsc/3.13.0) Will that work for you? > I am building petsc from source as part of development work on petsc itself so modules are unfortunately not useful here. > The files you sent me do not contain all the dependencies (other mod files) required to reproduce the error. > I'll attempt to build version 3.14.4 from scratch and recreate the failing symptom you are observing. > Yes, petsc uses an automated system to generate the fortran files from C which goes about 20 rabbit holes deeper than I was willing to dig. Let me know if you run into trouble configuring and building petsc, I can point you in the right direction. I’ve attached a “reconfigure” script with this email, it contains all of the arguments I used to configure petsc successfully on Lassen. If you place it into your $PETSC_DIR (i.e. the folder titled “petsc” and that contains a “configure” file) and run: > > $ python3 ./reconfigure-arch-linux-c-debug.py > > It should work. If not, you will have to > > $ ./configure —all-the-args —in-the-reconfigure —file > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > Cell: (312) 694-3391[attachment "reconfigure-arch-linux-c-debug.py" deleted by Roy Musselman/Rochester/Contr/IBM] > On Feb 18, 2021, at 15:07, Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> wrote: > Hi Jacob, > > The source file appears to come from the PETSc 3.14.4 library. The most recently built version available on the CORAL systems is 3.13.0. (ml load petsc/3.13.0) Will that work for you? > The files you sent me do not contain all the dependencies (other mod files) required to reproduce the error. > I'll attempt to build version 3.14.4 from scratch and recreate the failing symptom you are observing. > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Roy Musselman > IBM HPC Application Analyst at Lawrence Livermore National Lab > email: roymuss(a)us.ibm.com <mailto:[email protected]> > LLNL office: 925-422-6033 > Cell: 507-358-8895, Home: 507-281-9565 > > <graycol.gif>Roy Musselman---02/18/2021 11:18:20 AM---I'll take a look. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Roy Musselman > > From: Roy Musselman/Rochester/Contr/IBM > To: LC Hotline <lc-hotline(a)llnl.gov <mailto:[email protected]>> > Cc: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>> > Date: 02/18/2021 11:18 AM > Subject: Re: [EXTERNAL] FW: xlf90_r Internal Compiler Error > > > > > > I'll take a look. > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Roy Musselman > IBM HPC Application Analyst at Lawrence Livermore National Lab > email: roymuss(a)us.ibm.com <mailto:[email protected]> > LLNL office: 925-422-6033 > Cell: 507-358-8895, Home: 507-281-9565 > > > <graycol.gif>LC Hotline ---02/18/2021 11:03:55 AM---Hi John, Roy, Can you help this user with the problem that he is seeing when he tries to build with > > From: LC Hotline <lc-hotline(a)llnl.gov <mailto:[email protected]>> > To: "Gyllenhaal, John C." <gyllenhaal1(a)llnl.gov <mailto:[email protected]>>, Roy Musselman <roymuss(a)us.ibm.com <mailto:[email protected]>> > Date: 02/18/2021 11:03 AM > Subject: [EXTERNAL] FW: xlf90_r Internal Compiler Error > > > > Hi John, Roy, Can you help this user with the problem that he is... > This Message Is From an External Sender > This message came from outside your organization. > Hi John, Roy, > > Can you help this user with the problem that he is seeing when he tries to build with xlf90 on Lassen? > > Thanks, > Ryan > -- > LC Hotline > > From: Jacob Faibussowitsch <faibuss2(a)illinois.edu <mailto:[email protected]>> > Date: Wednesday, February 17, 2021 at 5:27 PM > To: LC Hotline <lc-hotline(a)llnl.gov <mailto:[email protected]>> > Subject: xlf90_r Internal Compiler Error > > Hello LC Support, > > While compiling my application on Lassen I seem have run afoul of the xlf90 mpi compiler wrapper with the following error: > > *** Error in `/usr/tce/packages/xl/xl-2020.11.12/xlf/16.1.1/exe/xlfentry': free(): invalid pointer: 0x0000200001740018 *** > > I’m fairly certain this isn’t my fault as this is code that compiles regularly on extensive CI/CD under various other compilers and machines, but you can never rule it out. I have included a verbose full log of my make run (which includes a comprehensive rundown of the environment) as well as a separate file containing the error message and stack trace from the compiler. Additionally I have also included the file which I believe is causing the error. Let me know if there is anything else I should send. > > P.S. My list of loaded modules: > > Currently Loaded Modules: > 1) StdEnv (S) 4) cuda/11.1.1 7) valgrind/3.16.1 > 2) clang/ibm-11.0.0 5) python/3.8.2 8) lapack/3.9.0-xl-2020.11.12 > 3) spectrum-mpi/rolling-release 6) cmake/3.18.0 9) hip/3.0.0 > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > Cell: (312) 694-3391[attachment "errorReport.zip" deleted by Roy Musselman/Rochester/Contr/IBM]

1 0

Re: [petsc-dev] Commit squashing in MR
by Scott Kruger 03 Mar '21

03 Mar '21

Attached is a little presentation that I gave to some fellow devs at Tech-X. It's a rehash of what has been discussed here, but the graphical nature might be useful. Scott On 2021-03-02 21:32, Jed Brown did write: > Satish Balay via petsc-dev <petsc-dev(a)mcs.anl.gov> writes: > > > On Wed, 3 Mar 2021, Blaise A Bourdin wrote: > > > >> Hi, > >> > >> This is not technically a petsc question. > >> It would be great to have a short section in the PETSc integration workflow document explaining how to squash commits in a MR for git-impaired developers like me. > >> > >> Anybody wants to pitch in, or explain me how to do this? > > > > To squash commits - I use the 'squash' action in 'git rebase -i HASH' and figure out the HASH to use from 'gitk main..branch' > > > > [as git rebase requires the commit prior to the first commit of interest] > > > > git provides many ways of modifying the branch (and the rebase topic is very generic) so I think its best to rely on proper git docs/tutorials > > [and its not really specific to petsc workflow] > > You can do it in one line, without changing the base: > > git rebase -i $(git merge-base main HEAD) > > > An alternative is > > git rebase -i main > > which gives you interactive rebase to replay on top of current 'main'. This does two things at once and changing the base for your branch is not always desirable. -- Scott Kruger Tech-X Corporation kruger(a)txcorp.com 5621 Arapahoe Ave, Suite A Phone: (720) 466-3196 Boulder, CO 80303 Fax: (303) 448-7756

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Barry Smith 03 Mar '21

03 Mar '21

>> 2) I can reproduce the src/mat/tests/ex242.c error (which explicitly uses ScaLAPACK, none of the above PC uses it explicitly, except PCBDDC/PCHPDDM when using MUMPS on “big” problems where root nodes are factorized using ScaLAPACK, see -mat_mumps_icntl_13) >> 3) I’m seeing that both on your machine and mine, PETSc BuildSystem insist on linking libmkl_blacs_intelmpi_lp64.so even though we supply explicitly libmkl_blacs_openmpi_lp64.so >> This for example yields a wrong Makefile.inc for MUMPS: >> $ cat arch-linux2-c-opt-ompi/externalpackages/MUMPS_5.3.5/Makefile.inc|grep blacs >> SCALAP = […] -lmkl_blacs_openmpi_lp64 >> LIBBLAS = […] -lmkl_blacs_intelmpi_lp64 -lgomp -ldl -lpthread -lm […] >> >> Despite what Barry says, I think PETSc is partially to blame as well (why use libmkl_blacs_intelmpi_lp64.so even though BuildSystem is capable of detecting we are using OpenMPI). >> I’ll try to fix this to see if it solves 2). > Okay, that's a very nice finding!!! Hope it will be "fixable" easily! > > The knowledge is there but the information may not be trivially available to make the right decisions. Parts of the BLAS/LAPACK checks use the "check everything" approach. For example # Look for Multi-Threaded MKL for MKL_C/Pardiso useCPardiso=0 usePardiso=0 if self.argDB['with-mkl_cpardiso'] or 'with-mkl_cpardiso-dir' in self.argDB or 'with-mkl_cpardiso-lib' in self.argDB: useCPardiso=1 mkl_blacs_64=[['mkl_blacs_intelmpi'+ILP64+''],['mkl_blacs_mpich'+ILP64+''],['mkl_blacs_sgimpt'+ILP64+''],['mkl_blacs_openmpi'+ILP64+'']] mkl_blacs_32=[['mkl_blacs_intelmpi'],['mkl_blacs_mpich'],['mkl_blacs_sgimpt'],['mkl_blacs_openmpi']] elif self.argDB['with-mkl_pardiso'] or 'with-mkl_pardiso-dir' in self.argDB or 'with-mkl_pardiso-lib' in self.argDB: usePardiso=1 mkl_blacs_64=[[]] mkl_blacs_32=[[]] if useCPardiso or usePardiso: self.logPrintBox('BLASLAPACK: Looking for Multithreaded MKL for C/Pardiso') for libdir in [os.path.join('lib','64'),os.path.join('lib','ia64'),os.path.join('lib','em64t'),os.path.join('lib','intel64'),'lib','64','ia64','em64t','intel64', os.path.join('lib','32'),os.path.join('lib','ia32'),'32','ia32','']: if not os.path.exists(os.path.join(dir,libdir)): self.logPrint('MKL Path not found.. skipping: '+os.path.join(dir,libdir)) else: self.log.write('Files and directories in that directory:\n'+str(os.listdir(os.path.join(dir,libdir)))+'\n') # iomp5 is provided by the Intel compilers on MacOS. Run source /opt/intel/bin/compilervars.sh intel64 to have it added to LIBRARY_PATH # then locate libimp5.dylib in the LIBRARY_PATH and copy it to os.path.join(dir,libdir) for i in mkl_blacs_64: yield ('User specified MKL-C/Pardiso Intel-Linux64', None, [os.path.join(dir,libdir,'libmkl_intel'+ILP64+'.a'),'mkl_core','mkl_intel_thread']+i+['iomp5','dl','pthread'],known,'yes') yield ('User specified MKL-C/Pardiso GNU-Linux64', None, [os.path.join(dir,libdir,'libmkl_intel'+ILP64+'.a'),'mkl_core','mkl_gnu_thread']+i+['gomp','dl','pthread'],known,'yes') yield ('User specified MKL-Pardiso Intel-Windows64', None, [os.path.join(dir,libdir,'mkl_core.lib'),'mkl_intel'+ILP64+'.lib','mkl_intel_thread.lib']+i+['libiomp5md.lib'],known,'yes') for i in mkl_blacs_32: yield ('User specified MKL-C/Pardiso Intel-Linux32', None, [os.path.join(dir,libdir,'libmkl_intel.a'),'mkl_core','mkl_intel_thread']+i+['iomp5','dl','pthread'],'32','yes') yield ('User specified MKL-C/Pardiso GNU-Linux32', None, [os.path.join(dir,libdir,'libmkl_intel.a'),'mkl_core','mkl_gnu_thread']+i+['gomp','dl','pthread'],'32','yes') yield ('User specified MKL-Pardiso Intel-Windows32', None, [os.path.join(dir,libdir,'mkl_core.lib'),'mkl_intel_c.lib','mkl_intel_thread.lib']+i+['libiomp5md.lib'],'32','yes') return The assumption is that the link will fail unless the correct libraries are in the list. But apparently this is not the case; it returns the first case that links but that case does not run which is why it appears to be producing "silly" results. If you set the right MPI and threading library, at these locations instead of trying all of them it might resolve the problems. if self.openmp.found: ITHREAD='intel_thread' ITHREADGNU='gnu_thread' ompthread = 'yes' else: ITHREAD='sequential' ITHREADGNU='sequential' ompthread = 'no' mkl_blacs_64=[['mkl_blacs_intelmpi'+ILP64+''],['mkl_blacs_mpich'+ILP64+''],['mkl_blacs_sgimpt'+ILP64+''],['mkl_blacs_openmpi'+ILP64+'']] mkl_blacs_32=[['mkl_blacs_intelmpi'],['mkl_blacs_mpich'],['mkl_blacs_sgimpt'],['mkl_blacs_openmpi']] > > On Mar 3, 2021, at 8:22 AM, Eric Chamberland <Eric.Chamberland(a)giref.ulaval.ca> wrote: > > Hi Pierre, > > On 2021-03-03 2:42 a.m., Pierre Jolivet wrote: >>> If it ends that there is a problem combining MKL + openMP that relies on linking configuration for example, should it be a good thing to have this (--with-openmp=1) tested into the pipelines (with external packages of course)? >>> >> As Barry said, there is not much (if any) OpenMP in PETSc. >> There is however some workers with the MKL (+ Intel compilers) turned on, but I don’t think we test MKL + GNU compilers (which I feel like is a very niche combination, hence not really worth testing, IMHO). > Ouch, this is my almost my personal working configuration and for most of our users too... and it worked well until I activated the OpenMP thing... > > We had good reasons to work with g++ or clang++ instead of intel compilers: > > - It is mandatory to pay to work with an intel compiler (didn't looked at OneAPI licensing yet, but it may have changed?) > > - No support of Intel compilers with iceccd (slow recompilation) > > - MKL was freely distributed, so it can be used with any compiler > > That doesn't mean we don't want to use intel compiler, but maybe we just want to to a specific delivery with it but continue to develop with g++ or clang++ (my personal choice). > > But I understand it is less straightforward to combine gcc and MKL than using native Intel tool-chain.... > I agree the MKL + GNU compilers is commonly used and should be tested and maintained in PETSc. >>> Does the guys who maintain all these libs are reading petsc-dev? ;) >>> >> I don’t think they are, but don’t worry, we do forward the appropriate messages to them :) > :) >> >> About yesterday’s failures… >> 1) I cannot reproduce any of the PCHYPRE/PCBDDC/PCHPDDM errors (sorry I didn’t bother putting the SuperLU_DIST tarball on my cluster) > Hmmm, maybe my environment variables may play a role into this? > > for comparisons considerations, we explicitly set: > > export MKL_CBWR=COMPATIBLE > export MKL_NUM_THREADS=1 > > but it would be surprising it helps reproduce a problem: they usually stabilize results... > > Merci, > > Eric > >> >> Thanks, >> Pierre >> >> http://joliv.et/irene-rome-configure.log <http://joliv.et/irene-rome-configure.log> >> $ /usr/bin/gmake -f gmakefile test test-fail=1 >> Using MAKEFLAGS: test-fail=1 >> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_baij.counts >> ok snes_tutorials-ex12_quad_hpddm_reuse_baij >> ok diff-snes_tutorials-ex12_quad_hpddm_reuse_baij >> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex50_tut_2.counts >> ok ksp_ksp_tutorials-ex50_tut_2 # SKIP PETSC_HAVE_SUPERLU_DIST requirement not met >> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_hypre.counts >> ok snes_tutorials-ex56_hypre >> ok diff-snes_tutorials-ex56_hypre >> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_elas.counts >> ok snes_tutorials-ex17_3d_q3_trig_elas >> ok diff-snes_tutorials-ex17_3d_q3_trig_elas >> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij.counts >> ok snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij >> ok diff-snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij >> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_tri_parmetis_hpddm_baij.counts >> ok snes_tutorials-ex12_tri_parmetis_hpddm_baij >> ok diff-snes_tutorials-ex12_tri_parmetis_hpddm_baij >> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_tut_3.counts >> ok snes_tutorials-ex19_tut_3 >> ok diff-snes_tutorials-ex19_tut_3 >> TEST arch-linux2-c-opt-ompi/tests/counts/mat_tests-ex242_3.counts >> not ok mat_tests-ex242_3 # Error code: 137 >> # [1]PETSC ERROR: ------------------------------------------------------------------------ >> # [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >> # [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> # [1]PETSC ERROR: or see https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind <https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind> >> # [1]PETSC ERROR: or try http://valgrind.org <http://valgrind.org/> on GNU/linux and Apple Mac OS X to find memory corruption errors >> # [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >> # [1]PETSC ERROR: to get more information on the crash. >> # [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> # [1]PETSC ERROR: Signal received >> # [1]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html <https://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting. >> # [1]PETSC ERROR: Petsc Development GIT revision: v3.14.4-733-g7ab9467ef9 GIT Date: 2021-03-02 16:15:11 +0000 >> # [2]PETSC ERROR: ------------------------------------------------------------------------ >> # [2]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >> # [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >> # [2]PETSC ERROR: or see https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind <https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind> >> # [2]PETSC ERROR: or try http://valgrind.org <http://valgrind.org/> on GNU/linux and Apple Mac OS X to find memory corruption errors >> # [2]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run >> # [2]PETSC ERROR: to get more information on the crash. >> # [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> # [2]PETSC ERROR: Signal received >> # [2]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html <https://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting. >> # [2]PETSC ERROR: Petsc Development GIT revision: v3.14.4-733-g7ab9467ef9 GIT Date: 2021-03-02 16:15:11 +0000 >> # [2]PETSC ERROR: /ccc/work/cont003/rndm/rndm/petsc/arch-linux2-c-opt-ompi/tests/mat/tests/runex242_3/../ex242 on a arch-linux2-c-opt-ompi named irene4047 by jolivetp Wed Mar 3 08:21:20 2021 >> # [2]PETSC ERROR: Configure options --download-hpddm --download-hpddm-commit=origin/main --download-hypre --download-metis --download-mumps --download-parmetis --download-ptscotch --download-slepc --download-slepc-commit=origin/main --download-tetgen --known-mpi-c-double-complex --known-mpi-int64_t --known-mpi-long-double --with-avx512-kernels=1 --with-blaslapack-dir=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64 --with-cc=mpicc --with-cxx=mpicxx --with-debugging=0 --with-fc=mpifort --with-fortran-bindings=0 --with-make-np=40 --with-mkl_cpardiso-dir=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281 --with-mkl_cpardiso=1 --with-mkl_pardiso-dir=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl --with-mkl_pardiso=1 --with-mpiexec=ccc_mprun --with-openmp=1 --with-packages-download-dir=/ccc/cont003/home/enseeiht/jolivetp/Dude/externalpackages/ --with-scalapack-include=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/include --with-scalapack-lib="[/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_scalapack_lp64.so,/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.so]" --with-scalar-type=real --with-x=0 COPTFLAGS="-O3 -fp-model fast -mavx2" CXXOPTFLAGS="-O3 -fp-model fast -mavx2" FOPTFLAGS="-O3 -fp-model fast -mavx2" PETSC_ARCH=arch-linux2-c-opt-ompi >> # [2]PETSC ERROR: #1 User provided function() line 0 in unknown file >> # [2]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. >> # [1]PETSC ERROR: /ccc/work/cont003/rndm/rndm/petsc/arch-linux2-c-opt-ompi/tests/mat/tests/runex242_3/../ex242 on a arch-linux2-c-opt-ompi named irene4047 by jolivetp Wed Mar 3 08:21:20 2021 >> # [1]PETSC ERROR: Configure options --download-hpddm --download-hpddm-commit=origin/main --download-hypre --download-metis --download-mumps --download-parmetis --download-ptscotch --download-slepc --download-slepc-commit=origin/main --download-tetgen --known-mpi-c-double-complex --known-mpi-int64_t --known-mpi-long-double --with-avx512-kernels=1 --with-blaslapack-dir=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64 --with-cc=mpicc --with-cxx=mpicxx --with-debugging=0 --with-fc=mpifort --with-fortran-bindings=0 --with-make-np=40 --with-mkl_cpardiso-dir=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281 --with-mkl_cpardiso=1 --with-mkl_pardiso-dir=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl --with-mkl_pardiso=1 --with-mpiexec=ccc_mprun --with-openmp=1 --with-packages-download-dir=/ccc/cont003/home/enseeiht/jolivetp/Dude/externalpackages/ --with-scalapack-include=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/include --with-scalapack-lib="[/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_scalapack_lp64.so,/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.so]" --with-scalar-type=real --with-x=0 COPTFLAGS="-O3 -fp-model fast -mavx2" CXXOPTFLAGS="-O3 -fp-model fast -mavx2" FOPTFLAGS="-O3 -fp-model fast -mavx2" PETSC_ARCH=arch-linux2-c-opt-ompi >> # [1]PETSC ERROR: #1 User provided function() line 0 in unknown file >> # [1]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. >> # -------------------------------------------------------------------------- >> # MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD >> # with errorcode 50176059. >> # >> # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> # You may or may not see output from other processes, depending on >> # exactly when Open MPI kills them. >> # -------------------------------------------------------------------------- >> # -------------------------------------------------------------------------- >> # MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD >> # with errorcode 50176059. >> # >> # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> # You may or may not see output from other processes, depending on >> # exactly when Open MPI kills them. >> # -------------------------------------------------------------------------- >> # srun: Job step aborted: Waiting up to 302 seconds for job step to finish. >> # slurmstepd-irene4047: error: *** STEP 1374176.36 ON irene4047 CANCELLED AT 2021-03-03T08:21:20 *** >> # srun: error: irene4047: task 0: Killed >> # srun: error: irene4047: tasks 1-2: Exited with exit code 16 >> ok mat_tests-ex242_3 # SKIP Command failed so no diff >> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_vlap.counts >> ok snes_tutorials-ex17_3d_q3_trig_vlap >> ok diff-snes_tutorials-ex17_3d_q3_trig_vlap >> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre.counts >> ok snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre >> ok diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre >> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex49_hypre_nullspace.counts >> ok ksp_ksp_tutorials-ex49_hypre_nullspace >> ok diff-ksp_ksp_tutorials-ex49_hypre_nullspace >> TEST arch-linux2-c-opt-ompi/tests/counts/ts_tutorials-ex18_p1p1_xper_ref.counts >> ok ts_tutorials-ex18_p1p1_xper_ref >> ok diff-ts_tutorials-ex18_p1p1_xper_ref >> TEST arch-linux2-c-opt-ompi/tests/counts/ts_tutorials-ex18_p1p1_xyper_ref.counts >> ok ts_tutorials-ex18_p1p1_xyper_ref >> ok diff-ts_tutorials-ex18_p1p1_xyper_ref >> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre.counts >> ok snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre >> ok diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre >> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex64_1.counts >> ok ksp_ksp_tutorials-ex64_1 # SKIP PETSC_HAVE_SUPERLU_DIST requirement not met >> >>> On 3 Mar 2021, at 6:21 AM, Eric Chamberland <Eric.Chamberland(a)giref.ulaval.ca <mailto:[email protected]>> wrote: >>> >>> Just started a discussion on the side: >>> >>> https://community.intel.com/t5/Intel-oneAPI-Math-Kernel-Library/Intel-MKL-L… <https://community.intel.com/t5/Intel-oneAPI-Math-Kernel-Library/Intel-MKL-L…> >>> Eric >>> >>> On 2021-03-02 3:50 p.m., Pierre Jolivet wrote: >>>> Hello Eric, >>>> src/mat/tests/ex237.c is a recent test with some code paths that should be disabled for “old” MKL versions. It’s tricky to check directly in the source (we do check in BuildSystem) because there is no such thing as PETSC_PKG_MKL_VERSION_LT, but I guess we can change if defined(PETSC_HAVE_MKL) to if defined(PETSC_HAVE_MKL) && defined(PETSC_HAVE_MKL_SPARSE_OPTIMIZE), I’ll make a MR, thanks for reporting this. >>>> >>>> For the other issues, I’m sensing this is a problem with gomp + intel_gnu_thread, but this is pure speculation… sorry. >>>> I’ll try to reproduce some of these problems if you are not given a more meaningful answer. >>>> >>>> Thanks, >>>> Pierre >>>> >>>>> On 2 Mar 2021, at 9:14 PM, Eric Chamberland <Eric.Chamberland(a)giref.ulaval.ca <mailto:[email protected]>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> It all started when I wanted to test PETSC/CUDA compatibility for our code. >>>>> >>>>> I had to activate --with-openmp to configure with --with-cuda=1 successfully. >>>>> >>>>> I then saw that PETSC_HAVE_OPENMP is used at least in MUMPS (and some other places). >>>>> >>>>> So, I configured and tested petsc with openmp activated, without CUDA. >>>>> >>>>> The first thing I see is that our code CI pipelines now fails for many tests. >>>>> >>>>> After looking deeper, it seems that PETSc itself fails many tests when I activate openmp! >>>>> >>>>> Here are all the configurations I have results for, after/before activating OpenMP for PETSc: >>>>> ============================================================================== >>>>> >>>>> ============================================================================== >>>>> >>>>> For petsc/master + OpenMPI 4.0.4 + MKL 2019.4.243: >>>>> >>>>> With OpenMP=1 >>>>> >>>>> https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.03.02.02h00m02s_m… <https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.03.02.02h00m02s_m…> >>>>> https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.03.02.02h00m02s_c… <https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.03.02.02h00m02s_c…> >>>>> # ------------- >>>>> # Summary >>>>> # ------------- >>>>> # FAILED snes_tutorials-ex12_quad_hpddm_reuse_baij diff-ksp_ksp_tests-ex33_superlu_dist_2 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 ksp_ksp_tutorials-ex50_tut_2 diff-ksp_ksp_tests-ex33_superlu_dist diff-snes_tutorials-ex56_hypre snes_tutorials-ex17_3d_q3_trig_elas snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij ksp_ksp_tutorials-ex5_superlu_dist_3 ksp_ksp_tutorials-ex5f_superlu_dist snes_tutorials-ex12_tri_parmetis_hpddm_baij diff-snes_tutorials-ex19_tut_3 mat_tests-ex242_3 snes_tutorials-ex17_3d_q3_trig_vlap ksp_ksp_tutorials-ex5f_superlu_dist_3 snes_tutorials-ex19_superlu_dist diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre diff-ksp_ksp_tutorials-ex49_hypre_nullspace ts_tutorials-ex18_p1p1_xper_ref ts_tutorials-ex18_p1p1_xyper_ref snes_tutorials-ex19_superlu_dist_2 ksp_ksp_tutorials-ex5_superlu_dist_2 diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre ksp_ksp_tutorials-ex64_1 ksp_ksp_tutorials-ex5_superlu_dist ksp_ksp_tutorials-ex5f_superlu_dist_2 >>>>> # success 8275/10003 tests (82.7%) >>>>> # failed 33/10003 tests (0.3%) >>>>> With OpenMP=0 >>>>> >>>>> https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.02.26.02h00m16s_m… <https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.02.26.02h00m16s_m…> >>>>> https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.02.26.02h00m16s_c… <https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.02.26.02h00m16s_c…> >>>>> # ------------- >>>>> # Summary >>>>> # ------------- >>>>> # FAILED tao_constrained_tutorials-tomographyADMM_6 snes_tutorials-ex17_3d_q3_trig_elas mat_tests-ex242_3 snes_tutorials-ex17_3d_q3_trig_vlap tao_leastsquares_tutorials-tomography_1 tao_constrained_tutorials-tomographyADMM_5 >>>>> # success 8262/9983 tests (82.8%) >>>>> # failed 6/9983 tests (0.1%) >>>>> ============================================================================== >>>>> >>>>> ============================================================================== >>>>> >>>>> For OpenMPI 3.1.x/master: >>>>> >>>>> With OpenMP=1: >>>>> >>>>> https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.03.01.22h00m01s_make_test.l… <https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.03.01.22h00m01s_make_test.l…> >>>>> https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.03.01.22h00m01s_configure.l… <https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.03.01.22h00m01s_configure.l…> >>>>> # ------------- >>>>> # Summary >>>>> # ------------- >>>>> # FAILED mat_tests-ex242_3 mat_tests-ex242_2 diff-mat_tests-ex219f_1 diff-dm_tutorials-ex11f90_1 ksp_ksp_tutorials-ex5_superlu_dist_3 diff-ksp_ksp_tutorials-ex49_hypre_nullspace ksp_ksp_tutorials-ex5f_superlu_dist_3 snes_tutorials-ex17_3d_q3_trig_vlap diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre diff-snes_tutorials-ex19_tut_3 diff-snes_tutorials-ex56_hypre diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre tao_leastsquares_tutorials-tomography_1 tao_constrained_tutorials-tomographyADMM_4 tao_constrained_tutorials-tomographyADMM_6 diff-tao_constrained_tutorials-toyf_1 >>>>> # success 8142/9765 tests (83.4%) >>>>> # failed 16/9765 tests (0.2%) >>>>> With OpenMP=0: >>>>> >>>>> https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.02.28.22h00m02s_make_test.l… <https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.02.28.22h00m02s_make_test.l…> >>>>> https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.02.28.22h00m02s_configure.l… <https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.02.28.22h00m02s_configure.l…> >>>>> # ------------- >>>>> # Summary >>>>> # ------------- >>>>> # FAILED mat_tests-ex242_3 mat_tests-ex242_2 diff-mat_tests-ex219f_1 diff-dm_tutorials-ex11f90_1 ksp_ksp_tutorials-ex56_2 snes_tutorials-ex17_3d_q3_trig_vlap tao_leastsquares_tutorials-tomography_1 tao_constrained_tutorials-tomographyADMM_4 diff-tao_constrained_tutorials-toyf_1 >>>>> # success 8151/9767 tests (83.5%) >>>>> # failed 9/9767 tests (0.1%) >>>>> ============================================================================== >>>>> >>>>> ============================================================================== >>>>> >>>>> For OpenMPI 4.0.x/master: >>>>> >>>>> With OpenMP=1: >>>>> >>>>> https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.03.01.20h00m01s_make_test.l… <https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.03.01.20h00m01s_make_test.l…> >>>>> https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.03.01.20h00m01s_configure.l… <https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.03.01.20h00m01s_configure.l…> >>>>> # FAILED snes_tutorials-ex17_3d_q3_trig_elas snes_tutorials-ex19_hypre ksp_ksp_tutorials-ex56_2 tao_leastsquares_tutorials-tomography_1 tao_constrained_tutorials-tomographyADMM_5 mat_tests-ex242_3 ksp_ksp_tutorials-ex55_hypre ksp_ksp_tutorials-ex5_superlu_dist_2 tao_constrained_tutorials-tomographyADMM_6 snes_tutorials-ex56_hypre snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre ksp_ksp_tutorials-ex5f_superlu_dist_3 ksp_ksp_tutorials-ex34_hyprestruct diff-ksp_ksp_tutorials-ex49_hypre_nullspace snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre ksp_ksp_tutorials-ex5f_superlu_dist ksp_ksp_tutorials-ex5f_superlu_dist_2 ksp_ksp_tutorials-ex5_superlu_dist snes_tutorials-ex19_tut_3 snes_tutorials-ex19_superlu_dist ksp_ksp_tutorials-ex50_tut_2 snes_tutorials-ex17_3d_q3_trig_vlap ksp_ksp_tutorials-ex5_superlu_dist_3 snes_tutorials-ex19_superlu_dist_2 tao_constrained_tutorials-tomographyADMM_4 ts_tutorials-ex26_2 >>>>> # success 8125/9753 tests (83.3%) >>>>> # failed 26/9753 tests (0.3%) >>>>> With OpenMP=0 >>>>> >>>>> https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.02.28.20h00m04s_make_test.l… <https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.02.28.20h00m04s_make_test.l…> >>>>> https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.02.28.20h00m04s_configure.l… <https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.02.28.20h00m04s_configure.l…> >>>>> # FAILED mat_tests-ex242_3 >>>>> # success 8174/9777 tests (83.6%) >>>>> # failed 1/9777 tests (0.0%) >>>>> >>>>> ============================================================================== >>>>> >>>>> ============================================================================== >>>>> >>>>> Is that known and normal? >>>>> >>>>> In all cases, I am using MKL and I suspect it may come from there... :/ >>>>> >>>>> I also saw a second problem, "make test" fails to compile petsc examples on older versions of MKL (but that's less important for me, I just upgraded to OneAPI to avoid this, but you may want to know): >>>>> >>>>> https://giref.ulaval.ca/~cmpgiref/dernier_ompi/2021.03.02.02h16m01s_make_te… <https://giref.ulaval.ca/~cmpgiref/dernier_ompi/2021.03.02.02h16m01s_make_te…> >>>>> https://giref.ulaval.ca/~cmpgiref/dernier_ompi/2021.03.02.02h16m01s_configu… <https://giref.ulaval.ca/~cmpgiref/dernier_ompi/2021.03.02.02h16m01s_configu…> >>>>> Thanks, >>>>> >>>>> Eric >>>>> >>>>> -- >>>>> Eric Chamberland, ing., M. Ing >>>>> Professionnel de recherche >>>>> GIREF/Université Laval >>>>> (418) 656-2131 poste 41 22 42 >>>> >>> -- >>> Eric Chamberland, ing., M. Ing >>> Professionnel de recherche >>> GIREF/Université Laval >>> (418) 656-2131 poste 41 22 42 >> > -- > Eric Chamberland, ing., M. Ing > Professionnel de recherche > GIREF/Université Laval > (418) 656-2131 poste 41 22 42

1 0

Re: [petsc-dev] Commit squashing in MR
by Zhang, Hong 03 Mar '21

03 Mar '21

Patrick, I need update petsc manual on DMNetwork, but do not know how to proceed. I tried your suggested steps: 1) go to the docs page you want to edit on docs.petsc.org<http://docs.petsc.org/> 2) select the version you want (usually "main") in the black ReadTheDocs box in the lower right 3) click "edit" in "on GitLab" and make your MR (name the branch with "docs-" to maybe get it to auto-build on ReadTheDocs, label with docs and docs-only) I do not understand 3). Can you give a tutorial demo in next petsc meeting? Hong ________________________________ From: petsc-dev <petsc-dev-bounces(a)mcs.anl.gov> on behalf of Patrick Sanan <patrick.sanan(a)gmail.com> Sent: Wednesday, March 3, 2021 12:23 AM To: Jed Brown <jed(a)jedbrown.org> Cc: Satish Balay via petsc-dev <petsc-dev(a)mcs.anl.gov> Subject: Re: [petsc-dev] Commit squashing in MR The whole section on git in the dev manual needs some attention. (It was moved there in the consolidation of docs we had scattered in various places, but hasn't been expertly updated yet). Ideal, I think, would be to find some good, external instructions and link to them, under the idea that we should only maintain things in our own docs that aren't adequately documented somewhere else. This might not be possible (since we had to create these instructions in the first place). There is a section on squashing but it's currently a bit buried, and the advice in this thread is probably more useful/current https://docs.petsc.org/en/main/developers/integration/#squashing-excessive-… If anyone wants to go in there and quickly update those docs, remember that you can do so all from web interfaces! This workflow still has some wrinkles, but for small changes I still think it's appealing: - go to the docs page you want to edit on docs.petsc.org<http://docs.petsc.org> - select the version you want (usually "main") in the black ReadTheDocs box in the lower right - click "edit" in "on GitLab" and make your MR (name the branch with "docs-" to maybe get it to auto-build on ReadTheDocs, label with docs and docs-only) - if you get feedback on your MR and need to update, or notice a typo, I *think* this will work: - click on the last commit of your new branch - find the offending file - click on "edit at @deadbeef123" - change the branch *back* to your branch in the pulldown - click "edit" - back in your MR, edit to "squash commits" You can get a partial preview with the usual "preview" button, though not everything is interpreted correctly (but for things like links, it works fine). If you want a full preview, you can 1. Build the Sphinx docs locally from your branch, either with - "make sphinx-docs-all LOC=$PETSC_DIR" (you may need to add PYTHON=python3, since this relies on Python 3.3+ for venv) - install the required Python packages yourself (e.g. pip install -r src/docs/sphinx_docs/requirements.txt), go to src/docs/sphinx_docs, run "make html", and look in _build/html 2. Build the Sphinx docs for your branch as a version on ReadTheDocs. There is currently an automation rule there that if your branch name has "docs-" in it, it should build (though I must admit I'm still not completely sure I understand exactly when RTD updates its information from GitLab). Or, if you have access, you can activate a new version yourself. Am 03.03.2021 um 05:32 schrieb Jed Brown <jed(a)jedbrown.org<mailto:[email protected]>>: Satish Balay via petsc-dev <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> writes: On Wed, 3 Mar 2021, Blaise A Bourdin wrote: Hi, This is not technically a petsc question. It would be great to have a short section in the PETSc integration workflow document explaining how to squash commits in a MR for git-impaired developers like me. Anybody wants to pitch in, or explain me how to do this? To squash commits - I use the 'squash' action in 'git rebase -i HASH' and figure out the HASH to use from 'gitk main..branch' [as git rebase requires the commit prior to the first commit of interest] git provides many ways of modifying the branch (and the rebase topic is very generic) so I think its best to rely on proper git docs/tutorials [and its not really specific to petsc workflow] You can do it in one line, without changing the base: git rebase -i $(git merge-base main HEAD) An alternative is git rebase -i main which gives you interactive rebase to replay on top of current 'main'. This does two things at once and changing the base for your branch is not always desirable.

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Eric Chamberland 03 Mar '21

03 Mar '21

Hi Pierre, On 2021-03-03 2:42 a.m., Pierre Jolivet wrote: >> >> If it ends that there is a problem combining MKL + openMP that relies >> on linking configuration for example, should it be a good thing to >> have this (--with-openmp=1) tested into the pipelines (with external >> packages of course)? >> > As Barry said, there is not much (if any) OpenMP in PETSc. > There is however some workers with the MKL (+ Intel compilers) turned > on, but I don’t think we test MKL + GNU compilers (which I feel like > is a very niche combination, hence not really worth testing, IMHO). Ouch, this is my almost my personal working configuration and for most of our users too... and it worked well until I activated the OpenMP thing... We had good reasons to work with g++ or clang++ instead of intel compilers: - It is mandatory to pay to work with an intel compiler (didn't looked at OneAPI licensing yet, but it may have changed?) - No support of Intel compilers with iceccd (slow recompilation) - MKL was freely distributed, so it can be used with any compiler That doesn't mean we don't want to use intel compiler, but maybe we just want to to a specific delivery with it but continue to develop with g++ or clang++ (my personal choice). But I understand it is less straightforward to combine gcc and MKL than using native Intel tool-chain.... >> Does the guys who maintain all these libs are reading petsc-dev? ;) >> > I don’t think they are, but don’t worry, we do forward the appropriate > messages to them :) :) > > About yesterday’s failures… > 1) I cannot reproduce any of the PCHYPRE/PCBDDC/PCHPDDM errors (sorry > I didn’t bother putting the SuperLU_DIST tarball on my cluster) Hmmm, maybe my environment variables may play a role into this? for comparisons considerations, we explicitly set: export MKL_CBWR=COMPATIBLE export MKL_NUM_THREADS=1 but it would be surprising it helps reproduce a problem: they usually stabilize results... > 2) I can reproduce the src/mat/tests/ex242.c error (which explicitly > uses ScaLAPACK, none of the above PC uses it explicitly, except > PCBDDC/PCHPDDM when using MUMPS on “big” problems where root nodes are > factorized using ScaLAPACK, see -mat_mumps_icntl_13) > 3) I’m seeing that both on your machine and mine, PETSc BuildSystem > insist on linking libmkl_blacs_intelmpi_lp64.so even though we supply > explicitly libmkl_blacs_openmpi_lp64.so > This for example yields a wrong Makefile.inc for MUMPS: > $ cat > arch-linux2-c-opt-ompi/externalpackages/MUMPS_5.3.5/Makefile.inc|grep > blacs > SCALAP = […] -lmkl_blacs_openmpi_lp64 > LIBBLAS = […] -lmkl_blacs_intelmpi_lp64 -lgomp -ldl -lpthread -lm […] > > Despite what Barry says, I think PETSc is partially to blame as well > (why use libmkl_blacs_intelmpi_lp64.so even though BuildSystem is > capable of detecting we are using OpenMPI). > I’ll try to fix this to see if it solves 2). Okay, that's a very nice finding!!! Hope it will be "fixable" easily! Merci, Eric > > Thanks, > Pierre > > http://joliv.et/irene-rome-configure.log > <http://joliv.et/irene-rome-configure.log> > $ /usr/bin/gmake -f gmakefile test test-fail=1 > Using MAKEFLAGS: test-fail=1 > TEST > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_baij.counts > ok snes_tutorials-ex12_quad_hpddm_reuse_baij > ok diff-snes_tutorials-ex12_quad_hpddm_reuse_baij > TEST > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex50_tut_2.counts > ok ksp_ksp_tutorials-ex50_tut_2 # SKIP PETSC_HAVE_SUPERLU_DIST > requirement not met > TEST > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_hypre.counts > ok snes_tutorials-ex56_hypre > ok diff-snes_tutorials-ex56_hypre > TEST > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_elas.counts > ok snes_tutorials-ex17_3d_q3_trig_elas > ok diff-snes_tutorials-ex17_3d_q3_trig_elas > TEST > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij.counts > ok snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij > ok diff-snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij > TEST > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_tri_parmetis_hpddm_baij.counts > ok snes_tutorials-ex12_tri_parmetis_hpddm_baij > ok diff-snes_tutorials-ex12_tri_parmetis_hpddm_baij > TEST > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_tut_3.counts > ok snes_tutorials-ex19_tut_3 > ok diff-snes_tutorials-ex19_tut_3 > TEST arch-linux2-c-opt-ompi/tests/counts/mat_tests-ex242_3.counts > not ok mat_tests-ex242_3 # Error code: 137 > #[1]PETSC ERROR: > ------------------------------------------------------------------------ > #[1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > #[1]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > #[1]PETSC ERROR: or see > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > <https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind> > #[1]PETSC ERROR: or try http://valgrind.org <http://valgrind.org> on > GNU/linux and Apple Mac OS X to find memory corruption errors > #[1]PETSC ERROR: configure using --with-debugging=yes, recompile, > link, and run > #[1]PETSC ERROR: to get more information on the crash. > #[1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > #[1]PETSC ERROR: Signal received > #[1]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html > <https://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble > shooting. > #[1]PETSC ERROR: Petsc Development GIT revision: > v3.14.4-733-g7ab9467ef9 GIT Date: 2021-03-02 16:15:11 +0000 > #[2]PETSC ERROR: > ------------------------------------------------------------------------ > #[2]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > #[2]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > #[2]PETSC ERROR: or see > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > <https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind> > #[2]PETSC ERROR: or try http://valgrind.org <http://valgrind.org> on > GNU/linux and Apple Mac OS X to find memory corruption errors > #[2]PETSC ERROR: configure using --with-debugging=yes, recompile, > link, and run > #[2]PETSC ERROR: to get more information on the crash. > #[2]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > #[2]PETSC ERROR: Signal received > #[2]PETSC ERROR: See > https://www.mcs.anl.gov/petsc/documentation/faq.html > <https://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble > shooting. > #[2]PETSC ERROR: Petsc Development GIT revision: > v3.14.4-733-g7ab9467ef9 GIT Date: 2021-03-02 16:15:11 +0000 > #[2]PETSC ERROR: > /ccc/work/cont003/rndm/rndm/petsc/arch-linux2-c-opt-ompi/tests/mat/tests/runex242_3/../ex242 > on a arch-linux2-c-opt-ompi named irene4047 by jolivetp Wed Mar 3 > 08:21:20 2021 > #[2]PETSC ERROR: Configure options --download-hpddm > --download-hpddm-commit=origin/main --download-hypre --download-metis > --download-mumps --download-parmetis --download-ptscotch > --download-slepc --download-slepc-commit=origin/main --download-tetgen > --known-mpi-c-double-complex --known-mpi-int64_t > --known-mpi-long-double --with-avx512-kernels=1 > --with-blaslapack-dir=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64 > --with-cc=mpicc --with-cxx=mpicxx --with-debugging=0 --with-fc=mpifort > --with-fortran-bindings=0 --with-make-np=40 > --with-mkl_cpardiso-dir=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281 > --with-mkl_cpardiso=1 > --with-mkl_pardiso-dir=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl > --with-mkl_pardiso=1 --with-mpiexec=ccc_mprun --with-openmp=1 > --with-packages-download-dir=/ccc/cont003/home/enseeiht/jolivetp/Dude/externalpackages/ > --with-scalapack-include=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/include > --with-scalapack-lib="[/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_scalapack_lp64.so,/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.so]" > --with-scalar-type=real --with-x=0 COPTFLAGS="-O3 -fp-model fast > -mavx2" CXXOPTFLAGS="-O3 -fp-model fast -mavx2" FOPTFLAGS="-O3 > -fp-model fast -mavx2" PETSC_ARCH=arch-linux2-c-opt-ompi > #[2]PETSC ERROR: #1 User provided function() line 0 in unknown file > #[2]PETSC ERROR: Run with -malloc_debug to check if memory corruption > is causing the crash. > #[1]PETSC ERROR: > /ccc/work/cont003/rndm/rndm/petsc/arch-linux2-c-opt-ompi/tests/mat/tests/runex242_3/../ex242 > on a arch-linux2-c-opt-ompi named irene4047 by jolivetp Wed Mar 3 > 08:21:20 2021 > #[1]PETSC ERROR: Configure options --download-hpddm > --download-hpddm-commit=origin/main --download-hypre --download-metis > --download-mumps --download-parmetis --download-ptscotch > --download-slepc --download-slepc-commit=origin/main --download-tetgen > --known-mpi-c-double-complex --known-mpi-int64_t > --known-mpi-long-double --with-avx512-kernels=1 > --with-blaslapack-dir=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64 > --with-cc=mpicc --with-cxx=mpicxx --with-debugging=0 --with-fc=mpifort > --with-fortran-bindings=0 --with-make-np=40 > --with-mkl_cpardiso-dir=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281 > --with-mkl_cpardiso=1 > --with-mkl_pardiso-dir=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl > --with-mkl_pardiso=1 --with-mpiexec=ccc_mprun --with-openmp=1 > --with-packages-download-dir=/ccc/cont003/home/enseeiht/jolivetp/Dude/externalpackages/ > --with-scalapack-include=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/include > --with-scalapack-lib="[/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_scalapack_lp64.so,/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.so]" > --with-scalar-type=real --with-x=0 COPTFLAGS="-O3 -fp-model fast > -mavx2" CXXOPTFLAGS="-O3 -fp-model fast -mavx2" FOPTFLAGS="-O3 > -fp-model fast -mavx2" PETSC_ARCH=arch-linux2-c-opt-ompi > #[1]PETSC ERROR: #1 User provided function() line 0 in unknown file > #[1]PETSC ERROR: Run with -malloc_debug to check if memory corruption > is causing the crash. > #-------------------------------------------------------------------------- > #MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD > #with errorcode 50176059. > # > #NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > #You may or may not see output from other processes, depending on > #exactly when Open MPI kills them. > #-------------------------------------------------------------------------- > #-------------------------------------------------------------------------- > #MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD > #with errorcode 50176059. > # > #NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > #You may or may not see output from other processes, depending on > #exactly when Open MPI kills them. > #-------------------------------------------------------------------------- > #srun: Job step aborted: Waiting up to 302 seconds for job step to finish. > #slurmstepd-irene4047: error: *** STEP 1374176.36 ON irene4047 > CANCELLED AT 2021-03-03T08:21:20 *** > #srun: error: irene4047: task 0: Killed > #srun: error: irene4047: tasks 1-2: Exited with exit code 16 > ok mat_tests-ex242_3 # SKIP Command failed so no diff > TEST > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_vlap.counts > ok snes_tutorials-ex17_3d_q3_trig_vlap > ok diff-snes_tutorials-ex17_3d_q3_trig_vlap > TEST > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre.counts > ok snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre > ok diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre > TEST > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex49_hypre_nullspace.counts > ok ksp_ksp_tutorials-ex49_hypre_nullspace > ok diff-ksp_ksp_tutorials-ex49_hypre_nullspace > TEST > arch-linux2-c-opt-ompi/tests/counts/ts_tutorials-ex18_p1p1_xper_ref.counts > ok ts_tutorials-ex18_p1p1_xper_ref > ok diff-ts_tutorials-ex18_p1p1_xper_ref > TEST > arch-linux2-c-opt-ompi/tests/counts/ts_tutorials-ex18_p1p1_xyper_ref.counts > ok ts_tutorials-ex18_p1p1_xyper_ref > ok diff-ts_tutorials-ex18_p1p1_xyper_ref > TEST > arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre.counts > ok snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre > ok diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre > TEST > arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex64_1.counts > ok ksp_ksp_tutorials-ex64_1 # SKIP PETSC_HAVE_SUPERLU_DIST > requirement not met > >> On 3 Mar 2021, at 6:21 AM, Eric Chamberland >> <Eric.Chamberland(a)giref.ulaval.ca >> <mailto:[email protected]>> wrote: >> >> Just started a discussion on the side: >> >> https://community.intel.com/t5/Intel-oneAPI-Math-Kernel-Library/Intel-MKL-L… >> >> Eric >> >> On 2021-03-02 3:50 p.m., Pierre Jolivet wrote: >>> Hello Eric, >>> src/mat/tests/ex237.c is a recent test with some code paths that >>> should be disabled for “old” MKL versions. It’s tricky to check >>> directly in the source (we do check in BuildSystem) because there is >>> no such thing as PETSC_PKG_MKL_VERSION_LT, but I guess we can change >>> if defined(PETSC_HAVE_MKL) to if defined(PETSC_HAVE_MKL) && >>> defined(PETSC_HAVE_MKL_SPARSE_OPTIMIZE), I’ll make a MR, thanks for >>> reporting this. >>> For the other issues, I’m sensing this is a problem with gomp + >>> intel_gnu_thread, but this is pure speculation… sorry. >>> I’ll try to reproduce some of these problems if you are not given a >>> more meaningful answer. >>> Thanks, >>> Pierre >>>> On 2 Mar 2021, at 9:14 PM, Eric Chamberland >>>> <Eric.Chamberland(a)giref.ulaval.ca >>>> <mailto:[email protected]>> wrote: >>>> >>>> Hi, >>>> >>>> It all started when I wanted to test PETSC/CUDA compatibility for >>>> our code. >>>> >>>> I had to activate --with-openmp to configure with --with-cuda=1 >>>> successfully. >>>> >>>> I then saw that PETSC_HAVE_OPENMP is used at least in MUMPS (and >>>> some other places). >>>> >>>> So, I configured and tested petsc with openmp activated, without CUDA. >>>> >>>> The first thing I see is that our code CI pipelines now fails for >>>> many tests. >>>> >>>> After looking deeper, it seems that PETSc itself fails many tests >>>> when I activate openmp! >>>> >>>> Here are all the configurations I have results for, after/before >>>> activating OpenMP for PETSc: >>>> >>>> ============================================================================== >>>> >>>> ============================================================================== >>>> >>>> For petsc/master + OpenMPI 4.0.4 + MKL 2019.4.243: >>>> >>>> With OpenMP=1 >>>> >>>> https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.03.02.02h00m02s_m… >>>> >>>> https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.03.02.02h00m02s_c… >>>> >>>> # ------------- >>>> # Summary >>>> # ------------- >>>> # FAILED snes_tutorials-ex12_quad_hpddm_reuse_baij diff-ksp_ksp_tests-ex33_superlu_dist_2 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 ksp_ksp_tutorials-ex50_tut_2 diff-ksp_ksp_tests-ex33_superlu_dist diff-snes_tutorials-ex56_hypre snes_tutorials-ex17_3d_q3_trig_elas snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij ksp_ksp_tutorials-ex5_superlu_dist_3 ksp_ksp_tutorials-ex5f_superlu_dist snes_tutorials-ex12_tri_parmetis_hpddm_baij diff-snes_tutorials-ex19_tut_3 mat_tests-ex242_3 snes_tutorials-ex17_3d_q3_trig_vlap ksp_ksp_tutorials-ex5f_superlu_dist_3 snes_tutorials-ex19_superlu_dist diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre diff-ksp_ksp_tutorials-ex49_hypre_nullspace ts_tutorials-ex18_p1p1_xper_ref ts_tutorials-ex18_p1p1_xyper_ref snes_tutorials-ex19_superlu_dist_2 ksp_ksp_tutorials-ex5_superlu_dist_2 diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre ksp_ksp_tutorials-ex64_1 ksp_ksp_tutorials-ex5_superlu_dist ksp_ksp_tutorials-ex5f_superlu_dist_2 >>>> # success 8275/10003 tests (82.7%) >>>> #*failed 33/10003* tests (0.3%) >>>> >>>> With OpenMP=0 >>>> >>>> https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.02.26.02h00m16s_m… >>>> >>>> https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.02.26.02h00m16s_c… >>>> >>>> # ------------- >>>> # Summary >>>> # ------------- >>>> # FAILED tao_constrained_tutorials-tomographyADMM_6 snes_tutorials-ex17_3d_q3_trig_elas mat_tests-ex242_3 snes_tutorials-ex17_3d_q3_trig_vlap tao_leastsquares_tutorials-tomography_1 tao_constrained_tutorials-tomographyADMM_5 >>>> # success 8262/9983 tests (82.8%) >>>> #*failed 6/9983* tests (0.1%) >>>> >>>> ============================================================================== >>>> >>>> ============================================================================== >>>> >>>> For OpenMPI 3.1.x/master: >>>> >>>> With OpenMP=1: >>>> >>>> https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.03.01.22h00m01s_make_test.l… >>>> >>>> https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.03.01.22h00m01s_configure.l… >>>> >>>> # ------------- >>>> # Summary >>>> # ------------- >>>> # FAILED mat_tests-ex242_3 mat_tests-ex242_2 diff-mat_tests-ex219f_1 diff-dm_tutorials-ex11f90_1 ksp_ksp_tutorials-ex5_superlu_dist_3 diff-ksp_ksp_tutorials-ex49_hypre_nullspace ksp_ksp_tutorials-ex5f_superlu_dist_3 snes_tutorials-ex17_3d_q3_trig_vlap diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre diff-snes_tutorials-ex19_tut_3 diff-snes_tutorials-ex56_hypre diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre tao_leastsquares_tutorials-tomography_1 tao_constrained_tutorials-tomographyADMM_4 tao_constrained_tutorials-tomographyADMM_6 diff-tao_constrained_tutorials-toyf_1 >>>> # success 8142/9765 tests (83.4%) >>>> #*failed 16/9765* tests (0.2%) >>>> >>>> With OpenMP=0: >>>> >>>> https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.02.28.22h00m02s_make_test.l… >>>> >>>> https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.02.28.22h00m02s_configure.l… >>>> >>>> # ------------- >>>> # Summary >>>> # ------------- >>>> # FAILED mat_tests-ex242_3 mat_tests-ex242_2 diff-mat_tests-ex219f_1 diff-dm_tutorials-ex11f90_1 ksp_ksp_tutorials-ex56_2 snes_tutorials-ex17_3d_q3_trig_vlap tao_leastsquares_tutorials-tomography_1 tao_constrained_tutorials-tomographyADMM_4 diff-tao_constrained_tutorials-toyf_1 >>>> # success 8151/9767 tests (83.5%) >>>> #*failed 9/9767* tests (0.1%) >>>> >>>> ============================================================================== >>>> >>>> ============================================================================== >>>> >>>> For OpenMPI 4.0.x/master: >>>> >>>> With OpenMP=1: >>>> >>>> https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.03.01.20h00m01s_make_test.l… >>>> >>>> https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.03.01.20h00m01s_configure.l… >>>> >>>> # FAILED snes_tutorials-ex17_3d_q3_trig_elas snes_tutorials-ex19_hypre ksp_ksp_tutorials-ex56_2 tao_leastsquares_tutorials-tomography_1 tao_constrained_tutorials-tomographyADMM_5 mat_tests-ex242_3 ksp_ksp_tutorials-ex55_hypre ksp_ksp_tutorials-ex5_superlu_dist_2 tao_constrained_tutorials-tomographyADMM_6 snes_tutorials-ex56_hypre snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre ksp_ksp_tutorials-ex5f_superlu_dist_3 ksp_ksp_tutorials-ex34_hyprestruct diff-ksp_ksp_tutorials-ex49_hypre_nullspace snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre ksp_ksp_tutorials-ex5f_superlu_dist ksp_ksp_tutorials-ex5f_superlu_dist_2 ksp_ksp_tutorials-ex5_superlu_dist snes_tutorials-ex19_tut_3 snes_tutorials-ex19_superlu_dist ksp_ksp_tutorials-ex50_tut_2 snes_tutorials-ex17_3d_q3_trig_vlap ksp_ksp_tutorials-ex5_superlu_dist_3 snes_tutorials-ex19_superlu_dist_2 tao_constrained_tutorials-tomographyADMM_4 ts_tutorials-ex26_2 >>>> # success 8125/9753 tests (83.3%) >>>> #*failed 26/9753* tests (0.3%) >>>> >>>> With OpenMP=0 >>>> >>>> https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.02.28.20h00m04s_make_test.l… >>>> >>>> https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.02.28.20h00m04s_configure.l… >>>> >>>> # FAILED mat_tests-ex242_3 >>>> # success 8174/9777 tests (83.6%) >>>> #*failed 1/9777* tests (0.0%) >>>> >>>> ============================================================================== >>>> >>>> ============================================================================== >>>> >>>> Is that known and normal? >>>> >>>> In all cases, I am using MKL and I suspect it may come from >>>> there... :/ >>>> >>>> I also saw a second problem, "make test" fails to compile petsc >>>> examples on older versions of MKL (but that's less important for >>>> me, I just upgraded to OneAPI to avoid this, but you may want to know): >>>> >>>> https://giref.ulaval.ca/~cmpgiref/dernier_ompi/2021.03.02.02h16m01s_make_te… >>>> >>>> https://giref.ulaval.ca/~cmpgiref/dernier_ompi/2021.03.02.02h16m01s_configu… >>>> >>>> Thanks, >>>> >>>> Eric >>>> >>>> -- >>>> Eric Chamberland, ing., M. Ing >>>> Professionnel de recherche >>>> GIREF/Université Laval >>>> (418) 656-2131 poste 41 22 42 >>> >> -- >> Eric Chamberland, ing., M. Ing >> Professionnel de recherche >> GIREF/Université Laval >> (418) 656-2131 poste 41 22 42 > -- Eric Chamberland, ing., M. Ing Professionnel de recherche GIREF/Université Laval (418) 656-2131 poste 41 22 42

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Pierre Jolivet 03 Mar '21

03 Mar '21

Here is a fix for PETSc disregarding what kind of MPI implementation is being used with cluster PARDISO: https://gitlab.com/petsc/petsc/-/merge_requests/3678 <https://gitlab.com/petsc/petsc/-/merge_requests/3678> This doesn’t fix src/mat/tests/ex242, whose backtrace looks suspicious to me… (gdb) #0 0x00002ac152c9c6ed in mkl_trans_avx2_mkl_domatcopy2_t () from /ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_avx2.so #1 0x00002ba345400dc7 in mkl_trans_avx2_mkl_domatcopy () from /ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_avx2.so #2 0x00002ba31564fac2 in dmmdatdl_ () from /ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_scalapack_lp64.so #3 0x00002ba3156d70de in PB_Cptran_DC () from /ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_scalapack_lp64.so #4 0x00002ba315740062 in pdtran_ () from /ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_scalapack_lp64.so #5 0x00002ba31370211b in MatTranspose_ScaLAPACK () from /ccc/work/cont003/rndm/rndm/petsc/arch-linux2-c-opt-ompi/lib/libpetsc.so.3.014 I’m now realizing that this is also segfaulting with BLACS IntelMPI, so it’s probably unrelated, and simply some Intel mess. (gdb) #0 0x00002ab818c4f3de in mkl_trans_def_mkl_domatcopy2_t () from /ccc/products/mkl-20.0.4/system/default/20.0.4/mkl/lib/intel64/libmkl_def.so #1 0x00002ab818c51247 in mkl_trans_def_mkl_domatcopy () from /ccc/products/mkl-20.0.4/system/default/20.0.4/mkl/lib/intel64/libmkl_def.so #2 0x00002ab76e0ba062 in dmmdatdl_ () from /ccc/products/mkl-20.0.4/system/default/20.0.4/mkl/lib/intel64/libmkl_scalapack_lp64.so #3 0x00002ab76e14157e in PB_Cptran_DC () from /ccc/products/mkl-20.0.4/system/default/20.0.4/mkl/lib/intel64/libmkl_scalapack_lp64.so #4 0x00002ab76e1aa502 in pdtran_ () from /ccc/products/mkl-20.0.4/system/default/20.0.4/mkl/lib/intel64/libmkl_scalapack_lp64.so #5 0x00002b7f87c833a1 in MatTranspose_ScaLAPACK () from /ccc/work/cont003/rndm/rndm/petsc/arch-linux2-c-opt-impi/lib/libpetsc.so.3.014 (Notice the switch from arch-linux2-c-opt-ompi to arch-linux2-c-opt-impi) Thanks, Pierre > On 3 Mar 2021, at 8:42 AM, Pierre Jolivet <pierre(a)joliv.et> wrote: > > >> If it ends that there is a problem combining MKL + openMP that relies on linking configuration for example, should it be a good thing to have this (--with-openmp=1) tested into the pipelines (with external packages of course)? >> > As Barry said, there is not much (if any) OpenMP in PETSc. > There is however some workers with the MKL (+ Intel compilers) turned on, but I don’t think we test MKL + GNU compilers (which I feel like is a very niche combination, hence not really worth testing, IMHO). > >> Does the guys who maintain all these libs are reading petsc-dev? ;) >> > I don’t think they are, but don’t worry, we do forward the appropriate messages to them :) > > About yesterday’s failures… > 1) I cannot reproduce any of the PCHYPRE/PCBDDC/PCHPDDM errors (sorry I didn’t bother putting the SuperLU_DIST tarball on my cluster) > 2) I can reproduce the src/mat/tests/ex242.c error (which explicitly uses ScaLAPACK, none of the above PC uses it explicitly, except PCBDDC/PCHPDDM when using MUMPS on “big” problems where root nodes are factorized using ScaLAPACK, see -mat_mumps_icntl_13) > 3) I’m seeing that both on your machine and mine, PETSc BuildSystem insist on linking libmkl_blacs_intelmpi_lp64.so even though we supply explicitly libmkl_blacs_openmpi_lp64.so > This for example yields a wrong Makefile.inc for MUMPS: > $ cat arch-linux2-c-opt-ompi/externalpackages/MUMPS_5.3.5/Makefile.inc|grep blacs > SCALAP = […] -lmkl_blacs_openmpi_lp64 > LIBBLAS = […] -lmkl_blacs_intelmpi_lp64 -lgomp -ldl -lpthread -lm […] > > Despite what Barry says, I think PETSc is partially to blame as well (why use libmkl_blacs_intelmpi_lp64.so even though BuildSystem is capable of detecting we are using OpenMPI). > I’ll try to fix this to see if it solves 2). > > Thanks, > Pierre > > http://joliv.et/irene-rome-configure.log <http://joliv.et/irene-rome-configure.log> > $ /usr/bin/gmake -f gmakefile test test-fail=1 > Using MAKEFLAGS: test-fail=1 > TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_baij.counts > ok snes_tutorials-ex12_quad_hpddm_reuse_baij > ok diff-snes_tutorials-ex12_quad_hpddm_reuse_baij > TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex50_tut_2.counts > ok ksp_ksp_tutorials-ex50_tut_2 # SKIP PETSC_HAVE_SUPERLU_DIST requirement not met > TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_hypre.counts > ok snes_tutorials-ex56_hypre > ok diff-snes_tutorials-ex56_hypre > TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_elas.counts > ok snes_tutorials-ex17_3d_q3_trig_elas > ok diff-snes_tutorials-ex17_3d_q3_trig_elas > TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij.counts > ok snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij > ok diff-snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij > TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_tri_parmetis_hpddm_baij.counts > ok snes_tutorials-ex12_tri_parmetis_hpddm_baij > ok diff-snes_tutorials-ex12_tri_parmetis_hpddm_baij > TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_tut_3.counts > ok snes_tutorials-ex19_tut_3 > ok diff-snes_tutorials-ex19_tut_3 > TEST arch-linux2-c-opt-ompi/tests/counts/mat_tests-ex242_3.counts > not ok mat_tests-ex242_3 # Error code: 137 > # [1]PETSC ERROR: ------------------------------------------------------------------------ > # [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > # [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > # [1]PETSC ERROR: or see https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind <https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind> > # [1]PETSC ERROR: or try http://valgrind.org <http://valgrind.org/> on GNU/linux and Apple Mac OS X to find memory corruption errors > # [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > # [1]PETSC ERROR: to get more information on the crash. > # [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > # [1]PETSC ERROR: Signal received > # [1]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html <https://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting. > # [1]PETSC ERROR: Petsc Development GIT revision: v3.14.4-733-g7ab9467ef9 GIT Date: 2021-03-02 16:15:11 +0000 > # [2]PETSC ERROR: ------------------------------------------------------------------------ > # [2]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > # [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > # [2]PETSC ERROR: or see https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind <https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind> > # [2]PETSC ERROR: or try http://valgrind.org <http://valgrind.org/> on GNU/linux and Apple Mac OS X to find memory corruption errors > # [2]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run > # [2]PETSC ERROR: to get more information on the crash. > # [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > # [2]PETSC ERROR: Signal received > # [2]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html <https://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting. > # [2]PETSC ERROR: Petsc Development GIT revision: v3.14.4-733-g7ab9467ef9 GIT Date: 2021-03-02 16:15:11 +0000 > # [2]PETSC ERROR: /ccc/work/cont003/rndm/rndm/petsc/arch-linux2-c-opt-ompi/tests/mat/tests/runex242_3/../ex242 on a arch-linux2-c-opt-ompi named irene4047 by jolivetp Wed Mar 3 08:21:20 2021 > # [2]PETSC ERROR: Configure options --download-hpddm --download-hpddm-commit=origin/main --download-hypre --download-metis --download-mumps --download-parmetis --download-ptscotch --download-slepc --download-slepc-commit=origin/main --download-tetgen --known-mpi-c-double-complex --known-mpi-int64_t --known-mpi-long-double --with-avx512-kernels=1 --with-blaslapack-dir=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64 --with-cc=mpicc --with-cxx=mpicxx --with-debugging=0 --with-fc=mpifort --with-fortran-bindings=0 --with-make-np=40 --with-mkl_cpardiso-dir=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281 --with-mkl_cpardiso=1 --with-mkl_pardiso-dir=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl --with-mkl_pardiso=1 --with-mpiexec=ccc_mprun --with-openmp=1 --with-packages-download-dir=/ccc/cont003/home/enseeiht/jolivetp/Dude/externalpackages/ --with-scalapack-include=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/include --with-scalapack-lib="[/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_scalapack_lp64.so,/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.so]" --with-scalar-type=real --with-x=0 COPTFLAGS="-O3 -fp-model fast -mavx2" CXXOPTFLAGS="-O3 -fp-model fast -mavx2" FOPTFLAGS="-O3 -fp-model fast -mavx2" PETSC_ARCH=arch-linux2-c-opt-ompi > # [2]PETSC ERROR: #1 User provided function() line 0 in unknown file > # [2]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. > # [1]PETSC ERROR: /ccc/work/cont003/rndm/rndm/petsc/arch-linux2-c-opt-ompi/tests/mat/tests/runex242_3/../ex242 on a arch-linux2-c-opt-ompi named irene4047 by jolivetp Wed Mar 3 08:21:20 2021 > # [1]PETSC ERROR: Configure options --download-hpddm --download-hpddm-commit=origin/main --download-hypre --download-metis --download-mumps --download-parmetis --download-ptscotch --download-slepc --download-slepc-commit=origin/main --download-tetgen --known-mpi-c-double-complex --known-mpi-int64_t --known-mpi-long-double --with-avx512-kernels=1 --with-blaslapack-dir=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64 --with-cc=mpicc --with-cxx=mpicxx --with-debugging=0 --with-fc=mpifort --with-fortran-bindings=0 --with-make-np=40 --with-mkl_cpardiso-dir=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281 --with-mkl_cpardiso=1 --with-mkl_pardiso-dir=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl --with-mkl_pardiso=1 --with-mpiexec=ccc_mprun --with-openmp=1 --with-packages-download-dir=/ccc/cont003/home/enseeiht/jolivetp/Dude/externalpackages/ --with-scalapack-include=/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/include --with-scalapack-lib="[/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_scalapack_lp64.so,/ccc/products/mkl-19.0.5.281/intel--19.0.5.281__openmpi--4.0.1/default/19.0.5.281/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.so]" --with-scalar-type=real --with-x=0 COPTFLAGS="-O3 -fp-model fast -mavx2" CXXOPTFLAGS="-O3 -fp-model fast -mavx2" FOPTFLAGS="-O3 -fp-model fast -mavx2" PETSC_ARCH=arch-linux2-c-opt-ompi > # [1]PETSC ERROR: #1 User provided function() line 0 in unknown file > # [1]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. > # -------------------------------------------------------------------------- > # MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD > # with errorcode 50176059. > # > # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > # You may or may not see output from other processes, depending on > # exactly when Open MPI kills them. > # -------------------------------------------------------------------------- > # -------------------------------------------------------------------------- > # MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD > # with errorcode 50176059. > # > # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > # You may or may not see output from other processes, depending on > # exactly when Open MPI kills them. > # -------------------------------------------------------------------------- > # srun: Job step aborted: Waiting up to 302 seconds for job step to finish. > # slurmstepd-irene4047: error: *** STEP 1374176.36 ON irene4047 CANCELLED AT 2021-03-03T08:21:20 *** > # srun: error: irene4047: task 0: Killed > # srun: error: irene4047: tasks 1-2: Exited with exit code 16 > ok mat_tests-ex242_3 # SKIP Command failed so no diff > TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_vlap.counts > ok snes_tutorials-ex17_3d_q3_trig_vlap > ok diff-snes_tutorials-ex17_3d_q3_trig_vlap > TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre.counts > ok snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre > ok diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre > TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex49_hypre_nullspace.counts > ok ksp_ksp_tutorials-ex49_hypre_nullspace > ok diff-ksp_ksp_tutorials-ex49_hypre_nullspace > TEST arch-linux2-c-opt-ompi/tests/counts/ts_tutorials-ex18_p1p1_xper_ref.counts > ok ts_tutorials-ex18_p1p1_xper_ref > ok diff-ts_tutorials-ex18_p1p1_xper_ref > TEST arch-linux2-c-opt-ompi/tests/counts/ts_tutorials-ex18_p1p1_xyper_ref.counts > ok ts_tutorials-ex18_p1p1_xyper_ref > ok diff-ts_tutorials-ex18_p1p1_xyper_ref > TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre.counts > ok snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre > ok diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre > TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex64_1.counts > ok ksp_ksp_tutorials-ex64_1 # SKIP PETSC_HAVE_SUPERLU_DIST requirement not met > >> On 3 Mar 2021, at 6:21 AM, Eric Chamberland <Eric.Chamberland(a)giref.ulaval.ca <mailto:[email protected]>> wrote: >> >> Just started a discussion on the side: >> >> https://community.intel.com/t5/Intel-oneAPI-Math-Kernel-Library/Intel-MKL-L… <https://community.intel.com/t5/Intel-oneAPI-Math-Kernel-Library/Intel-MKL-L…> >> Eric >> >> On 2021-03-02 3:50 p.m., Pierre Jolivet wrote: >>> Hello Eric, >>> src/mat/tests/ex237.c is a recent test with some code paths that should be disabled for “old” MKL versions. It’s tricky to check directly in the source (we do check in BuildSystem) because there is no such thing as PETSC_PKG_MKL_VERSION_LT, but I guess we can change if defined(PETSC_HAVE_MKL) to if defined(PETSC_HAVE_MKL) && defined(PETSC_HAVE_MKL_SPARSE_OPTIMIZE), I’ll make a MR, thanks for reporting this. >>> >>> For the other issues, I’m sensing this is a problem with gomp + intel_gnu_thread, but this is pure speculation… sorry. >>> I’ll try to reproduce some of these problems if you are not given a more meaningful answer. >>> >>> Thanks, >>> Pierre >>> >>>> On 2 Mar 2021, at 9:14 PM, Eric Chamberland <Eric.Chamberland(a)giref.ulaval.ca <mailto:[email protected]>> wrote: >>>> >>>> Hi, >>>> >>>> It all started when I wanted to test PETSC/CUDA compatibility for our code. >>>> >>>> I had to activate --with-openmp to configure with --with-cuda=1 successfully. >>>> >>>> I then saw that PETSC_HAVE_OPENMP is used at least in MUMPS (and some other places). >>>> >>>> So, I configured and tested petsc with openmp activated, without CUDA. >>>> >>>> The first thing I see is that our code CI pipelines now fails for many tests. >>>> >>>> After looking deeper, it seems that PETSc itself fails many tests when I activate openmp! >>>> >>>> Here are all the configurations I have results for, after/before activating OpenMP for PETSc: >>>> ============================================================================== >>>> >>>> ============================================================================== >>>> >>>> For petsc/master + OpenMPI 4.0.4 + MKL 2019.4.243: >>>> >>>> With OpenMP=1 >>>> >>>> https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.03.02.02h00m02s_m… <https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.03.02.02h00m02s_m…> >>>> https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.03.02.02h00m02s_c… <https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.03.02.02h00m02s_c…> >>>> # ------------- >>>> # Summary >>>> # ------------- >>>> # FAILED snes_tutorials-ex12_quad_hpddm_reuse_baij diff-ksp_ksp_tests-ex33_superlu_dist_2 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 ksp_ksp_tutorials-ex50_tut_2 diff-ksp_ksp_tests-ex33_superlu_dist diff-snes_tutorials-ex56_hypre snes_tutorials-ex17_3d_q3_trig_elas snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij ksp_ksp_tutorials-ex5_superlu_dist_3 ksp_ksp_tutorials-ex5f_superlu_dist snes_tutorials-ex12_tri_parmetis_hpddm_baij diff-snes_tutorials-ex19_tut_3 mat_tests-ex242_3 snes_tutorials-ex17_3d_q3_trig_vlap ksp_ksp_tutorials-ex5f_superlu_dist_3 snes_tutorials-ex19_superlu_dist diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre diff-ksp_ksp_tutorials-ex49_hypre_nullspace ts_tutorials-ex18_p1p1_xper_ref ts_tutorials-ex18_p1p1_xyper_ref snes_tutorials-ex19_superlu_dist_2 ksp_ksp_tutorials-ex5_superlu_dist_2 diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre ksp_ksp_tutorials-ex64_1 ksp_ksp_tutorials-ex5_superlu_dist ksp_ksp_tutorials-ex5f_superlu_dist_2 >>>> # success 8275/10003 tests (82.7%) >>>> # failed 33/10003 tests (0.3%) >>>> With OpenMP=0 >>>> >>>> https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.02.26.02h00m16s_m… <https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.02.26.02h00m16s_m…> >>>> https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.02.26.02h00m16s_c… <https://giref.ulaval.ca/~cmpgiref/petsc-master-debug/2021.02.26.02h00m16s_c…> >>>> # ------------- >>>> # Summary >>>> # ------------- >>>> # FAILED tao_constrained_tutorials-tomographyADMM_6 snes_tutorials-ex17_3d_q3_trig_elas mat_tests-ex242_3 snes_tutorials-ex17_3d_q3_trig_vlap tao_leastsquares_tutorials-tomography_1 tao_constrained_tutorials-tomographyADMM_5 >>>> # success 8262/9983 tests (82.8%) >>>> # failed 6/9983 tests (0.1%) >>>> ============================================================================== >>>> >>>> ============================================================================== >>>> >>>> For OpenMPI 3.1.x/master: >>>> >>>> With OpenMP=1: >>>> >>>> https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.03.01.22h00m01s_make_test.l… <https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.03.01.22h00m01s_make_test.l…> >>>> https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.03.01.22h00m01s_configure.l… <https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.03.01.22h00m01s_configure.l…> >>>> # ------------- >>>> # Summary >>>> # ------------- >>>> # FAILED mat_tests-ex242_3 mat_tests-ex242_2 diff-mat_tests-ex219f_1 diff-dm_tutorials-ex11f90_1 ksp_ksp_tutorials-ex5_superlu_dist_3 diff-ksp_ksp_tutorials-ex49_hypre_nullspace ksp_ksp_tutorials-ex5f_superlu_dist_3 snes_tutorials-ex17_3d_q3_trig_vlap diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre diff-snes_tutorials-ex19_tut_3 diff-snes_tutorials-ex56_hypre diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre tao_leastsquares_tutorials-tomography_1 tao_constrained_tutorials-tomographyADMM_4 tao_constrained_tutorials-tomographyADMM_6 diff-tao_constrained_tutorials-toyf_1 >>>> # success 8142/9765 tests (83.4%) >>>> # failed 16/9765 tests (0.2%) >>>> With OpenMP=0: >>>> >>>> https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.02.28.22h00m02s_make_test.l… <https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.02.28.22h00m02s_make_test.l…> >>>> https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.02.28.22h00m02s_configure.l… <https://giref.ulaval.ca/~cmpgiref/ompi_3.x/2021.02.28.22h00m02s_configure.l…> >>>> # ------------- >>>> # Summary >>>> # ------------- >>>> # FAILED mat_tests-ex242_3 mat_tests-ex242_2 diff-mat_tests-ex219f_1 diff-dm_tutorials-ex11f90_1 ksp_ksp_tutorials-ex56_2 snes_tutorials-ex17_3d_q3_trig_vlap tao_leastsquares_tutorials-tomography_1 tao_constrained_tutorials-tomographyADMM_4 diff-tao_constrained_tutorials-toyf_1 >>>> # success 8151/9767 tests (83.5%) >>>> # failed 9/9767 tests (0.1%) >>>> ============================================================================== >>>> >>>> ============================================================================== >>>> >>>> For OpenMPI 4.0.x/master: >>>> >>>> With OpenMP=1: >>>> >>>> https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.03.01.20h00m01s_make_test.l… <https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.03.01.20h00m01s_make_test.l…> >>>> https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.03.01.20h00m01s_configure.l… <https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.03.01.20h00m01s_configure.l…> >>>> # FAILED snes_tutorials-ex17_3d_q3_trig_elas snes_tutorials-ex19_hypre ksp_ksp_tutorials-ex56_2 tao_leastsquares_tutorials-tomography_1 tao_constrained_tutorials-tomographyADMM_5 mat_tests-ex242_3 ksp_ksp_tutorials-ex55_hypre ksp_ksp_tutorials-ex5_superlu_dist_2 tao_constrained_tutorials-tomographyADMM_6 snes_tutorials-ex56_hypre snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre ksp_ksp_tutorials-ex5f_superlu_dist_3 ksp_ksp_tutorials-ex34_hyprestruct diff-ksp_ksp_tutorials-ex49_hypre_nullspace snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre ksp_ksp_tutorials-ex5f_superlu_dist ksp_ksp_tutorials-ex5f_superlu_dist_2 ksp_ksp_tutorials-ex5_superlu_dist snes_tutorials-ex19_tut_3 snes_tutorials-ex19_superlu_dist ksp_ksp_tutorials-ex50_tut_2 snes_tutorials-ex17_3d_q3_trig_vlap ksp_ksp_tutorials-ex5_superlu_dist_3 snes_tutorials-ex19_superlu_dist_2 tao_constrained_tutorials-tomographyADMM_4 ts_tutorials-ex26_2 >>>> # success 8125/9753 tests (83.3%) >>>> # failed 26/9753 tests (0.3%) >>>> With OpenMP=0 >>>> >>>> https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.02.28.20h00m04s_make_test.l… <https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.02.28.20h00m04s_make_test.l…> >>>> https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.02.28.20h00m04s_configure.l… <https://giref.ulaval.ca/~cmpgiref/ompi_4.x/2021.02.28.20h00m04s_configure.l…> >>>> # FAILED mat_tests-ex242_3 >>>> # success 8174/9777 tests (83.6%) >>>> # failed 1/9777 tests (0.0%) >>>> >>>> ============================================================================== >>>> >>>> ============================================================================== >>>> >>>> Is that known and normal? >>>> >>>> In all cases, I am using MKL and I suspect it may come from there... :/ >>>> >>>> I also saw a second problem, "make test" fails to compile petsc examples on older versions of MKL (but that's less important for me, I just upgraded to OneAPI to avoid this, but you may want to know): >>>> >>>> https://giref.ulaval.ca/~cmpgiref/dernier_ompi/2021.03.02.02h16m01s_make_te… <https://giref.ulaval.ca/~cmpgiref/dernier_ompi/2021.03.02.02h16m01s_make_te…> >>>> https://giref.ulaval.ca/~cmpgiref/dernier_ompi/2021.03.02.02h16m01s_configu… <https://giref.ulaval.ca/~cmpgiref/dernier_ompi/2021.03.02.02h16m01s_configu…> >>>> Thanks, >>>> >>>> Eric >>>> >>>> -- >>>> Eric Chamberland, ing., M. Ing >>>> Professionnel de recherche >>>> GIREF/Université Laval >>>> (418) 656-2131 poste 41 22 42 >>> >> -- >> Eric Chamberland, ing., M. Ing >> Professionnel de recherche >> GIREF/Université Laval >> (418) 656-2131 poste 41 22 42 >

1 0

← Newer
1
...
19
20
21
22
23
24
25
26
Older →

Jump to page: