April 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] [EXTERNAL] Re: building on Fugaku
by Mark Adams 20 Apr '21

20 Apr '21

I settled on this: . /vol0004/apps/oss/spack/share/spack/setup-env.sh spack load [email protected]%[email protected] arch=linux-rhel8-a64fx spack load [email protected]%[email protected] arch=linux-rhel8-a64fx 'CC=mpicc', 'CXX=mpiCC', 'FC=mpif90', 'COPTFLAGS=-Ofast -march=armv8.2-a+sve -msve-vector-bits=512', 'CXXOPTFLAGS=-Ofast -march=armv8.2-a+sve -msve-vector-bits=512', Kokkos suggested this and I noticed that it helped: export OMP_PROC_BIND=spread export OMP_PLACES=threads I am getting great thread scaling but I seem to get no vectorization. I discussed this with Kokkos (Trott) today and he is not surprised. Auto vectorization is fragile. On Mon, Apr 19, 2021 at 8:26 PM Sreepathi, Sarat <sarat(a)ornl.gov> wrote: > My turn: did you folks figure out tips for performant hybrid MPI+OMP core > binding? I tried some from the documentation but that didn’t seem to help. > > > > -Sarat. > > > > *From:* Sreepathi, Sarat > *Sent:* Friday, April 16, 2021 3:02 PM > *To:* Mark Adams <mfadams(a)lbl.gov>; petsc-dev <petsc-dev(a)mcs.anl.gov> > *Cc:* Satish Balay <balay(a)mcs.anl.gov> > *Subject:* RE: [petsc-dev] [EXTERNAL] Re: building on Fugaku > > > > It’s 48 cores but there are 4 NUMA domains (CMGs). So, you may want to > experiment in hybrid mode (4x12 etc.) if possible. > > > > -Sarat. > > > > *From:* Mark Adams <mfadams(a)lbl.gov> > *Sent:* Friday, April 16, 2021 2:10 PM > *To:* petsc-dev <petsc-dev(a)mcs.anl.gov> > *Cc:* Satish Balay <balay(a)mcs.anl.gov>; Sreepathi, Sarat <sarat(a)ornl.gov> > *Subject:* Re: [petsc-dev] [EXTERNAL] Re: building on Fugaku > > > > Sarat, is there anything special that you do for Kokkos - OpenMP? > > > > Just set OMP_NUM_THREADS=48 ? > > > > Also, I am confused about the number of cores here. Is 48 or 64 per > node/socket? > > > > On Fri, Apr 16, 2021 at 2:03 PM Mark Adams <mfadams(a)lbl.gov> wrote: > > Cool, I have it running too. Need to add Sarat's flags and test ex2. > > > > On Fri, Apr 16, 2021 at 1:57 PM Satish Balay via petsc-dev < > petsc-dev(a)mcs.anl.gov> wrote: > > Mark, > > The following build works for me: > > Satish > > ---- > > pjsub --interact -L "node=1" -L "rscunit=rscunit_ft01" -L "elapse=1:00:00" > --sparam "wait-time=1200" > > . /vol0004/apps/oss/spack/share/spack/setup-env.sh > spack load fujitsu-mpi%gcc > spack load gcc(a)10.2.0 arch=linux-rhel8-a64fx > ./configure COPTFLAGS='-Ofast -march=armv8.2-a+sve -msve-vector-bits=512' > CXXOPTFLAGS='-Ofast -march=armv8.2-a+sve -msve-vector-bits=512' > FOPTFLAGS='-Ofast -march=armv8.2-a+sve -msve-vector-bits=512' > --with-openmp=1 --download-p4est --download-zlib --download-kokkos > --download-kokkos-kernels --download-kokkos-commit=origin/develop > --download-kokkos-kernels-commit=origin/develop > '--download-kokkos-cmake-arguments=-DBUILD_TESTING=OFF > -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON' > --download-cmake= > https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1.tar… > --download-fblaslapack=1 > make PETSC_DIR=/vol0004/ra010009/a04201/petsc.z > PETSC_ARCH=arch-linux-c-debug all > > > To test - redo job allocation using max-proc-per-node: > > login6$ pjsub --interact -L "node=1" -L "rscunit=rscunit_ft01" -L > "elapse=1:00:00" --sparam "wait-time=1200" --mpi "max-proc-per-node=16" > > [a04201@c31-3201c petsc.z]$ . > /vol0004/apps/oss/spack/share/spack/setup-env.sh > [a04201@c31-3201c petsc.z]$ spack load fujitsu-mpi%gcc > [a04201@c31-3201c petsc.z]$ spack load gcc(a)10.2.0 arch=linux-rhel8-a64fx > [a04201@c31-3201c petsc.z]$ make check > Running check examples to verify correct installation > Using PETSC_DIR=/vol0004/ra010009/a04201/petsc.z and > PETSC_ARCH=arch-linux-c-debug > C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process > C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes > C/C++ example src/snes/tutorials/ex3k run successfully with kokkos-kernels > Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process > Completed test examples > [a04201@c31-3201c petsc.z]$ > >

1 0

Re: [petsc-dev] [EXTERNAL] Re: building on Fugaku
by Sreepathi, Sarat 19 Apr '21

19 Apr '21

My turn: did you folks figure out tips for performant hybrid MPI+OMP core binding? I tried some from the documentation but that didn’t seem to help. -Sarat. From: Sreepathi, Sarat Sent: Friday, April 16, 2021 3:02 PM To: Mark Adams <mfadams(a)lbl.gov>; petsc-dev <petsc-dev(a)mcs.anl.gov> Cc: Satish Balay <balay(a)mcs.anl.gov> Subject: RE: [petsc-dev] [EXTERNAL] Re: building on Fugaku It’s 48 cores but there are 4 NUMA domains (CMGs). So, you may want to experiment in hybrid mode (4x12 etc.) if possible. -Sarat. From: Mark Adams <mfadams(a)lbl.gov<mailto:[email protected]>> Sent: Friday, April 16, 2021 2:10 PM To: petsc-dev <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> Cc: Satish Balay <balay(a)mcs.anl.gov<mailto:[email protected]>>; Sreepathi, Sarat <sarat(a)ornl.gov<mailto:[email protected]>> Subject: Re: [petsc-dev] [EXTERNAL] Re: building on Fugaku Sarat, is there anything special that you do for Kokkos - OpenMP? Just set OMP_NUM_THREADS=48 ? Also, I am confused about the number of cores here. Is 48 or 64 per node/socket? On Fri, Apr 16, 2021 at 2:03 PM Mark Adams <mfadams(a)lbl.gov<mailto:[email protected]>> wrote: Cool, I have it running too. Need to add Sarat's flags and test ex2. On Fri, Apr 16, 2021 at 1:57 PM Satish Balay via petsc-dev <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> wrote: Mark, The following build works for me: Satish ---- pjsub --interact -L "node=1" -L "rscunit=rscunit_ft01" -L "elapse=1:00:00" --sparam "wait-time=1200" . /vol0004/apps/oss/spack/share/spack/setup-env.sh spack load fujitsu-mpi%gcc spack load gcc(a)10.2.0<mailto:[email protected]> arch=linux-rhel8-a64fx ./configure COPTFLAGS='-Ofast -march=armv8.2-a+sve -msve-vector-bits=512' CXXOPTFLAGS='-Ofast -march=armv8.2-a+sve -msve-vector-bits=512' FOPTFLAGS='-Ofast -march=armv8.2-a+sve -msve-vector-bits=512' --with-openmp=1 --download-p4est --download-zlib --download-kokkos --download-kokkos-kernels --download-kokkos-commit=origin/develop --download-kokkos-kernels-commit=origin/develop '--download-kokkos-cmake-arguments=-DBUILD_TESTING=OFF -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON' --download-cmake=https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1.tar.gz --download-fblaslapack=1 make PETSC_DIR=/vol0004/ra010009/a04201/petsc.z PETSC_ARCH=arch-linux-c-debug all To test - redo job allocation using max-proc-per-node: login6$ pjsub --interact -L "node=1" -L "rscunit=rscunit_ft01" -L "elapse=1:00:00" --sparam "wait-time=1200" --mpi "max-proc-per-node=16" [a04201@c31-3201c petsc.z]$ . /vol0004/apps/oss/spack/share/spack/setup-env.sh [a04201@c31-3201c petsc.z]$ spack load fujitsu-mpi%gcc [a04201@c31-3201c petsc.z]$ spack load gcc(a)10.2.0<mailto:[email protected]> arch=linux-rhel8-a64fx [a04201@c31-3201c petsc.z]$ make check Running check examples to verify correct installation Using PETSC_DIR=/vol0004/ra010009/a04201/petsc.z and PETSC_ARCH=arch-linux-c-debug C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes C/C++ example src/snes/tutorials/ex3k run successfully with kokkos-kernels Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process Completed test examples [a04201@c31-3201c petsc.z]$

1 0

Re: [petsc-dev] how do people manage projects that require two or more PETSc branches
by Barry Smith 18 Apr '21

18 Apr '21

But often one may have the two (or more branches already) and even started MR on them so just using a single branch and splitting it appropriately after the fact might not work. Plus who has the patience to label each commit properly into which MR it will belong when they make each commit and not make a bad commit that includes code that belongs in multiple MR branches. I am trying Jed's approach. My sanity is nearly gone. > On Apr 18, 2021, at 11:56 AM, Satish Balay <balay(a)mcs.anl.gov> wrote: > > On Sat, 17 Apr 2021, Barry Smith wrote: > >> >> Say I have a project (a code whose source is not in the PETSc repository) that requires code from two PETSc branches (that may or may not yet be MR) but I do not want to make a single PETSc branch (since it would be incoherent). > > The primary issue is managing the dependencies between changes (commits, features). And when making a new change - (for each new change) evaluate and update, reorder the prior commits as needed. > > Attempting to avoid this usually results in a bad commit history - and I've been trying to avoid in petsc history. [currently a single merge commit - which is not cohesive - or easily debuggable is what I'm falling back on] > > Any work-flow below won't be able to mask the above problem easily. > > If the above problem is managed (requires constant rebasing/collapsing/rerordering/moving-commits etc] - then the workflow below is just book-keeping.. [one can aswell use a single branch and split into 2 branches when the features are complete] > > Satish > > >> I may possibly be needing to add code to both of the PETSc branches as I develop the project. I also need to do development sometimes on different machines. >> >> I can do >> >> git checkout branch1 >> git checkout -b combinedbranch >> git merge branch2 >> configure >> make >> use for a while >> >> but now I need to fix or add something to branch1 (which, of course, will be an iterative process as I write the code and fix it) and then add something in branch2. >> >> Doing the above procedure over and over again is tedious and prone to produce errors, editing in the wrong branch etc. >> >> Does anyone have advice on good work flows for this situation? >> >> Barry >> >

1 0

Re: [petsc-dev] how do people manage projects that require two or more PETSc branches
by Satish Balay 18 Apr '21

18 Apr '21

On Sat, 17 Apr 2021, Barry Smith wrote: > > Say I have a project (a code whose source is not in the PETSc repository) that requires code from two PETSc branches (that may or may not yet be MR) but I do not want to make a single PETSc branch (since it would be incoherent). The primary issue is managing the dependencies between changes (commits, features). And when making a new change - (for each new change) evaluate and update, reorder the prior commits as needed. Attempting to avoid this usually results in a bad commit history - and I've been trying to avoid in petsc history. [currently a single merge commit - which is not cohesive - or easily debuggable is what I'm falling back on] Any work-flow below won't be able to mask the above problem easily. If the above problem is managed (requires constant rebasing/collapsing/rerordering/moving-commits etc] - then the workflow below is just book-keeping.. [one can aswell use a single branch and split into 2 branches when the features are complete] Satish > I may possibly be needing to add code to both of the PETSc branches as I develop the project. I also need to do development sometimes on different machines. > > I can do > > git checkout branch1 > git checkout -b combinedbranch > git merge branch2 > configure > make > use for a while > > but now I need to fix or add something to branch1 (which, of course, will be an iterative process as I write the code and fix it) and then add something in branch2. > > Doing the above procedure over and over again is tedious and prone to produce errors, editing in the wrong branch etc. > > Does anyone have advice on good work flows for this situation? > > Barry >

1 0

Re: [petsc-dev] how do people manage projects that require two or more PETSc branches
by Matthew Knepley 18 Apr '21

18 Apr '21

On Sat, Apr 17, 2021 at 7:38 PM Jed Brown <jed(a)jedbrown.org> wrote: > One option is to work on the throw-away combined branch as above, keep > your commits for the two topics distinct, then cherry-pick out into the > separate topic branches. > > If you are committing in the topics and respinning the throw-away branch, > I would recommend rerere, so you don't have to repeat conflict resolution. > > https://git-scm.com/docs/git-rerere > This is what happens for PyLith. I work on a few topic branches that are all needed. I keep a branch, knepley/pylith, to hold all of this. In my work branches, I just pile the commits in until everything is working. That way merges are knepley/pylith are fast-forward. Then when these branches pass the CI and are ready to be merged to PETSc, I rebase them and make nice commits, then recreate knepley/pylith. That means everyone else in the project has to destroy the old knepley/pylith and get a new one, which is annoying, but I do not see a better way. It might be nice if git had an ephemeral branch feature, where you could say that some branch is the combination of a few others and every time they change, it changes too. Thanks, Matt > On Sat, Apr 17, 2021, at 5:02 PM, Jacob Faibussowitsch wrote: > > I have done this maybe only a handful of times (and not with petsc) but > take a look at git worktree https://git-scm.com/docs/git-worktree > > The basics of it is: > > 1. cd $PETSC_DIR — or wherever the root folder of your repo of choice is > 2. git worktree add ../myBranchName > > Then this recreates the entire src tree of the current branch in a > separate directory (namely $PETSC_DIR/../myBranchName). Not sure how useful > it is for petsc, given how dependent everything is on $PETSC_DIR but it's > occasionally useful. Its as if you had 2 separate clones of petsc, but they > both share the same .git folder (I think). > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > > On Apr 17, 2021, at 17:24, Barry Smith <bsmith(a)petsc.dev> wrote: > > > Say I have a project (a code whose source is not in the PETSc > repository) that requires code from two PETSc branches (that may or may not > yet be MR) but I do not want to make a single PETSc branch (since it would > be incoherent). I may possibly be needing to add code to both of the PETSc > branches as I develop the project. I also need to do development sometimes > on different machines. > > I can do > > git checkout branch1 > git checkout -b combinedbranch > git merge branch2 > configure > make > use for a while > > but now I need to fix or add something to branch1 (which, of course, will > be an iterative process as I write the code and fix it) and then add > something in branch2. > > Doing the above procedure over and over again is tedious and prone to > produce errors, editing in the wrong branch etc. > > Does anyone have advice on good work flows for this situation? > > Barry > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] how do people manage projects that require two or more PETSc branches
by Jed Brown 17 Apr '21

17 Apr '21

One option is to work on the throw-away combined branch as above, keep your commits for the two topics distinct, then cherry-pick out into the separate topic branches. If you are committing in the topics and respinning the throw-away branch, I would recommend rerere, so you don't have to repeat conflict resolution. https://git-scm.com/docs/git-rerere On Sat, Apr 17, 2021, at 5:02 PM, Jacob Faibussowitsch wrote: > I have done this maybe only a handful of times (and not with petsc) but take a look at git worktree https://git-scm.com/docs/git-worktree > > The basics of it is: > > 1. cd $PETSC_DIR — or wherever the root folder of your repo of choice is > 2. git worktree add ../myBranchName > > Then this recreates the entire src tree of the current branch in a separate directory (namely $PETSC_DIR/../myBranchName). Not sure how useful it is for petsc, given how dependent everything is on $PETSC_DIR but it's occasionally useful. Its as if you had 2 separate clones of petsc, but they both share the same .git folder (I think). > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > >> On Apr 17, 2021, at 17:24, Barry Smith <bsmith(a)petsc.dev> wrote: >> >> >> Say I have a project (a code whose source is not in the PETSc repository) that requires code from two PETSc branches (that may or may not yet be MR) but I do not want to make a single PETSc branch (since it would be incoherent). I may possibly be needing to add code to both of the PETSc branches as I develop the project. I also need to do development sometimes on different machines. >> >> I can do >> >> git checkout branch1 >> git checkout -b combinedbranch >> git merge branch2 >> configure >> make >> use for a while >> >> but now I need to fix or add something to branch1 (which, of course, will be an iterative process as I write the code and fix it) and then add something in branch2. >> >> Doing the above procedure over and over again is tedious and prone to produce errors, editing in the wrong branch etc. >> >> Does anyone have advice on good work flows for this situation? >> >> Barry

1 0

Re: [petsc-dev] how do people manage projects that require two or more PETSc branches
by Jacob Faibussowitsch 17 Apr '21

17 Apr '21

I have done this maybe only a handful of times (and not with petsc) but take a look at git worktree https://git-scm.com/docs/git-worktree <https://git-scm.com/docs/git-worktree> The basics of it is: 1. cd $PETSC_DIR — or wherever the root folder of your repo of choice is 2. git worktree add ../myBranchName Then this recreates the entire src tree of the current branch in a separate directory (namely $PETSC_DIR/../myBranchName). Not sure how useful it is for petsc, given how dependent everything is on $PETSC_DIR but it's occasionally useful. Its as if you had 2 separate clones of petsc, but they both share the same .git folder (I think). Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) > On Apr 17, 2021, at 17:24, Barry Smith <bsmith(a)petsc.dev> wrote: > > > Say I have a project (a code whose source is not in the PETSc repository) that requires code from two PETSc branches (that may or may not yet be MR) but I do not want to make a single PETSc branch (since it would be incoherent). I may possibly be needing to add code to both of the PETSc branches as I develop the project. I also need to do development sometimes on different machines. > > I can do > > git checkout branch1 > git checkout -b combinedbranch > git merge branch2 > configure > make > use for a while > > but now I need to fix or add something to branch1 (which, of course, will be an iterative process as I write the code and fix it) and then add something in branch2. > > Doing the above procedure over and over again is tedious and prone to produce errors, editing in the wrong branch etc. > > Does anyone have advice on good work flows for this situation? > > Barry >

1 0

how do people manage projects that require two or more PETSc branches
by Barry Smith 17 Apr '21

17 Apr '21

Say I have a project (a code whose source is not in the PETSc repository) that requires code from two PETSc branches (that may or may not yet be MR) but I do not want to make a single PETSc branch (since it would be incoherent). I may possibly be needing to add code to both of the PETSc branches as I develop the project. I also need to do development sometimes on different machines. I can do git checkout branch1 git checkout -b combinedbranch git merge branch2 configure make use for a while but now I need to fix or add something to branch1 (which, of course, will be an iterative process as I write the code and fix it) and then add something in branch2. Doing the above procedure over and over again is tedious and prone to produce errors, editing in the wrong branch etc. Does anyone have advice on good work flows for this situation? Barry

1 0

Re: [petsc-dev] Fortran macOS Linkage Warning
by Barry Smith 16 Apr '21

16 Apr '21

Better just to report the error then to make strange workarounds. https://gitlab.com/petsc/petsc/-/merge_requests/3884 <https://gitlab.com/petsc/petsc/-/merge_requests/3884> This complaint about MAOSX_DEPLOYMENT_TARGET has popped up forever. Previously when linking code PETSc ignored all "warning" messages so likely it just worked and you didn't feel the need to monkey with version min stuff. Anyways, hope this works and, if not, we may need some more filtering. Barry > On Apr 16, 2021, at 3:23 PM, Jacob Faibussowitsch <jacob.fai(a)gmail.com> wrote: > >> I am not sure what the point is here. Are you unable to configure? or make all? or make check PETSc? > > At least on my end, none of them are possible without setting MAOSX_DEPLOYMENT_TARGET or --LDFLAGS=-mmacosx-version-min=11.2. Configure fails when it tests fortran linkage capability. > > <configure.log> > > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > >> On Apr 16, 2021, at 15:03, Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >> >> >> I am not sure what the point is here. Are you unable to configure? or make all? or make check PETSc? >> >> If so that should clearly be reported as a bug with all relevant information supplied. >> >> No, we have never and do not ever intend for PETSc users even developers to know or care about MACOSX_DEPLOYMENT_TARGET >> >> Anyways if you are reporting a bug and I am guess the bug correctly the fix is to add "was built for newer macOS version" to the list of filtered link warnings. There will be a bunch of such warnings that will need to be added to the filtering in the next few months but once they are in place the filtering should not be an issue. >> >> >> Barry >> >> >> >>> On Apr 16, 2021, at 6:20 AM, Pierre Jolivet <pierre(a)joliv.et <mailto:[email protected]>> wrote: >>> >>> Barry, >>> That issue + this commit: >>> $ git bisect bad >>> 36c166f2bbb44c2987d8fb7385fbe9e2d1cfb0c8 is the first bad commit >>> commit 36c166f2bbb44c2987d8fb7385fbe9e2d1cfb0c8 >>> Author: Barry Smith <bsmith(a)mcs.anl.gov <mailto:[email protected]>> >>> Date: Sat Apr 10 13:40:10 2021 -0500 >>> makes it actually impossible to ./configure on macOS. >>> Do we really expect all macOS users to know about the variable MACOSX_DEPLOYMENT_TARGET? >>> >>> Thanks, >>> Pierre >>> >>> $ ./configure >>> =============================================================================== >>> Configuring PETSc to compile on your system >>> =============================================================================== >>> TESTING: checkFortranLibraries from config.compilers(config/BuildSystem/config/compilers.py:921) >>> ******************************************************************************* >>> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): >>> ------------------------------------------------------------------------------- >>> Fortran libraries cannot be used with C as linker >>> ******************************************************************************* >>> >>> Executing: mpicc -o /var/folders/_9/0qn1m4z92wzbhz9434vx39fc0000gn/T/petsc-bhav6feq/config.setCompilers/conftest -Wl,-bind_at_load -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first -Wl,-no_compact_unwind -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fno-stack-check -Qunused-arguments -fvisibility=hidden -g3 /var/folders/_9/0qn1m4z92wzbhz9434vx39fc0000gn/T/petsc-bhav6feq/config.setCompilers/conftest.o -lc++ -ldl -lc++ -ldl -Wl,-rpath,/usr/local/Cellar/mpich/3.4.1_1/lib -L/usr/local/Cellar/mpich/3.4.1_1/lib -lmpifort -lmpi -lpmpi -lgfortran -Wl,-rpath,/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -L/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -Wl,-rpath,/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10 -L/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10 -lgfortran -lquadmath -lm >>> Possible ERROR while running linker: >>> stderr: >>> ld: warning: dylib (/usr/local/Cellar/mpich/3.4.1_1/lib/libmpifort.dylib) was built for newer macOS version (11.2) than being linked (11.0) >>> ld: warning: dylib (/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/libgfortran.dylib) was built for newer macOS version (11.2) than being linked (11.0) >>> ld: warning: dylib (/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/libquadmath.dylib) was built for newer macOS version (11.2) than being linked (11.0) >>> ld: warning: dylib (/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/libquadmath.0.dylib) was built for newer macOS version (11.2) than being linked (11.0) >>> compilers: Cannot compile/link C with mpicc. >>> >>>> On 12 Mar 2021, at 6:16 PM, Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >>>> >>>> >>>> You should be able to edit lib/petsc/conf/rules and add it to the filter for .cu.PETSc .kokkos.cxx.PETSc .c.PETSc .cxx.PETSc: and .F.PETSc .F90.PETSc: >>>> >>>> I think it is fine to hide this warning; it is impossible for me to understand anyway :-(. >>>> >>>> Barry >>>> >>>> >>>> >>>> >>>>> On Mar 12, 2021, at 2:44 AM, Pierre Jolivet <pierre(a)joliv.et <mailto:[email protected]>> wrote: >>>>> >>>>> Sorry to dig up this old thread, but this is triggering false positives while make checking for people who don’t want to toy around with MACOSX_DEPLOYMENT_TARGET. >>>>> Could/should we filter this somehow, or have BuildSystem tinker with the environment variable itself? >>>>> >>>>> Thanks, >>>>> Pierre >>>>> >>>>> $ make check >>>>> Running check examples to verify correct installation >>>>> Using PETSC_DIR=/Users/jolivet/repo/petsc and PETSC_ARCH=arch-darwin-c-debug-real >>>>> *******************Error detected during compile or link!******************* >>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> >>>>> /Users/jolivet/repo/petsc/src/snes/tutorials ex19 >>>>> ********************************************************************************* >>>>> mpicc -Wl,-bind_at_load -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first -Wl,-no_compact_unwind -Wall -g -Wall -g -I/Users/jolivet/repo/petsc/include -I/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/include -I/opt/intel/mkl/include ex19.c -Wl,-rpath,/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib -L/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib -Wl,-rpath,/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib -L/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib -Wl,-rpath,/opt/intel/mkl/lib -L/opt/intel/mkl/lib -Wl,-rpath,/opt/intel/lib -L/opt/intel/lib -Wl,-rpath,/usr/local/Cellar/mpich/3.4.1_1/lib -L/usr/local/Cellar/mpich/3.4.1_1/lib -Wl,-rpath,/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -L/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -Wl,-rpath,/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10 -L/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10 -lpetsc -lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lmkl_scalapack_lp64 -lmkl_blacs_mpich_lp64 -lEl -lElSuiteSparse -lpmrrr -lp4est -lsc -liomp5 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lparmmg -lmmg -lmmg3d -lptesmumps -lptscotchparmetis -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -lparmetis -lmetis -ltriangle -ltet -lz -lX11 -lc++ -ldl -lmpifort -lmpi -lpmpi -lgfortran -lquadmath -lm -lc++ -ldl -o ex19 >>>>> ld: warning: dylib (/usr/local/Cellar/mpich/3.4.1_1/lib/libmpifort.dylib) was built for newer macOS version (11.1) than being linked (11.0) >>>>> ld: warning: dylib (/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/libgfortran.dylib) was built for newer macOS version (11.2) than being linked (11.0) >>>>> ld: warning: dylib (/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/libquadmath.dylib) was built for newer macOS version (11.2) than being linked (11.0) >>>>> ld: warning: dylib (/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/libquadmath.0.dylib) was built for newer macOS version (11.2) than being linked (11.0) >>>>> ld: warning: object file (/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib/libcmumps.a(cana_aux.o)) was built for newer macOS version (11.2) than being linked (11.0) >>>>> ld: warning: object file (/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib/libdmumps.a(dana_aux.o)) was built for newer macOS version (11.2) than being linked (11.0) >>>>> ld: warning: object file (/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib/libsmumps.a(sana_aux.o)) was built for newer macOS version (11.2) than being linked (11.0) >>>>> ld: warning: object file (/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib/libzmumps.a(zana_aux.o)) was built for newer macOS version (11.2) than being linked (11.0) >>>>> ld: warning: object file (/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib/libmumps_common.a(front_data_mgt_m.o)) was built for newer macOS version (11.2) than being linked (11.0) >>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process >>>>> >>>>> >>>>>> On 15 Feb 2021, at 6:08 PM, Jacob Faibussowitsch <jacob.fai(a)gmail.com <mailto:[email protected]>> wrote: >>>>>> >>>>>> Yes, this also seems to work on my end. >>>>>> >>>>>> Best regards, >>>>>> >>>>>> Jacob Faibussowitsch >>>>>> (Jacob Fai - booss - oh - vitch) >>>>>> Cell: (312) 694-3391 >>>>>> >>>>>>> On Feb 15, 2021, at 12:05, Satish Balay <balay(a)mcs.anl.gov <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> Does setting MACOSX_DEPLOYMENT_TARGET also work? >>>>>>> >>>>>>> Satish >>>>>>> >>>>>>> On Mon, 15 Feb 2021, Jacob Faibussowitsch wrote: >>>>>>> >>>>>>>> I think this problem ultimately stems from the fact that gcc/g++ and gfortran are not from the same people on macOS. It seems like clang is configured to build for 11.0 target, whilst gcc defaults to newest available target. And indeed if I change my compile command there is no warning: >>>>>>>> >>>>>>>> mkdir -p build && gfortran -c -o ./build/conftest.o conftest.F90 && gcc-10 -o ./build/libconftest.dylib ./build/conftest.o -dynamiclib -undefined dynamic_lookup && rm -rf ./build >>>>>>>> >>>>>>>> Note I now call gcc-10 as the linker which is the __actual__ gcc (not clang in a trench coat). >>>>>>>> >>>>>>>> Best regards, >>>>>>>> >>>>>>>> Jacob Faibussowitsch >>>>>>>> (Jacob Fai - booss - oh - vitch) >>>>>>>> Cell: (312) 694-3391 >>>>>>>> >>>>>>>>> On Feb 15, 2021, at 11:46, Jacob Faibussowitsch <jacob.fai(a)gmail.com <mailto:[email protected]>> wrote: >>>>>>>>> >>>>>>>>> Ok so if I pass -mmacosx-version-min=11.2 to the linker the warning disappears... >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> >>>>>>>>> Jacob Faibussowitsch >>>>>>>>> (Jacob Fai - booss - oh - vitch) >>>>>>>>> Cell: (312) 694-3391 >>>>>>>>> >>>>>>>>>> On Feb 15, 2021, at 11:41, Jacob Faibussowitsch <jacob.fai(a)gmail.com <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> wrote: >>>>>>>>>> >>>>>>>>>> I can reproduce the warning compiling just that simple code you listed. This is the compile command: >>>>>>>>>> >>>>>>>>>> mkdir -p build && gfortran -c -o ./build/conftest.o conftest.F90 && gcc -o ./build/libconftest.dylib ./build/conftest.o -dynamiclib -undefined dynamic_lookup && rm -rf ./build >>>>>>>>>> >>>>>>>>>> And error: >>>>>>>>>> ld: warning: object file (./build/conftest.o) was built for newer macOS version (11.2) than being linked (11.0) >>>>>>>>>> >>>>>>>>>> Note I recently uninstalled and reinstalled command line tools too, which does not seem to have fixed it. >>>>>>>>>> >>>>>>>>>> Best regards, >>>>>>>>>> >>>>>>>>>> Jacob Faibussowitsch >>>>>>>>>> (Jacob Fai - booss - oh - vitch) >>>>>>>>>> Cell: (312) 694-3391 >>>>>>>>>> >>>>>>>>>>> On Feb 15, 2021, at 11:30, Satish Balay <balay(a)mcs.anl.gov <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Here is the compile command and the warning: >>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> ===== Checking linker >>>>>>>>>>> Executing: gfortran -c -o /var/folders/9w/7dlszmmn6q1gd5yf3r9n79bc0000gn/T/petsc-dk9aiv7x/config.setCompilers/conftest.o -I/var/folders/9w/7dlszmmn6q1gd5yf3r9n79bc0000gn/T/petsc-dk9aiv7x/config.setCompilers /var/folders/9w/7dlszmmn6q1gd5yf3r9n79bc0000gn/T/petsc-dk9aiv7x/config.setCompilers/conftest.F90 >>>>>>>>>>> Successful compile: >>>>>>>>>>> Source: >>>>>>>>>>> function foo(a) >>>>>>>>>>> real:: a,x,bar >>>>>>>>>>> common /xx/ x >>>>>>>>>>> x=a >>>>>>>>>>> foo = bar(x) >>>>>>>>>>> end >>>>>>>>>>> Executing: gcc -o /var/folders/9w/7dlszmmn6q1gd5yf3r9n79bc0000gn/T/petsc-dk9aiv7x/config.setCompilers/libconftest.dylib -dynamiclib -single_module -undefined dynamic_lookup -multiply_defined suppress /var/folders/9w/7dlszmmn6q1gd5yf3r9n79bc0000gn/T/petsc-dk9aiv7x/config.setCompilers/conftest.o >>>>>>>>>>> Possible ERROR while running linker: >>>>>>>>>>> stderr: >>>>>>>>>>> ld: warning: object file (/var/folders/9w/7dlszmmn6q1gd5yf3r9n79bc0000gn/T/petsc-dk9aiv7x/config.setCompilers/conftest.o) was built for newer macOS version (11.2) than being linked (11.0) >>>>>>>>>>> Accepted FC PIC code without compiler flag >>>>>>>>>>> ============================================================================================= >>>>>>>>>>> <<<<<< >>>>>>>>>>> >>>>>>>>>>> Its likely due to the way brew built gfortran. You can try compiling a simple test code and see if you can reproduce this warning. >>>>>>>>>>> >>>>>>>>>>> There is a related issue at: >>>>>>>>>>> >>>>>>>>>>> https://github.com/spack/spack/issues/21644 <https://github.com/spack/spack/issues/21644> <https://github.com/spack/spack/issues/21644 <https://github.com/spack/spack/issues/21644>> >>>>>>>>>>> >>>>>>>>>>> Satish >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Sat, 13 Feb 2021, Jacob Faibussowitsch wrote: >>>>>>>>>>> >>>>>>>>>>>> Hello All, >>>>>>>>>>>> >>>>>>>>>>>> I’ve been getting these warnings for weeks, at first I figured it was due to gfortran brew package not being updated for Big Sur, but since it has persisted through a few gfortran brew updates I’m thinking it may be petsc related. This is on clean build on master. >>>>>>>>>>>> >>>>>>>>>>>> ld: warning: object file (master-arch-darwin-c-debug/obj/sys/classes/bag/f2003-src/fsrc/bagenum.o) was built for newer macOS version (11.2) than being linked (11.0) >>>>>>>>>>>> ld: warning: object file (master-arch-darwin-c-debug/obj/sys/f90-src/fsrc/f90_fwrap.o) was built for newer macOS version (11.2) than being linked (11.0) >>>>>>>>>>>> ld: warning: object file (master-arch-darwin-c-debug/obj/sys/objects/f2003-src/fsrc/optionenum.o) was built for newer macOS version (11.2) than being linked (11.0) >>>>>>>>>>>> ld: warning: object file (master-arch-darwin-c-debug/obj/sys/fsrc/somefort.o) was built for newer macOS version (11.2) than being linked (11.0) >>>>>>>>>>>> ld: warning: object file (master-arch-darwin-c-debug/obj/sys/f90-mod/petscsysmod.o) was built for newer macOS version (11.2) than being linked (11.0) >>>>>>>>>>>> >>>>>>>>>>>> Best regards, >>>>>>>>>>>> >>>>>>>>>>>> Jacob Faibussowitsch >>>>>>>>>>>> (Jacob Fai - booss - oh - vitch) >>>>>>>>>>>> Cell: (312) 694-3391 >>>>>> >>>>> >>>> >>> >> >

1 0

Re: [petsc-dev] p4est w/o MPI
by Barry Smith 16 Apr '21

16 Apr '21

Toby, PETSc's MPI-uni has an include file with the MPI names but it attempts to map them all immediately to private PETSc names; the intention is that MPI Uni won't have public MPI symbols in the libraries that can clash with other package's private MPI-uni-like symbols. We may have missed some, of course, that need to be fixed. The idea behind MPI-uni is to live harmoniously and orthogonally with everyone else's MPI-uni. Barry > On Apr 16, 2021, at 3:36 PM, Isaac, Tobin G <tisaac(a)cc.gatech.edu> wrote: > > Understandable, p4est has its own minimal pseudo-mpi interface, I'll work with Satish and the other p4est devs to a simple way to keep the MPIUNI / MPIP4EST from colliding. > > ________________________________________ > From: Barry Smith <bsmith(a)petsc.dev> > Sent: Friday, April 16, 2021 16:10 > To: Isaac, Tobin G > Cc: petsc-dev > Subject: Re: [petsc-dev] p4est w/o MPI > > > Toby, > > It was never our intention that MPIUNI support any other packages besides PETSc/SLEPc as an "mpi substitute". There are just so many fragile aspects to it. I think it is is best if p4est maintains its own non-MPI version (if it wants to have such as thing) and does not have it involve MPIUNI. > > Barry > > > > On Apr 16, 2021, at 12:53 PM, Isaac, Tobin G <tisaac(a)cc.gatech.edu<mailto:[email protected]>> wrote: > > Hmm. So if petsc always compiles mpiuni if --with-mpi=0, then it makes me think it should be possible to drop requires(MPI) from p4est, while still passing `--enable-mpi` to p4est. > > Maybe: if mpiuni doesn't provide some esoteric part of MPI that p4est uses, it will break. > > Toby Isaac, Assistant Professor, GTCSE > > ________________________________ > From: Satish Balay <balay(a)mcs.anl.gov<mailto:[email protected]>> > Sent: Friday, April 16, 2021 1:03:55 PM > To: Isaac, Tobin G <tisaac(a)cc.gatech.edu<mailto:[email protected]>> > Cc: petsc-dev <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> > Subject: Re: [petsc-dev] p4est w/o MPI > > Thanks! Another issue: > > mpiuni provides mpi.h. And I get the following errors when building PETSc. > > /home/balay/petsc/arch-linux-c-debug/include/sc.h:87:2: error: #error "mpi.h is included. Use --enable-mpi." > 87 | #error "mpi.h is included. Use --enable-mpi." > | ^~~~~ > > etc.. > > Satish > > > On Fri, 16 Apr 2021, Isaac, Tobin G wrote: > >> Yes, `--disable-mpi` should be accepted by p4est. >> >> ________________________________________ >> From: Satish Balay <balay(a)mcs.anl.gov<mailto:[email protected]>> >> Sent: Friday, April 16, 2021 11:47 >> To: Satish Balay via petsc-dev >> Cc: Isaac, Tobin G >> Subject: Re: [petsc-dev] p4est w/o MPI >> >> BTW: Is there no --disable-mpi option? >> >> Satish >> >> On Fri, 16 Apr 2021, Satish Balay via petsc-dev wrote: >> >>> MPI pacakge is a bit weird compared to others [as it does this funny thing with MPIUNI]. >>> >>> For regular packages - I guess I would use 'if package.found' >>> >>> I guess I should create an MR with this diff. >>> >>> Satish >>> >>> ---- >>> >>> diff --git a/config/BuildSystem/config/packages/p4est.py b/config/BuildSystem/config/packages/p4est.py >>> index e7108b0ab8..40d6797f02 100644 >>> --- a/config/BuildSystem/config/packages/p4est.py >>> +++ b/config/BuildSystem/config/packages/p4est.py >>> @@ -24,14 +24,16 @@ class Configure(config.package.GNUPackage): >>> self.blasLapack = framework.require('config.packages.BlasLapack',self) >>> self.zlib = framework.require('config.packages.zlib',self) >>> self.memalign = framework.argDB['with-memalign'] >>> - self.deps = [self.mpi,self.blasLapack,self.zlib] >>> + self.deps = [self.blasLapack,self.zlib] >>> + self.odeps = [self.mpi] >>> return >>> >>> def formGNUConfigureArgs(self): >>> args = config.package.GNUPackage.formGNUConfigureArgs(self) >>> if self.argDB['with-p4est-debugging']: >>> args.append('--enable-debug') >>> - args.append('--enable-mpi') >>> + if not self.mpi.usingMPIUni: >>> + args.append('--enable-mpi') >>> args.append('CPPFLAGS="'+self.headers.toStringNoDupes(self.dinclude)+'"') >>> args.append('LIBS="'+self.libraries.toString(self.dlib)+'"') >>> args.append('--enable-memalign='+self.memalign) >>> >>> On Fri, 16 Apr 2021, Isaac, Tobin G wrote: >>> >>>> Matt, this is the configure logic I would like, but I don't know how to do it: >>>> - configure p4est after MPI, but do not require MPI >>>> - if MPI is present, add the '--enable-mpi' flack in formGNUConfigureArgs(), otherwise omit it. >>>> >>>> ________________________________________ >>>> From: Isaac, Tobin G <tisaac(a)cc.gatech.edu<mailto:[email protected]>> >>>> Sent: Friday, April 16, 2021 11:03 >>>> To: Mark Adams; For users of the development version of PETSc >>>> Subject: Re: [petsc-dev] p4est w/o MPI >>>> >>>> p4est has a mode where it can compile without MPI, I don't know if PETSc is using it, will check. >>>> >>>> ________________________________________ >>>> From: petsc-dev <petsc-dev-bounces(a)mcs.anl.gov<mailto:[email protected]>> on behalf of Mark Adams <mfadams(a)lbl.gov<mailto:[email protected]>> >>>> Sent: Friday, April 16, 2021 10:23 >>>> To: For users of the development version of PETSc >>>> Subject: [petsc-dev] p4est w/o MPI >>>> >>>> I don't have MPI (Fugaku w/ gcc) and p4est seems to need it. Is there a work around? >>>> Thanks, >>>> Mark >>>> >>> >> > >

1 0