April 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] p4est w/o MPI
by Satish Balay 16 Apr '21

16 Apr '21

Sure that might work. Also just commenting out the 3 lines for that check got the build going for me. Or a simple change (but a bit intrusive): [balay@pj01 petsc]$ diff -Nru ./arch-linux-c-debug/include/sc.h~ ./arch-linux-c-debug/include/sc.h --- ./arch-linux-c-debug/include/sc.h~ 2021-04-16 12:59:20.803995227 -0500 +++ ./arch-linux-c-debug/include/sc.h 2021-04-16 13:00:06.490993496 -0500 @@ -83,7 +83,7 @@ #ifdef SC_ENABLE_MPI #include <mpi.h> #else -#ifdef MPI_SUCCESS +#if defined(MPI_SUCCESS) && !defined(MPIUNI_H) #error "mpi.h is included. Use --enable-mpi." #endif #endif Also getting test suite errors - Matt might have to check on. [attaching log] Satish On Fri, 16 Apr 2021, Isaac, Tobin G wrote: > Hmm. So if petsc always compiles mpiuni if --with-mpi=0, then it makes me think it should be possible to drop requires(MPI) from p4est, while still passing `--enable-mpi` to p4est. > > Maybe: if mpiuni doesn't provide some esoteric part of MPI that p4est uses, it will break. > > Toby Isaac, Assistant Professor, GTCSE > > ________________________________ > From: Satish Balay <balay(a)mcs.anl.gov> > Sent: Friday, April 16, 2021 1:03:55 PM > To: Isaac, Tobin G <tisaac(a)cc.gatech.edu> > Cc: petsc-dev <petsc-dev(a)mcs.anl.gov> > Subject: Re: [petsc-dev] p4est w/o MPI > > Thanks! Another issue: > > mpiuni provides mpi.h. And I get the following errors when building PETSc. > > /home/balay/petsc/arch-linux-c-debug/include/sc.h:87:2: error: #error "mpi.h is included. Use --enable-mpi." > 87 | #error "mpi.h is included. Use --enable-mpi." > | ^~~~~ > > etc.. > > Satish > > > On Fri, 16 Apr 2021, Isaac, Tobin G wrote: > > > Yes, `--disable-mpi` should be accepted by p4est. > > > > ________________________________________ > > From: Satish Balay <balay(a)mcs.anl.gov> > > Sent: Friday, April 16, 2021 11:47 > > To: Satish Balay via petsc-dev > > Cc: Isaac, Tobin G > > Subject: Re: [petsc-dev] p4est w/o MPI > > > > BTW: Is there no --disable-mpi option? > > > > Satish > > > > On Fri, 16 Apr 2021, Satish Balay via petsc-dev wrote: > > > > > MPI pacakge is a bit weird compared to others [as it does this funny thing with MPIUNI]. > > > > > > For regular packages - I guess I would use 'if package.found' > > > > > > I guess I should create an MR with this diff. > > > > > > Satish > > > > > > ---- > > > > > > diff --git a/config/BuildSystem/config/packages/p4est.py b/config/BuildSystem/config/packages/p4est.py > > > index e7108b0ab8..40d6797f02 100644 > > > --- a/config/BuildSystem/config/packages/p4est.py > > > +++ b/config/BuildSystem/config/packages/p4est.py > > > @@ -24,14 +24,16 @@ class Configure(config.package.GNUPackage): > > > self.blasLapack = framework.require('config.packages.BlasLapack',self) > > > self.zlib = framework.require('config.packages.zlib',self) > > > self.memalign = framework.argDB['with-memalign'] > > > - self.deps = [self.mpi,self.blasLapack,self.zlib] > > > + self.deps = [self.blasLapack,self.zlib] > > > + self.odeps = [self.mpi] > > > return > > > > > > def formGNUConfigureArgs(self): > > > args = config.package.GNUPackage.formGNUConfigureArgs(self) > > > if self.argDB['with-p4est-debugging']: > > > args.append('--enable-debug') > > > - args.append('--enable-mpi') > > > + if not self.mpi.usingMPIUni: > > > + args.append('--enable-mpi') > > > args.append('CPPFLAGS="'+self.headers.toStringNoDupes(self.dinclude)+'"') > > > args.append('LIBS="'+self.libraries.toString(self.dlib)+'"') > > > args.append('--enable-memalign='+self.memalign) > > > > > > On Fri, 16 Apr 2021, Isaac, Tobin G wrote: > > > > > > > Matt, this is the configure logic I would like, but I don't know how to do it: > > > > - configure p4est after MPI, but do not require MPI > > > > - if MPI is present, add the '--enable-mpi' flack in formGNUConfigureArgs(), otherwise omit it. > > > > > > > > ________________________________________ > > > > From: Isaac, Tobin G <tisaac(a)cc.gatech.edu> > > > > Sent: Friday, April 16, 2021 11:03 > > > > To: Mark Adams; For users of the development version of PETSc > > > > Subject: Re: [petsc-dev] p4est w/o MPI > > > > > > > > p4est has a mode where it can compile without MPI, I don't know if PETSc is using it, will check. > > > > > > > > ________________________________________ > > > > From: petsc-dev <petsc-dev-bounces(a)mcs.anl.gov> on behalf of Mark Adams <mfadams(a)lbl.gov> > > > > Sent: Friday, April 16, 2021 10:23 > > > > To: For users of the development version of PETSc > > > > Subject: [petsc-dev] p4est w/o MPI > > > > > > > > I don't have MPI (Fugaku w/ gcc) and p4est seems to need it. Is there a work around? > > > > Thanks, > > > > Mark > > > > > > > > > > >

1 0

Re: [petsc-dev] [EXTERNAL] Re: building on Fugaku
by Mark Adams 16 Apr '21

16 Apr '21

Cool, I have it running too. Need to add Sarat's flags and test ex2. On Fri, Apr 16, 2021 at 1:57 PM Satish Balay via petsc-dev < petsc-dev(a)mcs.anl.gov> wrote: > Mark, > > The following build works for me: > > Satish > > ---- > > pjsub --interact -L "node=1" -L "rscunit=rscunit_ft01" -L "elapse=1:00:00" > --sparam "wait-time=1200" > > . /vol0004/apps/oss/spack/share/spack/setup-env.sh > spack load fujitsu-mpi%gcc > spack load gcc(a)10.2.0 arch=linux-rhel8-a64fx > ./configure COPTFLAGS='-Ofast -march=armv8.2-a+sve -msve-vector-bits=512' > CXXOPTFLAGS='-Ofast -march=armv8.2-a+sve -msve-vector-bits=512' > FOPTFLAGS='-Ofast -march=armv8.2-a+sve -msve-vector-bits=512' > --with-openmp=1 --download-p4est --download-zlib --download-kokkos > --download-kokkos-kernels --download-kokkos-commit=origin/develop > --download-kokkos-kernels-commit=origin/develop > '--download-kokkos-cmake-arguments=-DBUILD_TESTING=OFF > -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON' > --download-cmake= > https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1.tar… > --download-fblaslapack=1 > make PETSC_DIR=/vol0004/ra010009/a04201/petsc.z > PETSC_ARCH=arch-linux-c-debug all > > > To test - redo job allocation using max-proc-per-node: > > login6$ pjsub --interact -L "node=1" -L "rscunit=rscunit_ft01" -L > "elapse=1:00:00" --sparam "wait-time=1200" --mpi "max-proc-per-node=16" > > [a04201@c31-3201c petsc.z]$ . > /vol0004/apps/oss/spack/share/spack/setup-env.sh > [a04201@c31-3201c petsc.z]$ spack load fujitsu-mpi%gcc > [a04201@c31-3201c petsc.z]$ spack load gcc(a)10.2.0 arch=linux-rhel8-a64fx > [a04201@c31-3201c petsc.z]$ make check > Running check examples to verify correct installation > Using PETSC_DIR=/vol0004/ra010009/a04201/petsc.z and > PETSC_ARCH=arch-linux-c-debug > C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process > C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes > C/C++ example src/snes/tutorials/ex3k run successfully with kokkos-kernels > Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process > Completed test examples > [a04201@c31-3201c petsc.z]$ >

1 0

Re: [petsc-dev] [EXTERNAL] Re: building on Fugaku
by Mark Adams 16 Apr '21

16 Apr '21

On Fri, Apr 16, 2021 at 12:31 PM Sreepathi, Sarat <sarat(a)ornl.gov> wrote: > Hi Mark, > > > > I couldn’t follow all the issues in the other thread. Anyway, you are in > luck. > > I have pushed for GNU compiler support with Fujitsu MPI since November > after hitting multiple issues. Finally, got an update from Fugaku helpdesk > earlier this week. > > As you are aware, you will probably get better performance with GNU than > Fujitsu on A64FX. My latest experiments confirmed that **again** for > climate workloads. > > > > See "Users Guide - Language and development environment Ver. 1.02" - > Chapter 4 for details. > > > https://www.fugaku.r-ccs.riken.jp/doc_root/en/user_guides/lang_1.02/GCC_MPI… > > > > Additionally, please note that you will need to `spack load gcc(a)10.2.0` > to load the latest GNU compiler. > > To summarize, you need to do the following on a compute node to get the > requisite environment. > > ``` > > . /vol0004/apps/oss/spack/share/spack/setup-env.sh > > spack load fujitsu-mpi%gcc > > spack load gcc(a)10.2.0 > > ``` > Yes, I go this this AM from Kokkos: . /vol0004/apps/oss/spack/share/spack/setup-env.sh spack load [email protected]%[email protected] arch=linux-rhel8-a64fx spack load [email protected]%[email protected] arch=linux-rhel8-a64fx Looks like the same thing + @8.3.1 arch=linux-rhel8-a64fx It has passed PETSc 'make check' Thanks, > Hope this helps (cc’ing petsc-dev), > Sarat. > > *From:* Mark Adams <mfadams(a)lbl.gov> > *Sent:* Friday, April 16, 2021 10:27 AM > *To:* Sreepathi, Sarat <sarat(a)ornl.gov> > *Subject:* Re: [EXTERNAL] Re: building on Fugaku > > > > Sarat, > > I am making progress. Kokkos is working w/o OMP, but Kokkos Kernels + OMP > is barfing and Kokkos people suggested GNU. > > But one of my third party libraries needs MPI. Do you know of some sort of > MPI library on Fugaku? > > It just has to work with one process. > > Thanks, > > Mark > > > > On Wed, Apr 14, 2021 at 3:21 PM Sreepathi, Sarat <sarat(a)ornl.gov> wrote: > > Hi Mark, > > > > From earlier notes, I used the Kokkos “develop” branch. > > > > cmake -DCMAKE_BUILD_TYPE=Release \ > -DCMAKE_INSTALL_PREFIX=${KOKKOS_SRC_DIR}/install \ > -DBUILD_TESTING=OFF \ > -DKokkos_ENABLE_LIBDL=OFF \ > -DKokkos_ENABLE_OPENMP=ON \ > -DKokkos_ENABLE_SERIAL=ON \ > -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON \ > .. > > -Sarat. > > > > *From:* Mark Adams <mfadams(a)lbl.gov> > *Sent:* Wednesday, April 14, 2021 3:09 PM > *To:* Sreepathi, Sarat <sarat(a)ornl.gov> > *Cc:* Richard Tran Mills <rtmills(a)anl.gov> > *Subject:* [EXTERNAL] Re: building on Fugaku > > > > Sarat, > > > > I have made some progress but am getting a C++ error: > > > > CMake Warning at cmake/kokkos_test_cxx_std.cmake:47 (MESSAGE): > C++14 is not supported as a compiler feature. We will choose custom > flags > for now, but this behavior has been deprecated. Please open an issue at > https://github.com/kokkos/kokkos/issues reporting that Fujitsu failed > for > 14, preferrably including your CMake command. > > > > I am using: > > > > 04:08 adams/landau-gpu-opt= ~/petsc$ mpiFCC --version > FCC (FCC) 4.5.0 20210304 > simulating gcc version 6.1 > > > > Do you have any ideas? > > > > Thanks, > > Mark > > > > On Wed, Apr 14, 2021 at 7:55 AM Mark Adams <mfadams(a)lbl.gov> wrote: > > Hi Sarat, > > > > My colleague, Rich Mills, and I are coming back to Fugaku after a while, > and I wanted to ask you how you build, with Kokkos, on Fugaku. > > I have tried Arm and Fugitzu compilers and was able to get Fugitzu to work > a few months ago, but I wanted to see what you use. Especially for Kokkos. > > Can you share with us your build details? > > > > Thanks, > > Mark > >

1 0

Re: [petsc-dev] [EXTERNAL] Re: building on Fugaku
by Satish Balay 16 Apr '21

16 Apr '21

Mark, The following build works for me: Satish ---- pjsub --interact -L "node=1" -L "rscunit=rscunit_ft01" -L "elapse=1:00:00" --sparam "wait-time=1200" . /vol0004/apps/oss/spack/share/spack/setup-env.sh spack load fujitsu-mpi%gcc spack load gcc(a)10.2.0 arch=linux-rhel8-a64fx ./configure COPTFLAGS='-Ofast -march=armv8.2-a+sve -msve-vector-bits=512' CXXOPTFLAGS='-Ofast -march=armv8.2-a+sve -msve-vector-bits=512' FOPTFLAGS='-Ofast -march=armv8.2-a+sve -msve-vector-bits=512' --with-openmp=1 --download-p4est --download-zlib --download-kokkos --download-kokkos-kernels --download-kokkos-commit=origin/develop --download-kokkos-kernels-commit=origin/develop '--download-kokkos-cmake-arguments=-DBUILD_TESTING=OFF -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON' --download-cmake=https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1.tar.gz --download-fblaslapack=1 make PETSC_DIR=/vol0004/ra010009/a04201/petsc.z PETSC_ARCH=arch-linux-c-debug all To test - redo job allocation using max-proc-per-node: login6$ pjsub --interact -L "node=1" -L "rscunit=rscunit_ft01" -L "elapse=1:00:00" --sparam "wait-time=1200" --mpi "max-proc-per-node=16" [a04201@c31-3201c petsc.z]$ . /vol0004/apps/oss/spack/share/spack/setup-env.sh [a04201@c31-3201c petsc.z]$ spack load fujitsu-mpi%gcc [a04201@c31-3201c petsc.z]$ spack load gcc(a)10.2.0 arch=linux-rhel8-a64fx [a04201@c31-3201c petsc.z]$ make check Running check examples to verify correct installation Using PETSC_DIR=/vol0004/ra010009/a04201/petsc.z and PETSC_ARCH=arch-linux-c-debug C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes C/C++ example src/snes/tutorials/ex3k run successfully with kokkos-kernels Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process Completed test examples [a04201@c31-3201c petsc.z]$

1 0

Re: [petsc-dev] p4est w/o MPI
by Isaac, Tobin G 16 Apr '21

16 Apr '21

Hmm. So if petsc always compiles mpiuni if --with-mpi=0, then it makes me think it should be possible to drop requires(MPI) from p4est, while still passing `--enable-mpi` to p4est. Maybe: if mpiuni doesn't provide some esoteric part of MPI that p4est uses, it will break. Toby Isaac, Assistant Professor, GTCSE ________________________________ From: Satish Balay <balay(a)mcs.anl.gov> Sent: Friday, April 16, 2021 1:03:55 PM To: Isaac, Tobin G <tisaac(a)cc.gatech.edu> Cc: petsc-dev <petsc-dev(a)mcs.anl.gov> Subject: Re: [petsc-dev] p4est w/o MPI Thanks! Another issue: mpiuni provides mpi.h. And I get the following errors when building PETSc. /home/balay/petsc/arch-linux-c-debug/include/sc.h:87:2: error: #error "mpi.h is included. Use --enable-mpi." 87 | #error "mpi.h is included. Use --enable-mpi." | ^~~~~ etc.. Satish On Fri, 16 Apr 2021, Isaac, Tobin G wrote: > Yes, `--disable-mpi` should be accepted by p4est. > > ________________________________________ > From: Satish Balay <balay(a)mcs.anl.gov> > Sent: Friday, April 16, 2021 11:47 > To: Satish Balay via petsc-dev > Cc: Isaac, Tobin G > Subject: Re: [petsc-dev] p4est w/o MPI > > BTW: Is there no --disable-mpi option? > > Satish > > On Fri, 16 Apr 2021, Satish Balay via petsc-dev wrote: > > > MPI pacakge is a bit weird compared to others [as it does this funny thing with MPIUNI]. > > > > For regular packages - I guess I would use 'if package.found' > > > > I guess I should create an MR with this diff. > > > > Satish > > > > ---- > > > > diff --git a/config/BuildSystem/config/packages/p4est.py b/config/BuildSystem/config/packages/p4est.py > > index e7108b0ab8..40d6797f02 100644 > > --- a/config/BuildSystem/config/packages/p4est.py > > +++ b/config/BuildSystem/config/packages/p4est.py > > @@ -24,14 +24,16 @@ class Configure(config.package.GNUPackage): > > self.blasLapack = framework.require('config.packages.BlasLapack',self) > > self.zlib = framework.require('config.packages.zlib',self) > > self.memalign = framework.argDB['with-memalign'] > > - self.deps = [self.mpi,self.blasLapack,self.zlib] > > + self.deps = [self.blasLapack,self.zlib] > > + self.odeps = [self.mpi] > > return > > > > def formGNUConfigureArgs(self): > > args = config.package.GNUPackage.formGNUConfigureArgs(self) > > if self.argDB['with-p4est-debugging']: > > args.append('--enable-debug') > > - args.append('--enable-mpi') > > + if not self.mpi.usingMPIUni: > > + args.append('--enable-mpi') > > args.append('CPPFLAGS="'+self.headers.toStringNoDupes(self.dinclude)+'"') > > args.append('LIBS="'+self.libraries.toString(self.dlib)+'"') > > args.append('--enable-memalign='+self.memalign) > > > > On Fri, 16 Apr 2021, Isaac, Tobin G wrote: > > > > > Matt, this is the configure logic I would like, but I don't know how to do it: > > > - configure p4est after MPI, but do not require MPI > > > - if MPI is present, add the '--enable-mpi' flack in formGNUConfigureArgs(), otherwise omit it. > > > > > > ________________________________________ > > > From: Isaac, Tobin G <tisaac(a)cc.gatech.edu> > > > Sent: Friday, April 16, 2021 11:03 > > > To: Mark Adams; For users of the development version of PETSc > > > Subject: Re: [petsc-dev] p4est w/o MPI > > > > > > p4est has a mode where it can compile without MPI, I don't know if PETSc is using it, will check. > > > > > > ________________________________________ > > > From: petsc-dev <petsc-dev-bounces(a)mcs.anl.gov> on behalf of Mark Adams <mfadams(a)lbl.gov> > > > Sent: Friday, April 16, 2021 10:23 > > > To: For users of the development version of PETSc > > > Subject: [petsc-dev] p4est w/o MPI > > > > > > I don't have MPI (Fugaku w/ gcc) and p4est seems to need it. Is there a work around? > > > Thanks, > > > Mark > > > > > >

1 0

Re: [petsc-dev] p4est w/o MPI
by Satish Balay 16 Apr '21

16 Apr '21

Thanks! Another issue: mpiuni provides mpi.h. And I get the following errors when building PETSc. /home/balay/petsc/arch-linux-c-debug/include/sc.h:87:2: error: #error "mpi.h is included. Use --enable-mpi." 87 | #error "mpi.h is included. Use --enable-mpi." | ^~~~~ etc.. Satish On Fri, 16 Apr 2021, Isaac, Tobin G wrote: > Yes, `--disable-mpi` should be accepted by p4est. > > ________________________________________ > From: Satish Balay <balay(a)mcs.anl.gov> > Sent: Friday, April 16, 2021 11:47 > To: Satish Balay via petsc-dev > Cc: Isaac, Tobin G > Subject: Re: [petsc-dev] p4est w/o MPI > > BTW: Is there no --disable-mpi option? > > Satish > > On Fri, 16 Apr 2021, Satish Balay via petsc-dev wrote: > > > MPI pacakge is a bit weird compared to others [as it does this funny thing with MPIUNI]. > > > > For regular packages - I guess I would use 'if package.found' > > > > I guess I should create an MR with this diff. > > > > Satish > > > > ---- > > > > diff --git a/config/BuildSystem/config/packages/p4est.py b/config/BuildSystem/config/packages/p4est.py > > index e7108b0ab8..40d6797f02 100644 > > --- a/config/BuildSystem/config/packages/p4est.py > > +++ b/config/BuildSystem/config/packages/p4est.py > > @@ -24,14 +24,16 @@ class Configure(config.package.GNUPackage): > > self.blasLapack = framework.require('config.packages.BlasLapack',self) > > self.zlib = framework.require('config.packages.zlib',self) > > self.memalign = framework.argDB['with-memalign'] > > - self.deps = [self.mpi,self.blasLapack,self.zlib] > > + self.deps = [self.blasLapack,self.zlib] > > + self.odeps = [self.mpi] > > return > > > > def formGNUConfigureArgs(self): > > args = config.package.GNUPackage.formGNUConfigureArgs(self) > > if self.argDB['with-p4est-debugging']: > > args.append('--enable-debug') > > - args.append('--enable-mpi') > > + if not self.mpi.usingMPIUni: > > + args.append('--enable-mpi') > > args.append('CPPFLAGS="'+self.headers.toStringNoDupes(self.dinclude)+'"') > > args.append('LIBS="'+self.libraries.toString(self.dlib)+'"') > > args.append('--enable-memalign='+self.memalign) > > > > On Fri, 16 Apr 2021, Isaac, Tobin G wrote: > > > > > Matt, this is the configure logic I would like, but I don't know how to do it: > > > - configure p4est after MPI, but do not require MPI > > > - if MPI is present, add the '--enable-mpi' flack in formGNUConfigureArgs(), otherwise omit it. > > > > > > ________________________________________ > > > From: Isaac, Tobin G <tisaac(a)cc.gatech.edu> > > > Sent: Friday, April 16, 2021 11:03 > > > To: Mark Adams; For users of the development version of PETSc > > > Subject: Re: [petsc-dev] p4est w/o MPI > > > > > > p4est has a mode where it can compile without MPI, I don't know if PETSc is using it, will check. > > > > > > ________________________________________ > > > From: petsc-dev <petsc-dev-bounces(a)mcs.anl.gov> on behalf of Mark Adams <mfadams(a)lbl.gov> > > > Sent: Friday, April 16, 2021 10:23 > > > To: For users of the development version of PETSc > > > Subject: [petsc-dev] p4est w/o MPI > > > > > > I don't have MPI (Fugaku w/ gcc) and p4est seems to need it. Is there a work around? > > > Thanks, > > > Mark > > > > > >

1 0

Re: [petsc-dev] [EXTERNAL] Re: building on Fugaku
by Satish Balay 16 Apr '21

16 Apr '21

Thanks! Are you able to use cmake from this spack install? I get this error: [a04201@a33-5211c ~]$ spack load cmake ==> Error: cmake matches multiple packages. Matching packages: ioepv7h [email protected]%[email protected] arch=linux-rhel8-skylake_avx512 bae6gkh [email protected]%[email protected] arch=linux-rhel8-a64fx Use a more specific spec. [a04201@a33-5211c ~]$ spack load [email protected]%[email protected] [WARN] xos LPG 2002 - Failed to map HugeTLBfs for data/bss: /bin/bash The e_type of elf header must be ET_EXEC when using libmpg. You can check it on your load module by readelf -h command. Satish On Fri, 16 Apr 2021, Sreepathi, Sarat via petsc-dev wrote: > For GNU in general, > -Ofast -march=armv8.2-a+sve -msve-vector-bits=512 > > You may want to use -O3 if you run into FP exceptions etc. or look into turning on/off specific optimizations. > -Sarat. > > -----Original Message----- > From: Satish Balay <balay(a)mcs.anl.gov> > Sent: Friday, April 16, 2021 12:41 PM > To: Sreepathi, Sarat <sarat(a)ornl.gov> > Cc: Mark Adams <mfadams(a)lbl.gov>; petsc-dev <petsc-dev(a)mcs.anl.gov> > Subject: Re: [petsc-dev] [EXTERNAL] Re: building on Fugaku > > Sarat, > > I can give this a try. I see the instructions say "Execute the following commands on the command line in an interactive job" > > What optimization flags do you use here? > > Satish > > On Fri, 16 Apr 2021, Sreepathi, Sarat via petsc-dev wrote: > > > Hi Mark, > > > > I couldn’t follow all the issues in the other thread. Anyway, you are in luck. > > > > I have pushed for GNU compiler support with Fujitsu MPI since November after hitting multiple issues. Finally, got an update from Fugaku helpdesk earlier this week. > > As you are aware, you will probably get better performance with GNU than Fujitsu on A64FX. My latest experiments confirmed that *again* for climate workloads. > > > > See "Users Guide - Language and development environment Ver. 1.02" - Chapter 4 for details. > > https://www.fugaku.r-ccs.riken.jp/doc_root/en/user_guides/lang_1.02/GC > > C_MPI/index.html > > > > Additionally, please note that you will need to `spack load gcc(a)10.2.0`<mailto:[email protected]%60> to load the latest GNU compiler. > > To summarize, you need to do the following on a compute node to get the requisite environment. > > ``` > > . /vol0004/apps/oss/spack/share/spack/setup-env.sh > > spack load fujitsu-mpi%gcc > > spack load gcc(a)10.2.0<mailto:[email protected]> ``` > > > > Hope this helps (cc’ing petsc-dev), > > Sarat. > > > > From: Mark Adams <mfadams(a)lbl.gov> > > Sent: Friday, April 16, 2021 10:27 AM > > To: Sreepathi, Sarat <sarat(a)ornl.gov> > > Subject: Re: [EXTERNAL] Re: building on Fugaku > > > > Sarat, > > I am making progress. Kokkos is working w/o OMP, but Kokkos Kernels + OMP is barfing and Kokkos people suggested GNU. > > But one of my third party libraries needs MPI. Do you know of some sort of MPI library on Fugaku? > > It just has to work with one process. > > Thanks, > > Mark > > > > On Wed, Apr 14, 2021 at 3:21 PM Sreepathi, Sarat <sarat(a)ornl.gov<mailto:[email protected]>> wrote: > > Hi Mark, > > > > From earlier notes, I used the Kokkos “develop” branch. > > > > cmake -DCMAKE_BUILD_TYPE=Release \ > > -DCMAKE_INSTALL_PREFIX=${KOKKOS_SRC_DIR}/install \ > > -DBUILD_TESTING=OFF \ > > -DKokkos_ENABLE_LIBDL=OFF \ > > -DKokkos_ENABLE_OPENMP=ON \ > > -DKokkos_ENABLE_SERIAL=ON \ > > -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON \ > > .. > > -Sarat. > > > > From: Mark Adams <mfadams(a)lbl.gov<mailto:[email protected]>> > > Sent: Wednesday, April 14, 2021 3:09 PM > > To: Sreepathi, Sarat <sarat(a)ornl.gov<mailto:[email protected]>> > > Cc: Richard Tran Mills <rtmills(a)anl.gov<mailto:[email protected]>> > > Subject: [EXTERNAL] Re: building on Fugaku > > > > Sarat, > > > > I have made some progress but am getting a C++ error: > > > > CMake Warning at cmake/kokkos_test_cxx_std.cmake:47 (MESSAGE): > > C++14 is not supported as a compiler feature. We will choose custom flags > > for now, but this behavior has been deprecated. Please open an issue at > > https://github.com/kokkos/kokkos/issues reporting that Fujitsu failed for > > 14, preferrably including your CMake command. > > > > I am using: > > > > 04:08 adams/landau-gpu-opt= ~/petsc$ mpiFCC --version FCC (FCC) 4.5.0 > > 20210304 simulating gcc version 6.1 > > > > Do you have any ideas? > > > > Thanks, > > Mark > > > > On Wed, Apr 14, 2021 at 7:55 AM Mark Adams <mfadams(a)lbl.gov<mailto:[email protected]>> wrote: > > Hi Sarat, > > > > My colleague, Rich Mills, and I are coming back to Fugaku after a while, and I wanted to ask you how you build, with Kokkos, on Fugaku. > > I have tried Arm and Fugitzu compilers and was able to get Fugitzu to work a few months ago, but I wanted to see what you use. Especially for Kokkos. > > Can you share with us your build details? > > > > Thanks, > > Mark > > >

1 0

Re: [petsc-dev] [EXTERNAL] Re: building on Fugaku
by Sreepathi, Sarat 16 Apr '21

16 Apr '21

For GNU in general, -Ofast -march=armv8.2-a+sve -msve-vector-bits=512 You may want to use -O3 if you run into FP exceptions etc. or look into turning on/off specific optimizations. -Sarat. -----Original Message----- From: Satish Balay <balay(a)mcs.anl.gov> Sent: Friday, April 16, 2021 12:41 PM To: Sreepathi, Sarat <sarat(a)ornl.gov> Cc: Mark Adams <mfadams(a)lbl.gov>; petsc-dev <petsc-dev(a)mcs.anl.gov> Subject: Re: [petsc-dev] [EXTERNAL] Re: building on Fugaku Sarat, I can give this a try. I see the instructions say "Execute the following commands on the command line in an interactive job" What optimization flags do you use here? Satish On Fri, 16 Apr 2021, Sreepathi, Sarat via petsc-dev wrote: > Hi Mark, > > I couldn’t follow all the issues in the other thread. Anyway, you are in luck. > > I have pushed for GNU compiler support with Fujitsu MPI since November after hitting multiple issues. Finally, got an update from Fugaku helpdesk earlier this week. > As you are aware, you will probably get better performance with GNU than Fujitsu on A64FX. My latest experiments confirmed that *again* for climate workloads. > > See "Users Guide - Language and development environment Ver. 1.02" - Chapter 4 for details. > https://www.fugaku.r-ccs.riken.jp/doc_root/en/user_guides/lang_1.02/GC > C_MPI/index.html > > Additionally, please note that you will need to `spack load gcc(a)10.2.0`<mailto:[email protected]%60> to load the latest GNU compiler. > To summarize, you need to do the following on a compute node to get the requisite environment. > ``` > . /vol0004/apps/oss/spack/share/spack/setup-env.sh > spack load fujitsu-mpi%gcc > spack load gcc(a)10.2.0<mailto:[email protected]> ``` > > Hope this helps (cc’ing petsc-dev), > Sarat. > > From: Mark Adams <mfadams(a)lbl.gov> > Sent: Friday, April 16, 2021 10:27 AM > To: Sreepathi, Sarat <sarat(a)ornl.gov> > Subject: Re: [EXTERNAL] Re: building on Fugaku > > Sarat, > I am making progress. Kokkos is working w/o OMP, but Kokkos Kernels + OMP is barfing and Kokkos people suggested GNU. > But one of my third party libraries needs MPI. Do you know of some sort of MPI library on Fugaku? > It just has to work with one process. > Thanks, > Mark > > On Wed, Apr 14, 2021 at 3:21 PM Sreepathi, Sarat <sarat(a)ornl.gov<mailto:[email protected]>> wrote: > Hi Mark, > > From earlier notes, I used the Kokkos “develop” branch. > > cmake -DCMAKE_BUILD_TYPE=Release \ > -DCMAKE_INSTALL_PREFIX=${KOKKOS_SRC_DIR}/install \ > -DBUILD_TESTING=OFF \ > -DKokkos_ENABLE_LIBDL=OFF \ > -DKokkos_ENABLE_OPENMP=ON \ > -DKokkos_ENABLE_SERIAL=ON \ > -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON \ > .. > -Sarat. > > From: Mark Adams <mfadams(a)lbl.gov<mailto:[email protected]>> > Sent: Wednesday, April 14, 2021 3:09 PM > To: Sreepathi, Sarat <sarat(a)ornl.gov<mailto:[email protected]>> > Cc: Richard Tran Mills <rtmills(a)anl.gov<mailto:[email protected]>> > Subject: [EXTERNAL] Re: building on Fugaku > > Sarat, > > I have made some progress but am getting a C++ error: > > CMake Warning at cmake/kokkos_test_cxx_std.cmake:47 (MESSAGE): > C++14 is not supported as a compiler feature. We will choose custom flags > for now, but this behavior has been deprecated. Please open an issue at > https://github.com/kokkos/kokkos/issues reporting that Fujitsu failed for > 14, preferrably including your CMake command. > > I am using: > > 04:08 adams/landau-gpu-opt= ~/petsc$ mpiFCC --version FCC (FCC) 4.5.0 > 20210304 simulating gcc version 6.1 > > Do you have any ideas? > > Thanks, > Mark > > On Wed, Apr 14, 2021 at 7:55 AM Mark Adams <mfadams(a)lbl.gov<mailto:[email protected]>> wrote: > Hi Sarat, > > My colleague, Rich Mills, and I are coming back to Fugaku after a while, and I wanted to ask you how you build, with Kokkos, on Fugaku. > I have tried Arm and Fugitzu compilers and was able to get Fugitzu to work a few months ago, but I wanted to see what you use. Especially for Kokkos. > Can you share with us your build details? > > Thanks, > Mark >

1 0

Re: [petsc-dev] p4est w/o MPI
by Isaac, Tobin G 16 Apr '21

16 Apr '21

Yes, `--disable-mpi` should be accepted by p4est. ________________________________________ From: Satish Balay <balay(a)mcs.anl.gov> Sent: Friday, April 16, 2021 11:47 To: Satish Balay via petsc-dev Cc: Isaac, Tobin G Subject: Re: [petsc-dev] p4est w/o MPI BTW: Is there no --disable-mpi option? Satish On Fri, 16 Apr 2021, Satish Balay via petsc-dev wrote: > MPI pacakge is a bit weird compared to others [as it does this funny thing with MPIUNI]. > > For regular packages - I guess I would use 'if package.found' > > I guess I should create an MR with this diff. > > Satish > > ---- > > diff --git a/config/BuildSystem/config/packages/p4est.py b/config/BuildSystem/config/packages/p4est.py > index e7108b0ab8..40d6797f02 100644 > --- a/config/BuildSystem/config/packages/p4est.py > +++ b/config/BuildSystem/config/packages/p4est.py > @@ -24,14 +24,16 @@ class Configure(config.package.GNUPackage): > self.blasLapack = framework.require('config.packages.BlasLapack',self) > self.zlib = framework.require('config.packages.zlib',self) > self.memalign = framework.argDB['with-memalign'] > - self.deps = [self.mpi,self.blasLapack,self.zlib] > + self.deps = [self.blasLapack,self.zlib] > + self.odeps = [self.mpi] > return > > def formGNUConfigureArgs(self): > args = config.package.GNUPackage.formGNUConfigureArgs(self) > if self.argDB['with-p4est-debugging']: > args.append('--enable-debug') > - args.append('--enable-mpi') > + if not self.mpi.usingMPIUni: > + args.append('--enable-mpi') > args.append('CPPFLAGS="'+self.headers.toStringNoDupes(self.dinclude)+'"') > args.append('LIBS="'+self.libraries.toString(self.dlib)+'"') > args.append('--enable-memalign='+self.memalign) > > On Fri, 16 Apr 2021, Isaac, Tobin G wrote: > > > Matt, this is the configure logic I would like, but I don't know how to do it: > > - configure p4est after MPI, but do not require MPI > > - if MPI is present, add the '--enable-mpi' flack in formGNUConfigureArgs(), otherwise omit it. > > > > ________________________________________ > > From: Isaac, Tobin G <tisaac(a)cc.gatech.edu> > > Sent: Friday, April 16, 2021 11:03 > > To: Mark Adams; For users of the development version of PETSc > > Subject: Re: [petsc-dev] p4est w/o MPI > > > > p4est has a mode where it can compile without MPI, I don't know if PETSc is using it, will check. > > > > ________________________________________ > > From: petsc-dev <petsc-dev-bounces(a)mcs.anl.gov> on behalf of Mark Adams <mfadams(a)lbl.gov> > > Sent: Friday, April 16, 2021 10:23 > > To: For users of the development version of PETSc > > Subject: [petsc-dev] p4est w/o MPI > > > > I don't have MPI (Fugaku w/ gcc) and p4est seems to need it. Is there a work around? > > Thanks, > > Mark > > >

1 0

Re: [petsc-dev] [EXTERNAL] Re: building on Fugaku
by Satish Balay 16 Apr '21

16 Apr '21

Sarat, I can give this a try. I see the instructions say "Execute the following commands on the command line in an interactive job" What optimization flags do you use here? Satish On Fri, 16 Apr 2021, Sreepathi, Sarat via petsc-dev wrote: > Hi Mark, > > I couldn’t follow all the issues in the other thread. Anyway, you are in luck. > > I have pushed for GNU compiler support with Fujitsu MPI since November after hitting multiple issues. Finally, got an update from Fugaku helpdesk earlier this week. > As you are aware, you will probably get better performance with GNU than Fujitsu on A64FX. My latest experiments confirmed that *again* for climate workloads. > > See "Users Guide - Language and development environment Ver. 1.02" - Chapter 4 for details. > https://www.fugaku.r-ccs.riken.jp/doc_root/en/user_guides/lang_1.02/GCC_MPI… > > Additionally, please note that you will need to `spack load gcc(a)10.2.0`<mailto:[email protected]%60> to load the latest GNU compiler. > To summarize, you need to do the following on a compute node to get the requisite environment. > ``` > . /vol0004/apps/oss/spack/share/spack/setup-env.sh > spack load fujitsu-mpi%gcc > spack load gcc(a)10.2.0<mailto:[email protected]> > ``` > > Hope this helps (cc’ing petsc-dev), > Sarat. > > From: Mark Adams <mfadams(a)lbl.gov> > Sent: Friday, April 16, 2021 10:27 AM > To: Sreepathi, Sarat <sarat(a)ornl.gov> > Subject: Re: [EXTERNAL] Re: building on Fugaku > > Sarat, > I am making progress. Kokkos is working w/o OMP, but Kokkos Kernels + OMP is barfing and Kokkos people suggested GNU. > But one of my third party libraries needs MPI. Do you know of some sort of MPI library on Fugaku? > It just has to work with one process. > Thanks, > Mark > > On Wed, Apr 14, 2021 at 3:21 PM Sreepathi, Sarat <sarat(a)ornl.gov<mailto:[email protected]>> wrote: > Hi Mark, > > From earlier notes, I used the Kokkos “develop” branch. > > cmake -DCMAKE_BUILD_TYPE=Release \ > -DCMAKE_INSTALL_PREFIX=${KOKKOS_SRC_DIR}/install \ > -DBUILD_TESTING=OFF \ > -DKokkos_ENABLE_LIBDL=OFF \ > -DKokkos_ENABLE_OPENMP=ON \ > -DKokkos_ENABLE_SERIAL=ON \ > -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON \ > .. > -Sarat. > > From: Mark Adams <mfadams(a)lbl.gov<mailto:[email protected]>> > Sent: Wednesday, April 14, 2021 3:09 PM > To: Sreepathi, Sarat <sarat(a)ornl.gov<mailto:[email protected]>> > Cc: Richard Tran Mills <rtmills(a)anl.gov<mailto:[email protected]>> > Subject: [EXTERNAL] Re: building on Fugaku > > Sarat, > > I have made some progress but am getting a C++ error: > > CMake Warning at cmake/kokkos_test_cxx_std.cmake:47 (MESSAGE): > C++14 is not supported as a compiler feature. We will choose custom flags > for now, but this behavior has been deprecated. Please open an issue at > https://github.com/kokkos/kokkos/issues reporting that Fujitsu failed for > 14, preferrably including your CMake command. > > I am using: > > 04:08 adams/landau-gpu-opt= ~/petsc$ mpiFCC --version > FCC (FCC) 4.5.0 20210304 > simulating gcc version 6.1 > > Do you have any ideas? > > Thanks, > Mark > > On Wed, Apr 14, 2021 at 7:55 AM Mark Adams <mfadams(a)lbl.gov<mailto:[email protected]>> wrote: > Hi Sarat, > > My colleague, Rich Mills, and I are coming back to Fugaku after a while, and I wanted to ask you how you build, with Kokkos, on Fugaku. > I have tried Arm and Fugitzu compilers and was able to get Fugitzu to work a few months ago, but I wanted to see what you use. Especially for Kokkos. > Can you share with us your build details? > > Thanks, > Mark >

1 0