April 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] -info from just one process?
by Barry Smith 22 Apr '21

22 Apr '21

It would be straightforward to add an option to -info to output only from a particular MPI_COMM_WORLD global rank. Maybe an extra :rank at the end of the info options? Open 128k files is nuts. Barry > On Apr 22, 2021, at 11:14 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > OK, so -info out.%p would do what I asked. Alas, what I really need it to reduce the output with 128K processes. > Thanks, > > On Thu, Apr 22, 2021 at 11:54 AM Pierre Jolivet <pierre(a)joliv.et <mailto:[email protected]>> wrote: > > >> On 22 Apr 2021, at 5:33 PM, Jacob Faibussowitsch <jacob.fai(a)gmail.com <mailto:[email protected]>> wrote: >> >> Not out of the box no. There is -info ::self which makes info only print from calls made on PETSC_COMM_SELF, but I don’t think that’s what you want. Best advice would be to print to a file and grep the file for any lines starting with the rank number you want. > > There is no need for grep, IMHO, just do -info file and then open file.%p > > Thanks, > Pierre > >> Best regards, >> >> Jacob Faibussowitsch >> (Jacob Fai - booss - oh - vitch) >> >>> On Apr 22, 2021, at 10:16, Mark Adams <mfadams(a)lbl.gov <mailto:[email protected]>> wrote: >>> >>> Can I make -info only print from one process? >>> Mark >> >

1 0

Re: [petsc-dev] -info from just one process?
by Mark Adams 22 Apr '21

22 Apr '21

OK, so -info out.%p would do what I asked. Alas, what I really need it to reduce the output with 128K processes. Thanks, On Thu, Apr 22, 2021 at 11:54 AM Pierre Jolivet <pierre(a)joliv.et> wrote: > > > On 22 Apr 2021, at 5:33 PM, Jacob Faibussowitsch <jacob.fai(a)gmail.com> > wrote: > > Not out of the box no. There is -info ::self which makes info only print > from calls made on PETSC_COMM_SELF, but I don’t think that’s what you want. > Best advice would be to print to a file and grep the file for any lines > starting with the rank number you want. > > > There is no need for grep, IMHO, just do -info file and then open file.%p > > Thanks, > Pierre > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > > On Apr 22, 2021, at 10:16, Mark Adams <mfadams(a)lbl.gov> wrote: > > Can I make -info only print from one process? > Mark > > > >

1 0

Re: [petsc-dev] -info from just one process?
by Pierre Jolivet 22 Apr '21

22 Apr '21

> On 22 Apr 2021, at 5:33 PM, Jacob Faibussowitsch <jacob.fai(a)gmail.com> wrote: > > Not out of the box no. There is -info ::self which makes info only print from calls made on PETSC_COMM_SELF, but I don’t think that’s what you want. Best advice would be to print to a file and grep the file for any lines starting with the rank number you want. There is no need for grep, IMHO, just do -info file and then open file.%p Thanks, Pierre > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > >> On Apr 22, 2021, at 10:16, Mark Adams <mfadams(a)lbl.gov <mailto:[email protected]>> wrote: >> >> Can I make -info only print from one process? >> Mark >

1 0

Re: [petsc-dev] -info from just one process?
by Jacob Faibussowitsch 22 Apr '21

22 Apr '21

Not out of the box no. There is -info ::self which makes info only print from calls made on PETSC_COMM_SELF, but I don’t think that’s what you want. Best advice would be to print to a file and grep the file for any lines starting with the rank number you want. Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) > On Apr 22, 2021, at 10:16, Mark Adams <mfadams(a)lbl.gov> wrote: > > Can I make -info only print from one process? > Mark

1 0

-info from just one process?
by Mark Adams 22 Apr '21

22 Apr '21

Can I make -info only print from one process? Mark

1 0

Re: [petsc-dev] Compilation errors with kokkos and --with-scalar-type complex
by Stefano Zampini 21 Apr '21

21 Apr '21

no cuda Il giorno mer 21 apr 2021 alle ore 18:11 Junchao Zhang < junchao.zhang(a)gmail.com> ha scritto: > > > > On Wed, Apr 21, 2021 at 5:23 AM Stefano Zampini <stefano.zampini(a)gmail.com> > wrote: > >> Incidentally, I found PETSc does not compile when configured using >> >> --with-scalar-type=complex --with-kokkos-dir=.... >> --with-kokkos_kernels-dir=... >> > With cuda or not? > >> >> Some fixes are trivial, others require some more thought. >> >> Thanks >> -- >> Stefano >> > -- Stefano

1 0

Re: [petsc-dev] Compilation errors with kokkos and --with-scalar-type complex
by Junchao Zhang 21 Apr '21

21 Apr '21

On Wed, Apr 21, 2021 at 5:23 AM Stefano Zampini <stefano.zampini(a)gmail.com> wrote: > Incidentally, I found PETSc does not compile when configured using > > --with-scalar-type=complex --with-kokkos-dir=.... > --with-kokkos_kernels-dir=... > With cuda or not? > > Some fixes are trivial, others require some more thought. > > Thanks > -- > Stefano >

1 0

Re: [petsc-dev] [EXTERNAL] Re: building on Fugaku
by Mark Adams 21 Apr '21

21 Apr '21

On Tue, Apr 20, 2021 at 9:06 PM Sreepathi, Sarat <sarat(a)ornl.gov> wrote: > Already tried those but it didn't help. I have been trying to experiment > with 48x1, 24x2 etc. and performance degraded for the climate workload. > I have problems even using all 48 cores on both my Kokkos Landau code and KK matrix-vector products (basically) in algebraic multigrid (AMG). For AMG using 8 (threads) x 4 (MPI) was best and thread speedup was moderate. I don't know how well KK vectorizes but in principle they should be able to make that work (they can write any code they want in KK). For Landau, I get great thread speedup, This code is MPI serial. I get the same throughput with 32x1, 16x2, 8x4 and 4x8. It looks like I am not getting any vectorization. With a large (10 species) test that I use as my test case, it runs very slow when I use all 48 cores in any configuration. With 2 species it does not die, just not great, but I have not looked at this in any detail. Let us know if you find anything. Thanks, Mark >

1 0

Compilation errors with kokkos and --with-scalar-type complex
by Stefano Zampini 21 Apr '21

21 Apr '21

Incidentally, I found PETSc does not compile when configured using --with-scalar-type=complex --with-kokkos-dir=.... --with-kokkos_kernels-dir=... Some fixes are trivial, others require some more thought. Thanks -- Stefano

1 0

Re: [petsc-dev] [EXTERNAL] Re: building on Fugaku
by Sreepathi, Sarat 20 Apr '21

20 Apr '21

Already tried those but it didn't help. I have been trying to experiment with 48x1, 24x2 etc. and performance degraded for the climate workload. ________________________________ From: Mark Adams <mfadams(a)lbl.gov> Sent: Tuesday, April 20, 2021 8:14:12 PM To: Sreepathi, Sarat <sarat(a)ornl.gov> Cc: petsc-dev <petsc-dev(a)mcs.anl.gov>; Satish Balay <balay(a)mcs.anl.gov> Subject: Re: [petsc-dev] [EXTERNAL] Re: building on Fugaku I settled on this: . /vol0004/apps/oss/spack/share/spack/setup-env.sh spack load [email protected]%[email protected] arch=linux-rhel8-a64fx spack load [email protected]%[email protected] arch=linux-rhel8-a64fx 'CC=mpicc', 'CXX=mpiCC', 'FC=mpif90', 'COPTFLAGS=-Ofast -march=armv8.2-a+sve -msve-vector-bits=512', 'CXXOPTFLAGS=-Ofast -march=armv8.2-a+sve -msve-vector-bits=512', Kokkos suggested this and I noticed that it helped: export OMP_PROC_BIND=spread export OMP_PLACES=threads I am getting great thread scaling but I seem to get no vectorization. I discussed this with Kokkos (Trott) today and he is not surprised. Auto vectorization is fragile. On Mon, Apr 19, 2021 at 8:26 PM Sreepathi, Sarat <sarat(a)ornl.gov<mailto:[email protected]>> wrote: My turn: did you folks figure out tips for performant hybrid MPI+OMP core binding? I tried some from the documentation but that didn’t seem to help. -Sarat. From: Sreepathi, Sarat Sent: Friday, April 16, 2021 3:02 PM To: Mark Adams <mfadams(a)lbl.gov<mailto:[email protected]>>; petsc-dev <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> Cc: Satish Balay <balay(a)mcs.anl.gov<mailto:[email protected]>> Subject: RE: [petsc-dev] [EXTERNAL] Re: building on Fugaku It’s 48 cores but there are 4 NUMA domains (CMGs). So, you may want to experiment in hybrid mode (4x12 etc.) if possible. -Sarat. From: Mark Adams <mfadams(a)lbl.gov<mailto:[email protected]>> Sent: Friday, April 16, 2021 2:10 PM To: petsc-dev <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> Cc: Satish Balay <balay(a)mcs.anl.gov<mailto:[email protected]>>; Sreepathi, Sarat <sarat(a)ornl.gov<mailto:[email protected]>> Subject: Re: [petsc-dev] [EXTERNAL] Re: building on Fugaku Sarat, is there anything special that you do for Kokkos - OpenMP? Just set OMP_NUM_THREADS=48 ? Also, I am confused about the number of cores here. Is 48 or 64 per node/socket? On Fri, Apr 16, 2021 at 2:03 PM Mark Adams <mfadams(a)lbl.gov<mailto:[email protected]>> wrote: Cool, I have it running too. Need to add Sarat's flags and test ex2. On Fri, Apr 16, 2021 at 1:57 PM Satish Balay via petsc-dev <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> wrote: Mark, The following build works for me: Satish ---- pjsub --interact -L "node=1" -L "rscunit=rscunit_ft01" -L "elapse=1:00:00" --sparam "wait-time=1200" . /vol0004/apps/oss/spack/share/spack/setup-env.sh spack load fujitsu-mpi%gcc spack load gcc(a)10.2.0<mailto:[email protected]> arch=linux-rhel8-a64fx ./configure COPTFLAGS='-Ofast -march=armv8.2-a+sve -msve-vector-bits=512' CXXOPTFLAGS='-Ofast -march=armv8.2-a+sve -msve-vector-bits=512' FOPTFLAGS='-Ofast -march=armv8.2-a+sve -msve-vector-bits=512' --with-openmp=1 --download-p4est --download-zlib --download-kokkos --download-kokkos-kernels --download-kokkos-commit=origin/develop --download-kokkos-kernels-commit=origin/develop '--download-kokkos-cmake-arguments=-DBUILD_TESTING=OFF -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON' --download-cmake=https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1.tar.gz --download-fblaslapack=1 make PETSC_DIR=/vol0004/ra010009/a04201/petsc.z PETSC_ARCH=arch-linux-c-debug all To test - redo job allocation using max-proc-per-node: login6$ pjsub --interact -L "node=1" -L "rscunit=rscunit_ft01" -L "elapse=1:00:00" --sparam "wait-time=1200" --mpi "max-proc-per-node=16" [a04201@c31-3201c petsc.z]$ . /vol0004/apps/oss/spack/share/spack/setup-env.sh [a04201@c31-3201c petsc.z]$ spack load fujitsu-mpi%gcc [a04201@c31-3201c petsc.z]$ spack load gcc(a)10.2.0<mailto:[email protected]> arch=linux-rhel8-a64fx [a04201@c31-3201c petsc.z]$ make check Running check examples to verify correct installation Using PETSC_DIR=/vol0004/ra010009/a04201/petsc.z and PETSC_ARCH=arch-linux-c-debug C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes C/C++ example src/snes/tutorials/ex3k run successfully with kokkos-kernels Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process Completed test examples [a04201@c31-3201c petsc.z]$

1 0