October 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] AMGx integration
by Matthew Knepley 19 Oct '21

19 Oct '21

On Tue, Oct 19, 2021 at 8:17 AM Mark Adams <mfadams(a)lbl.gov> wrote: > We (Matt Martineau and I) are ready to work on the AMGx integration and I > want to start by getting the AMGx download working. > > I'm looking for advice on how to proceed. > > Should I look at a simple example and clone it? > What build system does it use? Thanks, Matt > Thanks, > Mark > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] AMGx integration
by Stefano Zampini 19 Oct '21

19 Oct '21

Il giorno mar 19 ott 2021 alle ore 15:17 Mark Adams <mfadams(a)lbl.gov> ha scritto: > We (Matt Martineau and I) are ready to work on the AMGx integration and I > want to start by getting the AMGx download working. > > I'm looking for advice on how to proceed. > > Should I look at a simple example and clone it? > What do you mean? Starting from PC implementation and incrementally adding things? > > Thanks, > Mark > > -- Stefano

1 0

AMGx integration
by Mark Adams 19 Oct '21

19 Oct '21

We (Matt Martineau and I) are ready to work on the AMGx integration and I want to start by getting the AMGx download working. I'm looking for advice on how to proceed. Should I look at a simple example and clone it? Thanks, Mark

1 0

Re: [petsc-dev] Anyone run STREAMS on an Apple M1 system?
by Barry Smith 18 Oct '21

18 Oct '21

BTW: In the table Oct 18th, 2021 ASUS System Product Name <https://browser.geekbench.com/v3/cpu/8966621> Intel Core i9-11900K 3504 MHz (8 cores) Windows 32-bit 32 bit WTF? > On Oct 18, 2021, at 6:02 PM, Barry Smith <bsmith(a)petsc.dev> wrote: > > > Thanks, presumably we'll see the new Mac's there in a few days. BTW: the old streams benchmark page should point to this site; google is worthless. > > I get 24 on my Intel MacBook Pro and it also saturates with 1 core. > > Barry > > >> On Oct 18, 2021, at 4:48 PM, Jed Brown <jed(a)jedbrown.org> wrote: >> >> I don't have one, but this suggests it gets about 40 GB/s and can be saturated by a single core. I believe it uses two channels of LPDDR4X-4266, which has a theoretical peak of 68 GB/s. >> >> https://browser.geekbench.com/v3/cpu/8931693 >> >> The press release claims up to 400 GB/s on the Max using DDR5. I assume that's calculated based on 8 channels of LPDDR5-6400, which seems like a surprisingly big step and I'm skeptical of what will actually be realized. >> >> Barry Smith <bsmith(a)petsc.dev> writes: >> >>> Can anyone who owns an Apple M1 system run the MPI streams benchmark? Make sure the -O3 (or something) optimization flags are turned on. >>> >>> Thanks >>> >>> Barry >

1 0

Re: [petsc-dev] Anyone run STREAMS on an Apple M1 system?
by Barry Smith 18 Oct '21

18 Oct '21

Thanks, presumably we'll see the new Mac's there in a few days. BTW: the old streams benchmark page should point to this site; google is worthless. I get 24 on my Intel MacBook Pro and it also saturates with 1 core. Barry > On Oct 18, 2021, at 4:48 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > I don't have one, but this suggests it gets about 40 GB/s and can be saturated by a single core. I believe it uses two channels of LPDDR4X-4266, which has a theoretical peak of 68 GB/s. > > https://browser.geekbench.com/v3/cpu/8931693 > > The press release claims up to 400 GB/s on the Max using DDR5. I assume that's calculated based on 8 channels of LPDDR5-6400, which seems like a surprisingly big step and I'm skeptical of what will actually be realized. > > Barry Smith <bsmith(a)petsc.dev> writes: > >> Can anyone who owns an Apple M1 system run the MPI streams benchmark? Make sure the -O3 (or something) optimization flags are turned on. >> >> Thanks >> >> Barry

1 0

Re: [petsc-dev] Anyone run STREAMS on an Apple M1 system?
by Jed Brown 18 Oct '21

18 Oct '21

I don't have one, but this suggests it gets about 40 GB/s and can be saturated by a single core. I believe it uses two channels of LPDDR4X-4266, which has a theoretical peak of 68 GB/s. https://browser.geekbench.com/v3/cpu/8931693 The press release claims up to 400 GB/s on the Max using DDR5. I assume that's calculated based on 8 channels of LPDDR5-6400, which seems like a surprisingly big step and I'm skeptical of what will actually be realized. Barry Smith <bsmith(a)petsc.dev> writes: > Can anyone who owns an Apple M1 system run the MPI streams benchmark? Make sure the -O3 (or something) optimization flags are turned on. > > Thanks > > Barry

1 0

Anyone run STREAMS on an Apple M1 system?
by Barry Smith 18 Oct '21

18 Oct '21

Can anyone who owns an Apple M1 system run the MPI streams benchmark? Make sure the -O3 (or something) optimization flags are turned on. Thanks Barry

1 0

Re: [petsc-dev] snes/tests/ex13 hangs
by Mark Adams 17 Oct '21

17 Oct '21

Just an update. This is not deterministic. I am thinking that Permutters MPI might be an issue. On Sat, Oct 16, 2021 at 4:47 PM Mark Adams <mfadams(a)lbl.gov> wrote: > I am running snes/ex13 on Perlmutter and doing a scaling study in a > script. The first case runs fine: > > + srun -G 32 -n 256 --cpu-bind=cores --ntasks-per-core=1 > /global/homes/m/madams/mps-wrapper.sh ../ex13 -dm_plex_box_faces 4,8,8 > -petscpartitioner_simple_node_grid 2,2,2 *-dm_refine 3 *-dm_mat_type > aijcusparse -dm_vec_type cuda -dm_view -ksp_max_it 15 -log_view > > but the next case *with another levels of refinement*: > > + srun -G 32 -n 256 --cpu-bind=cores --ntasks-per-core=1 > /global/homes/m/madams/mps-wrapper.sh ../ex13 -dm_plex_box_faces 4,8,8 > -petscpartitioner_simple_node_grid 2,2,2 *-dm_refine 4 *-dm_mat_type > aijcusparse -dm_vec_type cuda -dm_view -ksp_max_it 15 -log_ > > hangs in BuildTwoSided. With log_trace I see this (grepping on 177). Args > appended. > > Any ideas? > Thanks > > > [177] 21.4441 Event begin: MatGetBrAoCol > [177] 21.4441 Event begin: SFSetUp > [177] 21.4441 Event begin: BuildTwoSided > [177] 21.5443 Event end: BuildTwoSided > [177] 21.5443 Event end: SFSetUp > [177] 21.5443 Event begin: MatAssemblyBegin > [177] 21.5444 Event end: MatAssemblyBegin > [177] 21.5444 Event begin: MatAssemblyEnd > [177] 21.5444 Event end: MatAssemblyEnd > [177] 21.5444 Event end: MatGetBrAoCol > [177] 21.5444 Event begin: MatGetLocalMat > [177] 21.5569 Event begin: MatCUSPARSCopyTo > [177] 21.557 Event end: MatCUSPARSCopyTo > [177] 21.557 Event begin: MatCUSPARSCopyTo > [177] 21.5571 Event end: MatCUSPARSCopyTo > [177] 21.5571 Event end: MatGetLocalMat > [177] 21.5571 Event begin: MatCUSPARSCopyTo > [177] 21.5571 Event end: MatCUSPARSCopyTo > [177] 21.5698 Event begin: MatCUSPARSCopyTo > [177] 21.5698 Event end: MatCUSPARSCopyTo > [177] 21.5827 Event begin: MatConvert > [177] 21.5954 Event end: MatConvert > [177] 21.5954 Event begin: MatCUSPARSCopyTo > [177] 21.5954 Event end: MatCUSPARSCopyTo > [177] 21.5954 Event begin: MatCUSPARSCopyTo > [177] 21.5955 Event end: MatCUSPARSCopyTo > [177] 21.6208 Event begin: SFSetGraph > [177] 21.6208 Event end: SFSetGraph > [177] 21.6208 Event begin: SFSetUp > [177] 21.6208 Event begin: BuildTwoSided > > > > #PETSc Option Table entries: > -benchmark_it 10 > -dm_distribute > -dm_mat_type aijcusparse > -dm_plex_box_faces 4,8,8 > -dm_plex_box_lower 0,0,0 > -dm_plex_box_upper 2,4,4 > -dm_plex_dim 3 > -dm_plex_simplex 0 > -dm_refine 3 > -dm_vec_type cuda > -dm_view > -ksp_max_it 15 > -ksp_monitor_short > -ksp_norm_type unpreconditioned > -ksp_rtol 1.e-12 > -ksp_type cg > -log_view > -matptap_via scalable > -mg_levels_esteig_ksp_max_it 5 > -mg_levels_esteig_ksp_type cg > -mg_levels_ksp_max_it 2 > -mg_levels_ksp_type richardson > -mg_levels_pc_type jacobi > -options_left > -pc_gamg_coarse_eq_limit 100 > -pc_gamg_coarse_grid_layout_type spread > -pc_gamg_esteig_ksp_max_it 5 > -pc_gamg_esteig_ksp_type cg > -pc_gamg_process_eq_limit 100 > -pc_gamg_repartition false > -pc_gamg_reuse_interpolation true > -pc_gamg_square_graph 1 > -pc_gamg_threshold 0.01 > -pc_gamg_threshold_scale .5 > -pc_type gamg > -petscpartitioner_simple_node_grid 2,2,2 > -petscpartitioner_simple_process_grid 2,4,4 > -petscpartitioner_type simple > -potential_petscspace_degree 2 > -snes_max_it 1 > -snes_rtol 1.e-8 > -snes_type ksponly > -use_gpu_aware_mpi 0 > #End of PETSc Option Table entries > Compiled without FORTRAN kernels > Compiled with full precision matrices (default) > sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 > sizeof(PetscScalar) 8 sizeof(PetscInt) 4 > Configure options: --CFLAGS=" -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 > -DLANDAU_MAX_Q=4" --CXXFLAGS=" -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 > -DLANDAU_MAX_Q=4" --CUDAFLAGS="-g -Xcompiler -rdynamic -DLANDAU_DIM=2 > -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_ > Q=4" --with-cc=cc --with-cxx=CC --with-fc=ftn > --with-cudac=/global/common/software/nersc/cos1.3/cuda/11.3.0/bin/nvcc > --FFLAGS=" -g " --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" > -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with > -mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 > PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda > ----------------------------------------- > Libraries compiled on 2021-10-16 18:33:45 on login02 > >

1 0

snes/tests/ex13 hangs
by Mark Adams 16 Oct '21

16 Oct '21

I am running snes/ex13 on Perlmutter and doing a scaling study in a script. The first case runs fine: + srun -G 32 -n 256 --cpu-bind=cores --ntasks-per-core=1 /global/homes/m/madams/mps-wrapper.sh ../ex13 -dm_plex_box_faces 4,8,8 -petscpartitioner_simple_node_grid 2,2,2 *-dm_refine 3 *-dm_mat_type aijcusparse -dm_vec_type cuda -dm_view -ksp_max_it 15 -log_view but the next case *with another levels of refinement*: + srun -G 32 -n 256 --cpu-bind=cores --ntasks-per-core=1 /global/homes/m/madams/mps-wrapper.sh ../ex13 -dm_plex_box_faces 4,8,8 -petscpartitioner_simple_node_grid 2,2,2 *-dm_refine 4 *-dm_mat_type aijcusparse -dm_vec_type cuda -dm_view -ksp_max_it 15 -log_ hangs in BuildTwoSided. With log_trace I see this (grepping on 177). Args appended. Any ideas? Thanks [177] 21.4441 Event begin: MatGetBrAoCol [177] 21.4441 Event begin: SFSetUp [177] 21.4441 Event begin: BuildTwoSided [177] 21.5443 Event end: BuildTwoSided [177] 21.5443 Event end: SFSetUp [177] 21.5443 Event begin: MatAssemblyBegin [177] 21.5444 Event end: MatAssemblyBegin [177] 21.5444 Event begin: MatAssemblyEnd [177] 21.5444 Event end: MatAssemblyEnd [177] 21.5444 Event end: MatGetBrAoCol [177] 21.5444 Event begin: MatGetLocalMat [177] 21.5569 Event begin: MatCUSPARSCopyTo [177] 21.557 Event end: MatCUSPARSCopyTo [177] 21.557 Event begin: MatCUSPARSCopyTo [177] 21.5571 Event end: MatCUSPARSCopyTo [177] 21.5571 Event end: MatGetLocalMat [177] 21.5571 Event begin: MatCUSPARSCopyTo [177] 21.5571 Event end: MatCUSPARSCopyTo [177] 21.5698 Event begin: MatCUSPARSCopyTo [177] 21.5698 Event end: MatCUSPARSCopyTo [177] 21.5827 Event begin: MatConvert [177] 21.5954 Event end: MatConvert [177] 21.5954 Event begin: MatCUSPARSCopyTo [177] 21.5954 Event end: MatCUSPARSCopyTo [177] 21.5954 Event begin: MatCUSPARSCopyTo [177] 21.5955 Event end: MatCUSPARSCopyTo [177] 21.6208 Event begin: SFSetGraph [177] 21.6208 Event end: SFSetGraph [177] 21.6208 Event begin: SFSetUp [177] 21.6208 Event begin: BuildTwoSided #PETSc Option Table entries: -benchmark_it 10 -dm_distribute -dm_mat_type aijcusparse -dm_plex_box_faces 4,8,8 -dm_plex_box_lower 0,0,0 -dm_plex_box_upper 2,4,4 -dm_plex_dim 3 -dm_plex_simplex 0 -dm_refine 3 -dm_vec_type cuda -dm_view -ksp_max_it 15 -ksp_monitor_short -ksp_norm_type unpreconditioned -ksp_rtol 1.e-12 -ksp_type cg -log_view -matptap_via scalable -mg_levels_esteig_ksp_max_it 5 -mg_levels_esteig_ksp_type cg -mg_levels_ksp_max_it 2 -mg_levels_ksp_type richardson -mg_levels_pc_type jacobi -options_left -pc_gamg_coarse_eq_limit 100 -pc_gamg_coarse_grid_layout_type spread -pc_gamg_esteig_ksp_max_it 5 -pc_gamg_esteig_ksp_type cg -pc_gamg_process_eq_limit 100 -pc_gamg_repartition false -pc_gamg_reuse_interpolation true -pc_gamg_square_graph 1 -pc_gamg_threshold 0.01 -pc_gamg_threshold_scale .5 -pc_type gamg -petscpartitioner_simple_node_grid 2,2,2 -petscpartitioner_simple_process_grid 2,4,4 -petscpartitioner_type simple -potential_petscspace_degree 2 -snes_max_it 1 -snes_rtol 1.e-8 -snes_type ksponly -use_gpu_aware_mpi 0 #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --CFLAGS=" -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4" --CXXFLAGS=" -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4" --CUDAFLAGS="-g -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_ Q=4" --with-cc=cc --with-cxx=CC --with-fc=ftn --with-cudac=/global/common/software/nersc/cos1.3/cuda/11.3.0/bin/nvcc --FFLAGS=" -g " --COPTFLAGS=" -O" --CXXOPTFLAGS=" -O" --FOPTFLAGS=" -O" --with-debugging=0 --with-cuda=1 --with-cuda-arch=80 --with -mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda ----------------------------------------- Libraries compiled on 2021-10-16 18:33:45 on login02

1 0

Re: [petsc-dev] Big in MF hessian and BNK?
by Dener, Alp 14 Oct '21

14 Oct '21

Hi Stefano, It’s odd that the matmult works for several iterations before crashing. I will dig into the code and see if we do something to the matrix that disrupts it’s state along the way. Only thing I can think of is that BNLS does attempt to do diagonal shifts when it detects ill conditioning and maybe that’s causing a bug with MFFD. The other Newton methods don’t do this because they don’t have a rigid descent direction requirement like the line search version does. Anyway thanks for pointing this out. And yes we should definitely add this to the tests. Alp Dener Argonne National Laboratory Mathematics and Computer Science https://alp.dener.me On Oct 14, 2021, at 10:19 AM, Stefano Zampini <stefano.zampini(a)gmail.com> wrote: Alp I have found this problem, reproducible with release (see below using src/tao/unconstrained/tutorials/minsurf1) I was trying MFFD hessian with bntr. Using assembled hessian is fine (tf-oneapi) [szampini@localhost tutorials]$ ./minsurf1 -tao_smonitor -tao_type bntr -mx 10 -my 8 -tao_bnk_max_cg_its 3 -tao_gatol 1.e-4 ---- Minimum Surface Area Problem ----- mx: 10 my: 8 iter = 0, Function value 1.45591, Residual: 0.21372 iter = 0, Function value 1.43469, Residual: 0.205909 iter = 0, Function value 1.43469, Residual: 0.205909 iter = 1, Function value 1.42058, Residual: 0.0881207 iter = 2, Function value 1.41987, Residual: 0.10019 iter = 3, Function value 1.41797, Residual: 0.0209656 iter = 1, Function value 1.41775, Residual: 0.000288154 iter = 0, Function value 1.41775, Residual: 0.000288154 iter = 1, Function value 1.41775, Residual: 0.000207788 iter = 2, Function value 1.41775, Residual: 0.000123638 iter = 3, Function value 1.41775, Residual: 0.000117563 iter = 2, Function value 1.41775, Residual: < 1.0e-6 Using MFFD hessian error (using bnls segefaults, we should try to check all possible Tao solvers affected by this bug?) (tf-oneapi) [szampini@localhost tutorials]$ ./minsurf1 -tao_smonitor -tao_type bntr -mx 10 -my 8 -tao_bnk_max_cg_its 3 -tao_gatol 1.e-4 -tao_mf_hessian ---- Minimum Surface Area Problem ----- mx: 10 my: 8 iter = 0, Function value 1.45591, Residual: 0.21372 iter = 0, Function value 1.43469, Residual: 0.205909 iter = 0, Function value 1.43469, Residual: 0.205909 iter = 1, Function value 1.42058, Residual: 0.0881207 iter = 2, Function value 1.41987, Residual: 0.10019 iter = 3, Function value 1.41797, Residual: 0.0209656 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: MatMFFDSetBase() has not been called, this is often caused by forgetting to call MatAssemblyBegin/End on the first Mat in the SNES compute function [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.16.0, unknown [0]PETSC ERROR: ./minsurf1 on a arch-conda-tf-oneapi-single-real named localhost.localdomain by szampini Thu Oct 14 18:11:23 2021 [0]PETSC ERROR: Configure options --with-blaslapack-dir=/home/szampini/Devel/miniforge/envs/tf-oneapi --download-thrust --LDFLAGS=-liomp5 --with-debugging=0 --with-openmp --with-precision=single --with-fc=0 --download-h2opus --download-slepc --download-slepc-commit=origin/release PETSC_ARCH=arch-conda-tf-oneapi-single-real PETSC_DIR=/home/szampini/Devel/miniforge/Devel/petsc [0]PETSC ERROR: #1 MatMult_MFFD() at /home/szampini/Devel/miniforge/Devel/petsc/src/mat/impls/mffd/mffd.c:333 [0]PETSC ERROR: #2 MatMult_Shell() at /home/szampini/Devel/miniforge/Devel/petsc/src/mat/impls/shell/shell.c:1066 [0]PETSC ERROR: #3 MatMult() at /home/szampini/Devel/miniforge/Devel/petsc/src/mat/interface/matrix.c:2439 [0]PETSC ERROR: #4 KSP_MatMult() at /home/szampini/Devel/miniforge/Devel/petsc/include/petsc/private/kspimpl.h:346 [0]PETSC ERROR: #5 KSPCGSolve_STCG() at /home/szampini/Devel/miniforge/Devel/petsc/src/ksp/ksp/impls/cg/stcg/stcg.c:183 [0]PETSC ERROR: #6 KSPSolve_Private() at /home/szampini/Devel/miniforge/Devel/petsc/src/ksp/ksp/interface/itfunc.c:914 [0]PETSC ERROR: #7 KSPSolve() at /home/szampini/Devel/miniforge/Devel/petsc/src/ksp/ksp/interface/itfunc.c:1086 [0]PETSC ERROR: #8 TaoBNKComputeStep() at /home/szampini/Devel/miniforge/Devel/petsc/src/tao/bound/impls/bnk/bnk.c:477 [0]PETSC ERROR: #9 TaoSolve_BNTR() at /home/szampini/Devel/miniforge/Devel/petsc/src/tao/bound/impls/bnk/bntr.c:139 [0]PETSC ERROR: #10 TaoSolve() at /home/szampini/Devel/miniforge/Devel/petsc/src/tao/interface/taosolver.c:227 [0]PETSC ERROR: #11 main() at minsurf1.c:110 [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -check_pointer_intensity 0 [0]PETSC ERROR: -mx 10 [0]PETSC ERROR: -my 8 [0]PETSC ERROR: -tao_bnk_max_cg_its 3 [0]PETSC ERROR: -tao_gatol 1.e-4 [0]PETSC ERROR: -tao_mf_hessian [0]PETSC ERROR: -tao_smonitor [0]PETSC ERROR: -tao_type bntr [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- Abort(73) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 73) - process 0 Using fd is fine (tf-oneapi) [szampini@localhost tutorials]$ ./minsurf1 -tao_smonitor -tao_type bntr -mx 10 -my 8 -tao_bnk_max_cg_its 3 -tao_gatol 1.e-4 -tao_fd_hessian ---- Minimum Surface Area Problem ----- mx: 10 my: 8 iter = 0, Function value 1.45591, Residual: 0.21372 iter = 0, Function value 1.43469, Residual: 0.205909 iter = 0, Function value 1.43469, Residual: 0.205909 iter = 1, Function value 1.42058, Residual: 0.0881207 iter = 2, Function value 1.41987, Residual: 0.10019 iter = 3, Function value 1.41797, Residual: 0.0209656 iter = 1, Function value 1.41775, Residual: 0.000342734 iter = 0, Function value 1.41775, Residual: 0.000342734 iter = 1, Function value 1.41775, Residual: 0.000275372 iter = 2, Function value 1.41775, Residual: 0.000148513 iter = 3, Function value 1.41775, Residual: 0.000150663 iter = 2, Function value 1.41775, Residual: < 1.0e-6 -- Stefano

1 0