Please send me exact instructions on how you are running the geoclaw example, when I run 

geoclaw/examples/bouss/radial_flat



it only runs with one level, but never errors, I changed setrun.py to amrdata.amr_levels_max = 5 but it made no difference.

  Thanks

   Barry



On Oct 24, 2024, at 12:17 PM, Praveen C <cpraveen@gmail.com> wrote:

Hello Barry

I use this script to install clawpack and required dependencies


See lines 125-132

You can use it like this

export CLAW=/path/to/where/you/want/clawpack
bash clawpack.sh v5.11.0

This will git pull clawpack and creates a conda env called “claw” and installs inside that.

I have not specified petsc version in this script, but latest miniforge should install petsc@3.22

Thank you
praveen

On 24 Oct 2024, at 9:37 PM, Barry Smith <bsmith@petsc.dev> wrote:


    Good, some helpful information with the runs you made.

   In the crash below it made more progress it was able to allocate multiple regions of shared memory and access them. I don't know why it would crash later.

   Can you tell me all the steps with miniforge (which seems to be related to the failure) you use? I've never used miniforge. 

   If I can get to an environment that reproduces the problem I can debug it and fix it.

   Barry


On Oct 24, 2024, at 11:49 AM, Praveen C <cpraveen@gmail.com> wrote:

I get this

$ mpiexec -n 3 ./ex89f  -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view  -mpi_linear_solver_server_minimum_count_per_rank 5
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash.
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_SELF
  Proc: [[47380,1],0]
  Errorcode: 59

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
prterun has exited due to process rank 0 with PID 0 on node MacMiniHome calling
"abort". This may have caused other processes in the application to be
terminated by signals sent by prterun (as reported here).
—————————————————————————————————————

and I have this after the code exits

$ ipcs -m
IPC status from <running system> as of Thu Oct 24 21:17:39 IST 2024
T     ID     KEY        MODE       OWNER    GROUP
Shared Memory:
m 1572864 0x0000000b --rw-rw-rw-  praveen    staff
m 524289 0x0000000c --rw-rw-rw-  praveen    staff
m 655362 0x0000000d --rw-rw-rw-  praveen    staff
m 262147 0x0000000e --rw-rw-rw-  praveen    staff
m 262148 0x0000000f --rw-rw-rw-  praveen    staff

m 393221 0x0000000a --rw-rw-rw-  praveen    staff

This is with petsc installed with miniforge, which I also use with clawpack. With spack installed petsc, I can run the ex89f example.

Thanks
praveen

On 24 Oct 2024, at 7:55 PM, Barry Smith <bsmith@petsc.dev> wrote:


   Ok, super strange, I've run many times on my Mac

   Can you please try to remove that allocated memory with ipcrm and then

   cd $PETSC_DIR/src/ksp/ksp/tutorials
   make ex89f
   mpiexec -n 3 ./ex89f  -n 20 -mpi_linear_solver_server -mpi_linear_solver_server -mpi_linear_solver_server_ksp_view -ksp_monitor -ksp_converged_reason -ksp_view  -mpi_linear_solver_server_minimum_count_per_rank 5 

  This does the same thing as the GeoClaw code but is much simpler.

   Barry


   

On Oct 23, 2024, at 10:55 PM, Praveen C <cpraveen@gmail.com> wrote:

I get very similar error on my mac with

$ gfortran -v
Using built-in specs.
COLLECT_GCC=gfortran
COLLECT_LTO_WRAPPER=/opt/homebrew/Caskroom/miniforge/base/envs/claw/libexec/gcc/arm64-apple-darwin20.0.0/13.2.0/lto-wrapper
Target: arm64-apple-darwin20.0.0
Configured with: ../configure --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --build=x86_64-apple-darwin13.4.0 --host=arm64-apple-darwin20.0.0 --target=arm64-apple-darwin20.0.0 --with-libiconv-prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-languages=fortran --disable-multilib --enable-checking=release --disable-bootstrap --disable-libssp --with-gmp=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpfr=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-mpc=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-isl=/opt/homebrew/Caskroom/miniforge/base/envs/claw --enable-darwin-at-rpath
Thread model: posix
Supported LTO compression algorithms: zlib
gcc version 13.2.0 (GCC) 

Before starting

$ ipcs -m
IPC status from <running system> as of Thu Oct 24 08:02:11 IST 2024
T     ID     KEY        MODE       OWNER    GROUP
Shared Memory:

and when I run the code

 Using a PETSc solver
 Using Bouss equations from the start
 rnode allocated...
 node allocated...
 listOfGrids allocated...
 Storage allocated...
 bndList allocated...
Gridding level   1 at t =  0.000000E+00:     4 grids with       10000 cells
   Setting initial dt to    2.9999999999999999E-002
  max threads set to            1
  
 Done reading data, starting computation ...  
  
 Total zeta at initial time:    39269.907650665169     
GEOCLAW: Frame    0 output files done at time t =  0.000000D+00

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Petsc has generated inconsistent data
[0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x130698000
[0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc!
[0]PETSC ERROR:   Option left: name:-ksp_type value: preonly source: file
[0]PETSC ERROR:   Option left: name:-mpi_ksp_max_it value: 200 source: file
[0]PETSC ERROR:   Option left: name:-mpi_ksp_reuse_preconditioner (no value) source: file
[0]PETSC ERROR:   Option left: name:-mpi_ksp_rtol value: 1.e-9 source: file
[0]PETSC ERROR:   Option left: name:-mpi_ksp_type value: gmres source: file
[0]PETSC ERROR:   Option left: name:-mpi_linear_solver_server_view (no value) source: file
[0]PETSC ERROR:   Option left: name:-mpi_pc_gamg_sym_graph value: true source: file
[0]PETSC ERROR:   Option left: name:-mpi_pc_gamg_symmetrize_graph value: true source: file
[0]PETSC ERROR:   Option left: name:-mpi_pc_type value: gamg source: file
[0]PETSC ERROR:   Option left: name:-pc_mpi_minimum_count_per_rank value: 5000 source: file
[0]PETSC ERROR:   Option left: name:-pc_type value: mpi source: file
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 
[0]PETSC ERROR: /Users/praveen/work/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH  on MacMiniHome.local by praveen Thu Oct 24 08:04:27 2024
[0]PETSC ERROR: Configure options: AR=arm64-apple-darwin20.0.0-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include  " CPPFLAGS="-D_FORTIFY_SOURCE=2 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include -mmacosx-version-min=11.0 -mmacosx-version-min=11.0" CXXFLAGS="-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include  " FFLAGS="-march=armv8.3-a -ftree-vectorize -fPIC -fno-stack-protector -O2 -pipe -isystem /opt/homebrew/Caskroom/miniforge/base/envs/claw/include  " LDFLAGS="-Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -L/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib" LIBS="-Wl,-rpath,/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.dylib --with-lapack-lib=liblapack.dylib --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/homebrew/Caskroom/miniforge/base/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/homebrew/Caskroom/miniforge/base/envs/claw --with-x=0 --with-scalar-type=real   --with-cuda=0 --with-batch --prefix=/opt/homebrew/Caskroom/miniforge/base/envs/claw
[0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/sys/utils/server.c:114
[0]PETSC ERROR: #2 PCMPISetMat() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:269
[0]PETSC ERROR: #3 PCSetUp_MPI() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/impls/mpi/pcmpi.c:853
[0]PETSC ERROR: #4 PCSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/pc/interface/precon.c:1071
[0]PETSC ERROR: #5 KSPSetUp() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:415
[0]PETSC ERROR: #6 KSPSolve_Private() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:826

[0]PETSC ERROR: #7 KSPSolve() at /Users/runner/miniforge3/conda-bld/petsc_1728030427805/work/src/ksp/ksp/interface/itfunc.c:1075

Code does not progress and I kill it

^CTraceback (most recent call last):
  File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 341, in <module>
    runclaw(*args)
  File "/Users/praveen/Applications/clawpack/clawutil/src/python/clawutil/runclaw.py", line 242, in runclaw
    proc = subprocess.check_call(cmd_split,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 408, in check_call
    retcode = call(*popenargs, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 391, in call
    return p.wait(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 1264, in wait
    return self._wait(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2053, in _wait
    (pid, sts) = self._try_wait(0)
                 ^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/claw/lib/python3.12/subprocess.py", line 2011, in _try_wait
    (pid, sts) = os.waitpid(self.pid, wait_flags)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
make[1]: *** [output] Interrupt: 2

make: *** [.output] Interrupt: 2

Now it says

$ ipcs -m        
IPC status from <running system> as of Thu Oct 24 08:05:06 IST 2024
T     ID     KEY        MODE       OWNER    GROUP
Shared Memory:

m 720896 0x0000000a --rw-rw-rw-  praveen    staff

Thanks
praveen

On 23 Oct 2024, at 8:26 PM, Barry Smith <bsmith@petsc.dev> wrote:


   Hmm,  so it is creating the first shared memory region with ID of 10 (A in hex) puts it in a linked list in PETSc but then when it tries to find it in the linked list it cannot find it.

   I don't know how to reproduce this or debug it remotely.

    Can you build on a completely different machine or with completely different compilers?

   Barry




On Oct 23, 2024, at 10:31 AM, Praveen C <cpraveen@gmail.com> wrote:

I get same error and now it shows

$ ipcs -m

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x0000000a 32788      praveen    666        240        6 

Note that the code seems to be still running after printing those error message, but it is not printing any progress which it should do.

Thanks
praveen

On 23 Oct 2024, at 7:56 PM, Barry Smith <bsmith@petsc.dev> wrote:


   Try 

ipcrm -m 11

ipcs -m

Try running the program again

If failed check 

ipcs -m

again



On Oct 23, 2024, at 10:20 AM, Praveen C <cpraveen@gmail.com> wrote:

Hello Barry

I see this

$ ipcs -m

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x0000000a 11         praveen    666        240        6  

and I am observing same error as below.

Thanks
praveen

On 23 Oct 2024, at 7:08 PM, Barry Smith <bsmith@petsc.dev> wrote:


  Please take a look at the notes in https://petsc.org/release/manualpages/Sys/PetscShmgetAllocateArray/  For some reason your program is not able to access/use the Unix shared memory; check if you are already using the shared memory (so it is not available for a new run) or the limits are too low to access enough memory.

   Barry

On Oct 23, 2024, at 8:23 AM, Praveen C <cpraveen@gmail.com> wrote:

Dear all

I am not able to run the boussinesq example from geoclaw using petsc@3.22.0

https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883f46572c58130d161986b8a87a57ca7816/examples/bouss__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MnHN6PMw$

It runs with petsc@3.21.6

The error I get is given below. After printing this, the code does not progress.

I use the following petsc options

# set min numbers of matrix rows per MPI rank  (default is 10000)
-mpi_linear_solve_minimum_count_per_rank 5000


# Krylov linear solver:
-mpi_linear_solver_server
-mpi_linear_solver_server_view
-ksp_type gmres
-ksp_max_it 200
-ksp_reuse_preconditioner
-ksp_rtol 1.e-9

# preconditioner:
-pc_type gamg

I installed petsc and other dependencies for clawpack using miniforge.

Thanks
pc

==> Use Bouss. in water deeper than    1.0000000000000000       Using a PETSc solver
Using Bouss equations from the start
rnode allocated...
node allocated...
listOfGrids allocated...
Storage allocated...
bndList allocated...
Gridding level   1 at t =  0.000000E+00:     4 grids with       10000 cells
 Setting initial dt to    2.9999999999999999E-002
max threads set to            6
  Done reading data, starting computation ...       Total zeta at initial time:    39269.907650665169      GEOCLAW: Frame    0 output files done at time t =  0.000000D+00

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Petsc has generated inconsistent data
[0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20
[0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc!
[0]PETSC ERROR:   Option left: name:-ksp_max_it value: 200 source: file
[0]PETSC ERROR:   Option left: name:-ksp_reuse_preconditioner (no value) source: file
[0]PETSC ERROR:   Option left: name:-ksp_rtol value: 1.e-9 source: file
[0]PETSC ERROR:   Option left: name:-ksp_type value: gmres source: file
[0]PETSC ERROR:   Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file
[0]PETSC ERROR:   Option left: name:-mpi_linear_solver_server_view (no value) source: file
[0]PETSC ERROR:   Option left: name:-pc_type value: gamg source: file
[0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!e3VQ4NHKmXGstRsQW5vtI7fmKfUT9zmJkMJcPbcvPyIjicyfJpNoMgx3wZ-qyGcKNSjIkNZkzilec8MvNjNo7A$  for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024  [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH  on euler by praveen Thu Oct 17 21:49:54 2024
[0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include  " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include  " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include   -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real   --with-cuda=0 --prefix=/opt/miniforge/envs/claw
[0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114
[0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269
[0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853
[0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071
[0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415
[0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826
[0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075