September 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Cannot locate file: share/petsc/datafiles/matrices/small
by Satish Balay 14 Sep '21

14 Sep '21

cc: Antonio On Tue, 14 Sep 2021, Matthew Knepley wrote: > Okay, we have to send this to OpenMPI. Volunteers? > > Maybe we should note this in the FAQ, or installation, so we remember how > to fix it if someone else asks? > > Thanks, > > Matt > > On Tue, Sep 14, 2021 at 2:35 AM Stefano Zampini <stefano.zampini(a)gmail.com> > wrote: > > > I can reproduce it even with OpenMPI 4.1.1 on a different machine (Ubuntu > > 18 + AMD milan + clang from AOCC) and it is definitely an OpenMPI bug in > > the vader BTL If I use tcp, everything runs smooth > > > > zampins@kanary:~/Devel/petsc$ cat > > /home/zampins/local/etc/openmpi-mca-params.conf | grep btl > > btl = tcp,self > > zampins@kanary:~/Devel/petsc$ make -f gmakefile.test > > vec_is_sf_tutorials-ex1_4 > > Using MAKEFLAGS: > > TEST arch-debug/tests/counts/vec_is_sf_tutorials-ex1_4.counts > > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create > > ok > > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create > > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic > > ok > > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic > > ok > > vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate > > ok > > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate > > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create > > ok > > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create > > ok > > vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic > > ok > > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic > > ok > > vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate > > ok > > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate > > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create > > ok > > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create > > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic > > ok > > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic > > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate > > ok > > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate > > > > > > zampins@kanary:~/Devel/petsc$ cat > > /home/zampins/local/etc/openmpi-mca-params.conf | grep btl > > btl = vader,tcp,self > > zampins@kanary:~/Devel/petsc$ make -f gmakefile.test > > vec_is_sf_tutorials-ex1_4 > > Using MAKEFLAGS: > > TEST arch-debug/tests/counts/vec_is_sf_tutorials-ex1_4.counts > > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create > > not ok > > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create > > # Error code: 1 > > # 43,46c43,46 > > # < [0] 0: 4001 2000 2002 3002 4002 > > # < [1] 0: 1001 3000 > > # < [2] 0: 2001 4000 > > # < [3] 0: 3001 1000 > > # --- > > # > [0] 0: 2002 2146435072 2 2146435072 38736240 > > # > [1] 0: 3000 2146435072 > > # > [2] 0: 2001 2146435072 > > # > [3] 0: 3001 2146435072 > > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic > > not ok > > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic > > # Error code: 1 > > # 43,46c43,46 > > # < [0] 0: 4001 2000 2002 3002 4002 > > # < [1] 0: 1001 3000 > > # < [2] 0: 2001 4000 > > # < [3] 0: 3001 1000 > > # --- > > # > [0] 0: 2002 2146435072 2 2146435072 0 > > # > [1] 0: 3000 2146435072 > > # > [2] 0: 2001 2146435072 > > # > [3] 0: 3001 2146435072 > > ok > > vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate > > ok > > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate > > # retrying > > vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create > > not ok > > vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create # > > Error code: 98 > > # [1]PETSC ERROR: --------------------- Error Message > > -------------------------------------------------------------- > > # [1]PETSC ERROR: General MPI error > > # [1]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > > # [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > > # [1]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b > > GIT Date: 2021-09-13 14:01:22 +0000 > > # [1]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by > > zampins Tue Sep 14 09:31:42 2021 > > # [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message > > -------------------------------------------------------------- > > # [2]PETSC ERROR: General MPI error > > # [2]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > > # [2]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > > # [2]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b > > GIT Date: 2021-09-13 14:01:22 +0000 > > # [2]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by > > zampins Tue Sep 14 09:31:42 2021 > > # [2]PETSC ERROR: Configure options > > --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 > > --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 > > --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 > > --with-hip-dir=/opt/rocm --with-hip=1 > > --with-hypre-dir=/home/zampins/local-petsc > > --with-kokkos-dir=/home/zampins/local-petsc > > --with-kokkos-kernels-dir=/home/zampins/local-petsc > > --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include > > --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" > > HIPFLAGS=--amdgpu-target=gfx908 > > HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > > # [2]PETSC ERROR: #1 PetscSFGetGroups() at > > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > > # [2]PETSC ERROR: #2 PetscSFSetUp_Window() at > > /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > > # [2]PETSC ERROR: #3 PetscSFSetUp() at > > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > > # [2]PETSC ERROR: #4 main() at > > /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > > # [2]PETSC ERROR: PETSc Option Table entries: > > # [2]PETSC ERROR: -sf_type window > > # [2]PETSC ERROR: -sf_window_flavor create > > # [2]PETSC ERROR: -sf_window_sync active > > # [2]PETSC ERROR: -test_gather > > # [2]PETSC ERROR: ----------------End of Error Message -------send entire > > error message to petsc-maint(a)mcs.anl.gov---------- > > # [3]PETSC ERROR: --------------------- Error Message > > -------------------------------------------------------------- > > # [3]PETSC ERROR: General MPI error > > # [3]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > > # [3]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > > # [3]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b > > GIT Date: 2021-09-13 14:01:22 +0000 > > # [3]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by > > zampins Tue Sep 14 09:31:42 2021 > > # [3]PETSC ERROR: Configure options > > --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 > > --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 > > --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 > > --with-hip-dir=/opt/rocm --with-hip=1 > > --with-hypre-dir=/home/zampins/local-petsc > > --with-kokkos-dir=/home/zampins/local-petsc > > --with-kokkos-kernels-dir=/home/zampins/local-petsc > > --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include > > --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" > > HIPFLAGS=--amdgpu-target=gfx908 > > HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > > # [3]PETSC ERROR: #1 PetscSFGetGroups() at > > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > > # [3]PETSC ERROR: #2 PetscSFSetUp_Window() at > > /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > > # [3]PETSC ERROR: #3 PetscSFSetUp() at > > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > > # [3]PETSC ERROR: #4 main() at > > /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > > # [3]PETSC ERROR: PETSc Option Table entries: > > # [3]PETSC ERROR: -sf_type window > > # [3]PETSC ERROR: -sf_window_flavor create > > # [3]PETSC ERROR: Configure options > > --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 > > --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 > > --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 > > --with-hip-dir=/opt/rocm --with-hip=1 > > --with-hypre-dir=/home/zampins/local-petsc > > --with-kokkos-dir=/home/zampins/local-petsc > > --with-kokkos-kernels-dir=/home/zampins/local-petsc > > --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include > > --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" > > HIPFLAGS=--amdgpu-target=gfx908 > > HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > > # [1]PETSC ERROR: #1 PetscSFGetGroups() at > > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > > # [1]PETSC ERROR: #2 PetscSFSetUp_Window() at > > /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > > # [1]PETSC ERROR: #3 PetscSFSetUp() at > > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > > # [1]PETSC ERROR: #4 main() at > > /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > > # [1]PETSC ERROR: PETSc Option Table entries: > > # [1]PETSC ERROR: -sf_type window > > # [1]PETSC ERROR: -sf_window_flavor create > > # [1]PETSC ERROR: -sf_window_sync active > > # [1]PETSC ERROR: -test_gather > > # [1]PETSC ERROR: ----------------End of Error Message -------send entire > > error message to petsc-maint(a)mcs.anl.gov---------- > > # -sf_window_sync active > > # [3]PETSC ERROR: -test_gather > > # [3]PETSC ERROR: ----------------End of Error Message -------send entire > > error message to petsc-maint(a)mcs.anl.gov---------- > > # > > -------------------------------------------------------------------------- > > # MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD > > # with errorcode 98. > > # > > # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > > # You may or may not see output from other processes, depending on > > # exactly when Open MPI kills them. > > # > > -------------------------------------------------------------------------- > > # [kanary.kaust.edu.sa:115527] 2 more processes have sent help message > > help-mpi-api.txt / mpi-abort > > # [kanary.kaust.edu.sa:115527] Set MCA parameter > > "orte_base_help_aggregate" to 0 to see all help / error messages > > ok vec_is_sf_tutorials-ex1_4 # SKIP Command failed so no diff > > # retrying > > vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic > > not ok > > vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic # > > Error code: 98 > > # [1]PETSC ERROR: --------------------- Error Message > > -------------------------------------------------------------- > > # [1]PETSC ERROR: General MPI error > > # [1]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > > # [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > > # [1]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b > > GIT Date: 2021-09-13 14:01:22 +0000 > > # [1]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by > > zampins Tue Sep 14 09:31:45 2021 > > # [1]PETSC ERROR: Configure options > > --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 > > --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 > > --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 > > --with-hip-dir=/opt/rocm --with-hip=1 > > --with-hypre-dir=/home/zampins/local-petsc > > --with-kokkos-dir=/home/zampins/local-petsc > > --with-kokkos-kernels-dir=/home/zampins/local-petsc > > --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include > > --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" > > HIPFLAGS=--amdgpu-target=gfx908 > > HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > > # [1]PETSC ERROR: #1 PetscSFGetGroups() at > > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > > # [1]PETSC ERROR: #2 PetscSFSetUp_Window() at > > /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > > # [1]PETSC ERROR: #3 PetscSFSetUp() at > > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > > # [1]PETSC ERROR: #4 main() at > > /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > > # [1]PETSC ERROR: PETSc Option Table entries: > > # [1]PETSC ERROR: -sf_type window > > # [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message > > -------------------------------------------------------------- > > # [2]PETSC ERROR: General MPI error > > # [2]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > > # [2]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > > # [2]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b > > GIT Date: 2021-09-13 14:01:22 +0000 > > # [2]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by > > zampins Tue Sep 14 09:31:45 2021 > > # [2]PETSC ERROR: Configure options > > --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 > > --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 > > --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 > > --with-hip-dir=/opt/rocm --with-hip=1 > > --with-hypre-dir=/home/zampins/local-petsc > > --with-kokkos-dir=/home/zampins/local-petsc > > --with-kokkos-kernels-dir=/home/zampins/local-petsc > > --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include > > --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" > > HIPFLAGS=--amdgpu-target=gfx908 > > HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > > # [2]PETSC ERROR: #1 PetscSFGetGroups() at > > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > > # [2]PETSC ERROR: #2 PetscSFSetUp_Window() at > > /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > > # [2]PETSC ERROR: #3 PetscSFSetUp() at > > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > > # [2]PETSC ERROR: #4 main() at > > /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > > # [2]PETSC ERROR: PETSc Option Table entries: > > # [2]PETSC ERROR: -sf_type window > > # [2]PETSC ERROR: -sf_window_flavor dynamic > > # [2]PETSC ERROR: -sf_window_sync active > > # [2]PETSC ERROR: -test_gather > > # [2]PETSC ERROR: ----------------End of Error Message -------send entire > > error message to petsc-maint(a)mcs.anl.gov---------- > > # [3]PETSC ERROR: --------------------- Error Message > > -------------------------------------------------------------- > > # [3]PETSC ERROR: General MPI error > > # [3]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > > # [3]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > > # [3]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b > > GIT Date: 2021-09-13 14:01:22 +0000 > > # [3]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by > > zampins Tue Sep 14 09:31:45 2021 > > # [3]PETSC ERROR: Configure options > > --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 > > --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 > > --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 > > --with-hip-dir=/opt/rocm --with-hip=1 > > --with-hypre-dir=/home/zampins/local-petsc > > --with-kokkos-dir=/home/zampins/local-petsc > > --with-kokkos-kernels-dir=/home/zampins/local-petsc > > --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include > > --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" > > HIPFLAGS=--amdgpu-target=gfx908 > > HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > > # [3]PETSC ERROR: #1 PetscSFGetGroups() at > > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > > # [3]PETSC ERROR: #2 PetscSFSetUp_Window() at > > /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > > # [3]PETSC ERROR: #3 PetscSFSetUp() at > > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > > # [3]PETSC ERROR: #4 main() at > > /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > > # [3]PETSC ERROR: PETSc Option Table entries: > > # [3]PETSC ERROR: -sf_type window > > # [3]PETSC ERROR: -sf_window_flavor dynamic > > # [3]PETSC ERROR: -sf_window_sync active > > # [3]PETSC ERROR: -test_gather > > # [3]PETSC ERROR: ----------------End of Error Message -------send entire > > error message to petsc-maint(a)mcs.anl.gov---------- > > # -sf_window_flavor dynamic > > # [1]PETSC ERROR: -sf_window_sync active > > # [1]PETSC ERROR: -test_gather > > # [1]PETSC ERROR: ----------------End of Error Message -------send entire > > error message to petsc-maint(a)mcs.anl.gov---------- > > # > > -------------------------------------------------------------------------- > > # MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD > > # with errorcode 98. > > # > > # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > > # You may or may not see output from other processes, depending on > > # exactly when Open MPI kills them. > > # > > -------------------------------------------------------------------------- > > # [kanary.kaust.edu.sa:115572] 2 more processes have sent help message > > help-mpi-api.txt / mpi-abort > > # [kanary.kaust.edu.sa:115572] Set MCA parameter > > "orte_base_help_aggregate" to 0 to see all help / error messages > > ok vec_is_sf_tutorials-ex1_4 # SKIP Command failed so no diff > > ok > > vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate > > ok > > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate > > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create > > not ok > > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create > > # Error code: 1 > > # 43,46c43,46 > > # < [0] 0: 4001 2000 2002 3002 4002 > > # < [1] 0: 1001 3000 > > # < [2] 0: 2001 4000 > > # < [3] 0: 3001 1000 > > # --- > > # > [0] 0: 4002 2146435072 2 2146435072 34619728 > > # > [1] 0: 3000 2146435072 > > # > [2] 0: 4000 2146435072 > > # > [3] 0: 3001 2146435072 > > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic > > not ok > > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic > > # Error code: 1 > > # 43,46c43,46 > > # < [0] 0: 4001 2000 2002 3002 4002 > > # < [1] 0: 1001 3000 > > # < [2] 0: 2001 4000 > > # < [3] 0: 3001 1000 > > # --- > > # > [0] 0: 4002 2146435072 2 2146435072 0 > > # > [1] 0: 3000 2146435072 > > # > [2] 0: 4000 2146435072 > > # > [3] 0: 3001 2146435072 > > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate > > ok > > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate > > > > > > > > Il giorno mar 14 set 2021 alle ore 07:44 Stefano Zampini < > > stefano.zampini(a)gmail.com> ha scritto: > > > >> I'll see if I can reproduce > >> > >> Il Mar 14 Set 2021, 06:58 Junchao Zhang <junchao.zhang(a)gmail.com> ha > >> scritto: > >> > >>> Hi, Stefano, > >>> Ping you again to see if you want to resolve this problem before > >>> petsc-3.16 > >>> > >>> --Junchao Zhang > >>> > >>> > >>> On Sun, Sep 12, 2021 at 3:06 PM Antonio T. sagitter < > >>> sagitter(a)fedoraproject.org> wrote: > >>> > >>>> Unfortunately, it's not possible. I must use the OpenMPI provided by > >>>> Fedora build-system (these rpm builds of PETSc are for Fedora's > >>>> repositories), downloading external software is not permitted. > >>>> > >>>> On 9/12/21 21:10, Pierre Jolivet wrote: > >>>> > > >>>> >> On 12 Sep 2021, at 8:56 PM, Matthew Knepley <knepley(a)gmail.com > >>>> >> <mailto:[email protected]>> wrote: > >>>> >> > >>>> >> On Sun, Sep 12, 2021 at 2:49 PM Antonio T. sagitter > >>>> >> <sagitter(a)fedoraproject.org <mailto:[email protected]>> > >>>> wrote: > >>>> >> > >>>> >> Those attached are configure.log/make.log from a MPI build in > >>>> >> Fedora 34 > >>>> >> x86_64 where the error below occurred. > >>>> >> > >>>> >> > >>>> >> This is OpenMPI 4.1.0. Is that the only MPI you build? My first > >>>> >> inclination is that this is an MPI implementation bug. > >>>> >> > >>>> >> Junchao, do we have an OpenMPI build in the CI? > >>>> > > >>>> > config/examples/arch-ci-linux-cuda-double-64idx.py: > >>>> > '--download-openmpi=1', > >>>> > config/examples/arch-ci-linux-pkgs-dbg-ftn-interfaces.py: > >>>> > '--download-openmpi=1', > >>>> > config/examples/arch-ci-linux-pkgs-opt.py: '--download-openmpi=1', > >>>> > > >>>> > config/BuildSystem/config/packages/OpenMPI.py uses version 4.1.0 as > >>>> well. > >>>> > I’m not sure PETSc is to blame here Antonio. You may want to try to > >>>> > ditch the OpenMPI shipped by your packet manager and try > >>>> > --download-openmpi as well, just for a quick sanity check. > >>>> > > >>>> > Thanks, > >>>> > Pierre > >>>> > > >>>> > >>>> -- > >>>> --- > >>>> Antonio Trande > >>>> Fedora Project > >>>> mailto: sagitter(a)fedoraproject.org > >>>> GPG key: 0x29FBC85D7A51CC2F > >>>> GPG key server: https://keyserver1.pgp.com/ > >>>> > >>> > > > > -- > > Stefano > > > > >

1 0

Re: [petsc-dev] Cannot locate file: share/petsc/datafiles/matrices/small
by Matthew Knepley 14 Sep '21

14 Sep '21

Okay, we have to send this to OpenMPI. Volunteers? Maybe we should note this in the FAQ, or installation, so we remember how to fix it if someone else asks? Thanks, Matt On Tue, Sep 14, 2021 at 2:35 AM Stefano Zampini <stefano.zampini(a)gmail.com> wrote: > I can reproduce it even with OpenMPI 4.1.1 on a different machine (Ubuntu > 18 + AMD milan + clang from AOCC) and it is definitely an OpenMPI bug in > the vader BTL If I use tcp, everything runs smooth > > zampins@kanary:~/Devel/petsc$ cat > /home/zampins/local/etc/openmpi-mca-params.conf | grep btl > btl = tcp,self > zampins@kanary:~/Devel/petsc$ make -f gmakefile.test > vec_is_sf_tutorials-ex1_4 > Using MAKEFLAGS: > TEST arch-debug/tests/counts/vec_is_sf_tutorials-ex1_4.counts > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create > ok > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic > ok > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic > ok > vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate > ok > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create > ok > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create > ok > vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic > ok > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic > ok > vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate > ok > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create > ok > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic > ok > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate > ok > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate > > > zampins@kanary:~/Devel/petsc$ cat > /home/zampins/local/etc/openmpi-mca-params.conf | grep btl > btl = vader,tcp,self > zampins@kanary:~/Devel/petsc$ make -f gmakefile.test > vec_is_sf_tutorials-ex1_4 > Using MAKEFLAGS: > TEST arch-debug/tests/counts/vec_is_sf_tutorials-ex1_4.counts > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create > not ok > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create > # Error code: 1 > # 43,46c43,46 > # < [0] 0: 4001 2000 2002 3002 4002 > # < [1] 0: 1001 3000 > # < [2] 0: 2001 4000 > # < [3] 0: 3001 1000 > # --- > # > [0] 0: 2002 2146435072 2 2146435072 38736240 > # > [1] 0: 3000 2146435072 > # > [2] 0: 2001 2146435072 > # > [3] 0: 3001 2146435072 > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic > not ok > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic > # Error code: 1 > # 43,46c43,46 > # < [0] 0: 4001 2000 2002 3002 4002 > # < [1] 0: 1001 3000 > # < [2] 0: 2001 4000 > # < [3] 0: 3001 1000 > # --- > # > [0] 0: 2002 2146435072 2 2146435072 0 > # > [1] 0: 3000 2146435072 > # > [2] 0: 2001 2146435072 > # > [3] 0: 3001 2146435072 > ok > vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate > ok > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate > # retrying > vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create > not ok > vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create # > Error code: 98 > # [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > # [1]PETSC ERROR: General MPI error > # [1]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > # [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > # [1]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b > GIT Date: 2021-09-13 14:01:22 +0000 > # [1]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by > zampins Tue Sep 14 09:31:42 2021 > # [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > # [2]PETSC ERROR: General MPI error > # [2]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > # [2]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > # [2]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b > GIT Date: 2021-09-13 14:01:22 +0000 > # [2]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by > zampins Tue Sep 14 09:31:42 2021 > # [2]PETSC ERROR: Configure options > --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 > --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 > --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 > --with-hip-dir=/opt/rocm --with-hip=1 > --with-hypre-dir=/home/zampins/local-petsc > --with-kokkos-dir=/home/zampins/local-petsc > --with-kokkos-kernels-dir=/home/zampins/local-petsc > --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include > --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" > HIPFLAGS=--amdgpu-target=gfx908 > HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > # [2]PETSC ERROR: #1 PetscSFGetGroups() at > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > # [2]PETSC ERROR: #2 PetscSFSetUp_Window() at > /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > # [2]PETSC ERROR: #3 PetscSFSetUp() at > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > # [2]PETSC ERROR: #4 main() at > /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > # [2]PETSC ERROR: PETSc Option Table entries: > # [2]PETSC ERROR: -sf_type window > # [2]PETSC ERROR: -sf_window_flavor create > # [2]PETSC ERROR: -sf_window_sync active > # [2]PETSC ERROR: -test_gather > # [2]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint(a)mcs.anl.gov---------- > # [3]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > # [3]PETSC ERROR: General MPI error > # [3]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > # [3]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > # [3]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b > GIT Date: 2021-09-13 14:01:22 +0000 > # [3]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by > zampins Tue Sep 14 09:31:42 2021 > # [3]PETSC ERROR: Configure options > --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 > --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 > --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 > --with-hip-dir=/opt/rocm --with-hip=1 > --with-hypre-dir=/home/zampins/local-petsc > --with-kokkos-dir=/home/zampins/local-petsc > --with-kokkos-kernels-dir=/home/zampins/local-petsc > --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include > --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" > HIPFLAGS=--amdgpu-target=gfx908 > HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > # [3]PETSC ERROR: #1 PetscSFGetGroups() at > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > # [3]PETSC ERROR: #2 PetscSFSetUp_Window() at > /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > # [3]PETSC ERROR: #3 PetscSFSetUp() at > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > # [3]PETSC ERROR: #4 main() at > /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > # [3]PETSC ERROR: PETSc Option Table entries: > # [3]PETSC ERROR: -sf_type window > # [3]PETSC ERROR: -sf_window_flavor create > # [3]PETSC ERROR: Configure options > --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 > --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 > --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 > --with-hip-dir=/opt/rocm --with-hip=1 > --with-hypre-dir=/home/zampins/local-petsc > --with-kokkos-dir=/home/zampins/local-petsc > --with-kokkos-kernels-dir=/home/zampins/local-petsc > --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include > --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" > HIPFLAGS=--amdgpu-target=gfx908 > HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > # [1]PETSC ERROR: #1 PetscSFGetGroups() at > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > # [1]PETSC ERROR: #2 PetscSFSetUp_Window() at > /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > # [1]PETSC ERROR: #3 PetscSFSetUp() at > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > # [1]PETSC ERROR: #4 main() at > /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > # [1]PETSC ERROR: PETSc Option Table entries: > # [1]PETSC ERROR: -sf_type window > # [1]PETSC ERROR: -sf_window_flavor create > # [1]PETSC ERROR: -sf_window_sync active > # [1]PETSC ERROR: -test_gather > # [1]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint(a)mcs.anl.gov---------- > # -sf_window_sync active > # [3]PETSC ERROR: -test_gather > # [3]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint(a)mcs.anl.gov---------- > # > -------------------------------------------------------------------------- > # MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD > # with errorcode 98. > # > # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > # You may or may not see output from other processes, depending on > # exactly when Open MPI kills them. > # > -------------------------------------------------------------------------- > # [kanary.kaust.edu.sa:115527] 2 more processes have sent help message > help-mpi-api.txt / mpi-abort > # [kanary.kaust.edu.sa:115527] Set MCA parameter > "orte_base_help_aggregate" to 0 to see all help / error messages > ok vec_is_sf_tutorials-ex1_4 # SKIP Command failed so no diff > # retrying > vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic > not ok > vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic # > Error code: 98 > # [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > # [1]PETSC ERROR: General MPI error > # [1]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > # [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > # [1]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b > GIT Date: 2021-09-13 14:01:22 +0000 > # [1]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by > zampins Tue Sep 14 09:31:45 2021 > # [1]PETSC ERROR: Configure options > --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 > --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 > --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 > --with-hip-dir=/opt/rocm --with-hip=1 > --with-hypre-dir=/home/zampins/local-petsc > --with-kokkos-dir=/home/zampins/local-petsc > --with-kokkos-kernels-dir=/home/zampins/local-petsc > --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include > --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" > HIPFLAGS=--amdgpu-target=gfx908 > HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > # [1]PETSC ERROR: #1 PetscSFGetGroups() at > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > # [1]PETSC ERROR: #2 PetscSFSetUp_Window() at > /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > # [1]PETSC ERROR: #3 PetscSFSetUp() at > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > # [1]PETSC ERROR: #4 main() at > /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > # [1]PETSC ERROR: PETSc Option Table entries: > # [1]PETSC ERROR: -sf_type window > # [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > # [2]PETSC ERROR: General MPI error > # [2]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > # [2]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > # [2]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b > GIT Date: 2021-09-13 14:01:22 +0000 > # [2]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by > zampins Tue Sep 14 09:31:45 2021 > # [2]PETSC ERROR: Configure options > --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 > --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 > --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 > --with-hip-dir=/opt/rocm --with-hip=1 > --with-hypre-dir=/home/zampins/local-petsc > --with-kokkos-dir=/home/zampins/local-petsc > --with-kokkos-kernels-dir=/home/zampins/local-petsc > --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include > --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" > HIPFLAGS=--amdgpu-target=gfx908 > HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > # [2]PETSC ERROR: #1 PetscSFGetGroups() at > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > # [2]PETSC ERROR: #2 PetscSFSetUp_Window() at > /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > # [2]PETSC ERROR: #3 PetscSFSetUp() at > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > # [2]PETSC ERROR: #4 main() at > /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > # [2]PETSC ERROR: PETSc Option Table entries: > # [2]PETSC ERROR: -sf_type window > # [2]PETSC ERROR: -sf_window_flavor dynamic > # [2]PETSC ERROR: -sf_window_sync active > # [2]PETSC ERROR: -test_gather > # [2]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint(a)mcs.anl.gov---------- > # [3]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > # [3]PETSC ERROR: General MPI error > # [3]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > # [3]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > # [3]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b > GIT Date: 2021-09-13 14:01:22 +0000 > # [3]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by > zampins Tue Sep 14 09:31:45 2021 > # [3]PETSC ERROR: Configure options > --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 > --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 > --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 > --with-hip-dir=/opt/rocm --with-hip=1 > --with-hypre-dir=/home/zampins/local-petsc > --with-kokkos-dir=/home/zampins/local-petsc > --with-kokkos-kernels-dir=/home/zampins/local-petsc > --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include > --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" > HIPFLAGS=--amdgpu-target=gfx908 > HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > # [3]PETSC ERROR: #1 PetscSFGetGroups() at > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > # [3]PETSC ERROR: #2 PetscSFSetUp_Window() at > /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > # [3]PETSC ERROR: #3 PetscSFSetUp() at > /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > # [3]PETSC ERROR: #4 main() at > /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > # [3]PETSC ERROR: PETSc Option Table entries: > # [3]PETSC ERROR: -sf_type window > # [3]PETSC ERROR: -sf_window_flavor dynamic > # [3]PETSC ERROR: -sf_window_sync active > # [3]PETSC ERROR: -test_gather > # [3]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint(a)mcs.anl.gov---------- > # -sf_window_flavor dynamic > # [1]PETSC ERROR: -sf_window_sync active > # [1]PETSC ERROR: -test_gather > # [1]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint(a)mcs.anl.gov---------- > # > -------------------------------------------------------------------------- > # MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD > # with errorcode 98. > # > # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > # You may or may not see output from other processes, depending on > # exactly when Open MPI kills them. > # > -------------------------------------------------------------------------- > # [kanary.kaust.edu.sa:115572] 2 more processes have sent help message > help-mpi-api.txt / mpi-abort > # [kanary.kaust.edu.sa:115572] Set MCA parameter > "orte_base_help_aggregate" to 0 to see all help / error messages > ok vec_is_sf_tutorials-ex1_4 # SKIP Command failed so no diff > ok > vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate > ok > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create > not ok > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create > # Error code: 1 > # 43,46c43,46 > # < [0] 0: 4001 2000 2002 3002 4002 > # < [1] 0: 1001 3000 > # < [2] 0: 2001 4000 > # < [3] 0: 3001 1000 > # --- > # > [0] 0: 4002 2146435072 2 2146435072 34619728 > # > [1] 0: 3000 2146435072 > # > [2] 0: 4000 2146435072 > # > [3] 0: 3001 2146435072 > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic > not ok > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic > # Error code: 1 > # 43,46c43,46 > # < [0] 0: 4001 2000 2002 3002 4002 > # < [1] 0: 1001 3000 > # < [2] 0: 2001 4000 > # < [3] 0: 3001 1000 > # --- > # > [0] 0: 4002 2146435072 2 2146435072 0 > # > [1] 0: 3000 2146435072 > # > [2] 0: 4000 2146435072 > # > [3] 0: 3001 2146435072 > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate > ok > diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate > > > > Il giorno mar 14 set 2021 alle ore 07:44 Stefano Zampini < > stefano.zampini(a)gmail.com> ha scritto: > >> I'll see if I can reproduce >> >> Il Mar 14 Set 2021, 06:58 Junchao Zhang <junchao.zhang(a)gmail.com> ha >> scritto: >> >>> Hi, Stefano, >>> Ping you again to see if you want to resolve this problem before >>> petsc-3.16 >>> >>> --Junchao Zhang >>> >>> >>> On Sun, Sep 12, 2021 at 3:06 PM Antonio T. sagitter < >>> sagitter(a)fedoraproject.org> wrote: >>> >>>> Unfortunately, it's not possible. I must use the OpenMPI provided by >>>> Fedora build-system (these rpm builds of PETSc are for Fedora's >>>> repositories), downloading external software is not permitted. >>>> >>>> On 9/12/21 21:10, Pierre Jolivet wrote: >>>> > >>>> >> On 12 Sep 2021, at 8:56 PM, Matthew Knepley <knepley(a)gmail.com >>>> >> <mailto:[email protected]>> wrote: >>>> >> >>>> >> On Sun, Sep 12, 2021 at 2:49 PM Antonio T. sagitter >>>> >> <sagitter(a)fedoraproject.org <mailto:[email protected]>> >>>> wrote: >>>> >> >>>> >> Those attached are configure.log/make.log from a MPI build in >>>> >> Fedora 34 >>>> >> x86_64 where the error below occurred. >>>> >> >>>> >> >>>> >> This is OpenMPI 4.1.0. Is that the only MPI you build? My first >>>> >> inclination is that this is an MPI implementation bug. >>>> >> >>>> >> Junchao, do we have an OpenMPI build in the CI? >>>> > >>>> > config/examples/arch-ci-linux-cuda-double-64idx.py: >>>> > '--download-openmpi=1', >>>> > config/examples/arch-ci-linux-pkgs-dbg-ftn-interfaces.py: >>>> > '--download-openmpi=1', >>>> > config/examples/arch-ci-linux-pkgs-opt.py: '--download-openmpi=1', >>>> > >>>> > config/BuildSystem/config/packages/OpenMPI.py uses version 4.1.0 as >>>> well. >>>> > I’m not sure PETSc is to blame here Antonio. You may want to try to >>>> > ditch the OpenMPI shipped by your packet manager and try >>>> > --download-openmpi as well, just for a quick sanity check. >>>> > >>>> > Thanks, >>>> > Pierre >>>> > >>>> >>>> -- >>>> --- >>>> Antonio Trande >>>> Fedora Project >>>> mailto: sagitter(a)fedoraproject.org >>>> GPG key: 0x29FBC85D7A51CC2F >>>> GPG key server: https://keyserver1.pgp.com/ >>>> >>> > > -- > Stefano > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] Cannot locate file: share/petsc/datafiles/matrices/small
by Stefano Zampini 14 Sep '21

14 Sep '21

I can reproduce it even with OpenMPI 4.1.1 on a different machine (Ubuntu 18 + AMD milan + clang from AOCC) and it is definitely an OpenMPI bug in the vader BTL If I use tcp, everything runs smooth zampins@kanary:~/Devel/petsc$ cat /home/zampins/local/etc/openmpi-mca-params.conf | grep btl btl = tcp,self zampins@kanary:~/Devel/petsc$ make -f gmakefile.test vec_is_sf_tutorials-ex1_4 Using MAKEFLAGS: TEST arch-debug/tests/counts/vec_is_sf_tutorials-ex1_4.counts ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate zampins@kanary:~/Devel/petsc$ cat /home/zampins/local/etc/openmpi-mca-params.conf | grep btl btl = vader,tcp,self zampins@kanary:~/Devel/petsc$ make -f gmakefile.test vec_is_sf_tutorials-ex1_4 Using MAKEFLAGS: TEST arch-debug/tests/counts/vec_is_sf_tutorials-ex1_4.counts ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create not ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create # Error code: 1 # 43,46c43,46 # < [0] 0: 4001 2000 2002 3002 4002 # < [1] 0: 1001 3000 # < [2] 0: 2001 4000 # < [3] 0: 3001 1000 # --- # > [0] 0: 2002 2146435072 2 2146435072 38736240 # > [1] 0: 3000 2146435072 # > [2] 0: 2001 2146435072 # > [3] 0: 3001 2146435072 ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic not ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic # Error code: 1 # 43,46c43,46 # < [0] 0: 4001 2000 2002 3002 4002 # < [1] 0: 1001 3000 # < [2] 0: 2001 4000 # < [3] 0: 3001 1000 # --- # > [0] 0: 2002 2146435072 2 2146435072 0 # > [1] 0: 3000 2146435072 # > [2] 0: 2001 2146435072 # > [3] 0: 3001 2146435072 ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate # retrying vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create not ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create # Error code: 98 # [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- # [1]PETSC ERROR: General MPI error # [1]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank # [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. # [1]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 # [1]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by zampins Tue Sep 14 09:31:42 2021 # [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- # [2]PETSC ERROR: General MPI error # [2]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank # [2]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. # [2]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 # [2]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by zampins Tue Sep 14 09:31:42 2021 # [2]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug # [2]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 # [2]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 # [2]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 # [2]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 # [2]PETSC ERROR: PETSc Option Table entries: # [2]PETSC ERROR: -sf_type window # [2]PETSC ERROR: -sf_window_flavor create # [2]PETSC ERROR: -sf_window_sync active # [2]PETSC ERROR: -test_gather # [2]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- # [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- # [3]PETSC ERROR: General MPI error # [3]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank # [3]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. # [3]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 # [3]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by zampins Tue Sep 14 09:31:42 2021 # [3]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug # [3]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 # [3]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 # [3]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 # [3]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 # [3]PETSC ERROR: PETSc Option Table entries: # [3]PETSC ERROR: -sf_type window # [3]PETSC ERROR: -sf_window_flavor create # [3]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug # [1]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 # [1]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 # [1]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 # [1]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 # [1]PETSC ERROR: PETSc Option Table entries: # [1]PETSC ERROR: -sf_type window # [1]PETSC ERROR: -sf_window_flavor create # [1]PETSC ERROR: -sf_window_sync active # [1]PETSC ERROR: -test_gather # [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- # -sf_window_sync active # [3]PETSC ERROR: -test_gather # [3]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- # -------------------------------------------------------------------------- # MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD # with errorcode 98. # # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. # You may or may not see output from other processes, depending on # exactly when Open MPI kills them. # -------------------------------------------------------------------------- # [kanary.kaust.edu.sa:115527] 2 more processes have sent help message help-mpi-api.txt / mpi-abort # [kanary.kaust.edu.sa:115527] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages ok vec_is_sf_tutorials-ex1_4 # SKIP Command failed so no diff # retrying vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic not ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic # Error code: 98 # [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- # [1]PETSC ERROR: General MPI error # [1]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank # [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. # [1]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 # [1]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by zampins Tue Sep 14 09:31:45 2021 # [1]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug # [1]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 # [1]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 # [1]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 # [1]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 # [1]PETSC ERROR: PETSc Option Table entries: # [1]PETSC ERROR: -sf_type window # [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- # [2]PETSC ERROR: General MPI error # [2]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank # [2]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. # [2]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 # [2]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by zampins Tue Sep 14 09:31:45 2021 # [2]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug # [2]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 # [2]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 # [2]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 # [2]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 # [2]PETSC ERROR: PETSc Option Table entries: # [2]PETSC ERROR: -sf_type window # [2]PETSC ERROR: -sf_window_flavor dynamic # [2]PETSC ERROR: -sf_window_sync active # [2]PETSC ERROR: -test_gather # [2]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- # [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- # [3]PETSC ERROR: General MPI error # [3]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank # [3]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. # [3]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 # [3]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by zampins Tue Sep 14 09:31:45 2021 # [3]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug # [3]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 # [3]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 # [3]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 # [3]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 # [3]PETSC ERROR: PETSc Option Table entries: # [3]PETSC ERROR: -sf_type window # [3]PETSC ERROR: -sf_window_flavor dynamic # [3]PETSC ERROR: -sf_window_sync active # [3]PETSC ERROR: -test_gather # [3]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- # -sf_window_flavor dynamic # [1]PETSC ERROR: -sf_window_sync active # [1]PETSC ERROR: -test_gather # [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- # -------------------------------------------------------------------------- # MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD # with errorcode 98. # # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. # You may or may not see output from other processes, depending on # exactly when Open MPI kills them. # -------------------------------------------------------------------------- # [kanary.kaust.edu.sa:115572] 2 more processes have sent help message help-mpi-api.txt / mpi-abort # [kanary.kaust.edu.sa:115572] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages ok vec_is_sf_tutorials-ex1_4 # SKIP Command failed so no diff ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create not ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create # Error code: 1 # 43,46c43,46 # < [0] 0: 4001 2000 2002 3002 4002 # < [1] 0: 1001 3000 # < [2] 0: 2001 4000 # < [3] 0: 3001 1000 # --- # > [0] 0: 4002 2146435072 2 2146435072 34619728 # > [1] 0: 3000 2146435072 # > [2] 0: 4000 2146435072 # > [3] 0: 3001 2146435072 ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic not ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic # Error code: 1 # 43,46c43,46 # < [0] 0: 4001 2000 2002 3002 4002 # < [1] 0: 1001 3000 # < [2] 0: 2001 4000 # < [3] 0: 3001 1000 # --- # > [0] 0: 4002 2146435072 2 2146435072 0 # > [1] 0: 3000 2146435072 # > [2] 0: 4000 2146435072 # > [3] 0: 3001 2146435072 ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate Il giorno mar 14 set 2021 alle ore 07:44 Stefano Zampini < stefano.zampini(a)gmail.com> ha scritto: > I'll see if I can reproduce > > Il Mar 14 Set 2021, 06:58 Junchao Zhang <junchao.zhang(a)gmail.com> ha > scritto: > >> Hi, Stefano, >> Ping you again to see if you want to resolve this problem before >> petsc-3.16 >> >> --Junchao Zhang >> >> >> On Sun, Sep 12, 2021 at 3:06 PM Antonio T. sagitter < >> sagitter(a)fedoraproject.org> wrote: >> >>> Unfortunately, it's not possible. I must use the OpenMPI provided by >>> Fedora build-system (these rpm builds of PETSc are for Fedora's >>> repositories), downloading external software is not permitted. >>> >>> On 9/12/21 21:10, Pierre Jolivet wrote: >>> > >>> >> On 12 Sep 2021, at 8:56 PM, Matthew Knepley <knepley(a)gmail.com >>> >> <mailto:[email protected]>> wrote: >>> >> >>> >> On Sun, Sep 12, 2021 at 2:49 PM Antonio T. sagitter >>> >> <sagitter(a)fedoraproject.org <mailto:[email protected]>> >>> wrote: >>> >> >>> >> Those attached are configure.log/make.log from a MPI build in >>> >> Fedora 34 >>> >> x86_64 where the error below occurred. >>> >> >>> >> >>> >> This is OpenMPI 4.1.0. Is that the only MPI you build? My first >>> >> inclination is that this is an MPI implementation bug. >>> >> >>> >> Junchao, do we have an OpenMPI build in the CI? >>> > >>> > config/examples/arch-ci-linux-cuda-double-64idx.py: >>> > '--download-openmpi=1', >>> > config/examples/arch-ci-linux-pkgs-dbg-ftn-interfaces.py: >>> > '--download-openmpi=1', >>> > config/examples/arch-ci-linux-pkgs-opt.py: '--download-openmpi=1', >>> > >>> > config/BuildSystem/config/packages/OpenMPI.py uses version 4.1.0 as >>> well. >>> > I’m not sure PETSc is to blame here Antonio. You may want to try to >>> > ditch the OpenMPI shipped by your packet manager and try >>> > --download-openmpi as well, just for a quick sanity check. >>> > >>> > Thanks, >>> > Pierre >>> > >>> >>> -- >>> --- >>> Antonio Trande >>> Fedora Project >>> mailto: sagitter(a)fedoraproject.org >>> GPG key: 0x29FBC85D7A51CC2F >>> GPG key server: https://keyserver1.pgp.com/ >>> >> -- Stefano

1 0

Re: [petsc-dev] Cannot locate file: share/petsc/datafiles/matrices/small
by Stefano Zampini 13 Sep '21

13 Sep '21

I'll see if I can reproduce Il Mar 14 Set 2021, 06:58 Junchao Zhang <junchao.zhang(a)gmail.com> ha scritto: > Hi, Stefano, > Ping you again to see if you want to resolve this problem before > petsc-3.16 > > --Junchao Zhang > > > On Sun, Sep 12, 2021 at 3:06 PM Antonio T. sagitter < > sagitter(a)fedoraproject.org> wrote: > >> Unfortunately, it's not possible. I must use the OpenMPI provided by >> Fedora build-system (these rpm builds of PETSc are for Fedora's >> repositories), downloading external software is not permitted. >> >> On 9/12/21 21:10, Pierre Jolivet wrote: >> > >> >> On 12 Sep 2021, at 8:56 PM, Matthew Knepley <knepley(a)gmail.com >> >> <mailto:[email protected]>> wrote: >> >> >> >> On Sun, Sep 12, 2021 at 2:49 PM Antonio T. sagitter >> >> <sagitter(a)fedoraproject.org <mailto:[email protected]>> >> wrote: >> >> >> >> Those attached are configure.log/make.log from a MPI build in >> >> Fedora 34 >> >> x86_64 where the error below occurred. >> >> >> >> >> >> This is OpenMPI 4.1.0. Is that the only MPI you build? My first >> >> inclination is that this is an MPI implementation bug. >> >> >> >> Junchao, do we have an OpenMPI build in the CI? >> > >> > config/examples/arch-ci-linux-cuda-double-64idx.py: >> > '--download-openmpi=1', >> > config/examples/arch-ci-linux-pkgs-dbg-ftn-interfaces.py: >> > '--download-openmpi=1', >> > config/examples/arch-ci-linux-pkgs-opt.py: '--download-openmpi=1', >> > >> > config/BuildSystem/config/packages/OpenMPI.py uses version 4.1.0 as >> well. >> > I’m not sure PETSc is to blame here Antonio. You may want to try to >> > ditch the OpenMPI shipped by your packet manager and try >> > --download-openmpi as well, just for a quick sanity check. >> > >> > Thanks, >> > Pierre >> > >> >> -- >> --- >> Antonio Trande >> Fedora Project >> mailto: sagitter(a)fedoraproject.org >> GPG key: 0x29FBC85D7A51CC2F >> GPG key server: https://keyserver1.pgp.com/ >> >

1 0

Re: [petsc-dev] Cannot locate file: share/petsc/datafiles/matrices/small
by Junchao Zhang 13 Sep '21

13 Sep '21

Hi, Stefano, Ping you again to see if you want to resolve this problem before petsc-3.16 --Junchao Zhang On Sun, Sep 12, 2021 at 3:06 PM Antonio T. sagitter < sagitter(a)fedoraproject.org> wrote: > Unfortunately, it's not possible. I must use the OpenMPI provided by > Fedora build-system (these rpm builds of PETSc are for Fedora's > repositories), downloading external software is not permitted. > > On 9/12/21 21:10, Pierre Jolivet wrote: > > > >> On 12 Sep 2021, at 8:56 PM, Matthew Knepley <knepley(a)gmail.com > >> <mailto:[email protected]>> wrote: > >> > >> On Sun, Sep 12, 2021 at 2:49 PM Antonio T. sagitter > >> <sagitter(a)fedoraproject.org <mailto:[email protected]>> wrote: > >> > >> Those attached are configure.log/make.log from a MPI build in > >> Fedora 34 > >> x86_64 where the error below occurred. > >> > >> > >> This is OpenMPI 4.1.0. Is that the only MPI you build? My first > >> inclination is that this is an MPI implementation bug. > >> > >> Junchao, do we have an OpenMPI build in the CI? > > > > config/examples/arch-ci-linux-cuda-double-64idx.py: > > '--download-openmpi=1', > > config/examples/arch-ci-linux-pkgs-dbg-ftn-interfaces.py: > > '--download-openmpi=1', > > config/examples/arch-ci-linux-pkgs-opt.py: '--download-openmpi=1', > > > > config/BuildSystem/config/packages/OpenMPI.py uses version 4.1.0 as well. > > I’m not sure PETSc is to blame here Antonio. You may want to try to > > ditch the OpenMPI shipped by your packet manager and try > > --download-openmpi as well, just for a quick sanity check. > > > > Thanks, > > Pierre > > > > -- > --- > Antonio Trande > Fedora Project > mailto: sagitter(a)fedoraproject.org > GPG key: 0x29FBC85D7A51CC2F > GPG key server: https://keyserver1.pgp.com/ >

1 0

Re: [petsc-dev] I have started a new position
by Zhang, Hong 13 Sep '21

13 Sep '21

https://www.simonsfoundation.org/people/barry-smith/ [https://simonsfoundation.imgix.net/wp-content/uploads/2021/09/08170010/Barr…]<https://www.simonsfoundation.org/people/barry-smith/> Barry Smith<https://www.simonsfoundation.org/people/barry-smith/> Barry Smith on Simons Foundation www.simonsfoundation.org ________________________________ From: Barry Smith <bsmith(a)petsc.dev> Sent: Monday, September 13, 2021 11:35 AM To: Zhang, Hong <hzhang(a)mcs.anl.gov> Cc: For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov> Subject: Re: [petsc-dev] I have started a new position The center for computational mathematics, I won't understand a thing in the other centers. Barry On Sep 13, 2021, at 11:15 AM, Zhang, Hong <hzhang(a)mcs.anl.gov<mailto:[email protected]>> wrote: Barry, https://en.wikipedia.org/wiki/Flatiron_Institute Flatiron Institute - Wikipedia<https://en.wikipedia.org/wiki/Flatiron_Institute> The Flatiron Institute is an internal research division of the Simons Foundation, launched in 2016. It comprises five centers for computational science: the Center for Computational Astrophysics (CCA); the Center for Computational Biology (CCB); the Center for Computational Quantum Physics (CCQ); the Center for Computational Mathematics (CCM); and the Center for Computational Neuroscience (CCN). en.wikipedia.org<http://en.wikipedia.org/> Which center you'll be working with? Hong ________________________________ From: petsc-dev <petsc-dev-bounces(a)mcs.anl.gov<mailto:[email protected]>> on behalf of Barry Smith <bsmith(a)petsc.dev<mailto:[email protected]>> Sent: Monday, September 13, 2021 9:24 AM To: For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> Subject: [petsc-dev] I have started a new position I have started a new research position at the Flatiron Institute and stopped my Argonne Associate position. The new position will allow me to continue to work on PETSc as well as work with new groups of people. Barry

1 0

Re: [petsc-dev] I have started a new position
by Barry Smith 13 Sep '21

13 Sep '21

The center for computational mathematics, I won't understand a thing in the other centers. Barry > On Sep 13, 2021, at 11:15 AM, Zhang, Hong <hzhang(a)mcs.anl.gov> wrote: > > Barry, > https://en.wikipedia.org/wiki/Flatiron_Institute <https://en.wikipedia.org/wiki/Flatiron_Institute> > Flatiron Institute - Wikipedia <https://en.wikipedia.org/wiki/Flatiron_Institute> > The Flatiron Institute is an internal research division of the Simons Foundation, launched in 2016. It comprises five centers for computational science: the Center for Computational Astrophysics (CCA); the Center for Computational Biology (CCB); the Center for Computational Quantum Physics (CCQ); the Center for Computational Mathematics (CCM); and the Center for Computational Neuroscience (CCN). > en.wikipedia.org <http://en.wikipedia.org/> > Which center you'll be working with? > Hong > From: petsc-dev <petsc-dev-bounces(a)mcs.anl.gov <mailto:[email protected]>> on behalf of Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> > Sent: Monday, September 13, 2021 9:24 AM > To: For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov <mailto:[email protected]>> > Subject: [petsc-dev] I have started a new position > > > I have started a new research position at the Flatiron Institute and stopped my Argonne Associate position. The new position will allow me to continue to work on PETSc as well as work with new groups of people. > > Barry

1 0

Re: [petsc-dev] I have started a new position
by Matthew Knepley 13 Sep '21

13 Sep '21

You can see Barry in the NA group. Matt On Mon, Sep 13, 2021 at 11:15 AM Zhang, Hong via petsc-dev < petsc-dev(a)mcs.anl.gov> wrote: > Barry, > https://en.wikipedia.org/wiki/Flatiron_Institute > Flatiron Institute - Wikipedia > <https://en.wikipedia.org/wiki/Flatiron_Institute> > The Flatiron Institute is an internal research division of the Simons > Foundation, launched in 2016. It comprises five centers for computational > science: the Center for Computational Astrophysics (CCA); the Center for > Computational Biology (CCB); the Center for Computational Quantum Physics > (CCQ); the Center for Computational Mathematics (CCM); and the Center for > Computational Neuroscience (CCN). > en.wikipedia.org > Which center you'll be working with? > Hong > ------------------------------ > *From:* petsc-dev <petsc-dev-bounces(a)mcs.anl.gov> on behalf of Barry > Smith <bsmith(a)petsc.dev> > *Sent:* Monday, September 13, 2021 9:24 AM > *To:* For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov > > > *Subject:* [petsc-dev] I have started a new position > > > I have started a new research position at the Flatiron Institute and > stopped my Argonne Associate position. The new position will allow me to > continue to work on PETSc as well as work with new groups of people. > > Barry > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] I have started a new position
by Zhang, Hong 13 Sep '21

13 Sep '21

Barry, https://en.wikipedia.org/wiki/Flatiron_Institute Flatiron Institute - Wikipedia<https://en.wikipedia.org/wiki/Flatiron_Institute> The Flatiron Institute is an internal research division of the Simons Foundation, launched in 2016. It comprises five centers for computational science: the Center for Computational Astrophysics (CCA); the Center for Computational Biology (CCB); the Center for Computational Quantum Physics (CCQ); the Center for Computational Mathematics (CCM); and the Center for Computational Neuroscience (CCN). en.wikipedia.org Which center you'll be working with? Hong ________________________________ From: petsc-dev <petsc-dev-bounces(a)mcs.anl.gov> on behalf of Barry Smith <bsmith(a)petsc.dev> Sent: Monday, September 13, 2021 9:24 AM To: For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov> Subject: [petsc-dev] I have started a new position I have started a new research position at the Flatiron Institute and stopped my Argonne Associate position. The new position will allow me to continue to work on PETSc as well as work with new groups of people. Barry

1 0

I have started a new position
by Barry Smith 13 Sep '21

13 Sep '21

I have started a new research position at the Flatiron Institute and stopped my Argonne Associate position. The new position will allow me to continue to work on PETSc as well as work with new groups of people. Barry

1 0