September 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] make test check-test-errors returning $? = 0
by Matthew Knepley 23 Sep '21

23 Sep '21

On Thu, Sep 23, 2021 at 5:07 AM Pierre Jolivet <pierre(a)joliv.et> wrote: > Hello, > When a tests fails because the timeout limit is reached, check-test-errors > still returns a 0 exit code. > Is this intended? Any way to bypass this and return something different > than 0, as it could return false negative in CI tests? > This is the intended behavior. We ignore timeouts, but report them. I would not have a problem with a flag to convert these to errors, but I would leave this default. Thanks, Matt > Thanks, > Pierre > > $ make -f gmakefile test s='ksp_ksp_tutorials-ex2_1' EXTRA_OPTIONS="-m 200 > -n 200" check-test-errors 2>&1 > /dev/null; echo $? > make: *** [check-test-errors] Error 1 > 2 > $ make -f gmakefile test s='ksp_ksp_tutorials-ex2_1' EXTRA_OPTIONS="-m 200 > -n 200" check-test-errors TIMEOUT=1 2>&1 > /dev/null; echo $? > 0 -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

make test check-test-errors returning $? = 0
by Pierre Jolivet 23 Sep '21

23 Sep '21

Hello, When a tests fails because the timeout limit is reached, check-test-errors still returns a 0 exit code. Is this intended? Any way to bypass this and return something different than 0, as it could return false negative in CI tests? Thanks, Pierre $ make -f gmakefile test s='ksp_ksp_tutorials-ex2_1' EXTRA_OPTIONS="-m 200 -n 200" check-test-errors 2>&1 > /dev/null; echo $? make: *** [check-test-errors] Error 1 2 $ make -f gmakefile test s='ksp_ksp_tutorials-ex2_1' EXTRA_OPTIONS="-m 200 -n 200" check-test-errors TIMEOUT=1 2>&1 > /dev/null; echo $? 0

1 0

Re: [petsc-dev] https://www.nsf.gov/pubs/2021/nsf21617/nsf21617.htm
by Matthew Knepley 20 Sep '21

20 Sep '21

On Sat, Sep 18, 2021 at 1:02 PM Barry Smith <bsmith(a)petsc.dev> wrote: > > > - > > The Program Description has been updated to introduce a new project > class on “Transition to Sustainability.” This solicitation welcomes > Elements, Framework Implementations, and Transition to Sustainability > proposals. > - The description of the “Framework Implementations” class has been > revised to encourage proposals for sharable and reusable multi-disciplinary > cyberinfrastructure. > > > - A new “CI Professional Mentoring and/or Professional Development > Plan” requirement has been introduced. > > > - Proposals may now include requests for high-throughput computing > resources through the Partnership to Advance Throughput Computing (PATh) > project supported by NSF. > > > - The Programmatic Areas of Interest section has been revised to > reflect the most recent programmatic priority areas for the collaborating > NSF directorates and divisions with respect to the CSSI solicitation. PIs > are strongly encouraged to contact program officer(s) from the list of > Cognizant Program Officers in the division(s) that typically support the > scientists and engineers who would make use of the proposed > cyberinfrastructure, to gain insight into the priorities for the relevant > areas of science and engineering to which their proposals may be responsive. > > I would be open to a Frameworks proposal. I think it would need to have a focused goal, something like one thrust of our DOE proposal, rather than be a broad "PETSc" proposal. Thanks, Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

https://www.nsf.gov/pubs/2021/nsf21617/nsf21617.htm
by Barry Smith 18 Sep '21

18 Sep '21

The Program Description has been updated to introduce a new project class on “Transition to Sustainability.” This solicitation welcomes Elements, Framework Implementations, and Transition to Sustainability proposals. The description of the “Framework Implementations” class has been revised to encourage proposals for sharable and reusable multi-disciplinary cyberinfrastructure. A new “CI Professional Mentoring and/or Professional Development Plan” requirement has been introduced. Proposals may now include requests for high-throughput computing resources through the Partnership to Advance Throughput Computing (PATh) project supported by NSF. The Programmatic Areas of Interest section has been revised to reflect the most recent programmatic priority areas for the collaborating NSF directorates and divisions with respect to the CSSI solicitation. PIs are strongly encouraged to contact program officer(s) from the list of Cognizant Program Officers in the division(s) that typically support the scientists and engineers who would make use of the proposed cyberinfrastructure, to gain insight into the priorities for the relevant areas of science and engineering to which their proposals may be responsive.

1 0

Re: [petsc-dev] Cannot locate file: share/petsc/datafiles/matrices/small
by Junchao Zhang 14 Sep '21

14 Sep '21

Yes, we can turn it off. The code without real use is just a maintenance burden. --Junchao Zhang On Tue, Sep 14, 2021 at 10:45 AM Barry Smith <bsmith(a)petsc.dev> wrote: > > Ok, so it could be a bug in PETSc, but if it appears with particular MPI > implementations shouldn't we turn off the support in those cases we know it > will fail? > > Barry > > > On Sep 14, 2021, at 11:10 AM, Junchao Zhang <junchao.zhang(a)gmail.com> > wrote: > > MPI one-sided is tricky and needs careful synchronization (like OpenMP). > An incorrect code could work in one interface but fail in another. > > --Junchao Zhang > > > On Tue, Sep 14, 2021 at 10:01 AM Barry Smith <bsmith(a)petsc.dev> wrote: > >> >> It sounds reproducible and related to using a particular versions of >> OpenMPI and even particular interfaces. >> >> Barry >> >> On Tue, Sep 14, 2021 at 2:35 AM Stefano Zampini < >> stefano.zampini(a)gmail.com> wrote: >>> >>> I can reproduce it even with OpenMPI 4.1.1 on a different machine >>> (Ubuntu 18 + AMD milan + clang from AOCC) and it is definitely an OpenMPI >>> bug in the vader BTL If I use tcp, everything runs smooth >>> >>> >>> >>> >> On Sep 14, 2021, at 10:54 AM, Junchao Zhang <junchao.zhang(a)gmail.com> >> wrote: >> >> Without a standalone & valid mpi example to reproduce the error, we are >> not assured to say it is an OpenMPI bug. >> >> --Junchao Zhang >> >> >> On Tue, Sep 14, 2021 at 6:17 AM Matthew Knepley <knepley(a)gmail.com> >> wrote: >> >>> Okay, we have to send this to OpenMPI. Volunteers? >>> >>> Maybe we should note this in the FAQ, or installation, so we remember >>> how to fix it if someone else asks? >>> >>> Thanks, >>> >>> Matt >>> >>> On Tue, Sep 14, 2021 at 2:35 AM Stefano Zampini < >>> stefano.zampini(a)gmail.com> wrote: >>> >>>> I can reproduce it even with OpenMPI 4.1.1 on a different machine >>>> (Ubuntu 18 + AMD milan + clang from AOCC) and it is definitely an OpenMPI >>>> bug in the vader BTL If I use tcp, everything runs smooth >>>> >>>> zampins@kanary:~/Devel/petsc$ cat >>>> /home/zampins/local/etc/openmpi-mca-params.conf | grep btl >>>> btl = tcp,self >>>> zampins@kanary:~/Devel/petsc$ make -f gmakefile.test >>>> vec_is_sf_tutorials-ex1_4 >>>> Using MAKEFLAGS: >>>> TEST arch-debug/tests/counts/vec_is_sf_tutorials-ex1_4.counts >>>> ok >>>> vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create >>>> ok >>>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create >>>> ok >>>> vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic >>>> ok >>>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic >>>> ok >>>> vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate >>>> ok >>>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate >>>> ok >>>> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create >>>> ok >>>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create >>>> ok >>>> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic >>>> ok >>>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic >>>> ok >>>> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate >>>> ok >>>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate >>>> ok >>>> vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create >>>> ok >>>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create >>>> ok >>>> vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic >>>> ok >>>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic >>>> ok >>>> vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate >>>> ok >>>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate >>>> >>>> >>>> zampins@kanary:~/Devel/petsc$ cat >>>> /home/zampins/local/etc/openmpi-mca-params.conf | grep btl >>>> btl = vader,tcp,self >>>> zampins@kanary:~/Devel/petsc$ make -f gmakefile.test >>>> vec_is_sf_tutorials-ex1_4 >>>> Using MAKEFLAGS: >>>> TEST arch-debug/tests/counts/vec_is_sf_tutorials-ex1_4.counts >>>> ok >>>> vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create >>>> not ok >>>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create >>>> # Error code: 1 >>>> # 43,46c43,46 >>>> # < [0] 0: 4001 2000 2002 3002 4002 >>>> # < [1] 0: 1001 3000 >>>> # < [2] 0: 2001 4000 >>>> # < [3] 0: 3001 1000 >>>> # --- >>>> # > [0] 0: 2002 2146435072 2 2146435072 38736240 >>>> # > [1] 0: 3000 2146435072 >>>> # > [2] 0: 2001 2146435072 >>>> # > [3] 0: 3001 2146435072 >>>> ok >>>> vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic >>>> not ok >>>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic >>>> # Error code: 1 >>>> # 43,46c43,46 >>>> # < [0] 0: 4001 2000 2002 3002 4002 >>>> # < [1] 0: 1001 3000 >>>> # < [2] 0: 2001 4000 >>>> # < [3] 0: 3001 1000 >>>> # --- >>>> # > [0] 0: 2002 2146435072 2 2146435072 0 >>>> # > [1] 0: 3000 2146435072 >>>> # > [2] 0: 2001 2146435072 >>>> # > [3] 0: 3001 2146435072 >>>> ok >>>> vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate >>>> ok >>>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate >>>> # retrying >>>> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create >>>> not ok >>>> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create # >>>> Error code: 98 >>>> # [1]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> # [1]PETSC ERROR: General MPI error >>>> # [1]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >>>> # [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >>>> shooting. >>>> # [1]PETSC ERROR: Petsc Development GIT revision: >>>> v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 >>>> # [1]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by >>>> zampins Tue Sep 14 09:31:42 2021 >>>> # [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> # [2]PETSC ERROR: General MPI error >>>> # [2]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >>>> # [2]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >>>> shooting. >>>> # [2]PETSC ERROR: Petsc Development GIT revision: >>>> v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 >>>> # [2]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by >>>> zampins Tue Sep 14 09:31:42 2021 >>>> # [2]PETSC ERROR: Configure options >>>> --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 >>>> --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 >>>> --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 >>>> --with-hip-dir=/opt/rocm --with-hip=1 >>>> --with-hypre-dir=/home/zampins/local-petsc >>>> --with-kokkos-dir=/home/zampins/local-petsc >>>> --with-kokkos-kernels-dir=/home/zampins/local-petsc >>>> --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include >>>> --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" >>>> HIPFLAGS=--amdgpu-target=gfx908 >>>> HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >>>> # [2]PETSC ERROR: #1 PetscSFGetGroups() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >>>> # [2]PETSC ERROR: #2 PetscSFSetUp_Window() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >>>> # [2]PETSC ERROR: #3 PetscSFSetUp() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >>>> # [2]PETSC ERROR: #4 main() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >>>> # [2]PETSC ERROR: PETSc Option Table entries: >>>> # [2]PETSC ERROR: -sf_type window >>>> # [2]PETSC ERROR: -sf_window_flavor create >>>> # [2]PETSC ERROR: -sf_window_sync active >>>> # [2]PETSC ERROR: -test_gather >>>> # [2]PETSC ERROR: ----------------End of Error Message -------send >>>> entire error message to petsc-maint(a)mcs.anl.gov---------- >>>> # [3]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> # [3]PETSC ERROR: General MPI error >>>> # [3]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >>>> # [3]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >>>> shooting. >>>> # [3]PETSC ERROR: Petsc Development GIT revision: >>>> v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 >>>> # [3]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by >>>> zampins Tue Sep 14 09:31:42 2021 >>>> # [3]PETSC ERROR: Configure options >>>> --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 >>>> --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 >>>> --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 >>>> --with-hip-dir=/opt/rocm --with-hip=1 >>>> --with-hypre-dir=/home/zampins/local-petsc >>>> --with-kokkos-dir=/home/zampins/local-petsc >>>> --with-kokkos-kernels-dir=/home/zampins/local-petsc >>>> --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include >>>> --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" >>>> HIPFLAGS=--amdgpu-target=gfx908 >>>> HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >>>> # [3]PETSC ERROR: #1 PetscSFGetGroups() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >>>> # [3]PETSC ERROR: #2 PetscSFSetUp_Window() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >>>> # [3]PETSC ERROR: #3 PetscSFSetUp() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >>>> # [3]PETSC ERROR: #4 main() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >>>> # [3]PETSC ERROR: PETSc Option Table entries: >>>> # [3]PETSC ERROR: -sf_type window >>>> # [3]PETSC ERROR: -sf_window_flavor create >>>> # [3]PETSC ERROR: Configure options >>>> --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 >>>> --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 >>>> --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 >>>> --with-hip-dir=/opt/rocm --with-hip=1 >>>> --with-hypre-dir=/home/zampins/local-petsc >>>> --with-kokkos-dir=/home/zampins/local-petsc >>>> --with-kokkos-kernels-dir=/home/zampins/local-petsc >>>> --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include >>>> --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" >>>> HIPFLAGS=--amdgpu-target=gfx908 >>>> HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >>>> # [1]PETSC ERROR: #1 PetscSFGetGroups() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >>>> # [1]PETSC ERROR: #2 PetscSFSetUp_Window() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >>>> # [1]PETSC ERROR: #3 PetscSFSetUp() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >>>> # [1]PETSC ERROR: #4 main() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >>>> # [1]PETSC ERROR: PETSc Option Table entries: >>>> # [1]PETSC ERROR: -sf_type window >>>> # [1]PETSC ERROR: -sf_window_flavor create >>>> # [1]PETSC ERROR: -sf_window_sync active >>>> # [1]PETSC ERROR: -test_gather >>>> # [1]PETSC ERROR: ----------------End of Error Message -------send >>>> entire error message to petsc-maint(a)mcs.anl.gov---------- >>>> # -sf_window_sync active >>>> # [3]PETSC ERROR: -test_gather >>>> # [3]PETSC ERROR: ----------------End of Error Message -------send >>>> entire error message to petsc-maint(a)mcs.anl.gov---------- >>>> # >>>> -------------------------------------------------------------------------- >>>> # MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD >>>> # with errorcode 98. >>>> # >>>> # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>> # You may or may not see output from other processes, depending on >>>> # exactly when Open MPI kills them. >>>> # >>>> -------------------------------------------------------------------------- >>>> # [kanary.kaust.edu.sa:115527] 2 more processes have sent help message >>>> help-mpi-api.txt / mpi-abort >>>> # [kanary.kaust.edu.sa:115527] Set MCA parameter >>>> "orte_base_help_aggregate" to 0 to see all help / error messages >>>> ok vec_is_sf_tutorials-ex1_4 # SKIP Command failed so no diff >>>> # retrying >>>> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic >>>> not ok >>>> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic # >>>> Error code: 98 >>>> # [1]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> # [1]PETSC ERROR: General MPI error >>>> # [1]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >>>> # [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >>>> shooting. >>>> # [1]PETSC ERROR: Petsc Development GIT revision: >>>> v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 >>>> # [1]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by >>>> zampins Tue Sep 14 09:31:45 2021 >>>> # [1]PETSC ERROR: Configure options >>>> --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 >>>> --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 >>>> --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 >>>> --with-hip-dir=/opt/rocm --with-hip=1 >>>> --with-hypre-dir=/home/zampins/local-petsc >>>> --with-kokkos-dir=/home/zampins/local-petsc >>>> --with-kokkos-kernels-dir=/home/zampins/local-petsc >>>> --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include >>>> --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" >>>> HIPFLAGS=--amdgpu-target=gfx908 >>>> HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >>>> # [1]PETSC ERROR: #1 PetscSFGetGroups() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >>>> # [1]PETSC ERROR: #2 PetscSFSetUp_Window() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >>>> # [1]PETSC ERROR: #3 PetscSFSetUp() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >>>> # [1]PETSC ERROR: #4 main() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >>>> # [1]PETSC ERROR: PETSc Option Table entries: >>>> # [1]PETSC ERROR: -sf_type window >>>> # [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> # [2]PETSC ERROR: General MPI error >>>> # [2]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >>>> # [2]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >>>> shooting. >>>> # [2]PETSC ERROR: Petsc Development GIT revision: >>>> v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 >>>> # [2]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by >>>> zampins Tue Sep 14 09:31:45 2021 >>>> # [2]PETSC ERROR: Configure options >>>> --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 >>>> --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 >>>> --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 >>>> --with-hip-dir=/opt/rocm --with-hip=1 >>>> --with-hypre-dir=/home/zampins/local-petsc >>>> --with-kokkos-dir=/home/zampins/local-petsc >>>> --with-kokkos-kernels-dir=/home/zampins/local-petsc >>>> --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include >>>> --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" >>>> HIPFLAGS=--amdgpu-target=gfx908 >>>> HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >>>> # [2]PETSC ERROR: #1 PetscSFGetGroups() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >>>> # [2]PETSC ERROR: #2 PetscSFSetUp_Window() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >>>> # [2]PETSC ERROR: #3 PetscSFSetUp() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >>>> # [2]PETSC ERROR: #4 main() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >>>> # [2]PETSC ERROR: PETSc Option Table entries: >>>> # [2]PETSC ERROR: -sf_type window >>>> # [2]PETSC ERROR: -sf_window_flavor dynamic >>>> # [2]PETSC ERROR: -sf_window_sync active >>>> # [2]PETSC ERROR: -test_gather >>>> # [2]PETSC ERROR: ----------------End of Error Message -------send >>>> entire error message to petsc-maint(a)mcs.anl.gov---------- >>>> # [3]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> # [3]PETSC ERROR: General MPI error >>>> # [3]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >>>> # [3]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >>>> shooting. >>>> # [3]PETSC ERROR: Petsc Development GIT revision: >>>> v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 >>>> # [3]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by >>>> zampins Tue Sep 14 09:31:45 2021 >>>> # [3]PETSC ERROR: Configure options >>>> --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 >>>> --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 >>>> --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 >>>> --with-hip-dir=/opt/rocm --with-hip=1 >>>> --with-hypre-dir=/home/zampins/local-petsc >>>> --with-kokkos-dir=/home/zampins/local-petsc >>>> --with-kokkos-kernels-dir=/home/zampins/local-petsc >>>> --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include >>>> --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" >>>> HIPFLAGS=--amdgpu-target=gfx908 >>>> HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >>>> # [3]PETSC ERROR: #1 PetscSFGetGroups() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >>>> # [3]PETSC ERROR: #2 PetscSFSetUp_Window() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >>>> # [3]PETSC ERROR: #3 PetscSFSetUp() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >>>> # [3]PETSC ERROR: #4 main() at >>>> /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >>>> # [3]PETSC ERROR: PETSc Option Table entries: >>>> # [3]PETSC ERROR: -sf_type window >>>> # [3]PETSC ERROR: -sf_window_flavor dynamic >>>> # [3]PETSC ERROR: -sf_window_sync active >>>> # [3]PETSC ERROR: -test_gather >>>> # [3]PETSC ERROR: ----------------End of Error Message -------send >>>> entire error message to petsc-maint(a)mcs.anl.gov---------- >>>> # -sf_window_flavor dynamic >>>> # [1]PETSC ERROR: -sf_window_sync active >>>> # [1]PETSC ERROR: -test_gather >>>> # [1]PETSC ERROR: ----------------End of Error Message -------send >>>> entire error message to petsc-maint(a)mcs.anl.gov---------- >>>> # >>>> -------------------------------------------------------------------------- >>>> # MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD >>>> # with errorcode 98. >>>> # >>>> # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>> # You may or may not see output from other processes, depending on >>>> # exactly when Open MPI kills them. >>>> # >>>> -------------------------------------------------------------------------- >>>> # [kanary.kaust.edu.sa:115572] 2 more processes have sent help message >>>> help-mpi-api.txt / mpi-abort >>>> # [kanary.kaust.edu.sa:115572] Set MCA parameter >>>> "orte_base_help_aggregate" to 0 to see all help / error messages >>>> ok vec_is_sf_tutorials-ex1_4 # SKIP Command failed so no diff >>>> ok >>>> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate >>>> ok >>>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate >>>> ok >>>> vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create >>>> not ok >>>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create >>>> # Error code: 1 >>>> # 43,46c43,46 >>>> # < [0] 0: 4001 2000 2002 3002 4002 >>>> # < [1] 0: 1001 3000 >>>> # < [2] 0: 2001 4000 >>>> # < [3] 0: 3001 1000 >>>> # --- >>>> # > [0] 0: 4002 2146435072 2 2146435072 34619728 >>>> # > [1] 0: 3000 2146435072 >>>> # > [2] 0: 4000 2146435072 >>>> # > [3] 0: 3001 2146435072 >>>> ok >>>> vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic >>>> not ok >>>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic >>>> # Error code: 1 >>>> # 43,46c43,46 >>>> # < [0] 0: 4001 2000 2002 3002 4002 >>>> # < [1] 0: 1001 3000 >>>> # < [2] 0: 2001 4000 >>>> # < [3] 0: 3001 1000 >>>> # --- >>>> # > [0] 0: 4002 2146435072 2 2146435072 0 >>>> # > [1] 0: 3000 2146435072 >>>> # > [2] 0: 4000 2146435072 >>>> # > [3] 0: 3001 2146435072 >>>> ok >>>> vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate >>>> ok >>>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate >>>> >>>> >>>> >>>> Il giorno mar 14 set 2021 alle ore 07:44 Stefano Zampini < >>>> stefano.zampini(a)gmail.com> ha scritto: >>>> >>>>> I'll see if I can reproduce >>>>> >>>>> Il Mar 14 Set 2021, 06:58 Junchao Zhang <junchao.zhang(a)gmail.com> ha >>>>> scritto: >>>>> >>>>>> Hi, Stefano, >>>>>> Ping you again to see if you want to resolve this problem before >>>>>> petsc-3.16 >>>>>> >>>>>> --Junchao Zhang >>>>>> >>>>>> >>>>>> On Sun, Sep 12, 2021 at 3:06 PM Antonio T. sagitter < >>>>>> sagitter(a)fedoraproject.org> wrote: >>>>>> >>>>>>> Unfortunately, it's not possible. I must use the OpenMPI provided by >>>>>>> Fedora build-system (these rpm builds of PETSc are for Fedora's >>>>>>> repositories), downloading external software is not permitted. >>>>>>> >>>>>>> On 9/12/21 21:10, Pierre Jolivet wrote: >>>>>>> > >>>>>>> >> On 12 Sep 2021, at 8:56 PM, Matthew Knepley <knepley(a)gmail.com >>>>>>> >> <mailto:[email protected]>> wrote: >>>>>>> >> >>>>>>> >> On Sun, Sep 12, 2021 at 2:49 PM Antonio T. sagitter >>>>>>> >> <sagitter(a)fedoraproject.org <mailto:[email protected]>> >>>>>>> wrote: >>>>>>> >> >>>>>>> >> Those attached are configure.log/make.log from a MPI build in >>>>>>> >> Fedora 34 >>>>>>> >> x86_64 where the error below occurred. >>>>>>> >> >>>>>>> >> >>>>>>> >> This is OpenMPI 4.1.0. Is that the only MPI you build? My first >>>>>>> >> inclination is that this is an MPI implementation bug. >>>>>>> >> >>>>>>> >> Junchao, do we have an OpenMPI build in the CI? >>>>>>> > >>>>>>> > config/examples/arch-ci-linux-cuda-double-64idx.py: >>>>>>> > '--download-openmpi=1', >>>>>>> > config/examples/arch-ci-linux-pkgs-dbg-ftn-interfaces.py: >>>>>>> > '--download-openmpi=1', >>>>>>> > config/examples/arch-ci-linux-pkgs-opt.py: '--download-openmpi=1', >>>>>>> > >>>>>>> > config/BuildSystem/config/packages/OpenMPI.py uses version 4.1.0 >>>>>>> as well. >>>>>>> > I’m not sure PETSc is to blame here Antonio. You may want to try >>>>>>> to >>>>>>> > ditch the OpenMPI shipped by your packet manager and try >>>>>>> > --download-openmpi as well, just for a quick sanity check. >>>>>>> > >>>>>>> > Thanks, >>>>>>> > Pierre >>>>>>> > >>>>>>> >>>>>>> -- >>>>>>> --- >>>>>>> Antonio Trande >>>>>>> Fedora Project >>>>>>> mailto: sagitter(a)fedoraproject.org >>>>>>> GPG key: 0x29FBC85D7A51CC2F >>>>>>> GPG key server: https://keyserver1.pgp.com/ >>>>>>> >>>>>> >>>> >>>> -- >>>> Stefano >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> <http://www.cse.buffalo.edu/~knepley/> >>> >> >> >

1 0

Re: [petsc-dev] Cannot locate file: share/petsc/datafiles/matrices/small
by Barry Smith 14 Sep '21

14 Sep '21

Ok, so it could be a bug in PETSc, but if it appears with particular MPI implementations shouldn't we turn off the support in those cases we know it will fail? Barry > On Sep 14, 2021, at 11:10 AM, Junchao Zhang <junchao.zhang(a)gmail.com> wrote: > > MPI one-sided is tricky and needs careful synchronization (like OpenMP). An incorrect code could work in one interface but fail in another. > > --Junchao Zhang > > > On Tue, Sep 14, 2021 at 10:01 AM Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: > > It sounds reproducible and related to using a particular versions of OpenMPI and even particular interfaces. > > Barry > > On Tue, Sep 14, 2021 at 2:35 AM Stefano Zampini <stefano.zampini(a)gmail.com <mailto:[email protected]>> wrote: > I can reproduce it even with OpenMPI 4.1.1 on a different machine (Ubuntu 18 + AMD milan + clang from AOCC) and it is definitely an OpenMPI bug in the vader BTL If I use tcp, everything runs smooth > > > > >> On Sep 14, 2021, at 10:54 AM, Junchao Zhang <junchao.zhang(a)gmail.com <mailto:[email protected]>> wrote: >> >> Without a standalone & valid mpi example to reproduce the error, we are not assured to say it is an OpenMPI bug. >> >> --Junchao Zhang >> >> >> On Tue, Sep 14, 2021 at 6:17 AM Matthew Knepley <knepley(a)gmail.com <mailto:[email protected]>> wrote: >> Okay, we have to send this to OpenMPI. Volunteers? >> >> Maybe we should note this in the FAQ, or installation, so we remember how to fix it if someone else asks? >> >> Thanks, >> >> Matt >> >> On Tue, Sep 14, 2021 at 2:35 AM Stefano Zampini <stefano.zampini(a)gmail.com <mailto:[email protected]>> wrote: >> I can reproduce it even with OpenMPI 4.1.1 on a different machine (Ubuntu 18 + AMD milan + clang from AOCC) and it is definitely an OpenMPI bug in the vader BTL If I use tcp, everything runs smooth >> >> zampins@kanary:~/Devel/petsc$ cat /home/zampins/local/etc/openmpi-mca-params.conf | grep btl >> btl = tcp,self >> zampins@kanary:~/Devel/petsc$ make -f gmakefile.test vec_is_sf_tutorials-ex1_4 >> Using MAKEFLAGS: >> TEST arch-debug/tests/counts/vec_is_sf_tutorials-ex1_4.counts >> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create >> ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create >> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic >> ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic >> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate >> ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate >> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create >> ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create >> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic >> ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic >> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate >> ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate >> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create >> ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create >> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic >> ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic >> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate >> ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate >> >> >> zampins@kanary:~/Devel/petsc$ cat /home/zampins/local/etc/openmpi-mca-params.conf | grep btl >> btl = vader,tcp,self >> zampins@kanary:~/Devel/petsc$ make -f gmakefile.test vec_is_sf_tutorials-ex1_4 >> Using MAKEFLAGS: >> TEST arch-debug/tests/counts/vec_is_sf_tutorials-ex1_4.counts >> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create >> not ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create # Error code: 1 >> # 43,46c43,46 >> # < [0] 0: 4001 2000 2002 3002 4002 >> # < [1] 0: 1001 3000 >> # < [2] 0: 2001 4000 >> # < [3] 0: 3001 1000 >> # --- >> # > [0] 0: 2002 2146435072 2 2146435072 38736240 >> # > [1] 0: 3000 2146435072 >> # > [2] 0: 2001 2146435072 >> # > [3] 0: 3001 2146435072 >> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic >> not ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic # Error code: 1 >> # 43,46c43,46 >> # < [0] 0: 4001 2000 2002 3002 4002 >> # < [1] 0: 1001 3000 >> # < [2] 0: 2001 4000 >> # < [3] 0: 3001 1000 >> # --- >> # > [0] 0: 2002 2146435072 2 2146435072 0 >> # > [1] 0: 3000 2146435072 >> # > [2] 0: 2001 2146435072 >> # > [3] 0: 3001 2146435072 >> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate >> ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate >> # retrying vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create >> not ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create # Error code: 98 >> # [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> # [1]PETSC ERROR: General MPI error >> # [1]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >> # [1]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting. >> # [1]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 >> # [1]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa <http://kanary.kaust.edu.sa/> by zampins Tue Sep 14 09:31:42 2021 >> # [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> # [2]PETSC ERROR: General MPI error >> # [2]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >> # [2]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting. >> # [2]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 >> # [2]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa <http://kanary.kaust.edu.sa/> by zampins Tue Sep 14 09:31:42 2021 >> # [2]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >> # [2]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >> # [2]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >> # [2]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >> # [2]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >> # [2]PETSC ERROR: PETSc Option Table entries: >> # [2]PETSC ERROR: -sf_type window >> # [2]PETSC ERROR: -sf_window_flavor create >> # [2]PETSC ERROR: -sf_window_sync active >> # [2]PETSC ERROR: -test_gather >> # [2]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov <mailto:[email protected]>---------- >> # [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> # [3]PETSC ERROR: General MPI error >> # [3]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >> # [3]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting. >> # [3]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 >> # [3]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa <http://kanary.kaust.edu.sa/> by zampins Tue Sep 14 09:31:42 2021 >> # [3]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >> # [3]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >> # [3]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >> # [3]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >> # [3]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >> # [3]PETSC ERROR: PETSc Option Table entries: >> # [3]PETSC ERROR: -sf_type window >> # [3]PETSC ERROR: -sf_window_flavor create >> # [3]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >> # [1]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >> # [1]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >> # [1]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >> # [1]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >> # [1]PETSC ERROR: PETSc Option Table entries: >> # [1]PETSC ERROR: -sf_type window >> # [1]PETSC ERROR: -sf_window_flavor create >> # [1]PETSC ERROR: -sf_window_sync active >> # [1]PETSC ERROR: -test_gather >> # [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov <mailto:[email protected]>---------- >> # -sf_window_sync active >> # [3]PETSC ERROR: -test_gather >> # [3]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov <mailto:[email protected]>---------- >> # -------------------------------------------------------------------------- >> # MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD >> # with errorcode 98. >> # >> # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> # You may or may not see output from other processes, depending on >> # exactly when Open MPI kills them. >> # -------------------------------------------------------------------------- >> # [kanary.kaust.edu.sa:115527 <http://kanary.kaust.edu.sa:115527>] 2 more processes have sent help message help-mpi-api.txt / mpi-abort >> # [kanary.kaust.edu.sa:115527 <http://kanary.kaust.edu.sa:115527>] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages >> ok vec_is_sf_tutorials-ex1_4 # SKIP Command failed so no diff >> # retrying vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic >> not ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic # Error code: 98 >> # [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> # [1]PETSC ERROR: General MPI error >> # [1]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >> # [1]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting. >> # [1]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 >> # [1]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa <http://kanary.kaust.edu.sa/> by zampins Tue Sep 14 09:31:45 2021 >> # [1]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >> # [1]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >> # [1]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >> # [1]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >> # [1]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >> # [1]PETSC ERROR: PETSc Option Table entries: >> # [1]PETSC ERROR: -sf_type window >> # [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> # [2]PETSC ERROR: General MPI error >> # [2]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >> # [2]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting. >> # [2]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 >> # [2]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa <http://kanary.kaust.edu.sa/> by zampins Tue Sep 14 09:31:45 2021 >> # [2]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >> # [2]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >> # [2]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >> # [2]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >> # [2]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >> # [2]PETSC ERROR: PETSc Option Table entries: >> # [2]PETSC ERROR: -sf_type window >> # [2]PETSC ERROR: -sf_window_flavor dynamic >> # [2]PETSC ERROR: -sf_window_sync active >> # [2]PETSC ERROR: -test_gather >> # [2]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov <mailto:[email protected]>---------- >> # [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >> # [3]PETSC ERROR: General MPI error >> # [3]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >> # [3]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting. >> # [3]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 >> # [3]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa <http://kanary.kaust.edu.sa/> by zampins Tue Sep 14 09:31:45 2021 >> # [3]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >> # [3]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >> # [3]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >> # [3]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >> # [3]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >> # [3]PETSC ERROR: PETSc Option Table entries: >> # [3]PETSC ERROR: -sf_type window >> # [3]PETSC ERROR: -sf_window_flavor dynamic >> # [3]PETSC ERROR: -sf_window_sync active >> # [3]PETSC ERROR: -test_gather >> # [3]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov <mailto:[email protected]>---------- >> # -sf_window_flavor dynamic >> # [1]PETSC ERROR: -sf_window_sync active >> # [1]PETSC ERROR: -test_gather >> # [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov <mailto:[email protected]>---------- >> # -------------------------------------------------------------------------- >> # MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD >> # with errorcode 98. >> # >> # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> # You may or may not see output from other processes, depending on >> # exactly when Open MPI kills them. >> # -------------------------------------------------------------------------- >> # [kanary.kaust.edu.sa:115572 <http://kanary.kaust.edu.sa:115572>] 2 more processes have sent help message help-mpi-api.txt / mpi-abort >> # [kanary.kaust.edu.sa:115572 <http://kanary.kaust.edu.sa:115572>] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages >> ok vec_is_sf_tutorials-ex1_4 # SKIP Command failed so no diff >> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate >> ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate >> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create >> not ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create # Error code: 1 >> # 43,46c43,46 >> # < [0] 0: 4001 2000 2002 3002 4002 >> # < [1] 0: 1001 3000 >> # < [2] 0: 2001 4000 >> # < [3] 0: 3001 1000 >> # --- >> # > [0] 0: 4002 2146435072 2 2146435072 34619728 >> # > [1] 0: 3000 2146435072 >> # > [2] 0: 4000 2146435072 >> # > [3] 0: 3001 2146435072 >> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic >> not ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic # Error code: 1 >> # 43,46c43,46 >> # < [0] 0: 4001 2000 2002 3002 4002 >> # < [1] 0: 1001 3000 >> # < [2] 0: 2001 4000 >> # < [3] 0: 3001 1000 >> # --- >> # > [0] 0: 4002 2146435072 2 2146435072 0 >> # > [1] 0: 3000 2146435072 >> # > [2] 0: 4000 2146435072 >> # > [3] 0: 3001 2146435072 >> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate >> ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate >> >> >> >> Il giorno mar 14 set 2021 alle ore 07:44 Stefano Zampini <stefano.zampini(a)gmail.com <mailto:[email protected]>> ha scritto: >> I'll see if I can reproduce >> >> Il Mar 14 Set 2021, 06:58 Junchao Zhang <junchao.zhang(a)gmail.com <mailto:[email protected]>> ha scritto: >> Hi, Stefano, >> Ping you again to see if you want to resolve this problem before petsc-3.16 >> >> --Junchao Zhang >> >> >> On Sun, Sep 12, 2021 at 3:06 PM Antonio T. sagitter <sagitter(a)fedoraproject.org <mailto:[email protected]>> wrote: >> Unfortunately, it's not possible. I must use the OpenMPI provided by >> Fedora build-system (these rpm builds of PETSc are for Fedora's >> repositories), downloading external software is not permitted. >> >> On 9/12/21 21:10, Pierre Jolivet wrote: >> > >> >> On 12 Sep 2021, at 8:56 PM, Matthew Knepley <knepley(a)gmail.com <mailto:[email protected]> >> >> <mailto:[email protected] <mailto:[email protected]>>> wrote: >> >> >> >> On Sun, Sep 12, 2021 at 2:49 PM Antonio T. sagitter >> >> <sagitter(a)fedoraproject.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> wrote: >> >> >> >> Those attached are configure.log/make.log from a MPI build in >> >> Fedora 34 >> >> x86_64 where the error below occurred. >> >> >> >> >> >> This is OpenMPI 4.1.0. Is that the only MPI you build? My first >> >> inclination is that this is an MPI implementation bug. >> >> >> >> Junchao, do we have an OpenMPI build in the CI? >> > >> > config/examples/arch-ci-linux-cuda-double-64idx.py: >> > '--download-openmpi=1', >> > config/examples/arch-ci-linux-pkgs-dbg-ftn-interfaces.py: >> > '--download-openmpi=1', >> > config/examples/arch-ci-linux-pkgs-opt.py: '--download-openmpi=1', >> > >> > config/BuildSystem/config/packages/OpenMPI.py uses version 4.1.0 as well. >> > I’m not sure PETSc is to blame here Antonio. You may want to try to >> > ditch the OpenMPI shipped by your packet manager and try >> > --download-openmpi as well, just for a quick sanity check. >> > >> > Thanks, >> > Pierre >> > >> >> -- >> --- >> Antonio Trande >> Fedora Project >> mailto: sagitter(a)fedoraproject.org <mailto:[email protected]> >> GPG key: 0x29FBC85D7A51CC2F >> GPG key server: https://keyserver1.pgp.com/ <https://keyserver1.pgp.com/> >> >> >> -- >> Stefano >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/> >

1 0

Re: [petsc-dev] Cannot locate file: share/petsc/datafiles/matrices/small
by Junchao Zhang 14 Sep '21

14 Sep '21

MPI one-sided is tricky and needs careful synchronization (like OpenMP). An incorrect code could work in one interface but fail in another. --Junchao Zhang On Tue, Sep 14, 2021 at 10:01 AM Barry Smith <bsmith(a)petsc.dev> wrote: > > It sounds reproducible and related to using a particular versions of > OpenMPI and even particular interfaces. > > Barry > > On Tue, Sep 14, 2021 at 2:35 AM Stefano Zampini < > stefano.zampini(a)gmail.com> wrote: >> >> I can reproduce it even with OpenMPI 4.1.1 on a different machine (Ubuntu >> 18 + AMD milan + clang from AOCC) and it is definitely an OpenMPI bug in >> the vader BTL If I use tcp, everything runs smooth >> >> >> >> > On Sep 14, 2021, at 10:54 AM, Junchao Zhang <junchao.zhang(a)gmail.com> > wrote: > > Without a standalone & valid mpi example to reproduce the error, we are > not assured to say it is an OpenMPI bug. > > --Junchao Zhang > > > On Tue, Sep 14, 2021 at 6:17 AM Matthew Knepley <knepley(a)gmail.com> wrote: > >> Okay, we have to send this to OpenMPI. Volunteers? >> >> Maybe we should note this in the FAQ, or installation, so we remember how >> to fix it if someone else asks? >> >> Thanks, >> >> Matt >> >> On Tue, Sep 14, 2021 at 2:35 AM Stefano Zampini < >> stefano.zampini(a)gmail.com> wrote: >> >>> I can reproduce it even with OpenMPI 4.1.1 on a different machine >>> (Ubuntu 18 + AMD milan + clang from AOCC) and it is definitely an OpenMPI >>> bug in the vader BTL If I use tcp, everything runs smooth >>> >>> zampins@kanary:~/Devel/petsc$ cat >>> /home/zampins/local/etc/openmpi-mca-params.conf | grep btl >>> btl = tcp,self >>> zampins@kanary:~/Devel/petsc$ make -f gmakefile.test >>> vec_is_sf_tutorials-ex1_4 >>> Using MAKEFLAGS: >>> TEST arch-debug/tests/counts/vec_is_sf_tutorials-ex1_4.counts >>> ok >>> vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create >>> ok >>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create >>> ok >>> vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic >>> ok >>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic >>> ok >>> vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate >>> ok >>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate >>> ok >>> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create >>> ok >>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create >>> ok >>> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic >>> ok >>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic >>> ok >>> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate >>> ok >>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate >>> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create >>> ok >>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create >>> ok >>> vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic >>> ok >>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic >>> ok >>> vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate >>> ok >>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate >>> >>> >>> zampins@kanary:~/Devel/petsc$ cat >>> /home/zampins/local/etc/openmpi-mca-params.conf | grep btl >>> btl = vader,tcp,self >>> zampins@kanary:~/Devel/petsc$ make -f gmakefile.test >>> vec_is_sf_tutorials-ex1_4 >>> Using MAKEFLAGS: >>> TEST arch-debug/tests/counts/vec_is_sf_tutorials-ex1_4.counts >>> ok >>> vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create >>> not ok >>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create >>> # Error code: 1 >>> # 43,46c43,46 >>> # < [0] 0: 4001 2000 2002 3002 4002 >>> # < [1] 0: 1001 3000 >>> # < [2] 0: 2001 4000 >>> # < [3] 0: 3001 1000 >>> # --- >>> # > [0] 0: 2002 2146435072 2 2146435072 38736240 >>> # > [1] 0: 3000 2146435072 >>> # > [2] 0: 2001 2146435072 >>> # > [3] 0: 3001 2146435072 >>> ok >>> vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic >>> not ok >>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic >>> # Error code: 1 >>> # 43,46c43,46 >>> # < [0] 0: 4001 2000 2002 3002 4002 >>> # < [1] 0: 1001 3000 >>> # < [2] 0: 2001 4000 >>> # < [3] 0: 3001 1000 >>> # --- >>> # > [0] 0: 2002 2146435072 2 2146435072 0 >>> # > [1] 0: 3000 2146435072 >>> # > [2] 0: 2001 2146435072 >>> # > [3] 0: 3001 2146435072 >>> ok >>> vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate >>> ok >>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate >>> # retrying >>> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create >>> not ok >>> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create # >>> Error code: 98 >>> # [1]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> # [1]PETSC ERROR: General MPI error >>> # [1]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >>> # [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >>> shooting. >>> # [1]PETSC ERROR: Petsc Development GIT revision: >>> v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 >>> # [1]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by >>> zampins Tue Sep 14 09:31:42 2021 >>> # [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> # [2]PETSC ERROR: General MPI error >>> # [2]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >>> # [2]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >>> shooting. >>> # [2]PETSC ERROR: Petsc Development GIT revision: >>> v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 >>> # [2]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by >>> zampins Tue Sep 14 09:31:42 2021 >>> # [2]PETSC ERROR: Configure options >>> --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 >>> --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 >>> --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 >>> --with-hip-dir=/opt/rocm --with-hip=1 >>> --with-hypre-dir=/home/zampins/local-petsc >>> --with-kokkos-dir=/home/zampins/local-petsc >>> --with-kokkos-kernels-dir=/home/zampins/local-petsc >>> --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include >>> --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" >>> HIPFLAGS=--amdgpu-target=gfx908 >>> HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >>> # [2]PETSC ERROR: #1 PetscSFGetGroups() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >>> # [2]PETSC ERROR: #2 PetscSFSetUp_Window() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >>> # [2]PETSC ERROR: #3 PetscSFSetUp() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >>> # [2]PETSC ERROR: #4 main() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >>> # [2]PETSC ERROR: PETSc Option Table entries: >>> # [2]PETSC ERROR: -sf_type window >>> # [2]PETSC ERROR: -sf_window_flavor create >>> # [2]PETSC ERROR: -sf_window_sync active >>> # [2]PETSC ERROR: -test_gather >>> # [2]PETSC ERROR: ----------------End of Error Message -------send >>> entire error message to petsc-maint(a)mcs.anl.gov---------- >>> # [3]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> # [3]PETSC ERROR: General MPI error >>> # [3]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >>> # [3]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >>> shooting. >>> # [3]PETSC ERROR: Petsc Development GIT revision: >>> v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 >>> # [3]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by >>> zampins Tue Sep 14 09:31:42 2021 >>> # [3]PETSC ERROR: Configure options >>> --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 >>> --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 >>> --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 >>> --with-hip-dir=/opt/rocm --with-hip=1 >>> --with-hypre-dir=/home/zampins/local-petsc >>> --with-kokkos-dir=/home/zampins/local-petsc >>> --with-kokkos-kernels-dir=/home/zampins/local-petsc >>> --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include >>> --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" >>> HIPFLAGS=--amdgpu-target=gfx908 >>> HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >>> # [3]PETSC ERROR: #1 PetscSFGetGroups() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >>> # [3]PETSC ERROR: #2 PetscSFSetUp_Window() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >>> # [3]PETSC ERROR: #3 PetscSFSetUp() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >>> # [3]PETSC ERROR: #4 main() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >>> # [3]PETSC ERROR: PETSc Option Table entries: >>> # [3]PETSC ERROR: -sf_type window >>> # [3]PETSC ERROR: -sf_window_flavor create >>> # [3]PETSC ERROR: Configure options >>> --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 >>> --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 >>> --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 >>> --with-hip-dir=/opt/rocm --with-hip=1 >>> --with-hypre-dir=/home/zampins/local-petsc >>> --with-kokkos-dir=/home/zampins/local-petsc >>> --with-kokkos-kernels-dir=/home/zampins/local-petsc >>> --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include >>> --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" >>> HIPFLAGS=--amdgpu-target=gfx908 >>> HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >>> # [1]PETSC ERROR: #1 PetscSFGetGroups() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >>> # [1]PETSC ERROR: #2 PetscSFSetUp_Window() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >>> # [1]PETSC ERROR: #3 PetscSFSetUp() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >>> # [1]PETSC ERROR: #4 main() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >>> # [1]PETSC ERROR: PETSc Option Table entries: >>> # [1]PETSC ERROR: -sf_type window >>> # [1]PETSC ERROR: -sf_window_flavor create >>> # [1]PETSC ERROR: -sf_window_sync active >>> # [1]PETSC ERROR: -test_gather >>> # [1]PETSC ERROR: ----------------End of Error Message -------send >>> entire error message to petsc-maint(a)mcs.anl.gov---------- >>> # -sf_window_sync active >>> # [3]PETSC ERROR: -test_gather >>> # [3]PETSC ERROR: ----------------End of Error Message -------send >>> entire error message to petsc-maint(a)mcs.anl.gov---------- >>> # >>> -------------------------------------------------------------------------- >>> # MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD >>> # with errorcode 98. >>> # >>> # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>> # You may or may not see output from other processes, depending on >>> # exactly when Open MPI kills them. >>> # >>> -------------------------------------------------------------------------- >>> # [kanary.kaust.edu.sa:115527] 2 more processes have sent help message >>> help-mpi-api.txt / mpi-abort >>> # [kanary.kaust.edu.sa:115527] Set MCA parameter >>> "orte_base_help_aggregate" to 0 to see all help / error messages >>> ok vec_is_sf_tutorials-ex1_4 # SKIP Command failed so no diff >>> # retrying >>> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic >>> not ok >>> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic # >>> Error code: 98 >>> # [1]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> # [1]PETSC ERROR: General MPI error >>> # [1]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >>> # [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >>> shooting. >>> # [1]PETSC ERROR: Petsc Development GIT revision: >>> v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 >>> # [1]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by >>> zampins Tue Sep 14 09:31:45 2021 >>> # [1]PETSC ERROR: Configure options >>> --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 >>> --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 >>> --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 >>> --with-hip-dir=/opt/rocm --with-hip=1 >>> --with-hypre-dir=/home/zampins/local-petsc >>> --with-kokkos-dir=/home/zampins/local-petsc >>> --with-kokkos-kernels-dir=/home/zampins/local-petsc >>> --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include >>> --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" >>> HIPFLAGS=--amdgpu-target=gfx908 >>> HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >>> # [1]PETSC ERROR: #1 PetscSFGetGroups() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >>> # [1]PETSC ERROR: #2 PetscSFSetUp_Window() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >>> # [1]PETSC ERROR: #3 PetscSFSetUp() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >>> # [1]PETSC ERROR: #4 main() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >>> # [1]PETSC ERROR: PETSc Option Table entries: >>> # [1]PETSC ERROR: -sf_type window >>> # [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> # [2]PETSC ERROR: General MPI error >>> # [2]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >>> # [2]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >>> shooting. >>> # [2]PETSC ERROR: Petsc Development GIT revision: >>> v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 >>> # [2]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by >>> zampins Tue Sep 14 09:31:45 2021 >>> # [2]PETSC ERROR: Configure options >>> --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 >>> --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 >>> --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 >>> --with-hip-dir=/opt/rocm --with-hip=1 >>> --with-hypre-dir=/home/zampins/local-petsc >>> --with-kokkos-dir=/home/zampins/local-petsc >>> --with-kokkos-kernels-dir=/home/zampins/local-petsc >>> --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include >>> --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" >>> HIPFLAGS=--amdgpu-target=gfx908 >>> HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >>> # [2]PETSC ERROR: #1 PetscSFGetGroups() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >>> # [2]PETSC ERROR: #2 PetscSFSetUp_Window() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >>> # [2]PETSC ERROR: #3 PetscSFSetUp() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >>> # [2]PETSC ERROR: #4 main() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >>> # [2]PETSC ERROR: PETSc Option Table entries: >>> # [2]PETSC ERROR: -sf_type window >>> # [2]PETSC ERROR: -sf_window_flavor dynamic >>> # [2]PETSC ERROR: -sf_window_sync active >>> # [2]PETSC ERROR: -test_gather >>> # [2]PETSC ERROR: ----------------End of Error Message -------send >>> entire error message to petsc-maint(a)mcs.anl.gov---------- >>> # [3]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> # [3]PETSC ERROR: General MPI error >>> # [3]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >>> # [3]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >>> shooting. >>> # [3]PETSC ERROR: Petsc Development GIT revision: >>> v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 >>> # [3]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by >>> zampins Tue Sep 14 09:31:45 2021 >>> # [3]PETSC ERROR: Configure options >>> --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 >>> --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 >>> --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 >>> --with-hip-dir=/opt/rocm --with-hip=1 >>> --with-hypre-dir=/home/zampins/local-petsc >>> --with-kokkos-dir=/home/zampins/local-petsc >>> --with-kokkos-kernels-dir=/home/zampins/local-petsc >>> --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include >>> --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" >>> HIPFLAGS=--amdgpu-target=gfx908 >>> HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >>> # [3]PETSC ERROR: #1 PetscSFGetGroups() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >>> # [3]PETSC ERROR: #2 PetscSFSetUp_Window() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >>> # [3]PETSC ERROR: #3 PetscSFSetUp() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >>> # [3]PETSC ERROR: #4 main() at >>> /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >>> # [3]PETSC ERROR: PETSc Option Table entries: >>> # [3]PETSC ERROR: -sf_type window >>> # [3]PETSC ERROR: -sf_window_flavor dynamic >>> # [3]PETSC ERROR: -sf_window_sync active >>> # [3]PETSC ERROR: -test_gather >>> # [3]PETSC ERROR: ----------------End of Error Message -------send >>> entire error message to petsc-maint(a)mcs.anl.gov---------- >>> # -sf_window_flavor dynamic >>> # [1]PETSC ERROR: -sf_window_sync active >>> # [1]PETSC ERROR: -test_gather >>> # [1]PETSC ERROR: ----------------End of Error Message -------send >>> entire error message to petsc-maint(a)mcs.anl.gov---------- >>> # >>> -------------------------------------------------------------------------- >>> # MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD >>> # with errorcode 98. >>> # >>> # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>> # You may or may not see output from other processes, depending on >>> # exactly when Open MPI kills them. >>> # >>> -------------------------------------------------------------------------- >>> # [kanary.kaust.edu.sa:115572] 2 more processes have sent help message >>> help-mpi-api.txt / mpi-abort >>> # [kanary.kaust.edu.sa:115572] Set MCA parameter >>> "orte_base_help_aggregate" to 0 to see all help / error messages >>> ok vec_is_sf_tutorials-ex1_4 # SKIP Command failed so no diff >>> ok >>> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate >>> ok >>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate >>> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create >>> not ok >>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create >>> # Error code: 1 >>> # 43,46c43,46 >>> # < [0] 0: 4001 2000 2002 3002 4002 >>> # < [1] 0: 1001 3000 >>> # < [2] 0: 2001 4000 >>> # < [3] 0: 3001 1000 >>> # --- >>> # > [0] 0: 4002 2146435072 2 2146435072 34619728 >>> # > [1] 0: 3000 2146435072 >>> # > [2] 0: 4000 2146435072 >>> # > [3] 0: 3001 2146435072 >>> ok >>> vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic >>> not ok >>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic >>> # Error code: 1 >>> # 43,46c43,46 >>> # < [0] 0: 4001 2000 2002 3002 4002 >>> # < [1] 0: 1001 3000 >>> # < [2] 0: 2001 4000 >>> # < [3] 0: 3001 1000 >>> # --- >>> # > [0] 0: 4002 2146435072 2 2146435072 0 >>> # > [1] 0: 3000 2146435072 >>> # > [2] 0: 4000 2146435072 >>> # > [3] 0: 3001 2146435072 >>> ok >>> vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate >>> ok >>> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate >>> >>> >>> >>> Il giorno mar 14 set 2021 alle ore 07:44 Stefano Zampini < >>> stefano.zampini(a)gmail.com> ha scritto: >>> >>>> I'll see if I can reproduce >>>> >>>> Il Mar 14 Set 2021, 06:58 Junchao Zhang <junchao.zhang(a)gmail.com> ha >>>> scritto: >>>> >>>>> Hi, Stefano, >>>>> Ping you again to see if you want to resolve this problem before >>>>> petsc-3.16 >>>>> >>>>> --Junchao Zhang >>>>> >>>>> >>>>> On Sun, Sep 12, 2021 at 3:06 PM Antonio T. sagitter < >>>>> sagitter(a)fedoraproject.org> wrote: >>>>> >>>>>> Unfortunately, it's not possible. I must use the OpenMPI provided by >>>>>> Fedora build-system (these rpm builds of PETSc are for Fedora's >>>>>> repositories), downloading external software is not permitted. >>>>>> >>>>>> On 9/12/21 21:10, Pierre Jolivet wrote: >>>>>> > >>>>>> >> On 12 Sep 2021, at 8:56 PM, Matthew Knepley <knepley(a)gmail.com >>>>>> >> <mailto:[email protected]>> wrote: >>>>>> >> >>>>>> >> On Sun, Sep 12, 2021 at 2:49 PM Antonio T. sagitter >>>>>> >> <sagitter(a)fedoraproject.org <mailto:[email protected]>> >>>>>> wrote: >>>>>> >> >>>>>> >> Those attached are configure.log/make.log from a MPI build in >>>>>> >> Fedora 34 >>>>>> >> x86_64 where the error below occurred. >>>>>> >> >>>>>> >> >>>>>> >> This is OpenMPI 4.1.0. Is that the only MPI you build? My first >>>>>> >> inclination is that this is an MPI implementation bug. >>>>>> >> >>>>>> >> Junchao, do we have an OpenMPI build in the CI? >>>>>> > >>>>>> > config/examples/arch-ci-linux-cuda-double-64idx.py: >>>>>> > '--download-openmpi=1', >>>>>> > config/examples/arch-ci-linux-pkgs-dbg-ftn-interfaces.py: >>>>>> > '--download-openmpi=1', >>>>>> > config/examples/arch-ci-linux-pkgs-opt.py: '--download-openmpi=1', >>>>>> > >>>>>> > config/BuildSystem/config/packages/OpenMPI.py uses version 4.1.0 as >>>>>> well. >>>>>> > I’m not sure PETSc is to blame here Antonio. You may want to try to >>>>>> > ditch the OpenMPI shipped by your packet manager and try >>>>>> > --download-openmpi as well, just for a quick sanity check. >>>>>> > >>>>>> > Thanks, >>>>>> > Pierre >>>>>> > >>>>>> >>>>>> -- >>>>>> --- >>>>>> Antonio Trande >>>>>> Fedora Project >>>>>> mailto: sagitter(a)fedoraproject.org >>>>>> GPG key: 0x29FBC85D7A51CC2F >>>>>> GPG key server: https://keyserver1.pgp.com/ >>>>>> >>>>> >>> >>> -- >>> Stefano >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> > >

1 0

Re: [petsc-dev] Cannot locate file: share/petsc/datafiles/matrices/small
by Barry Smith 14 Sep '21

14 Sep '21

It sounds reproducible and related to using a particular versions of OpenMPI and even particular interfaces. Barry On Tue, Sep 14, 2021 at 2:35 AM Stefano Zampini <stefano.zampini(a)gmail.com <mailto:[email protected]>> wrote: I can reproduce it even with OpenMPI 4.1.1 on a different machine (Ubuntu 18 + AMD milan + clang from AOCC) and it is definitely an OpenMPI bug in the vader BTL If I use tcp, everything runs smooth > On Sep 14, 2021, at 10:54 AM, Junchao Zhang <junchao.zhang(a)gmail.com> wrote: > > Without a standalone & valid mpi example to reproduce the error, we are not assured to say it is an OpenMPI bug. > > --Junchao Zhang > > > On Tue, Sep 14, 2021 at 6:17 AM Matthew Knepley <knepley(a)gmail.com <mailto:[email protected]>> wrote: > Okay, we have to send this to OpenMPI. Volunteers? > > Maybe we should note this in the FAQ, or installation, so we remember how to fix it if someone else asks? > > Thanks, > > Matt > > On Tue, Sep 14, 2021 at 2:35 AM Stefano Zampini <stefano.zampini(a)gmail.com <mailto:[email protected]>> wrote: > I can reproduce it even with OpenMPI 4.1.1 on a different machine (Ubuntu 18 + AMD milan + clang from AOCC) and it is definitely an OpenMPI bug in the vader BTL If I use tcp, everything runs smooth > > zampins@kanary:~/Devel/petsc$ cat /home/zampins/local/etc/openmpi-mca-params.conf | grep btl > btl = tcp,self > zampins@kanary:~/Devel/petsc$ make -f gmakefile.test vec_is_sf_tutorials-ex1_4 > Using MAKEFLAGS: > TEST arch-debug/tests/counts/vec_is_sf_tutorials-ex1_4.counts > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate > > > zampins@kanary:~/Devel/petsc$ cat /home/zampins/local/etc/openmpi-mca-params.conf | grep btl > btl = vader,tcp,self > zampins@kanary:~/Devel/petsc$ make -f gmakefile.test vec_is_sf_tutorials-ex1_4 > Using MAKEFLAGS: > TEST arch-debug/tests/counts/vec_is_sf_tutorials-ex1_4.counts > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create > not ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create # Error code: 1 > # 43,46c43,46 > # < [0] 0: 4001 2000 2002 3002 4002 > # < [1] 0: 1001 3000 > # < [2] 0: 2001 4000 > # < [3] 0: 3001 1000 > # --- > # > [0] 0: 2002 2146435072 2 2146435072 38736240 > # > [1] 0: 3000 2146435072 > # > [2] 0: 2001 2146435072 > # > [3] 0: 3001 2146435072 > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic > not ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic # Error code: 1 > # 43,46c43,46 > # < [0] 0: 4001 2000 2002 3002 4002 > # < [1] 0: 1001 3000 > # < [2] 0: 2001 4000 > # < [3] 0: 3001 1000 > # --- > # > [0] 0: 2002 2146435072 2 2146435072 0 > # > [1] 0: 3000 2146435072 > # > [2] 0: 2001 2146435072 > # > [3] 0: 3001 2146435072 > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate > # retrying vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create > not ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create # Error code: 98 > # [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > # [1]PETSC ERROR: General MPI error > # [1]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > # [1]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting. > # [1]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 > # [1]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa <http://kanary.kaust.edu.sa/> by zampins Tue Sep 14 09:31:42 2021 > # [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > # [2]PETSC ERROR: General MPI error > # [2]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > # [2]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting. > # [2]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 > # [2]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa <http://kanary.kaust.edu.sa/> by zampins Tue Sep 14 09:31:42 2021 > # [2]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > # [2]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > # [2]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > # [2]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > # [2]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > # [2]PETSC ERROR: PETSc Option Table entries: > # [2]PETSC ERROR: -sf_type window > # [2]PETSC ERROR: -sf_window_flavor create > # [2]PETSC ERROR: -sf_window_sync active > # [2]PETSC ERROR: -test_gather > # [2]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- > # [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > # [3]PETSC ERROR: General MPI error > # [3]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > # [3]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting. > # [3]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 > # [3]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa <http://kanary.kaust.edu.sa/> by zampins Tue Sep 14 09:31:42 2021 > # [3]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > # [3]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > # [3]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > # [3]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > # [3]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > # [3]PETSC ERROR: PETSc Option Table entries: > # [3]PETSC ERROR: -sf_type window > # [3]PETSC ERROR: -sf_window_flavor create > # [3]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > # [1]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > # [1]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > # [1]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > # [1]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > # [1]PETSC ERROR: PETSc Option Table entries: > # [1]PETSC ERROR: -sf_type window > # [1]PETSC ERROR: -sf_window_flavor create > # [1]PETSC ERROR: -sf_window_sync active > # [1]PETSC ERROR: -test_gather > # [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- > # -sf_window_sync active > # [3]PETSC ERROR: -test_gather > # [3]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- > # -------------------------------------------------------------------------- > # MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD > # with errorcode 98. > # > # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > # You may or may not see output from other processes, depending on > # exactly when Open MPI kills them. > # -------------------------------------------------------------------------- > # [kanary.kaust.edu.sa:115527 <http://kanary.kaust.edu.sa:115527>] 2 more processes have sent help message help-mpi-api.txt / mpi-abort > # [kanary.kaust.edu.sa:115527 <http://kanary.kaust.edu.sa:115527>] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages > ok vec_is_sf_tutorials-ex1_4 # SKIP Command failed so no diff > # retrying vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic > not ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic # Error code: 98 > # [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > # [1]PETSC ERROR: General MPI error > # [1]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > # [1]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting. > # [1]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 > # [1]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa <http://kanary.kaust.edu.sa/> by zampins Tue Sep 14 09:31:45 2021 > # [1]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > # [1]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > # [1]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > # [1]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > # [1]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > # [1]PETSC ERROR: PETSc Option Table entries: > # [1]PETSC ERROR: -sf_type window > # [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > # [2]PETSC ERROR: General MPI error > # [2]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > # [2]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting. > # [2]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 > # [2]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa <http://kanary.kaust.edu.sa/> by zampins Tue Sep 14 09:31:45 2021 > # [2]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > # [2]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > # [2]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > # [2]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > # [2]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > # [2]PETSC ERROR: PETSc Option Table entries: > # [2]PETSC ERROR: -sf_type window > # [2]PETSC ERROR: -sf_window_flavor dynamic > # [2]PETSC ERROR: -sf_window_sync active > # [2]PETSC ERROR: -test_gather > # [2]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- > # [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > # [3]PETSC ERROR: General MPI error > # [3]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > # [3]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting. > # [3]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 > # [3]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa <http://kanary.kaust.edu.sa/> by zampins Tue Sep 14 09:31:45 2021 > # [3]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > # [3]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > # [3]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > # [3]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > # [3]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > # [3]PETSC ERROR: PETSc Option Table entries: > # [3]PETSC ERROR: -sf_type window > # [3]PETSC ERROR: -sf_window_flavor dynamic > # [3]PETSC ERROR: -sf_window_sync active > # [3]PETSC ERROR: -test_gather > # [3]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- > # -sf_window_flavor dynamic > # [1]PETSC ERROR: -sf_window_sync active > # [1]PETSC ERROR: -test_gather > # [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- > # -------------------------------------------------------------------------- > # MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD > # with errorcode 98. > # > # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > # You may or may not see output from other processes, depending on > # exactly when Open MPI kills them. > # -------------------------------------------------------------------------- > # [kanary.kaust.edu.sa:115572 <http://kanary.kaust.edu.sa:115572>] 2 more processes have sent help message help-mpi-api.txt / mpi-abort > # [kanary.kaust.edu.sa:115572 <http://kanary.kaust.edu.sa:115572>] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages > ok vec_is_sf_tutorials-ex1_4 # SKIP Command failed so no diff > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create > not ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create # Error code: 1 > # 43,46c43,46 > # < [0] 0: 4001 2000 2002 3002 4002 > # < [1] 0: 1001 3000 > # < [2] 0: 2001 4000 > # < [3] 0: 3001 1000 > # --- > # > [0] 0: 4002 2146435072 2 2146435072 34619728 > # > [1] 0: 3000 2146435072 > # > [2] 0: 4000 2146435072 > # > [3] 0: 3001 2146435072 > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic > not ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic # Error code: 1 > # 43,46c43,46 > # < [0] 0: 4001 2000 2002 3002 4002 > # < [1] 0: 1001 3000 > # < [2] 0: 2001 4000 > # < [3] 0: 3001 1000 > # --- > # > [0] 0: 4002 2146435072 2 2146435072 0 > # > [1] 0: 3000 2146435072 > # > [2] 0: 4000 2146435072 > # > [3] 0: 3001 2146435072 > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate > > > > Il giorno mar 14 set 2021 alle ore 07:44 Stefano Zampini <stefano.zampini(a)gmail.com <mailto:[email protected]>> ha scritto: > I'll see if I can reproduce > > Il Mar 14 Set 2021, 06:58 Junchao Zhang <junchao.zhang(a)gmail.com <mailto:[email protected]>> ha scritto: > Hi, Stefano, > Ping you again to see if you want to resolve this problem before petsc-3.16 > > --Junchao Zhang > > > On Sun, Sep 12, 2021 at 3:06 PM Antonio T. sagitter <sagitter(a)fedoraproject.org <mailto:[email protected]>> wrote: > Unfortunately, it's not possible. I must use the OpenMPI provided by > Fedora build-system (these rpm builds of PETSc are for Fedora's > repositories), downloading external software is not permitted. > > On 9/12/21 21:10, Pierre Jolivet wrote: > > > >> On 12 Sep 2021, at 8:56 PM, Matthew Knepley <knepley(a)gmail.com <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>>> wrote: > >> > >> On Sun, Sep 12, 2021 at 2:49 PM Antonio T. sagitter > >> <sagitter(a)fedoraproject.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> wrote: > >> > >> Those attached are configure.log/make.log from a MPI build in > >> Fedora 34 > >> x86_64 where the error below occurred. > >> > >> > >> This is OpenMPI 4.1.0. Is that the only MPI you build? My first > >> inclination is that this is an MPI implementation bug. > >> > >> Junchao, do we have an OpenMPI build in the CI? > > > > config/examples/arch-ci-linux-cuda-double-64idx.py: > > '--download-openmpi=1', > > config/examples/arch-ci-linux-pkgs-dbg-ftn-interfaces.py: > > '--download-openmpi=1', > > config/examples/arch-ci-linux-pkgs-opt.py: '--download-openmpi=1', > > > > config/BuildSystem/config/packages/OpenMPI.py uses version 4.1.0 as well. > > I’m not sure PETSc is to blame here Antonio. You may want to try to > > ditch the OpenMPI shipped by your packet manager and try > > --download-openmpi as well, just for a quick sanity check. > > > > Thanks, > > Pierre > > > > -- > --- > Antonio Trande > Fedora Project > mailto: sagitter(a)fedoraproject.org <mailto:[email protected]> > GPG key: 0x29FBC85D7A51CC2F > GPG key server: https://keyserver1.pgp.com/ <https://keyserver1.pgp.com/> > > > -- > Stefano > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] Cannot locate file: share/petsc/datafiles/matrices/small
by Junchao Zhang 14 Sep '21

14 Sep '21

Without a standalone & valid mpi example to reproduce the error, we are not assured to say it is an OpenMPI bug. --Junchao Zhang On Tue, Sep 14, 2021 at 6:17 AM Matthew Knepley <knepley(a)gmail.com> wrote: > Okay, we have to send this to OpenMPI. Volunteers? > > Maybe we should note this in the FAQ, or installation, so we remember how > to fix it if someone else asks? > > Thanks, > > Matt > > On Tue, Sep 14, 2021 at 2:35 AM Stefano Zampini <stefano.zampini(a)gmail.com> > wrote: > >> I can reproduce it even with OpenMPI 4.1.1 on a different machine (Ubuntu >> 18 + AMD milan + clang from AOCC) and it is definitely an OpenMPI bug in >> the vader BTL If I use tcp, everything runs smooth >> >> zampins@kanary:~/Devel/petsc$ cat >> /home/zampins/local/etc/openmpi-mca-params.conf | grep btl >> btl = tcp,self >> zampins@kanary:~/Devel/petsc$ make -f gmakefile.test >> vec_is_sf_tutorials-ex1_4 >> Using MAKEFLAGS: >> TEST arch-debug/tests/counts/vec_is_sf_tutorials-ex1_4.counts >> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create >> ok >> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create >> ok >> vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic >> ok >> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic >> ok >> vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate >> ok >> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate >> ok >> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create >> ok >> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create >> ok >> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic >> ok >> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic >> ok >> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate >> ok >> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate >> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create >> ok >> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create >> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic >> ok >> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic >> ok >> vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate >> ok >> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate >> >> >> zampins@kanary:~/Devel/petsc$ cat >> /home/zampins/local/etc/openmpi-mca-params.conf | grep btl >> btl = vader,tcp,self >> zampins@kanary:~/Devel/petsc$ make -f gmakefile.test >> vec_is_sf_tutorials-ex1_4 >> Using MAKEFLAGS: >> TEST arch-debug/tests/counts/vec_is_sf_tutorials-ex1_4.counts >> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create >> not ok >> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create >> # Error code: 1 >> # 43,46c43,46 >> # < [0] 0: 4001 2000 2002 3002 4002 >> # < [1] 0: 1001 3000 >> # < [2] 0: 2001 4000 >> # < [3] 0: 3001 1000 >> # --- >> # > [0] 0: 2002 2146435072 2 2146435072 38736240 >> # > [1] 0: 3000 2146435072 >> # > [2] 0: 2001 2146435072 >> # > [3] 0: 3001 2146435072 >> ok >> vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic >> not ok >> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic >> # Error code: 1 >> # 43,46c43,46 >> # < [0] 0: 4001 2000 2002 3002 4002 >> # < [1] 0: 1001 3000 >> # < [2] 0: 2001 4000 >> # < [3] 0: 3001 1000 >> # --- >> # > [0] 0: 2002 2146435072 2 2146435072 0 >> # > [1] 0: 3000 2146435072 >> # > [2] 0: 2001 2146435072 >> # > [3] 0: 3001 2146435072 >> ok >> vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate >> ok >> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate >> # retrying >> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create >> not ok >> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create # >> Error code: 98 >> # [1]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> # [1]PETSC ERROR: General MPI error >> # [1]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >> # [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >> shooting. >> # [1]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b >> GIT Date: 2021-09-13 14:01:22 +0000 >> # [1]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by >> zampins Tue Sep 14 09:31:42 2021 >> # [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> # [2]PETSC ERROR: General MPI error >> # [2]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >> # [2]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >> shooting. >> # [2]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b >> GIT Date: 2021-09-13 14:01:22 +0000 >> # [2]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by >> zampins Tue Sep 14 09:31:42 2021 >> # [2]PETSC ERROR: Configure options >> --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 >> --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 >> --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 >> --with-hip-dir=/opt/rocm --with-hip=1 >> --with-hypre-dir=/home/zampins/local-petsc >> --with-kokkos-dir=/home/zampins/local-petsc >> --with-kokkos-kernels-dir=/home/zampins/local-petsc >> --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include >> --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" >> HIPFLAGS=--amdgpu-target=gfx908 >> HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >> # [2]PETSC ERROR: #1 PetscSFGetGroups() at >> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >> # [2]PETSC ERROR: #2 PetscSFSetUp_Window() at >> /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >> # [2]PETSC ERROR: #3 PetscSFSetUp() at >> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >> # [2]PETSC ERROR: #4 main() at >> /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >> # [2]PETSC ERROR: PETSc Option Table entries: >> # [2]PETSC ERROR: -sf_type window >> # [2]PETSC ERROR: -sf_window_flavor create >> # [2]PETSC ERROR: -sf_window_sync active >> # [2]PETSC ERROR: -test_gather >> # [2]PETSC ERROR: ----------------End of Error Message -------send entire >> error message to petsc-maint(a)mcs.anl.gov---------- >> # [3]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> # [3]PETSC ERROR: General MPI error >> # [3]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >> # [3]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >> shooting. >> # [3]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b >> GIT Date: 2021-09-13 14:01:22 +0000 >> # [3]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by >> zampins Tue Sep 14 09:31:42 2021 >> # [3]PETSC ERROR: Configure options >> --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 >> --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 >> --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 >> --with-hip-dir=/opt/rocm --with-hip=1 >> --with-hypre-dir=/home/zampins/local-petsc >> --with-kokkos-dir=/home/zampins/local-petsc >> --with-kokkos-kernels-dir=/home/zampins/local-petsc >> --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include >> --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" >> HIPFLAGS=--amdgpu-target=gfx908 >> HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >> # [3]PETSC ERROR: #1 PetscSFGetGroups() at >> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >> # [3]PETSC ERROR: #2 PetscSFSetUp_Window() at >> /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >> # [3]PETSC ERROR: #3 PetscSFSetUp() at >> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >> # [3]PETSC ERROR: #4 main() at >> /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >> # [3]PETSC ERROR: PETSc Option Table entries: >> # [3]PETSC ERROR: -sf_type window >> # [3]PETSC ERROR: -sf_window_flavor create >> # [3]PETSC ERROR: Configure options >> --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 >> --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 >> --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 >> --with-hip-dir=/opt/rocm --with-hip=1 >> --with-hypre-dir=/home/zampins/local-petsc >> --with-kokkos-dir=/home/zampins/local-petsc >> --with-kokkos-kernels-dir=/home/zampins/local-petsc >> --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include >> --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" >> HIPFLAGS=--amdgpu-target=gfx908 >> HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >> # [1]PETSC ERROR: #1 PetscSFGetGroups() at >> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >> # [1]PETSC ERROR: #2 PetscSFSetUp_Window() at >> /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >> # [1]PETSC ERROR: #3 PetscSFSetUp() at >> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >> # [1]PETSC ERROR: #4 main() at >> /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >> # [1]PETSC ERROR: PETSc Option Table entries: >> # [1]PETSC ERROR: -sf_type window >> # [1]PETSC ERROR: -sf_window_flavor create >> # [1]PETSC ERROR: -sf_window_sync active >> # [1]PETSC ERROR: -test_gather >> # [1]PETSC ERROR: ----------------End of Error Message -------send entire >> error message to petsc-maint(a)mcs.anl.gov---------- >> # -sf_window_sync active >> # [3]PETSC ERROR: -test_gather >> # [3]PETSC ERROR: ----------------End of Error Message -------send entire >> error message to petsc-maint(a)mcs.anl.gov---------- >> # >> -------------------------------------------------------------------------- >> # MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD >> # with errorcode 98. >> # >> # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> # You may or may not see output from other processes, depending on >> # exactly when Open MPI kills them. >> # >> -------------------------------------------------------------------------- >> # [kanary.kaust.edu.sa:115527] 2 more processes have sent help message >> help-mpi-api.txt / mpi-abort >> # [kanary.kaust.edu.sa:115527] Set MCA parameter >> "orte_base_help_aggregate" to 0 to see all help / error messages >> ok vec_is_sf_tutorials-ex1_4 # SKIP Command failed so no diff >> # retrying >> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic >> not ok >> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic # >> Error code: 98 >> # [1]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> # [1]PETSC ERROR: General MPI error >> # [1]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >> # [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >> shooting. >> # [1]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b >> GIT Date: 2021-09-13 14:01:22 +0000 >> # [1]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by >> zampins Tue Sep 14 09:31:45 2021 >> # [1]PETSC ERROR: Configure options >> --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 >> --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 >> --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 >> --with-hip-dir=/opt/rocm --with-hip=1 >> --with-hypre-dir=/home/zampins/local-petsc >> --with-kokkos-dir=/home/zampins/local-petsc >> --with-kokkos-kernels-dir=/home/zampins/local-petsc >> --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include >> --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" >> HIPFLAGS=--amdgpu-target=gfx908 >> HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >> # [1]PETSC ERROR: #1 PetscSFGetGroups() at >> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >> # [1]PETSC ERROR: #2 PetscSFSetUp_Window() at >> /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >> # [1]PETSC ERROR: #3 PetscSFSetUp() at >> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >> # [1]PETSC ERROR: #4 main() at >> /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >> # [1]PETSC ERROR: PETSc Option Table entries: >> # [1]PETSC ERROR: -sf_type window >> # [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> # [2]PETSC ERROR: General MPI error >> # [2]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >> # [2]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >> shooting. >> # [2]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b >> GIT Date: 2021-09-13 14:01:22 +0000 >> # [2]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by >> zampins Tue Sep 14 09:31:45 2021 >> # [2]PETSC ERROR: Configure options >> --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 >> --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 >> --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 >> --with-hip-dir=/opt/rocm --with-hip=1 >> --with-hypre-dir=/home/zampins/local-petsc >> --with-kokkos-dir=/home/zampins/local-petsc >> --with-kokkos-kernels-dir=/home/zampins/local-petsc >> --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include >> --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" >> HIPFLAGS=--amdgpu-target=gfx908 >> HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >> # [2]PETSC ERROR: #1 PetscSFGetGroups() at >> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >> # [2]PETSC ERROR: #2 PetscSFSetUp_Window() at >> /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >> # [2]PETSC ERROR: #3 PetscSFSetUp() at >> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >> # [2]PETSC ERROR: #4 main() at >> /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >> # [2]PETSC ERROR: PETSc Option Table entries: >> # [2]PETSC ERROR: -sf_type window >> # [2]PETSC ERROR: -sf_window_flavor dynamic >> # [2]PETSC ERROR: -sf_window_sync active >> # [2]PETSC ERROR: -test_gather >> # [2]PETSC ERROR: ----------------End of Error Message -------send entire >> error message to petsc-maint(a)mcs.anl.gov---------- >> # [3]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> # [3]PETSC ERROR: General MPI error >> # [3]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank >> # [3]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >> shooting. >> # [3]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b >> GIT Date: 2021-09-13 14:01:22 +0000 >> # [3]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa by >> zampins Tue Sep 14 09:31:45 2021 >> # [3]PETSC ERROR: Configure options >> --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 >> --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 >> --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 >> --with-hip-dir=/opt/rocm --with-hip=1 >> --with-hypre-dir=/home/zampins/local-petsc >> --with-kokkos-dir=/home/zampins/local-petsc >> --with-kokkos-kernels-dir=/home/zampins/local-petsc >> --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include >> --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" >> HIPFLAGS=--amdgpu-target=gfx908 >> HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug >> # [3]PETSC ERROR: #1 PetscSFGetGroups() at >> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 >> # [3]PETSC ERROR: #2 PetscSFSetUp_Window() at >> /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 >> # [3]PETSC ERROR: #3 PetscSFSetUp() at >> /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 >> # [3]PETSC ERROR: #4 main() at >> /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 >> # [3]PETSC ERROR: PETSc Option Table entries: >> # [3]PETSC ERROR: -sf_type window >> # [3]PETSC ERROR: -sf_window_flavor dynamic >> # [3]PETSC ERROR: -sf_window_sync active >> # [3]PETSC ERROR: -test_gather >> # [3]PETSC ERROR: ----------------End of Error Message -------send entire >> error message to petsc-maint(a)mcs.anl.gov---------- >> # -sf_window_flavor dynamic >> # [1]PETSC ERROR: -sf_window_sync active >> # [1]PETSC ERROR: -test_gather >> # [1]PETSC ERROR: ----------------End of Error Message -------send entire >> error message to petsc-maint(a)mcs.anl.gov---------- >> # >> -------------------------------------------------------------------------- >> # MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD >> # with errorcode 98. >> # >> # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> # You may or may not see output from other processes, depending on >> # exactly when Open MPI kills them. >> # >> -------------------------------------------------------------------------- >> # [kanary.kaust.edu.sa:115572] 2 more processes have sent help message >> help-mpi-api.txt / mpi-abort >> # [kanary.kaust.edu.sa:115572] Set MCA parameter >> "orte_base_help_aggregate" to 0 to see all help / error messages >> ok vec_is_sf_tutorials-ex1_4 # SKIP Command failed so no diff >> ok >> vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate >> ok >> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate >> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create >> not ok >> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create >> # Error code: 1 >> # 43,46c43,46 >> # < [0] 0: 4001 2000 2002 3002 4002 >> # < [1] 0: 1001 3000 >> # < [2] 0: 2001 4000 >> # < [3] 0: 3001 1000 >> # --- >> # > [0] 0: 4002 2146435072 2 2146435072 34619728 >> # > [1] 0: 3000 2146435072 >> # > [2] 0: 4000 2146435072 >> # > [3] 0: 3001 2146435072 >> ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic >> not ok >> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic >> # Error code: 1 >> # 43,46c43,46 >> # < [0] 0: 4001 2000 2002 3002 4002 >> # < [1] 0: 1001 3000 >> # < [2] 0: 2001 4000 >> # < [3] 0: 3001 1000 >> # --- >> # > [0] 0: 4002 2146435072 2 2146435072 0 >> # > [1] 0: 3000 2146435072 >> # > [2] 0: 4000 2146435072 >> # > [3] 0: 3001 2146435072 >> ok >> vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate >> ok >> diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate >> >> >> >> Il giorno mar 14 set 2021 alle ore 07:44 Stefano Zampini < >> stefano.zampini(a)gmail.com> ha scritto: >> >>> I'll see if I can reproduce >>> >>> Il Mar 14 Set 2021, 06:58 Junchao Zhang <junchao.zhang(a)gmail.com> ha >>> scritto: >>> >>>> Hi, Stefano, >>>> Ping you again to see if you want to resolve this problem before >>>> petsc-3.16 >>>> >>>> --Junchao Zhang >>>> >>>> >>>> On Sun, Sep 12, 2021 at 3:06 PM Antonio T. sagitter < >>>> sagitter(a)fedoraproject.org> wrote: >>>> >>>>> Unfortunately, it's not possible. I must use the OpenMPI provided by >>>>> Fedora build-system (these rpm builds of PETSc are for Fedora's >>>>> repositories), downloading external software is not permitted. >>>>> >>>>> On 9/12/21 21:10, Pierre Jolivet wrote: >>>>> > >>>>> >> On 12 Sep 2021, at 8:56 PM, Matthew Knepley <knepley(a)gmail.com >>>>> >> <mailto:[email protected]>> wrote: >>>>> >> >>>>> >> On Sun, Sep 12, 2021 at 2:49 PM Antonio T. sagitter >>>>> >> <sagitter(a)fedoraproject.org <mailto:[email protected]>> >>>>> wrote: >>>>> >> >>>>> >> Those attached are configure.log/make.log from a MPI build in >>>>> >> Fedora 34 >>>>> >> x86_64 where the error below occurred. >>>>> >> >>>>> >> >>>>> >> This is OpenMPI 4.1.0. Is that the only MPI you build? My first >>>>> >> inclination is that this is an MPI implementation bug. >>>>> >> >>>>> >> Junchao, do we have an OpenMPI build in the CI? >>>>> > >>>>> > config/examples/arch-ci-linux-cuda-double-64idx.py: >>>>> > '--download-openmpi=1', >>>>> > config/examples/arch-ci-linux-pkgs-dbg-ftn-interfaces.py: >>>>> > '--download-openmpi=1', >>>>> > config/examples/arch-ci-linux-pkgs-opt.py: '--download-openmpi=1', >>>>> > >>>>> > config/BuildSystem/config/packages/OpenMPI.py uses version 4.1.0 as >>>>> well. >>>>> > I’m not sure PETSc is to blame here Antonio. You may want to try to >>>>> > ditch the OpenMPI shipped by your packet manager and try >>>>> > --download-openmpi as well, just for a quick sanity check. >>>>> > >>>>> > Thanks, >>>>> > Pierre >>>>> > >>>>> >>>>> -- >>>>> --- >>>>> Antonio Trande >>>>> Fedora Project >>>>> mailto: sagitter(a)fedoraproject.org >>>>> GPG key: 0x29FBC85D7A51CC2F >>>>> GPG key server: https://keyserver1.pgp.com/ >>>>> >>>> >> >> -- >> Stefano >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> >

1 0

Re: [petsc-dev] Cannot locate file: share/petsc/datafiles/matrices/small
by Barry Smith 14 Sep '21

14 Sep '21

Have mpi.py turn off the use of MPI windows for these versions of OpenMPI. Note this is already standard practice, we have the comment "Skip buggy MPICH versions" and a variety of checks based on bad versions. Barry > On Sep 14, 2021, at 7:17 AM, Matthew Knepley <knepley(a)gmail.com> wrote: > > Okay, we have to send this to OpenMPI. Volunteers? > > Maybe we should note this in the FAQ, or installation, so we remember how to fix it if someone else asks? > > Thanks, > > Matt > > On Tue, Sep 14, 2021 at 2:35 AM Stefano Zampini <stefano.zampini(a)gmail.com <mailto:[email protected]>> wrote: > I can reproduce it even with OpenMPI 4.1.1 on a different machine (Ubuntu 18 + AMD milan + clang from AOCC) and it is definitely an OpenMPI bug in the vader BTL If I use tcp, everything runs smooth > > zampins@kanary:~/Devel/petsc$ cat /home/zampins/local/etc/openmpi-mca-params.conf | grep btl > btl = tcp,self > zampins@kanary:~/Devel/petsc$ make -f gmakefile.test vec_is_sf_tutorials-ex1_4 > Using MAKEFLAGS: > TEST arch-debug/tests/counts/vec_is_sf_tutorials-ex1_4.counts > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate > > > zampins@kanary:~/Devel/petsc$ cat /home/zampins/local/etc/openmpi-mca-params.conf | grep btl > btl = vader,tcp,self > zampins@kanary:~/Devel/petsc$ make -f gmakefile.test vec_is_sf_tutorials-ex1_4 > Using MAKEFLAGS: > TEST arch-debug/tests/counts/vec_is_sf_tutorials-ex1_4.counts > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create > not ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-create # Error code: 1 > # 43,46c43,46 > # < [0] 0: 4001 2000 2002 3002 4002 > # < [1] 0: 1001 3000 > # < [2] 0: 2001 4000 > # < [3] 0: 3001 1000 > # --- > # > [0] 0: 2002 2146435072 2 2146435072 38736240 > # > [1] 0: 3000 2146435072 > # > [2] 0: 2001 2146435072 > # > [3] 0: 3001 2146435072 > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic > not ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-dynamic # Error code: 1 > # 43,46c43,46 > # < [0] 0: 4001 2000 2002 3002 4002 > # < [1] 0: 1001 3000 > # < [2] 0: 2001 4000 > # < [3] 0: 3001 1000 > # --- > # > [0] 0: 2002 2146435072 2 2146435072 0 > # > [1] 0: 3000 2146435072 > # > [2] 0: 2001 2146435072 > # > [3] 0: 3001 2146435072 > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-fence_sf_window_flavor-allocate > # retrying vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create > not ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-create # Error code: 98 > # [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > # [1]PETSC ERROR: General MPI error > # [1]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > # [1]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting. > # [1]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 > # [1]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa <http://kanary.kaust.edu.sa/> by zampins Tue Sep 14 09:31:42 2021 > # [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > # [2]PETSC ERROR: General MPI error > # [2]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > # [2]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting. > # [2]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 > # [2]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa <http://kanary.kaust.edu.sa/> by zampins Tue Sep 14 09:31:42 2021 > # [2]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > # [2]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > # [2]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > # [2]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > # [2]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > # [2]PETSC ERROR: PETSc Option Table entries: > # [2]PETSC ERROR: -sf_type window > # [2]PETSC ERROR: -sf_window_flavor create > # [2]PETSC ERROR: -sf_window_sync active > # [2]PETSC ERROR: -test_gather > # [2]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- > # [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > # [3]PETSC ERROR: General MPI error > # [3]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > # [3]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting. > # [3]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 > # [3]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa <http://kanary.kaust.edu.sa/> by zampins Tue Sep 14 09:31:42 2021 > # [3]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > # [3]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > # [3]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > # [3]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > # [3]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > # [3]PETSC ERROR: PETSc Option Table entries: > # [3]PETSC ERROR: -sf_type window > # [3]PETSC ERROR: -sf_window_flavor create > # [3]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > # [1]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > # [1]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > # [1]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > # [1]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > # [1]PETSC ERROR: PETSc Option Table entries: > # [1]PETSC ERROR: -sf_type window > # [1]PETSC ERROR: -sf_window_flavor create > # [1]PETSC ERROR: -sf_window_sync active > # [1]PETSC ERROR: -test_gather > # [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- > # -sf_window_sync active > # [3]PETSC ERROR: -test_gather > # [3]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- > # -------------------------------------------------------------------------- > # MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD > # with errorcode 98. > # > # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > # You may or may not see output from other processes, depending on > # exactly when Open MPI kills them. > # -------------------------------------------------------------------------- > # [kanary.kaust.edu.sa:115527 <http://kanary.kaust.edu.sa:115527>] 2 more processes have sent help message help-mpi-api.txt / mpi-abort > # [kanary.kaust.edu.sa:115527 <http://kanary.kaust.edu.sa:115527>] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages > ok vec_is_sf_tutorials-ex1_4 # SKIP Command failed so no diff > # retrying vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic > not ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-dynamic # Error code: 98 > # [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > # [1]PETSC ERROR: General MPI error > # [1]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > # [1]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting. > # [1]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 > # [1]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa <http://kanary.kaust.edu.sa/> by zampins Tue Sep 14 09:31:45 2021 > # [1]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > # [1]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > # [1]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > # [1]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > # [1]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > # [1]PETSC ERROR: PETSc Option Table entries: > # [1]PETSC ERROR: -sf_type window > # [1]PETSC ERROR: [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > # [2]PETSC ERROR: General MPI error > # [2]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > # [2]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting. > # [2]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 > # [2]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa <http://kanary.kaust.edu.sa/> by zampins Tue Sep 14 09:31:45 2021 > # [2]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > # [2]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > # [2]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > # [2]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > # [2]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > # [2]PETSC ERROR: PETSc Option Table entries: > # [2]PETSC ERROR: -sf_type window > # [2]PETSC ERROR: -sf_window_flavor dynamic > # [2]PETSC ERROR: -sf_window_sync active > # [2]PETSC ERROR: -test_gather > # [2]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- > # [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > # [3]PETSC ERROR: General MPI error > # [3]PETSC ERROR: MPI error 6 MPI_ERR_RANK: invalid rank > # [3]PETSC ERROR: See https://petsc.org/release/faq/ <https://petsc.org/release/faq/> for trouble shooting. > # [3]PETSC ERROR: Petsc Development GIT revision: v3.15.4-783-g168bb9f76b GIT Date: 2021-09-13 14:01:22 +0000 > # [3]PETSC ERROR: ../ex1 on a arch-debug named kanary.kaust.edu.sa <http://kanary.kaust.edu.sa/> by zampins Tue Sep 14 09:31:45 2021 > # [3]PETSC ERROR: Configure options --with-cc=/home/zampins/local/bin/mpicc --with-cxx-dialect=c++14 --with-cxx=/home/zampins/local/bin/mpicxx --with-debugging=1 --with-fc=/home/zampins/local/bin/mpifort --with-fortran-bindings=0 --with-hip-dir=/opt/rocm --with-hip=1 --with-hypre-dir=/home/zampins/local-petsc --with-kokkos-dir=/home/zampins/local-petsc --with-kokkos-kernels-dir=/home/zampins/local-petsc --with-blaslapack-include=/home/zampins/local-aocl/aocc/3.0-6/include --with-blaslapack-lib="[/home/zampins/local-aocl/aocc/3.0-6/lib/liblapacke.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libflame.a,/home/zampins/local-aocl/aocc/3.0-6/lib/libblis-mt.a]" HIPFLAGS=--amdgpu-target=gfx908 HIPPPFLAGS=-I/home/zampins/local-petsc/include PETSC_ARCH=arch-debug > # [3]PETSC ERROR: #1 PetscSFGetGroups() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:1166 > # [3]PETSC ERROR: #2 PetscSFSetUp_Window() at /home/zampins/Devel/petsc/src/vec/is/sf/impls/window/sfwindow.c:708 > # [3]PETSC ERROR: #3 PetscSFSetUp() at /home/zampins/Devel/petsc/src/vec/is/sf/interface/sf.c:318 > # [3]PETSC ERROR: #4 main() at /home/zampins/Devel/petsc/src/vec/is/sf/tutorials/ex1.c:172 > # [3]PETSC ERROR: PETSc Option Table entries: > # [3]PETSC ERROR: -sf_type window > # [3]PETSC ERROR: -sf_window_flavor dynamic > # [3]PETSC ERROR: -sf_window_sync active > # [3]PETSC ERROR: -test_gather > # [3]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- > # -sf_window_flavor dynamic > # [1]PETSC ERROR: -sf_window_sync active > # [1]PETSC ERROR: -test_gather > # [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- > # -------------------------------------------------------------------------- > # MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD > # with errorcode 98. > # > # NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > # You may or may not see output from other processes, depending on > # exactly when Open MPI kills them. > # -------------------------------------------------------------------------- > # [kanary.kaust.edu.sa:115572 <http://kanary.kaust.edu.sa:115572>] 2 more processes have sent help message help-mpi-api.txt / mpi-abort > # [kanary.kaust.edu.sa:115572 <http://kanary.kaust.edu.sa:115572>] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages > ok vec_is_sf_tutorials-ex1_4 # SKIP Command failed so no diff > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-active_sf_window_flavor-allocate > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create > not ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-create # Error code: 1 > # 43,46c43,46 > # < [0] 0: 4001 2000 2002 3002 4002 > # < [1] 0: 1001 3000 > # < [2] 0: 2001 4000 > # < [3] 0: 3001 1000 > # --- > # > [0] 0: 4002 2146435072 2 2146435072 34619728 > # > [1] 0: 3000 2146435072 > # > [2] 0: 4000 2146435072 > # > [3] 0: 3001 2146435072 > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic > not ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-dynamic # Error code: 1 > # 43,46c43,46 > # < [0] 0: 4001 2000 2002 3002 4002 > # < [1] 0: 1001 3000 > # < [2] 0: 2001 4000 > # < [3] 0: 3001 1000 > # --- > # > [0] 0: 4002 2146435072 2 2146435072 0 > # > [1] 0: 3000 2146435072 > # > [2] 0: 4000 2146435072 > # > [3] 0: 3001 2146435072 > ok vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate > ok diff-vec_is_sf_tutorials-ex1_4+sf_window_sync-lock_sf_window_flavor-allocate > > > > Il giorno mar 14 set 2021 alle ore 07:44 Stefano Zampini <stefano.zampini(a)gmail.com <mailto:[email protected]>> ha scritto: > I'll see if I can reproduce > > Il Mar 14 Set 2021, 06:58 Junchao Zhang <junchao.zhang(a)gmail.com <mailto:[email protected]>> ha scritto: > Hi, Stefano, > Ping you again to see if you want to resolve this problem before petsc-3.16 > > --Junchao Zhang > > > On Sun, Sep 12, 2021 at 3:06 PM Antonio T. sagitter <sagitter(a)fedoraproject.org <mailto:[email protected]>> wrote: > Unfortunately, it's not possible. I must use the OpenMPI provided by > Fedora build-system (these rpm builds of PETSc are for Fedora's > repositories), downloading external software is not permitted. > > On 9/12/21 21:10, Pierre Jolivet wrote: > > > >> On 12 Sep 2021, at 8:56 PM, Matthew Knepley <knepley(a)gmail.com <mailto:[email protected]> > >> <mailto:[email protected] <mailto:[email protected]>>> wrote: > >> > >> On Sun, Sep 12, 2021 at 2:49 PM Antonio T. sagitter > >> <sagitter(a)fedoraproject.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> wrote: > >> > >> Those attached are configure.log/make.log from a MPI build in > >> Fedora 34 > >> x86_64 where the error below occurred. > >> > >> > >> This is OpenMPI 4.1.0. Is that the only MPI you build? My first > >> inclination is that this is an MPI implementation bug. > >> > >> Junchao, do we have an OpenMPI build in the CI? > > > > config/examples/arch-ci-linux-cuda-double-64idx.py: > > '--download-openmpi=1', > > config/examples/arch-ci-linux-pkgs-dbg-ftn-interfaces.py: > > '--download-openmpi=1', > > config/examples/arch-ci-linux-pkgs-opt.py: '--download-openmpi=1', > > > > config/BuildSystem/config/packages/OpenMPI.py uses version 4.1.0 as well. > > I’m not sure PETSc is to blame here Antonio. You may want to try to > > ditch the OpenMPI shipped by your packet manager and try > > --download-openmpi as well, just for a quick sanity check. > > > > Thanks, > > Pierre > > > > -- > --- > Antonio Trande > Fedora Project > mailto: sagitter(a)fedoraproject.org <mailto:[email protected]> > GPG key: 0x29FBC85D7A51CC2F > GPG key server: https://keyserver1.pgp.com/ <https://keyserver1.pgp.com/> > > > -- > Stefano > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0