Perhaps we can back one step: Use your mpicc to build a "hello world" mpi test, then run it on a compute node (with GPU) to see if it works. If no, then your MPI environment has problems; If yes, then use it to build petsc (turn on petsc's gpu support, --with-cuda --with-cudac=nvcc), and then your code. --Junchao Zhang On Fri, Oct 7, 2022 at 10:45 PM Rob Kudyba <[email protected]> wrote:
The error changes now and at an earlier place, 66% vs 70%: make LDFLAGS="-Wl,--copy-dt-needed-entries" Consolidate compiler generated dependencies of target fmt [ 12%] Built target fmt Consolidate compiler generated dependencies of target richdem [ 37%] Built target richdem Consolidate compiler generated dependencies of target wtm [ 62%] Built target wtm Consolidate compiler generated dependencies of target wtm.x [ 66%] Linking CXX executable wtm.x /usr/bin/ld: libwtm.a(transient_groundwater.cpp.o): undefined reference to symbol 'MPI_Abort' /path/to/openmpi-4.1.1_ucx_cuda_11.0.3_support/lib/libmpi.so.40: error adding symbols: DSO missing from command line collect2: error: ld returned 1 exit status make[2]: *** [CMakeFiles/wtm.x.dir/build.make:103: wtm.x] Error 1 make[1]: *** [CMakeFiles/Makefile2:225: CMakeFiles/wtm.x.dir/all] Error 2 make: *** [Makefile:136: all] Error 2
So perhaps PET_Sc is now being found. Any other suggestions?
On Fri, Oct 7, 2022 at 11:18 PM Rob Kudyba <[email protected]> wrote:
Thanks for the quick reply. I added these options to make and make check
still produce the warnings so I used the command like this: make PETSC_DIR=/path/to/petsc PETSC_ARCH=arch-linux-c-debug MPIEXEC="mpiexec -mca orte_base_help_aggregate 0 --mca opal_warn_on_missing_libcuda 0 -mca pml ucx --mca btl '^openib'" check Running check examples to verify correct installation Using PETSC_DIR=/path/to/petsc and PETSC_ARCH=arch-linux-c-debug C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes Completed test examples
Could be useful for the FAQ.
You mentioned you had "OpenMPI 4.1.1 with CUDA aware", so I think a workable mpicc should automatically find cuda libraries. Maybe you unloaded cuda libraries?
Oh let me clarify, OpenMPI is CUDA aware however this code and the node where PET_Sc is compiling does not have a GPU, hence not needed and using the MPIEXEC option worked during the 'check' to suppress the warning.
I'm not trying to use PetSC to compile and linking appears to go awry:
[ 58%] Building CXX object CMakeFiles/wtm.dir/src/update_effective_storativity.cpp.o [ 62%] Linking CXX static library libwtm.a [ 62%] Built target wtm [ 66%] Building CXX object CMakeFiles/wtm.x.dir/src/WTM.cpp.o [ 70%] Linking CXX executable wtm.x /usr/bin/ld: cannot find -lpetsc collect2: error: ld returned 1 exit status make[2]: *** [CMakeFiles/wtm.x.dir/build.make:103: wtm.x] Error 1 make[1]: *** [CMakeFiles/Makefile2:269: CMakeFiles/wtm.x.dir/all] Error 2 make: *** [Makefile:136: all] Error 2
It seems cmake could not find petsc. Look at $PETSC_DIR/share/petsc/CMakeLists.txt and try to modify your CMakeLists.txt.
There is an explicit reference to the path in CMakeLists.txt: # NOTE: You may need to update this path to identify PETSc's location set(ENV{PKG_CONFIG_PATH} "$ENV{PKG_CONFIG_PATH}:/path/to/petsc/arch-linux-cxx-debug/lib/pkgconfig/") pkg_check_modules(PETSC PETSc>=3.17.1 IMPORTED_TARGET REQUIRED) message(STATUS "Found PETSc ${PETSC_VERSION}") add_subdirectory(common/richdem EXCLUDE_FROM_ALL) add_subdirectory(common/fmt EXCLUDE_FROM_ALL)
And that exists: ls /path/to/petsc/arch-linux-cxx-debug/lib/pkgconfig/ petsc.pc PETSc.pc
Is there an environment variable I'm missing? I've seen the suggestion
<https://www.mail-archive.com/[email protected]&q=subject:%22%5C%5Bpetsc%5C-users%5C%5D+CMake+error+in+PETSc%22&o=newest&f=1> to add it to LD_LIBRARY_PATH which I did with export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PETSC_DIR/$PETSC_ARCH/lib and that points to:
ls -l /path/to/petsc/arch-linux-c-debug/lib total 83732 lrwxrwxrwx 1 rk3199 user 18 Oct 7 13:56 libpetsc.so -> libpetsc.so.3.18.0 lrwxrwxrwx 1 rk3199 user 18 Oct 7 13:56 libpetsc.so.3.18 -> libpetsc.so.3.18.0 -rwxr-xr-x 1 rk3199 user 85719200 Oct 7 13:56 libpetsc.so.3.18.0 drwxr-xr-x 3 rk3199 user 4096 Oct 6 10:22 petsc drwxr-xr-x 2 rk3199 user 4096 Oct 6 10:23 pkgconfig
Anything else to check?
If modifying CMakeLists.txt does not work, you can try export LIBRARY_PATH=$LIBRARY_PATH:$PETSC_DIR/$PETSC_ARCH/lib LD_LIBRARY_PATHis is for run time, but the error happened at link time,
Yes that's what I already had. Any other debug that I can provide?
On Fri, Oct 7, 2022 at 1:53 PM Satish Balay <[email protected]> wrote:
you can try
make PETSC_DIR=/path/to/petsc PETSC_ARCH=arch-linux-c-debug MPIEXEC="mpiexec -mca orte_base_help_aggregate 0 --mca opal_warn_on_missing_libcuda 0 -mca pml ucx --mca btl '^openib'"
Wrt configure - it can be set with --with-mpiexec option - its saved in PETSC_ARCH/lib/petsc/conf/petscvariables
Satish
On Fri, 7 Oct 2022, Rob Kudyba wrote:
We are on RHEL 8, using modules that we can load/unload various version of packages/libraries, and I have OpenMPI 4.1.1 with CUDA aware loaded along with GDAL 3.3.0, GCC 10.2.0, and cmake 3.22.1
make PETSC_DIR=/path/to/petsc PETSC_ARCH=arch-linux-c-debug check fails with the below errors, Running check examples to verify correct installation
Using PETSC_DIR=/path/to/petsc and PETSC_ARCH=arch-linux-c-debug Possible error running C/C++ src/snes/tutorials/ex19 with 1 MPI process See https://petsc.org/release/faq/
--------------------------------------------------------------------------
The library attempted to open the following supporting CUDA libraries, but each of them failed. CUDA-aware support is disabled. libcuda.so.1: cannot open shared object file: No such file or directory libcuda.dylib: cannot open shared object file: No such file or directory /usr/lib64/libcuda.so.1: cannot open shared object file: No such file or directory /usr/lib64/libcuda.dylib: cannot open shared object file: No such file or directory If you are not interested in CUDA-aware support, then run with --mca opal_warn_on_missing_libcuda 0 to suppress this message. If you are interested in CUDA-aware support, then try setting LD_LIBRARY_PATH to the location of libcuda.so.1 to get passed this issue.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.
Local host: g117 Local device: mlx5_0
--------------------------------------------------------------------------
lid velocity = 0.0016, prandtl # = 1., grashof # = 1. Number of SNES iterations = 2 Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes See https://petsc.org/release/faq/
The library attempted to open the following supporting CUDA libraries, but each of them failed. CUDA-aware support is disabled. libcuda.so.1: cannot open shared object file: No such file or directory libcuda.dylib: cannot open shared object file: No such file or directory /usr/lib64/libcuda.so.1: cannot open shared object file: No such file or directory /usr/lib64/libcuda.dylib: cannot open shared object file: No such file or directory If you are not interested in CUDA-aware support, then run with --mca opal_warn_on_missing_libcuda 0 to suppress this message. If you are interested in CUDA-aware support, then try setting LD_LIBRARY_PATH to the locationof libcuda.so.1 to get passed this issue.
WARNING: There was an error initializing an OpenFabrics device.
Local host: xxx Local device: mlx5_0
lid velocity = 0.0016, prandtl # = 1., grashof # = 1. Number of SNES iterations = 2 [g117:4162783] 1 more process has sent help message help-mpi-common-cuda.txt / dlopen failed [g117:4162783] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages [g117:4162783] 1 more process has sent help message help-mpi-btl-openib.txt / error in device init Completed test examples Error while running make check gmake[1]: *** [makefile:149: check] Error 1 make: *** [GNUmakefile:17: check] Error 2
Where is $MPI_RUN set? I'd like to be able to pass options such as --mca orte_base_help_aggregate 0 --mca opal_warn_on_missing_libcuda 0 -mca pml ucx --mca btl '^openib' which will help me troubleshoot and hide unneeded warnings.
Thanks, Rob