June 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Error of petsc-3.15 with python-3.10
by Antonio T. sagitter 14 Jun '21

14 Jun '21

Hi again. I can report to you a temporary fix from Fedora's Python team: https://src.fedoraproject.org/rpms/petsc/blob/rawhide/f/petsc-3.15.0-workar… Bug reported to Python upstream: https://bugs.python.org/issue44351 On 5/31/21 8:12 PM, Matthew Knepley wrote: > Stack trace? I don't think we have any 3.10 instances running right now, > so it would be nice to know what went wrong. > > Thanks, > > Matt > > On Mon, May 31, 2021 at 2:07 PM Antonio T. sagitter > <sagitter(a)fedoraproject.org <mailto:[email protected]>> wrote: > > Hello! > > Do you need other info? > > > -- > --- > Antonio Trande > Fedora Project > mailto: sagitter(a)fedoraproject.org <mailto:[email protected]> > GPG key: 0x29FBC85D7A51CC2F > GPG key server: https://keys.gnupg.net/ <https://keys.gnupg.net/> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/> -- --- Antonio Trande Fedora Project mailto: sagitter(a)fedoraproject.org GPG key: 0x29FBC85D7A51CC2F GPG key server: https://keys.gnupg.net/

1 0

PETSc community white paper meeting 3pm Friday
by Barry Smith 10 Jun '21

10 Jun '21

I know Friday at 3 pm is not a great time but my schedule is too uncertain earlier in the day. https://www.overleaf.com/read/xbgxvzydzynp <https://www.overleaf.com/read/xbgxvzydzynp> Send me email with your Overleaf email address if you would like to edit the document Topic: PETSc community white paper Zoom Meeting Time: Jun 11, 2021 03:00 PM Central Time (US and Canada) Join Zoom Meeting https://us05web.zoom.us/j/86443184450?pwd=eTlsWSt6UzlIMW0yYnFEeFNid3MzUT09 Meeting ID: 864 4318 4450 Passcode: bqp2Ti

1 0

Re: [petsc-dev] Strange "Warnings" When Compiling With -flto
by Matthew Knepley 09 Jun '21

09 Jun '21

On Wed, Jun 9, 2021 at 5:07 PM Jacob Faibussowitsch <jacob.fai(a)gmail.com> wrote: > Hello All, > > I’m getting these strange “warnings” (if you can even call them that), > about struct vector alignments all over the shop when compiling with link > time optimization enabled on summit. For reference this is with > spectrum-mpi wrapping gcc v10.2.0. Here are a few examples: > > /autofs/nccs-svm1_home1/jfaibussow/petsc/src/mat/impls/baij/seq/baij.c: In > function 'MatDuplicateNoCreate_SeqBAIJ': > /autofs/nccs-svm1_home1/jfaibussow/petsc/src/mat/impls/baij/seq/baij.c:3250:21: > note: the layout of aggregates containing vectors with 8-byte alignment has > changed in GCC 5 > 3250 | c->bs2 = a->bs2; > | ^ > > autofs/nccs-svm1_home1/jfaibussow/petsc/src/dm/dt/dualspace/impls/lagrange/dspacelagrange.c: > In function 'PetscDualSpaceSetUp_Lagrange': > /autofs/nccs-svm1_home1/jfaibussow/petsc/src/dm/dt/dualspace/impls/lagrange/dspacelagrange.c:2291:15: > note: the layout of aggregates containing vectors with 8-byte alignment has > changed in GCC 5 > 2291 | sp->spdim = sp->spintdim = spdim; > | ^ > > /autofs/nccs-svm1_home1/jfaibussow/petsc/src/mat/impls/sell/mpi/mpisell.c: > In function 'MatDuplicate_MPISELL': > /autofs/nccs-svm1_home1/jfaibussow/petsc/src/mat/impls/sell/mpi/mpisell.c:1232:27: > note: the layout of aggregates containing vectors with 8-byte alignment has > changed in GCC 5 > 1232 | a->size = oldmat->size; > | ^ > > There doesn’t seem to be anything the compiler wants you to do, which is > strange. It just tells you about it. Googling it has not yielded useful > info either. Is this our own buggy code at fault or just an internal > compiler logging so rare that they forgot to turn it off? > There seems to be a discussion of the option to turn these off here: https://stackoverflow.com/questions/52020305/what-exactly-does-gccs-wpsabi-… Thanks, Matt > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Strange "Warnings" When Compiling With -flto
by Jacob Faibussowitsch 09 Jun '21

09 Jun '21

Hello All, I’m getting these strange “warnings” (if you can even call them that), about struct vector alignments all over the shop when compiling with link time optimization enabled on summit. For reference this is with spectrum-mpi wrapping gcc v10.2.0. Here are a few examples: /autofs/nccs-svm1_home1/jfaibussow/petsc/src/mat/impls/baij/seq/baij.c: In function 'MatDuplicateNoCreate_SeqBAIJ': /autofs/nccs-svm1_home1/jfaibussow/petsc/src/mat/impls/baij/seq/baij.c:3250:21: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5 3250 | c->bs2 = a->bs2; | ^ autofs/nccs-svm1_home1/jfaibussow/petsc/src/dm/dt/dualspace/impls/lagrange/dspacelagrange.c: In function 'PetscDualSpaceSetUp_Lagrange': /autofs/nccs-svm1_home1/jfaibussow/petsc/src/dm/dt/dualspace/impls/lagrange/dspacelagrange.c:2291:15: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5 2291 | sp->spdim = sp->spintdim = spdim; | ^ /autofs/nccs-svm1_home1/jfaibussow/petsc/src/mat/impls/sell/mpi/mpisell.c: In function 'MatDuplicate_MPISELL': /autofs/nccs-svm1_home1/jfaibussow/petsc/src/mat/impls/sell/mpi/mpisell.c:1232:27: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5 1232 | a->size = oldmat->size; | ^ There doesn’t seem to be anything the compiler wants you to do, which is strange. It just tells you about it. Googling it has not yielded useful info either. Is this our own buggy code at fault or just an internal compiler logging so rare that they forgot to turn it off? Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch)

1 0

Tom Cortese no longer at NCSA
by tom cortese 07 Jun '21

07 Jun '21

Hello all, While I have hardly ever posted to this PETSc developer's email list, I have been "lurking", and reading with interest, for the past several years. Two jobs ago, when I was working as a DoD contractor for the PET program, I was installing PETSc on all sorts of platforms and operating systems (IBM, Cray, HP, etc...), and the PETSc team was very helpful in resolving bugs and other issues (I'm sure that I had access to a much wider variety of HPC systems than the PETSc developers did, and as a result I was able to shake out some bugs that might have remained uncovered for who knows how long). After moving to NCSA, i only had one computer architecture to deal with - the marvelous Blue Waters - and I was no longer responsible for installing PETSc (someone at Cray did that, I think), but I would still occasinally support users who were using PETSc. My five-year contract at NCSA just ran out, after eight years (the NSF kept extending my contract by additional one-year increments, because they were happy with the work that NCSA and Blue Waters were accomplishing), but a year or so ago the funding agency changed from NSF to NGA, and my contract was not renewed (sniff sniff). So, please excuse me for asking this here (and if this is not the appropriate place, please re-direct me -- I absolutely don't want to be spamming anybody!), but I find myself looking for another position, and I was hoping that maybe something might be available at Argonne, and/or MCS, and/or PETSc. I am mildly encouraged by the fact that a few other people I know are there (Victor Anisimov, JaeHyuk Kwack and I used to be in the same group at NCSA, and Scott Parker and I had the same advisor in Theoretical & Applied Mechanics at the University of Illinois). Are there any openings? Thanks, -Tom Cortese

1 0

Re: [petsc-dev] looks like I forgot fortran stuff for the band solver
by Mark Adams 07 Jun '21

07 Jun '21

I think I just had an old .c file in my directory. On Mon, Jun 7, 2021 at 3:49 PM Mark Adams <mfadams(a)lbl.gov> wrote: > Makefile contains file not on filesystem: > /gpfs/alpine/csc314/scratch/adams/petsc/src/mat/impls/aij/seq/seqcusparse/ftn-auto: > ['aijcusparse.bandf.c'] >

1 0

looks like I forgot fortran stuff for the band solver
by Mark Adams 07 Jun '21

07 Jun '21

Makefile contains file not on filesystem: /gpfs/alpine/csc314/scratch/adams/petsc/src/mat/impls/aij/seq/seqcusparse/ftn-auto: ['aijcusparse.bandf.c']

1 0

Tight deadline. Preparation of white paper on PETSc Community Development, invitation to join. Due June 18, 2021
by Barry Smith 06 Jun '21

06 Jun '21

An announcement of the Collegeville Scientific Computing Software Teams workshop https://collegeville.github.io/CW21/ <https://collegeville.github.io/CW21/> has recently appeared. For the development of a PETSc community white paper submission for workshop, I have created an empty Overleaf document https://www.overleaf.com/read/xbgxvzydzynp. Everyone is welcome to help develop the white paper and presentation. Please email me your Overleaf email address if you are interested. I do not think Overleaf has yet a mechanism to request access directly from the URL. Deadlines, white paper June 18. Recorded presentation July 9. Please use the Overleaf comment feature and the text in the document for all discussion of the text. If that does not serve our communication purposes we can add a dedicated Slack channel for discussion of the white paper. We will have a zoom meeting the week of June 6th to discuss possible content. Please add your availability to the document. Additional white paper submissions are also possible for those interested in focusing on specific aspects of the team development of scientific software. There is a possibility of expanding the accepted white papers at a later date for submission to a Journal. Barry

1 0

Re: [petsc-dev] kokkos test fail in branch
by Mark Adams 06 Jun '21

06 Jun '21

Thanks, let's wait then. Not urgent. On Sat, Jun 5, 2021 at 10:52 PM Junchao Zhang <junchao.zhang(a)gmail.com> wrote: > This problem was fixed in > https://gitlab.com/petsc/petsc/-/merge_requests/4056, and is waiting for > !3411 <https://gitlab.com/petsc/petsc/-/merge_requests/3411> :) > --Junchao Zhang > > > On Sat, Jun 5, 2021 at 9:42 PM Barry Smith <bsmith(a)petsc.dev> wrote: > >> >> Looks like the MPI libraries are not being passed to the NVCC internal >> compiler (gcc). This would normally be setup in MPI.py, please send >> configure.log >> >> Barry >> >> >> On Jun 5, 2021, at 1:18 PM, Mark Adams <mfadams(a)lbl.gov> wrote: >> >> Ah, there was an old one there. I removed it and I can not compile ex3k. >> I thought check always rebuilt executables. Now I get a link error: >> >> 14:12 barry/2020-11-11/cleanup-matsetvaluesdevice *= >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make >> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >> PETSC_ARCH=arch-summit-opt-gnu-kokkos-notpl-cuda10 ex3k >> >> PATH=/sw/sources/lsf-tools/2.0/summit/bin:/sw/summit/xalt/1.2.1/bin:/sw/summit/forge/20.0.1/bin:/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/bin:/sw/summit/gcc/6.4.0/bin:/sw/summit/cuda/10.1.243/bin:/autofs/nccs-svm1_sw/summit/.swci/0-core/opt/spack/20180914/linux-rhel7-ppc64le/gcc-4.8.5/cmake-3.20.2-24ualfzy6em6ws5sbiu7rlgcuionodrm/bin:/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-4.8.5/darshan-runtime-3.1.7-cnvxicgf5j4ap64qi6v5gxp67hmrjz43/bin:/sw/sources/hpss/bin:/opt/ibm/spectrumcomputing/lsf/ >> 10.1.0.9/linux3.10-glibc2.17-ppc64le-csm/etc:/opt/ibm/spectrumcomputing/lsf/10.1.0.9/linux3.10-glibc2.17-ppc64le-csm/bin:/opt/ibm/csm/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/ibm/flightlog/bin:/opt/ibutils/bin:/opt/ibm/spectrum_mpi/jsm_pmix/bin:/opt/puppetlabs/bin:/usr/lpp/mmfs/bin:`dirname >> <http://10.1.0.9/linux3.10-glibc2.17-ppc64le-csm/etc:/opt/ibm/spectrumcomput…> >> nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=gcc >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/bin/nvcc_wrapper >> --expt-extended-lambda -Xcompiler -rdynamic -lineinfo -DLANDAU_DIM=2 >> -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 >> -Xcompiler -fPIC -O3 -gencode arch=compute_70,code=sm_70 >> -I/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/include >> -Wno-deprecated-gpu-targets >> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include >> -I/sw/summit/cuda/10.1.243/include >> -I/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/include >> -fPIC -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC >> -DLANDAU_MAX_Q=4 -Werror=maybe-uninitialized -O0 ex3k.kokkos.cxx >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib >> -L/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/netlib-lapack-3.8.0-wcabdyqhdi5rooxbkqa6x5d7hxyxwdkm/lib64 >> -Wl,-rpath,/sw/summit/cuda/10.1.243/lib64 -L/sw/summit/cuda/10.1.243/lib64 >> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lp4est -lsc -lblas >> -llapack -ltriangle -lm -lz -lcudart -lcufft -lcublas -lcusparse -lcusolver >> -lcurand -lstdc++ -ldl -o ex3k >> nvcc_wrapper - *warning* you have set multiple optimization flags (-O*), >> only the last is used because nvcc can only accept a single optimization >> setting. >> >> */usr/bin/ld: /tmp/tmpxft_0000e0c2_00000000-10_ex3k.kokkos.o: undefined >> reference to symbol 'ompi_mpi_comm_self'*/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/lib/libmpi_ibm.so.3: >> error adding symbols: DSO missing from command line >> collect2: error: ld returned 1 exit status >> make: *** [ex3k] Error 1 >> >> On Sat, Jun 5, 2021 at 12:16 PM Junchao Zhang <junchao.zhang(a)gmail.com> >> wrote: >> >>> $rm ex3k >>> $make ex3k >>> and run again? >>> >>> --Junchao Zhang >>> >>> >>> On Sat, Jun 5, 2021 at 10:25 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>> >>>> This is posted in Barry's MR, but I get this error with Kokkos-cuda on >>>> Summit. Failing to open a shared lib. >>>> Thoughts? >>>> Mark >>>> >>>> 11:15 barry/2020-11-11/cleanup-matsetvaluesdevice= >>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make >>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>> PETSC_ARCH=arch-summit-opt-gnu-kokkos-notpl-cuda10 check >>>> Running check examples to verify correct installation >>>> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and >>>> PETSC_ARCH=arch-summit-opt-gnu-kokkos-notpl-cuda10 >>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI >>>> process >>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >>>> processes >>>> C/C++ example src/snes/tutorials/ex19 run successfully with cuda >>>> gmake[3]: [runex3k_kokkos] Error 127 (ignored) >>>> 1,25c1,2 >>>> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 >>>> < Vec Object: Exact Solution 2 MPI processes >>>> < type: mpikokkos >>>> < Process [0] >>>> < 0. >>>> < 0.015625 >>>> < 0.125 >>>> < Process [1] >>>> < 0.421875 >>>> < 1. >>>> < Vec Object: Forcing function 2 MPI processes >>>> < type: mpikokkos >>>> < Process [0] >>>> < 1e-72 >>>> < 1.50024 >>>> < 3.01563 >>>> < Process [1] >>>> < 4.67798 >>>> < 7. >>>> < 0 SNES Function norm 5.414682427127e+00 >>>> < 1 SNES Function norm 2.952582418265e-01 >>>> < 2 SNES Function norm 4.502293658739e-04 >>>> < 3 SNES Function norm 1.389665806646e-09 >>>> < Number of SNES iterations = 3 >>>> < Norm of error 1.49752e-10 Iterations 3 >>>> --- >>>> >>>> >>>> *> ./ex3k: error while loading shared libraries: libpetsc.so.3.015: >>>> cannot open shared object file: No such file or directory> ./ex3k: error >>>> while loading shared libraries: libpetsc.so.3.015: cannot open shared >>>> object file: No such file or directory* >>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials >>>> >>> >>

1 0

Re: [petsc-dev] kokkos test fail in branch
by Junchao Zhang 05 Jun '21

05 Jun '21

This problem was fixed in https://gitlab.com/petsc/petsc/-/merge_requests/4056, and is waiting for !3411 <https://gitlab.com/petsc/petsc/-/merge_requests/3411> :) --Junchao Zhang On Sat, Jun 5, 2021 at 9:42 PM Barry Smith <bsmith(a)petsc.dev> wrote: > > Looks like the MPI libraries are not being passed to the NVCC internal > compiler (gcc). This would normally be setup in MPI.py, please send > configure.log > > Barry > > > On Jun 5, 2021, at 1:18 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > Ah, there was an old one there. I removed it and I can not compile ex3k. I > thought check always rebuilt executables. Now I get a link error: > > 14:12 barry/2020-11-11/cleanup-matsetvaluesdevice *= > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make > PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > PETSC_ARCH=arch-summit-opt-gnu-kokkos-notpl-cuda10 ex3k > > PATH=/sw/sources/lsf-tools/2.0/summit/bin:/sw/summit/xalt/1.2.1/bin:/sw/summit/forge/20.0.1/bin:/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/bin:/sw/summit/gcc/6.4.0/bin:/sw/summit/cuda/10.1.243/bin:/autofs/nccs-svm1_sw/summit/.swci/0-core/opt/spack/20180914/linux-rhel7-ppc64le/gcc-4.8.5/cmake-3.20.2-24ualfzy6em6ws5sbiu7rlgcuionodrm/bin:/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-4.8.5/darshan-runtime-3.1.7-cnvxicgf5j4ap64qi6v5gxp67hmrjz43/bin:/sw/sources/hpss/bin:/opt/ibm/spectrumcomputing/lsf/ > 10.1.0.9/linux3.10-glibc2.17-ppc64le-csm/etc:/opt/ibm/spectrumcomputing/lsf/10.1.0.9/linux3.10-glibc2.17-ppc64le-csm/bin:/opt/ibm/csm/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/ibm/flightlog/bin:/opt/ibutils/bin:/opt/ibm/spectrum_mpi/jsm_pmix/bin:/opt/puppetlabs/bin:/usr/lpp/mmfs/bin:`dirname > <http://10.1.0.9/linux3.10-glibc2.17-ppc64le-csm/etc:/opt/ibm/spectrumcomput…> > nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=gcc > /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/bin/nvcc_wrapper > --expt-extended-lambda -Xcompiler -rdynamic -lineinfo -DLANDAU_DIM=2 > -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 > -Xcompiler -fPIC -O3 -gencode arch=compute_70,code=sm_70 > -I/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/include > -Wno-deprecated-gpu-targets > -I/gpfs/alpine/csc314/scratch/adams/petsc/include > -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include > -I/sw/summit/cuda/10.1.243/include > -I/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/include > -fPIC -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC > -DLANDAU_MAX_Q=4 -Werror=maybe-uninitialized -O0 ex3k.kokkos.cxx > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib > -L/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/netlib-lapack-3.8.0-wcabdyqhdi5rooxbkqa6x5d7hxyxwdkm/lib64 > -Wl,-rpath,/sw/summit/cuda/10.1.243/lib64 -L/sw/summit/cuda/10.1.243/lib64 > -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lp4est -lsc -lblas > -llapack -ltriangle -lm -lz -lcudart -lcufft -lcublas -lcusparse -lcusolver > -lcurand -lstdc++ -ldl -o ex3k > nvcc_wrapper - *warning* you have set multiple optimization flags (-O*), > only the last is used because nvcc can only accept a single optimization > setting. > > */usr/bin/ld: /tmp/tmpxft_0000e0c2_00000000-10_ex3k.kokkos.o: undefined > reference to symbol 'ompi_mpi_comm_self'*/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/lib/libmpi_ibm.so.3: > error adding symbols: DSO missing from command line > collect2: error: ld returned 1 exit status > make: *** [ex3k] Error 1 > > On Sat, Jun 5, 2021 at 12:16 PM Junchao Zhang <junchao.zhang(a)gmail.com> > wrote: > >> $rm ex3k >> $make ex3k >> and run again? >> >> --Junchao Zhang >> >> >> On Sat, Jun 5, 2021 at 10:25 AM Mark Adams <mfadams(a)lbl.gov> wrote: >> >>> This is posted in Barry's MR, but I get this error with Kokkos-cuda on >>> Summit. Failing to open a shared lib. >>> Thoughts? >>> Mark >>> >>> 11:15 barry/2020-11-11/cleanup-matsetvaluesdevice= >>> /gpfs/alpine/csc314/scratch/adams/petsc$ make >>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>> PETSC_ARCH=arch-summit-opt-gnu-kokkos-notpl-cuda10 check >>> Running check examples to verify correct installation >>> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and >>> PETSC_ARCH=arch-summit-opt-gnu-kokkos-notpl-cuda10 >>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process >>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >>> processes >>> C/C++ example src/snes/tutorials/ex19 run successfully with cuda >>> gmake[3]: [runex3k_kokkos] Error 127 (ignored) >>> 1,25c1,2 >>> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 >>> < Vec Object: Exact Solution 2 MPI processes >>> < type: mpikokkos >>> < Process [0] >>> < 0. >>> < 0.015625 >>> < 0.125 >>> < Process [1] >>> < 0.421875 >>> < 1. >>> < Vec Object: Forcing function 2 MPI processes >>> < type: mpikokkos >>> < Process [0] >>> < 1e-72 >>> < 1.50024 >>> < 3.01563 >>> < Process [1] >>> < 4.67798 >>> < 7. >>> < 0 SNES Function norm 5.414682427127e+00 >>> < 1 SNES Function norm 2.952582418265e-01 >>> < 2 SNES Function norm 4.502293658739e-04 >>> < 3 SNES Function norm 1.389665806646e-09 >>> < Number of SNES iterations = 3 >>> < Norm of error 1.49752e-10 Iterations 3 >>> --- >>> >>> >>> *> ./ex3k: error while loading shared libraries: libpetsc.so.3.015: >>> cannot open shared object file: No such file or directory> ./ex3k: error >>> while loading shared libraries: libpetsc.so.3.015: cannot open shared >>> object file: No such file or directory* >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials >>> >> >

1 0