October 2018 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Error running on Titan with GPUs & GNU
by Smith, Barry F. 30 Oct '18

30 Oct '18

> On Oct 29, 2018, at 8:09 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > And a debug build seems to work: Well ok. Are there newer versions of the Gnu compiler for this system? Are there any other compilers on the system that would likely be less buggy? IBM compilers? If this simple code generates a gross error with optimization who's to say how many more subtle bugs may be induced in the library by the buggy optimizer (there may be none but IMHO probability says there will be others). Is there any chance that valgrind runs on this machine; you could run the optimized version through it and see what it says. Barry > > 21:04 1 master= /lustre/atlas/proj-shared/geo127/petsc$ make PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda PETSC_ARCH="" test > Running test examples to verify correct installation > Using PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda and PETSC_ARCH= > *******************Error detected during compile or link!******************* > See http://www.mcs.anl.gov/petsc/documentation/faq.html > /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials ex19 > ********************************************************************************* > cc -o ex19.o -c -g -I/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/include `pwd`/ex19.c > cc -g -o ex19 ex19.o -L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib -Wl,-rpath,/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib -L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib -lpetsc -lHYPRE -lflapack -lfblas -lparmetis -lmetis -ldl > /lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(dlimpl.o): In function `PetscDLOpen': > /lustre/atlas1/geo127/proj-shared/petsc/src/sys/dll/dlimpl.c:108: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking > /lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(send.o): In function `PetscOpenSocket': > /lustre/atlas1/geo127/proj-shared/petsc/src/sys/classes/viewer/impls/socket/send.c:108: warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking > true ex19 > rm ex19.o > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process > See http://www.mcs.anl.gov/petsc/documentation/faq.html > lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > Number of SNES iterations = 2 > Application 19081049 resources: utime ~1s, stime ~1s, Rss ~17112, inblocks ~36504, outblocks ~111043 > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes > See http://www.mcs.anl.gov/petsc/documentation/faq.html > lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > Number of SNES iterations = 2 > Application 19081050 resources: utime ~1s, stime ~1s, Rss ~19816, inblocks ~36527, outblocks ~111043 > 5a6 > > Application 19081051 resources: utime ~1s, stime ~0s, Rss ~13864, inblocks ~36527, outblocks ~111043 > /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials > Possible problem with ex19_hypre, diffs above > ========================================= > *******************Error detected during compile or link!******************* > See http://www.mcs.anl.gov/petsc/documentation/faq.html > /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials ex5f > ********************************************************* > ftn -c -g -I/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/include -o ex5f.o ex5f.F90 > ftn -g -o ex5f ex5f.o -L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib -Wl,-rpath,/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib -L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib -lpetsc -lHYPRE -lflapack -lfblas -lparmetis -lmetis -ldl > /lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(dlimpl.o): In function `PetscDLOpen': > /lustre/atlas1/geo127/proj-shared/petsc/src/sys/dll/dlimpl.c:108: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking > /lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(send.o): In function `PetscOpenSocket': > /lustre/atlas1/geo127/proj-shared/petsc/src/sys/classes/viewer/impls/socket/send.c:108: warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking > rm ex5f.o > Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process > See http://www.mcs.anl.gov/petsc/documentation/faq.html > Number of SNES iterations = 4 > Application 19081055 resources: utime ~1s, stime ~0s, Rss ~12760, inblocks ~36800, outblocks ~111983 > Completed test examples > 21:06 master= /lustre/atlas/proj-shared/geo127/petsc$

1 0

Re: [petsc-dev] Error running on Titan with GPUs & GNU
by Mark Adams 29 Oct '18

29 Oct '18

And a debug build seems to work: 21:04 1 master= /lustre/atlas/proj-shared/geo127/petsc$ make PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda PETSC_ARCH="" test Running test examples to verify correct installation Using PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda and PETSC_ARCH= *******************Error detected during compile or link!******************* See http://www.mcs.anl.gov/petsc/documentation/faq.html /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials ex19 ********************************************************************************* cc -o ex19.o -c -g -I/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/include `pwd`/ex19.c cc -g -o ex19 ex19.o -L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib -Wl,-rpath,/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib -L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib -lpetsc -lHYPRE -lflapack -lfblas -lparmetis -lmetis -ldl /lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(dlimpl.o): In function `PetscDLOpen': /lustre/atlas1/geo127/proj-shared/petsc/src/sys/dll/dlimpl.c:108: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(send.o): In function `PetscOpenSocket': /lustre/atlas1/geo127/proj-shared/petsc/src/sys/classes/viewer/impls/socket/send.c:108: warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking true ex19 rm ex19.o Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process See http://www.mcs.anl.gov/petsc/documentation/faq.html lid velocity = 0.0016, prandtl # = 1., grashof # = 1. Number of SNES iterations = 2 Application 19081049 resources: utime ~1s, stime ~1s, Rss ~17112, inblocks ~36504, outblocks ~111043 Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes See http://www.mcs.anl.gov/petsc/documentation/faq.html lid velocity = 0.0016, prandtl # = 1., grashof # = 1. Number of SNES iterations = 2 Application 19081050 resources: utime ~1s, stime ~1s, Rss ~19816, inblocks ~36527, outblocks ~111043 5a6 > Application 19081051 resources: utime ~1s, stime ~0s, Rss ~13864, inblocks ~36527, outblocks ~111043 /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials Possible problem with ex19_hypre, diffs above ========================================= *******************Error detected during compile or link!******************* See http://www.mcs.anl.gov/petsc/documentation/faq.html /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials ex5f ********************************************************* ftn -c -g -I/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/include -o ex5f.o ex5f.F90 ftn -g -o ex5f ex5f.o -L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib -Wl,-rpath,/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib -L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib -lpetsc -lHYPRE -lflapack -lfblas -lparmetis -lmetis -ldl /lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(dlimpl.o): In function `PetscDLOpen': /lustre/atlas1/geo127/proj-shared/petsc/src/sys/dll/dlimpl.c:108: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(send.o): In function `PetscOpenSocket': /lustre/atlas1/geo127/proj-shared/petsc/src/sys/classes/viewer/impls/socket/send.c:108: warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking rm ex5f.o Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process See http://www.mcs.anl.gov/petsc/documentation/faq.html Number of SNES iterations = 4 Application 19081055 resources: utime ~1s, stime ~0s, Rss ~12760, inblocks ~36800, outblocks ~111983 Completed test examples 21:06 master= /lustre/atlas/proj-shared/geo127/petsc$

1 0

Re: [petsc-dev] Error running on Titan with GPUs & GNU
by Mark Adams 29 Oct '18

29 Oct '18

On Mon, Oct 29, 2018 at 6:55 PM Smith, Barry F. <bsmith(a)mcs.anl.gov> wrote: > > Here is the code > > > PetscStackCallBLAS("LAPACKgeev",LAPACKgeev_("N","N",&bn,R,&bN,realpart,imagpart,&sdummy,&idummy,&sdummy,&idummy,work,&lwork,&lierr)); > if (lierr) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"Error in LAPACK > routine %d",(int)lierr); > > What is unfathomable is that it prints (int) lierr of 0 but then the if > () test should not be satisfied. > > Do a ./configure with debugging turned on, could be an optimizing > compiler error. > Configuring debug now. Note, I was able to run ex56 (ksp) which does not use GMRES. This error was from a GMRES method so maybe this is an isolated problem. > > Barry > > > > On Oct 29, 2018, at 3:56 PM, Mark Adams via petsc-dev < > petsc-dev(a)mcs.anl.gov> wrote: > > > > I get this error running the tests using GPUs. An error in an LAPACK > routine. > > > > 16:50 master= /lustre/atlas/proj-shared/geo127/petsc$ make > PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda > PETSC_ARCH="" test > > Running test examples to verify correct installation > > Using > PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda > and PETSC_ARCH= > > *******************Error detected during compile or > link!******************* > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials ex19 > > > ********************************************************************************* > > cc -o ex19.o -c -O > -I/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/include > `pwd`/ex19.c > > cc -O -o ex19 ex19.o > -L/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib > -Wl,-rpath,/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib > -L/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib > -lpetsc -lHYPRE -lparmetis -lmetis -ldl > > > /lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib/libpetsc.a(dlimpl.o): > In function `PetscDLOpen': > > dlimpl.c:(.text+0x3b): warning: Using 'dlopen' in statically linked > applications requires at runtime the shared libraries from the glibc > version used for linking > > > /lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib/libpetsc.a(send.o): > In function `PetscOpenSocket': > > send.c:(.text+0x3be): warning: Using 'gethostbyname' in statically > linked applications requires at runtime the shared libraries from the glibc > version used for linking > > true ex19 > > rm ex19.o > > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI > process > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > > Number of SNES iterations = 2 > > Application 19079964 resources: utime ~1s, stime ~1s, Rss ~29412, > inblocks ~37563, outblocks ~131654 > > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI > processes > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > > [1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > --------------------- Error Message > -------------------------------------------------------------- > > [1]PETSC ERROR: [0]PETSC ERROR: Error in external library > > Error in external library > > [1]PETSC ERROR: [0]PETSC ERROR: Error in LAPACK routine 0 > > Error in LAPACK routine 0 > > [1]PETSC ERROR: [0]PETSC ERROR: See > http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble > shooting. > > [1]PETSC ERROR: [0]PETSC ERROR: Petsc Development GIT revision: > v3.10.2-461-g0ed19bb123 GIT Date: 2018-10-29 13:43:53 +0100 > > Petsc Development GIT revision: v3.10.2-461-g0ed19bb123 GIT Date: > 2018-10-29 13:43:53 +0100 > > [1]PETSC ERROR: [0]PETSC ERROR: ./ex19 on a named nid16438 by adams Mon > Oct 29 16:52:05 2018 > > ./ex19 on a named nid16438 by adams Mon Oct 29 16:52:05 2018 > > [1]PETSC ERROR: [0]PETSC ERROR: Configure options --with-cudac=1 > --with-batch=0 > --prefix=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda > --download-hypre --download-metis --download-parmetis --with-cc=cc > --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 > --with-fc=ftn --with-fortranlib-autodetect=0 --with-shared-libraries=0 > --known-mpi-shared-libraries=1 --with-mpiexec=aprun --with-x=0 > --with-64-bit-indices --with-debugging=0 > PETSC_ARCH=arch-titan-opt64idx-gnu-cuda > PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc > > Configure options --with-cudac=1 --with-batch=0 > --prefix=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda > --download-hypre --download-metis --download-parmetis --with-cc=cc > --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 > --with-fc=ftn --with-fortranlib-autodetect=0 --with-shared-libraries=0 > --known-mpi-shared-libraries=1 --with-mpiexec=aprun --with-x=0 > --with-64-bit-indices --with-debugging=0 > PETSC_ARCH=arch-titan-opt64idx-gnu-cuda > PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc > > [1]PETSC ERROR: [0]PETSC ERROR: #1 KSPComputeEigenvalues_GMRES() line > 144 in > /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/gmres/gmreig.c > > #1 KSPComputeEigenvalues_GMRES() line 144 in > /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/gmres/gmreig.c > > [1]PETSC ERROR: [0]PETSC ERROR: #2 KSPComputeEigenvalues() line 132 in > /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/interface/itfunc.c > > #2 KSPComputeEigenvalues() line 132 in > /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/interface/itfunc.c > > [1]PETSC ERROR: [0]PETSC ERROR: #3 > KSPChebyshevComputeExtremeEigenvalues_Private() line 288 in > /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/cheby/cheby.c > > #3 KSPChebyshevComputeExtremeEigenvalues_Private() line 288 in > /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/cheby/cheby.c > > [1]PETSC ERROR: [0]PETSC ERROR: #4 KSPSolve_Chebyshev() line 390 in > /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/cheby/cheby.c > > #4 KSPSolve_Chebyshev() line 390 in > /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/cheby/cheby.c > > [1]PETSC ERROR: [0]PETSC ERROR: #5 KSPSolve() line 780 in > /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/interface/itfunc.c > > #5 KSPSolve() line 780 in > /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/interface/itfunc.c > > [1]PETSC ERROR: [0]PETSC ERROR: #6 PCMGMCycle_Private() line 20 in > /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/pc/impls/mg/mg.c > > #6 PCMGMCycle_Private() line 20 in > /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/pc/impls/mg/mg.c > > [1]PETSC ERROR: [0]PETSC ERROR: #7 PCApply_MG() line 377 in > /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/pc/impls/mg/mg.c > > #7 PCApply_MG() line 377 in > /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/pc/impls/mg/mg.c > > [1]PETSC ERROR: [0]PETSC ERROR: #8 PCApply() line 462 in > /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/pc/interface/precon.c > > #8 PCApply() line 462 in > /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/pc/interface/precon.c > > [1]PETSC ERROR: [0]PETSC ERROR: #9 KSP_PCApply() line 281 in > /lustre/atlas/proj-shared/geo127/petsc/include/petsc/private/kspimpl.h > > #9 KSP_PCApply() line 281 in > /lustre/atlas/proj-shared/geo127/petsc/include/petsc/private/kspimpl.h > > [1]PETSC ERROR: [0]PETSC ERROR: #10 KSPFGMRESCycle() line 166 in > /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > > #10 KSPFGMRESCycle() line 166 in > /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > > [1]PETSC ERROR: [0]PETSC ERROR: #11 KSPSolve_FGMRES() line 291 in > /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > > #11 KSPSolve_FGMRES() line 291 in > /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > > [1]PETSC ERROR: [0]PETSC ERROR: #12 KSPSolve() line 780 in > /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/interface/itfunc.c > > #12 KSPSolve() line 780 in > /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/interface/itfunc.c > > [1]PETSC ERROR: [0]PETSC ERROR: #13 SNESSolve_NEWTONLS() line 224 in > /lustre/atlas1/geo127/proj-shared/petsc/src/snes/impls/ls/ls.c > > #13 SNESSolve_NEWTONLS() line 224 in > /lustre/atlas1/geo127/proj-shared/petsc/src/snes/impls/ls/ls.c > > [1]PETSC ERROR: [0]PETSC ERROR: #14 SNESSolve() line 4396 in > /lustre/atlas1/geo127/proj-shared/petsc/src/snes/interface/snes.c > > #14 SNESSolve() line 4396 in > /lustre/atlas1/geo127/proj-shared/petsc/src/snes/interface/snes.c > > [1]PETSC ERROR: [0]PETSC ERROR: #15 main() line 161 in > /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials/ex19.c > > #15 main() line 161 in > /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials/ex19.c > > [1]PETSC ERROR: [0]PETSC ERROR: PETSc Option Table entries: > > PETSc Option Table entries: > > [1]PETSC ERROR: [0]PETSC ERROR: -da_refine 3 > > -da_refine 3 > > [1]PETSC ERROR: [0]PETSC ERROR: -ksp_type fgmres > > -ksp_type fgmres > > [1]PETSC ERROR: [0]PETSC ERROR: -pc_type mg > > -pc_type mg > > [1]PETSC ERROR: [0]PETSC ERROR: ----------------End of Error Message > -------send entire error message to petsc-maint(a)mcs.anl.gov---------- > > ----------------End of Error Message -------send entire error message to > petsc-maint(a)mcs.anl.gov---------- > > Rank 0 [Mon Oct 29 16:52:05 2018] [c7-7c1s4n2] application called > MPI_Abort(MPI_COMM_WORLD, 76) - process 0 > > Rank 1 [Mon Oct 29 16:52:05 2018] [c7-7c1s4n2] application called > MPI_Abort(MPI_COMM_WORLD, 76) - process 1 > > _pmiu_daemon(SIGCHLD): [NID 16438] [c7-7c1s4n2] [Mon Oct 29 16:52:05 > 2018] PE RANK 1 exit signal Aborted > > Application 19079965 exit codes: 134 > > Application 19079965 resources: utime ~1s, stime ~1s, Rss ~29412, > inblocks ~37571, outblocks ~131660 > > 5a6 > > > Application 19079968 resources: utime ~1s, stime ~1s, Rss ~29412, > inblocks ~37586, outblocks ~131654 > > > >

1 0

Re: [petsc-dev] Plex - Metis warnings
by Mark Adams 29 Oct '18

29 Oct '18

I was able to run ksp ex56 manually. This machine requires an inscrutable workflow. I built the code in my home directory, copied the executable to a working directory and ran there. I have seen my colleagues do this copy business that is clearly not in PETSc tests. I don't understand it but I can I can hand if off to my users with some assurance that it _can_ work! Thanks, Mark On Mon, Oct 29, 2018 at 7:23 PM Balay, Satish <balay(a)mcs.anl.gov> wrote: > On Mon, 29 Oct 2018, Mark Adams via petsc-dev wrote: > > > > > > > > > > Pushing language C > > > Popping language C > > > Executing: cc -o /tmp/petsc-yiGfSd/config.packages.MPI/conftest -O > > > /tmp/petsc-yiGfSd/config.packages.MPI/conftest.o -ldl > > > Testing executable /tmp/petsc-yiGfSd/config.packages.MPI/conftest to > see > > > if it can be run > > > Executing: /tmp/petsc-yiGfSd/config.packages.MPI/conftest > > > Executing: /tmp/petsc-yiGfSd/config.packages.MPI/conftest > > > ERROR while running executable: Could not execute > > > "['/tmp/petsc-yiGfSd/config.packages.MPI/conftest']": > > > [Mon Oct 29 17:44:45 2018] [unknown] Fatal error in MPI_Init: Other MPI > > > error, error stack: > > > MPIR_Init_thread(537): > > > MPID_Init(249).......: channel initialization failed > > > MPID_Init(638).......: PMI2 init failed: 1 > > > > > > <<<<<<< > > > > > > So all MPI tests fail on frontend? And you need to use --with-batch? > > > > > > > I ran the tests from an interactive shell. I guess I should run configure > > from an interactive shell. but configure and make seemed to work. > > >>>> > Executing: aprun /tmp/petsc-yiGfSd/config.libraries/conftest > Executing: aprun /tmp/petsc-yiGfSd/config.libraries/conftest > stdout: XALT Error: unable to find aprun > ERROR while running executable: Could not execute "['aprun > /tmp/petsc-yiGfSd/config.libraries/conftest']": > XALT Error: unable to find aprun > > Are you able to use 'aprun' via interactive nodes to run MPI jobs? Somehow > configure is unable to use aprun. > > Satish >

1 0

Re: [petsc-dev] Plex - Metis warnings
by Balay, Satish 29 Oct '18

29 Oct '18

On Mon, 29 Oct 2018, Mark Adams via petsc-dev wrote: > > > > > > Pushing language C > > Popping language C > > Executing: cc -o /tmp/petsc-yiGfSd/config.packages.MPI/conftest -O > > /tmp/petsc-yiGfSd/config.packages.MPI/conftest.o -ldl > > Testing executable /tmp/petsc-yiGfSd/config.packages.MPI/conftest to see > > if it can be run > > Executing: /tmp/petsc-yiGfSd/config.packages.MPI/conftest > > Executing: /tmp/petsc-yiGfSd/config.packages.MPI/conftest > > ERROR while running executable: Could not execute > > "['/tmp/petsc-yiGfSd/config.packages.MPI/conftest']": > > [Mon Oct 29 17:44:45 2018] [unknown] Fatal error in MPI_Init: Other MPI > > error, error stack: > > MPIR_Init_thread(537): > > MPID_Init(249).......: channel initialization failed > > MPID_Init(638).......: PMI2 init failed: 1 > > > > <<<<<<< > > > > So all MPI tests fail on frontend? And you need to use --with-batch? > > > > I ran the tests from an interactive shell. I guess I should run configure > from an interactive shell. but configure and make seemed to work. >>>> Executing: aprun /tmp/petsc-yiGfSd/config.libraries/conftest Executing: aprun /tmp/petsc-yiGfSd/config.libraries/conftest stdout: XALT Error: unable to find aprun ERROR while running executable: Could not execute "['aprun /tmp/petsc-yiGfSd/config.libraries/conftest']": XALT Error: unable to find aprun Are you able to use 'aprun' via interactive nodes to run MPI jobs? Somehow configure is unable to use aprun. Satish

1 0

Re: [petsc-dev] [petsc-users] Convergence of AMG
by Smith, Barry F. 29 Oct '18

29 Oct '18

Great, I just didn't understand the name. > On Oct 29, 2018, at 3:30 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > On Mon, Oct 29, 2018 at 2:35 PM Smith, Barry F. <bsmith(a)mcs.anl.gov> wrote: > > Why not just stop it once it is equal to or less than the minimum values set by the person. > > That is what it does now. It stops when it is below the value given. > > Thus you need not "backtrack" by removing levels but the user still has some control over preventing a "tiny" coarse problem. For example in this case if the user set a minimum of 1000 it would end up with 642 unknowns on the coarse level > > Yes, that is what it would do now. I thought you wanted something different. > > which is likely better than 6 or 54. > > > > Barry > > > > On Oct 29, 2018, at 8:27 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > > > On Sun, Oct 28, 2018 at 4:54 PM Smith, Barry F. <bsmith(a)mcs.anl.gov> wrote: > > > > Moved a question not needed in the public discussions to petsc-dev to ask Mark. > > > > > > Mark, > > > > PCGAMGSetCoarseEqLim - Set maximum number of equations on coarsest grid > > > > Is there a way to set the minimum number of equations on the coarse grid also? This particular case goes down to 6, 54 and 642 unknowns on the coarsest grids when I'm guessing it would be better to stop at 642 unknowns for the coarsest level. > > > > No, because I don't know how it is going to coarsen I did not want to bother with backtracking (I do when there is an error on the coarse grid so it would be easy to add this but I don't think it is worth the clutter). > > >

1 0

Re: [petsc-dev] Error running on Titan with GPUs & GNU
by Smith, Barry F. 29 Oct '18

29 Oct '18

Here is the code PetscStackCallBLAS("LAPACKgeev",LAPACKgeev_("N","N",&bn,R,&bN,realpart,imagpart,&sdummy,&idummy,&sdummy,&idummy,work,&lwork,&lierr)); if (lierr) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"Error in LAPACK routine %d",(int)lierr); What is unfathomable is that it prints (int) lierr of 0 but then the if () test should not be satisfied. Do a ./configure with debugging turned on, could be an optimizing compiler error. Barry > On Oct 29, 2018, at 3:56 PM, Mark Adams via petsc-dev <petsc-dev(a)mcs.anl.gov> wrote: > > I get this error running the tests using GPUs. An error in an LAPACK routine. > > 16:50 master= /lustre/atlas/proj-shared/geo127/petsc$ make PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda PETSC_ARCH="" test > Running test examples to verify correct installation > Using PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda and PETSC_ARCH= > *******************Error detected during compile or link!******************* > See http://www.mcs.anl.gov/petsc/documentation/faq.html > /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials ex19 > ********************************************************************************* > cc -o ex19.o -c -O -I/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/include `pwd`/ex19.c > cc -O -o ex19 ex19.o -L/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib -Wl,-rpath,/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib -L/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib -lpetsc -lHYPRE -lparmetis -lmetis -ldl > /lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib/libpetsc.a(dlimpl.o): In function `PetscDLOpen': > dlimpl.c:(.text+0x3b): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking > /lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib/libpetsc.a(send.o): In function `PetscOpenSocket': > send.c:(.text+0x3be): warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking > true ex19 > rm ex19.o > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process > See http://www.mcs.anl.gov/petsc/documentation/faq.html > lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > Number of SNES iterations = 2 > Application 19079964 resources: utime ~1s, stime ~1s, Rss ~29412, inblocks ~37563, outblocks ~131654 > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes > See http://www.mcs.anl.gov/petsc/documentation/faq.html > lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > [1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > --------------------- Error Message -------------------------------------------------------------- > [1]PETSC ERROR: [0]PETSC ERROR: Error in external library > Error in external library > [1]PETSC ERROR: [0]PETSC ERROR: Error in LAPACK routine 0 > Error in LAPACK routine 0 > [1]PETSC ERROR: [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [1]PETSC ERROR: [0]PETSC ERROR: Petsc Development GIT revision: v3.10.2-461-g0ed19bb123 GIT Date: 2018-10-29 13:43:53 +0100 > Petsc Development GIT revision: v3.10.2-461-g0ed19bb123 GIT Date: 2018-10-29 13:43:53 +0100 > [1]PETSC ERROR: [0]PETSC ERROR: ./ex19 on a named nid16438 by adams Mon Oct 29 16:52:05 2018 > ./ex19 on a named nid16438 by adams Mon Oct 29 16:52:05 2018 > [1]PETSC ERROR: [0]PETSC ERROR: Configure options --with-cudac=1 --with-batch=0 --prefix=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda --download-hypre --download-metis --download-parmetis --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-fc=ftn --with-fortranlib-autodetect=0 --with-shared-libraries=0 --known-mpi-shared-libraries=1 --with-mpiexec=aprun --with-x=0 --with-64-bit-indices --with-debugging=0 PETSC_ARCH=arch-titan-opt64idx-gnu-cuda PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc > Configure options --with-cudac=1 --with-batch=0 --prefix=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda --download-hypre --download-metis --download-parmetis --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-fc=ftn --with-fortranlib-autodetect=0 --with-shared-libraries=0 --known-mpi-shared-libraries=1 --with-mpiexec=aprun --with-x=0 --with-64-bit-indices --with-debugging=0 PETSC_ARCH=arch-titan-opt64idx-gnu-cuda PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc > [1]PETSC ERROR: [0]PETSC ERROR: #1 KSPComputeEigenvalues_GMRES() line 144 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/gmres/gmreig.c > #1 KSPComputeEigenvalues_GMRES() line 144 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/gmres/gmreig.c > [1]PETSC ERROR: [0]PETSC ERROR: #2 KSPComputeEigenvalues() line 132 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/interface/itfunc.c > #2 KSPComputeEigenvalues() line 132 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: [0]PETSC ERROR: #3 KSPChebyshevComputeExtremeEigenvalues_Private() line 288 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/cheby/cheby.c > #3 KSPChebyshevComputeExtremeEigenvalues_Private() line 288 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/cheby/cheby.c > [1]PETSC ERROR: [0]PETSC ERROR: #4 KSPSolve_Chebyshev() line 390 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/cheby/cheby.c > #4 KSPSolve_Chebyshev() line 390 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/cheby/cheby.c > [1]PETSC ERROR: [0]PETSC ERROR: #5 KSPSolve() line 780 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/interface/itfunc.c > #5 KSPSolve() line 780 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: [0]PETSC ERROR: #6 PCMGMCycle_Private() line 20 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/pc/impls/mg/mg.c > #6 PCMGMCycle_Private() line 20 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/pc/impls/mg/mg.c > [1]PETSC ERROR: [0]PETSC ERROR: #7 PCApply_MG() line 377 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/pc/impls/mg/mg.c > #7 PCApply_MG() line 377 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/pc/impls/mg/mg.c > [1]PETSC ERROR: [0]PETSC ERROR: #8 PCApply() line 462 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/pc/interface/precon.c > #8 PCApply() line 462 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/pc/interface/precon.c > [1]PETSC ERROR: [0]PETSC ERROR: #9 KSP_PCApply() line 281 in /lustre/atlas/proj-shared/geo127/petsc/include/petsc/private/kspimpl.h > #9 KSP_PCApply() line 281 in /lustre/atlas/proj-shared/geo127/petsc/include/petsc/private/kspimpl.h > [1]PETSC ERROR: [0]PETSC ERROR: #10 KSPFGMRESCycle() line 166 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > #10 KSPFGMRESCycle() line 166 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > [1]PETSC ERROR: [0]PETSC ERROR: #11 KSPSolve_FGMRES() line 291 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > #11 KSPSolve_FGMRES() line 291 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c > [1]PETSC ERROR: [0]PETSC ERROR: #12 KSPSolve() line 780 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/interface/itfunc.c > #12 KSPSolve() line 780 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/interface/itfunc.c > [1]PETSC ERROR: [0]PETSC ERROR: #13 SNESSolve_NEWTONLS() line 224 in /lustre/atlas1/geo127/proj-shared/petsc/src/snes/impls/ls/ls.c > #13 SNESSolve_NEWTONLS() line 224 in /lustre/atlas1/geo127/proj-shared/petsc/src/snes/impls/ls/ls.c > [1]PETSC ERROR: [0]PETSC ERROR: #14 SNESSolve() line 4396 in /lustre/atlas1/geo127/proj-shared/petsc/src/snes/interface/snes.c > #14 SNESSolve() line 4396 in /lustre/atlas1/geo127/proj-shared/petsc/src/snes/interface/snes.c > [1]PETSC ERROR: [0]PETSC ERROR: #15 main() line 161 in /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials/ex19.c > #15 main() line 161 in /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials/ex19.c > [1]PETSC ERROR: [0]PETSC ERROR: PETSc Option Table entries: > PETSc Option Table entries: > [1]PETSC ERROR: [0]PETSC ERROR: -da_refine 3 > -da_refine 3 > [1]PETSC ERROR: [0]PETSC ERROR: -ksp_type fgmres > -ksp_type fgmres > [1]PETSC ERROR: [0]PETSC ERROR: -pc_type mg > -pc_type mg > [1]PETSC ERROR: [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- > ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- > Rank 0 [Mon Oct 29 16:52:05 2018] [c7-7c1s4n2] application called MPI_Abort(MPI_COMM_WORLD, 76) - process 0 > Rank 1 [Mon Oct 29 16:52:05 2018] [c7-7c1s4n2] application called MPI_Abort(MPI_COMM_WORLD, 76) - process 1 > _pmiu_daemon(SIGCHLD): [NID 16438] [c7-7c1s4n2] [Mon Oct 29 16:52:05 2018] PE RANK 1 exit signal Aborted > Application 19079965 exit codes: 134 > Application 19079965 resources: utime ~1s, stime ~1s, Rss ~29412, inblocks ~37571, outblocks ~131660 > 5a6 > > Application 19079968 resources: utime ~1s, stime ~1s, Rss ~29412, inblocks ~37586, outblocks ~131654 >

1 0

Re: [petsc-dev] Plex - Metis warnings
by Mark Adams 29 Oct '18

29 Oct '18

> > > Pushing language C > Popping language C > Executing: cc -o /tmp/petsc-yiGfSd/config.packages.MPI/conftest -O > /tmp/petsc-yiGfSd/config.packages.MPI/conftest.o -ldl > Testing executable /tmp/petsc-yiGfSd/config.packages.MPI/conftest to see > if it can be run > Executing: /tmp/petsc-yiGfSd/config.packages.MPI/conftest > Executing: /tmp/petsc-yiGfSd/config.packages.MPI/conftest > ERROR while running executable: Could not execute > "['/tmp/petsc-yiGfSd/config.packages.MPI/conftest']": > [Mon Oct 29 17:44:45 2018] [unknown] Fatal error in MPI_Init: Other MPI > error, error stack: > MPIR_Init_thread(537): > MPID_Init(249).......: channel initialization failed > MPID_Init(638).......: PMI2 init failed: 1 > > <<<<<<< > > So all MPI tests fail on frontend? And you need to use --with-batch? > I ran the tests from an interactive shell. I guess I should run configure from an interactive shell. but configure and make seemed to work. > > Satish >

1 0

Re: [petsc-dev] Plex - Metis warnings
by Balay, Satish 29 Oct '18

29 Oct '18

On Mon, 29 Oct 2018, Mark Adams via petsc-dev wrote: > On Mon, Oct 29, 2018 at 5:26 PM Balay, Satish <balay(a)mcs.anl.gov> wrote: > > > Can you send corresponding logs? >>>>> Source: #include "confdefs.h" #include "conffix.h" #ifdef PETSC_HAVE_STDLIB_H #include <stdlib.h> #endif #include <mpi.h> int main() { MPI_Aint size; int ierr; MPI_Init(0,0); ierr = MPI_Type_extent(MPI_INT64_T, &size); if(ierr || (size == 0)) exit(1); MPI_Finalize(); ; return 0; } Pushing language C Popping language C Executing: cc -o /tmp/petsc-yiGfSd/config.packages.MPI/conftest -O /tmp/petsc-yiGfSd/config.packages.MPI/conftest.o -ldl Testing executable /tmp/petsc-yiGfSd/config.packages.MPI/conftest to see if it can be run Executing: /tmp/petsc-yiGfSd/config.packages.MPI/conftest Executing: /tmp/petsc-yiGfSd/config.packages.MPI/conftest ERROR while running executable: Could not execute "['/tmp/petsc-yiGfSd/config.packages.MPI/conftest']": [Mon Oct 29 17:44:45 2018] [unknown] Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(537): MPID_Init(249).......: channel initialization failed MPID_Init(638).......: PMI2 init failed: 1 <<<<<<< So all MPI tests fail on frontend? And you need to use --with-batch? Satish

1 0

Re: [petsc-dev] Error running on Titan with GPUs & GNU
by Mark Adams 29 Oct '18

29 Oct '18

Still getting this error with downloaded lapack. I sent the logs on the other thread. 18:02 master= /lustre/atlas/proj-shared/geo127/petsc$ make PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda PETSC_ARCH="" test Running test examples to verify correct installation Using PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda and PETSC_ARCH= *******************Error detected during compile or link!******************* See http://www.mcs.anl.gov/petsc/documentation/faq.html /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials ex19 ********************************************************************************* cc -o ex19.o -c -O -I/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/include `pwd`/ex19.c cc -O -o ex19 ex19.o -L/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib -Wl,-rpath,/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib -L/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib -lpetsc -lHYPRE -lflapack -lfblas -lparmetis -lmetis -ldl /lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib/libpetsc.a(dlimpl.o): In function `PetscDLOpen': dlimpl.c:(.text+0x3b): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib/libpetsc.a(send.o): In function `PetscOpenSocket': send.c:(.text+0x3be): warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking true ex19 rm ex19.o Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process See http://www.mcs.anl.gov/petsc/documentation/faq.html lid velocity = 0.0016, prandtl # = 1., grashof # = 1. Number of SNES iterations = 2 Application 19080270 resources: utime ~0s, stime ~1s, Rss ~72056, inblocks ~19397, outblocks ~51049 Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes See http://www.mcs.anl.gov/petsc/documentation/faq.html lid velocity = 0.0016, prandtl # = 1., grashof # = 1. [1]PETSC ERROR: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: [0]PETSC ERROR: Error in external library Error in external library [1]PETSC ERROR: [0]PETSC ERROR: Error in LAPACK routine 0 Error in LAPACK routine 0 [1]PETSC ERROR: [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: [0]PETSC ERROR: Petsc Development GIT revision: v3.10.2-461-g0ed19bb123 GIT Date: 2018-10-29 13:43:53 +0100 Petsc Development GIT revision: v3.10.2-461-g0ed19bb123 GIT Date: 2018-10-29 13:43:53 +0100 [1]PETSC ERROR: [0]PETSC ERROR: ./ex19 on a named nid08331 by adams Mon Oct 29 18:07:59 2018 ./ex19 on a named nid08331 by adams Mon Oct 29 18:07:59 2018 [1]PETSC ERROR: [0]PETSC ERROR: Configure options --with-cudac=1 --with-batch=0 --prefix=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda --download-hypre --download-metis --download-parmetis --download-fblaslapack --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-fc=ftn --with-fortranlib-autodetect=0 --with-shared-libraries=0 --known-mpi-shared-libraries=1 --with-mpiexec=aprun --with-x=0 --with-64-bit-indices --with-debugging=0 PETSC_ARCH=arch-titan-opt64idx-gnu-cuda PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc Configure options --with-cudac=1 --with-batch=0 --prefix=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda --download-hypre --download-metis --download-parmetis --download-fblaslapack --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-fc=ftn --with-fortranlib-autodetect=0 --with-shared-libraries=0 --known-mpi-shared-libraries=1 --with-mpiexec=aprun --with-x=0 --with-64-bit-indices --with-debugging=0 PETSC_ARCH=arch-titan-opt64idx-gnu-cuda PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc [1]PETSC ERROR: [0]PETSC ERROR: #1 KSPComputeEigenvalues_GMRES() line 144 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/gmres/gmreig.c #1 KSPComputeEigenvalues_GMRES() line 144 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/gmres/gmreig.c [1]PETSC ERROR: #2 KSPComputeEigenvalues() line 132 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [1]PETSC ERROR: #2 KSPComputeEigenvalues() line 132 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/interface/itfunc.c #3 KSPChebyshevComputeExtremeEigenvalues_Private() line 288 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/cheby/cheby.c [0]PETSC ERROR: [1]PETSC ERROR: #3 KSPChebyshevComputeExtremeEigenvalues_Private() line 288 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/cheby/cheby.c #4 KSPSolve_Chebyshev() line 390 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/cheby/cheby.c [0]PETSC ERROR: [1]PETSC ERROR: #4 KSPSolve_Chebyshev() line 390 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/cheby/cheby.c #5 KSPSolve() line 780 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [1]PETSC ERROR: #5 KSPSolve() line 780 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/interface/itfunc.c #6 PCMGMCycle_Private() line 20 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: [1]PETSC ERROR: #6 PCMGMCycle_Private() line 20 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/pc/impls/mg/mg.c #7 PCApply_MG() line 377 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: [1]PETSC ERROR: #7 PCApply_MG() line 377 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/pc/impls/mg/mg.c #8 PCApply() line 462 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: [1]PETSC ERROR: #8 PCApply() line 462 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/pc/interface/precon.c #9 KSP_PCApply() line 281 in /lustre/atlas/proj-shared/geo127/petsc/include/petsc/private/kspimpl.h [0]PETSC ERROR: [1]PETSC ERROR: #9 KSP_PCApply() line 281 in /lustre/atlas/proj-shared/geo127/petsc/include/petsc/private/kspimpl.h #10 KSPFGMRESCycle() line 166 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c [0]PETSC ERROR: [1]PETSC ERROR: #10 KSPFGMRESCycle() line 166 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c #11 KSPSolve_FGMRES() line 291 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c [0]PETSC ERROR: [1]PETSC ERROR: #11 KSPSolve_FGMRES() line 291 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c #12 KSPSolve() line 780 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [1]PETSC ERROR: #12 KSPSolve() line 780 in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/interface/itfunc.c #13 SNESSolve_NEWTONLS() line 224 in /lustre/atlas1/geo127/proj-shared/petsc/src/snes/impls/ls/ls.c [0]PETSC ERROR: [1]PETSC ERROR: #13 SNESSolve_NEWTONLS() line 224 in /lustre/atlas1/geo127/proj-shared/petsc/src/snes/impls/ls/ls.c #14 SNESSolve() line 4396 in /lustre/atlas1/geo127/proj-shared/petsc/src/snes/interface/snes.c [0]PETSC ERROR: [1]PETSC ERROR: #14 SNESSolve() line 4396 in /lustre/atlas1/geo127/proj-shared/petsc/src/snes/interface/snes.c #15 main() line 161 in /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials/ex19.c [0]PETSC ERROR: [1]PETSC ERROR: #15 main() line 161 in /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials/ex19.c PETSc Option Table entries: [0]PETSC ERROR: [1]PETSC ERROR: PETSc Option Table entries: -da_refine 3 [0]PETSC ERROR: [1]PETSC ERROR: -da_refine 3 -ksp_type fgmres [0]PETSC ERROR: [1]PETSC ERROR: -ksp_type fgmres -pc_type mg [0]PETSC ERROR: -pc_type mg [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- Rank 0 [Mon Oct 29 18:07:59 2018] [c20-3c1s5n3] application called MPI_Abort(MPI_COMM_WORLD, 76) - process 0 Rank 1 [Mon Oct 29 18:07:59 2018] [c20-3c1s5n3] application called MPI_Abort(MPI_COMM_WORLD, 76) - process 1 _pmiu_daemon(SIGCHLD): [NID 08331] [c20-3c1s5n3] [Mon Oct 29 18:07:59 2018] PE RANK 1 exit signal Aborted Application 19080271 exit codes: 134 Application 19080271 resources: utime ~0s, stime ~1s, Rss ~72056, inblocks ~19405, outblocks ~51055 5a6 > Application 19080272 resources: utime ~1s, stime ~1s, Rss ~72056, inblocks ~19420, outblocks ~51049 /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials Possible problem with ex19_hypre, diffs above ========================================= *******************Error detected during compile or link!******************* See http://www.mcs.anl.gov/petsc/documentation/faq.html /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials ex5f ********************************************************* ftn -c -O -I/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/include -o ex5f.o ex5f.F90 ftn -O -o ex5f ex5f.o -L/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib -Wl,-rpath,/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib -L/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib -lpetsc -lHYPRE -lflapack -lfblas -lparmetis -lmetis -ldl /lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib/libpetsc.a(dlimpl.o): In function `PetscDLOpen': dlimpl.c:(.text+0x3b): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib/libpetsc.a(send.o): In function `PetscOpenSocket': send.c:(.text+0x3be): warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking rm ex5f.o Possible error running Fortran example src/snes/examples/tutorials/ex5f with 1 MPI process

1 0