April 2018 - petsc-users - mailman3.cels.anl.gov

Nodes coordinates in distributed dmplex
by Alejandro D Otero 16 Jul '22

16 Jul '22

Hi, sorry if this is an obvious question, but I cannot figure out how to recover finite element nodes coordinates once I have distributed a mesh stored as a dmplex. I am using petsc4py as interface to petsc rutines. I first created a dmplex using: dm.createFromCellList() In a sequential run I got the coordinates with: Coords = dm.getCoordinates() which gave a sequential vector with the coordinates of the mesh nodes. When I distribute the mesh with: dm.distribute() each mpi process has it own dm but the indexing of the vector resulting from getCoordinates() or getCoordinatesLocal() seems not consistent with the local numbering of the cells and nodes. Which is the correct way of doing this in PETSc philosophy? Thanks in advance, Alejandro

4 9

MatGetDiagonal
by Nachiket Gokhale 09 Dec '19

09 Dec '19

Any chance of making this work in serial? http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetDiago… Not a show stopper, I am trying to get the diagonal of some small projected, dense matrices (which come from large sparse matrices). I am running in serial because 1) Since my projected matrices are small, and 2) PETSc does not do certain matrix multiplications involving a dense matrix in parallel, -Nachiket

4 5

GAMG scaling
by Mark Adams 24 Dec '18

24 Dec '18

(Hong), what is the current state of optimizing RAP for scaling? Nate, is driving 3D elasticity problems at scaling with GAMG and we are working out performance problems. They are hitting problems at ~1.5B dof problems on a basic Cray (XC30 I think). Thanks, Mark

7 28

Problems with DMDAVecGetArrayF90 + Intel
by Randall Mackie 15 May '18

15 May '18

Dear PETSc users, I’m curious if anyone else experiences problems using DMDAVecGetArrayF90 in conjunction with Intel compilers? We have had many problems (typically 11 SEGV segmentation violations) when PETSc is compiled in optimize mode (with various combinations of options). These same codes run valgrind clean with gfortran, so I assume this is an Intel bug, but before we submit a bug report I wanted to see if anyone else had similar experiences? We have basically gone back and replaced our calls to DMDAVecGetArrayF90 with calls to VecGetArrayF90 and pass those pointers into a “local” subroutine that works fine. In case anyone is curious, the attached test code shows this behavior when PETSc is compiled with the following options: ./configure \ --with-clean=1 \ --with-debugging=0 \ --with-fortran=1 \ --with-64-bit-indices \ --download-mpich=../mpich-3.3a2.tar.gz \ --with-blas-lapack-dir=/opt/intel/mkl \ --with-cc=icc \ --with-fc=ifort \ --with-cxx=icc \ --FOPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \ --COPTFLAGS='-O2 -xSSSE3 -axcore-avx2' \ --CXXOPTFLAGS='-O2 -xSSSE3 -axcore-avx2’ \ Thanks, Randy M.

6 26

monitoring the convergence of fieldsplit 0 and 1
by Klaij, Christiaan 15 May '18

15 May '18

A question when using PCFieldSplit: for each linear iteration of the system, how many iterations for fielsplit 0 and 1? One way to find out is to run with -ksp_monitor, -fieldsplit_0_ksp_monitor and -fieldsplit_0_ksp_monitor. This gives the complete convergence history. Another way, suggested by Matt, is to use -ksp_monitor, -fieldsplit_0_ksp_converged_reason and -fieldsplit_1_ksp_converged_reason. This gives only the totals for fieldsplit 0 and 1 (but without saying for which one). Both ways require to somehow process the output, which is a bit inconvenient. Could KSPGetResidualHistory perhaps return (some) information on the subsystems' convergence for processing inside the code? Chris dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:[email protected] T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl

4 19

understanding gmres output
by Ganesh Diwan 01 May '18

01 May '18

Dear Petsc developers, I am trying to learn to use PETSc GMRES with petsc4py and also trying to follow the codes in petsc/src/ksp/ksp/examples/tutorials. Here is the Python code I use to read a linear system and solve it with GMRES in PETSc (petsc4py) # gmrestest.py # usual imports import numpy as np import scipy.io as sio import scipy from sys import getrefcount from petsc4py import PETSc # read the matrix data mat_contents = sio.loadmat('data.mat') mat_A = mat_contents['A'] vec_b = mat_contents['b'] n = mat_contents['n'] vec_x = mat_contents['x'] mat_nnzA = mat_contents['nnzA'] # form the petsc matrices x = PETSc.Vec().createWithArray(vec_x) b = PETSc.Vec().createWithArray(vec_b) p1=mat_A.indptr p2=mat_A.indices p3=mat_A.data A = PETSc.Mat().createAIJ(size=mat_A.shape,csr=(p1,p2,p3)) A = scipy.transpose(A) # transpose the csr format as my matrix is column major originally A.assemblyBegin() A.assemblyEnd() # solve ksp = PETSc.KSP() ksp.create(PETSc.COMM_WORLD) ksp.setType('gmres') ksp.setOperators(A) ksp.setFromOptions() rtol = 1e-4 ksp.setTolerances(rtol, 1e-5, 10000., 10000) ksp.view() ksp.setConvergenceHistory() ksp.solve(b, x) # print print 'iterations = ', ksp.getIterationNumber() print 'residual = ', '{:.2e}'.format(ksp.getResidualNorm())# %.2E ksp.getResidualNorm() print 'norm(b)*rtol = ', '{:.2e}'.format(PETSc.Vec.norm(b)*rtol)# %.2E PETSc.Vec.norm(b)*rtol print 'converge reason# = ', ksp.getConvergedReason() print 'residuals at each iter = ', ksp.getConvergenceHistory() # Here is the output from the above code for a linear system of dimension 100 by 100. KSP Object: 1 MPI processes type: gmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=0.0001, absolute=1e-05, divergence=10000. left preconditioning using DEFAULT norm type for convergence test PC Object: 1 MPI processes type: ilu PC has not been set up so information may be incomplete out-of-place factorization 0 levels of fill tolerance for zero pivot 2.22045e-14 matrix ordering: natural linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=100, cols=100 total: nonzeros=2704, allocated nonzeros=2704 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 25 nodes, limit used is 5 iterations = 5 residual = 1.38e-03 norm(b)*rtol = 1.99e-01 converge reason# = 2 residuals at each iter = [ 2.05677686e+01 4.97916031e+00 4.82888782e-01 1.16849581e-01 8.87159777e-03 1.37992327e-03] Sorry if this sounds stupid, but I am trying to understand the output from PETSc by contrasting it with Matlab GMRES. I see that the residual < norm(b)*rtol for the 4th iteration itself, I am not sure then why GMRES continues for one more iteration. Secondly, how does one get total number of iterations? For eg. let's say if it takes 3 outer iterations each with the default restart of 30, then one would expect the length of residual vector to be 150. In the above specific example, it took GMRES a total of 5 iterations to converge. Does it mean the convergence was achieved without having to restart? Thank you, Ganesh

3 3

Re: [petsc-users] petsc-users Digest, Vol 112, Issue 141
by Afrah Najib 30 Apr '18

30 Apr '18

This word is wrong burden ... pardon me ... I am sorry ... There is no edit here On 30 April 2018 at 22:31, <petsc-users-request(a)mcs.anl.gov> wrote: > Send petsc-users mailing list submissions to > petsc-users(a)mcs.anl.gov > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.mcs.anl.gov/mailman/listinfo/petsc-users > or, via email, send a message with subject or body 'help' to > petsc-users-request(a)mcs.anl.gov > > You can reach the person managing the list at > petsc-users-owner(a)mcs.anl.gov > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of petsc-users digest..." > > > Today's Topics: > > 1. Fwd: PDSLin hybrid solver performance with PETSc (Afrah Najib) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 30 Apr 2018 22:31:04 +0300 > From: Afrah Najib <afrah.nacib(a)gmail.com> > To: PETSc <petsc-users(a)mcs.anl.gov> > Subject: [petsc-users] Fwd: PDSLin hybrid solver performance with > PETSc > Message-ID: > <CAPo_9=D8p0fC2fzqHCeMPg2DyZU-vdxEZ-FaYiHU_od-togaHA@mail. > gmail.com> > Content-Type: text/plain; charset="utf-8" > > I beg your burden, how can I use those tools of PETSc in PDSLin program ... > > valgrind is working in such "mpirun -n $n valgrind ./dtest_pdslin > $inputFIle $matrixFile " > > but these options are not working > > ---------- Forwarded message ---------- > From: Smith, Barry F. <bsmith(a)mcs.anl.gov> > Date: 30 April 2018 at 18:39 > Subject: Re: [petsc-users] PDSLin hybrid solver performance with PETSc > To: Afrah Najib <afrah.nacib(a)gmail.com> > Cc: PETSc <petsc-users(a)mcs.anl.gov>, "Xiaoye S. Li" <xsli(a)lbl.gov>, > Ichitaro Yamazaki <iyamazak(a)icl.utk.edu>, Ichitaro Yamazaki < > ic.yamazaki(a)gmail.com> > > > > Looks like there is a divide by a variable that is zero in pdslin where > the code doesn't first check if the value is not zero. > > You can try using the option -start_in_debugger and in each debugger > window use the debugger option to catch floating point exceptions and then > type cont to continue the run. Yes this is annoying when you have so many > debugger windows but is the only simple way to determine immediately the > line of code that is producing the floating point exception. > > Barry > > > > On Apr 30, 2018, at 6:20 AM, Afrah Najib <afrah.nacib(a)gmail.com> wrote: > > > > Hi all, > > > > I am testing PDSLin hybrid solver performance with PETSc krylov method on > a SPD matrix called audik_1 matrix(format is ijv full matrix ordered by > rows) > > https://sparse.tamu.edu/GHS_psdef/audikw_1 > > > > I am also running on cluster with Intel(R) Xeon(R) CPU E5-2680 v4 @ > 2.40GHz processor and 28 core , 128 ; 512GB per node > > > > but it takes a lot of time for Schur complement factorization. > > > > Compiler Subdomains processors Precond TmatrixPartition > TmatrixDist Tfactorise TcompApproSchur TprepApproSchur > Tfactoschur Tprecond_Total Tsolve > > gcc 8 128 Sparse 23.49532 0.6009915 6.059546 > 212.8243 0.04027608 38.74885 257.6796 > 13.69609 > > gcc 8 16 Sparse 11.864 0.4385703 27.23499 > 1390.153 0.5523289 55.20706 1473.207 4.213825 > > gcc 8 32 Sparse 9.912792 0.2592737 16.79125 > 934.3571 0.2372452 49.39175 1000.813 37.2029 > > gcc 8 64 Sparse 12.61929 0.2336042 8.2316 > 632.839 0.1113939 52.21551 693.4153 13.94034 > > > > My PETSc in stallation configuration is like this: > > > > export PETSC_DIR=/X/petsc > > export PETSC_ARCH=arch-real > > > > ./configure PETSC_DIR=/X/petsc \ > > prefix=/X/petsc/petsc-install \ > > PETSC_ARCH=arch-real \ > > --with-debugging=0 \ > > --with-cc=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc \ > > --with-cxx=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicxx \ > > --with-fc=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 \ > > --with-valgrind=1 \ > > --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 \ > > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 \ > > --with-blas-lapack-dir=/Y/progs/intel/compilers_and_ > libraries_2017.0.098/linux/mkl/lib/intel64. > > > > My slurm scrpt content is like this: > > > > #!/bin/bash -l > > #SBATCH -J "Paudik16" > > #SBATCH -A X > > #SBATCH -p X > > #SBATCH --exclusive > > #SBATCH -o Paudik_8_2_16.%j.out > > #SBATCH -e Paudik_8_2_16.%j.err > > > > #SBATCH -N 8 > > #SBATCH --ntasks-per-node=2 > > #SBATCH --cpus-per-task=1 > > > > export KMP_AFFINITY=verbose,disabled > > export MKL_NUM_THREADS=${SLURM_CPUS_PER_TASK} > > export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} > > > > module purge > > > > module load gcc/7.1.0 > > module load mpi/openmpi-3.0.0-gcc-7.1.0 > > module load intel/mkl_2017.0.098 > > > > file=input.in > > matrixFile=~/dataDir/ijv/audikw_1.ijv > > > > mpirun -np $SLURM_NTASKS ./dtest_pdslin $file $matrixFile > > > > > > > > I attached an output of one file along with the input configuration of > PDSLin. > > > > Any idea of how to enhance these results. > > > > Note in PDSLin, I tried with parallel Schur permutation but I got the > following error : > > > > [1]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [1]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > > [1]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [1]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS > /X to find memory corruption errors > > [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [1]PETSC ERROR: to get more information on the crash. > > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > ------------------------------------------------------------ > -------------- > > MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD > > with errorcode 59. > > > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > > You may or may not see output from other processes, depending on > > e/Xactly when Open MPI kills them. > > ------------------------------------------------------------ > -------------- > > [13]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [13]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > > [13]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [13]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [13]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS /X to find memory corruption errors > > [13]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [13]PETSC ERROR: to get more information on the crash. > > [11]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [11]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > > [11]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [11]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [11]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS /X to find memory corruption errors > > [11]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [11]PETSC ERROR: to get more information on the crash. > > [3]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [3]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > > [3]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [3]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS > /X to find memory corruption errors > > [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [3]PETSC ERROR: to get more information on the crash. > > [3]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [3]PETSC ERROR: Signal received > > [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [3]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [3]PETSC ERROR: [9]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [9]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > > [9]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [9]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [9]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS > /X to find memory corruption errors > > [9]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [9]PETSC ERROR: to get more information on the crash. > > [7]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [7]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > > [7]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [7]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS > /X to find memory corruption errors > > [7]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [7]PETSC ERROR: to get more information on the crash. > > [5]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [5]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > > [5]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [5]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS > /X to find memory corruption errors > > [5]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [5]PETSC ERROR: to get more information on the crash. > > [5]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [5]PETSC ERROR: Signal received > > [5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [5]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [5]PETSC ERROR: [15]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [15]PETSC ERROR: Caught signal number 8 FPE: Floating Point > E/Xception,probably divide by zero > > [15]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [15]PETSC ERROR: [13]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [13]PETSC ERROR: Signal received > > [13]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [13]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [13]PETSC ERROR: ./dtest_pdslin on a named s060 by afarea Mon Apr 30 > 05:29:53 2018 > > [13]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [11]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [11]PETSC ERROR: Signal received > > [11]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [11]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [11]PETSC ERROR: ./dtest_pdslin on a named s054 by afarea Mon Apr 30 > 05:29:53 2018 > > [11]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > ./dtest_pdslin on a named s034 by afarea Mon Apr 30 05:29:53 2018 > > [3]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [3]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [9]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [9]PETSC ERROR: Signal received > > [9]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [9]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [9]PETSC ERROR: ./dtest_pdslin on a named s044 by afarea Mon Apr 30 > 05:29:53 2018 > > [9]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [7]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [7]PETSC ERROR: Signal received > > [7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [7]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [7]PETSC ERROR: ./dtest_pdslin on a named s036 by afarea Mon Apr 30 > 05:29:53 2018 > > [7]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [7]PETSC ERROR: #1 User provided function() line 0 in unknown file > > ./dtest_pdslin on a named s035 by afarea Mon Apr 30 05:29:53 2018 > > [5]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [5]PETSC ERROR: #1 User provided function() line 0 in unknown file > > configure using --with-debugging=yes, recompile, link, and run > > [15]PETSC ERROR: to get more information on the crash. > > [15]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [15]PETSC ERROR: Signal received > > [15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [15]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [15]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 > 05:29:53 2018 > > [13]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [11]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [9]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [15]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [15]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [1]PETSC ERROR: Signal received > > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [1]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [1]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 > 05:29:53 2018 > > [1]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [1]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [1]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 > 05:29:53 2018 > > [1]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [6]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [6]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > > [6]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [6]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS > /X to find memory corruption errors > > [6]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [6]PETSC ERROR: to get more information on the crash. > > [6]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [6]PETSC ERROR: Signal received > > [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [6]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [6]PETSC ERROR: ./dtest_pdslin on a named s036 by afarea Mon Apr 30 > 05:29:53 2018 > > [6]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [6]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [4]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [4]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > > [4]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [4]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS > /X to find memory corruption errors > > [4]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [4]PETSC ERROR: to get more information on the crash. > > [4]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [4]PETSC ERROR: Signal received > > [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [4]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [4]PETSC ERROR: ./dtest_pdslin on a named s035 by afarea Mon Apr 30 > 05:29:53 2018 > > [4]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [4]PETSC ERROR: ./dtest_pdslin on a named s035 by afarea Mon Apr 30 > 05:29:53 2018 > > [4]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [4]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [0]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > > [0]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [0]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS > /X to find memory corruption errors > > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [0]PETSC ERROR: to get more information on the crash. > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Signal received > > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [0]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 > 05:29:53 2018 > > [0]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [10]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [10]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > > [10]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [10]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [10]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS /X to find memory corruption errors > > [10]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [10]PETSC ERROR: to get more information on the crash. > > [10]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [10]PETSC ERROR: Signal received > > [10]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [10]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [10]PETSC ERROR: ./dtest_pdslin on a named s054 by afarea Mon Apr 30 > 05:29:53 2018 > > > > [8]PETSC ERROR: ./dtest_pdslin on a named s044 by afarea Mon Apr 30 > 05:29:53 2018 > > [8]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [8]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [14]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > > [14]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [14]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [14]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS /X to find memory corruption errors > > [14]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [14]PETSC ERROR: to get more information on the crash. > > [14]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [14]PETSC ERROR: Signal received > > [14]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [14]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [14]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 > 05:29:53 2018 > > [14]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [14]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [15]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [15]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > > [15]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [15]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS /X to find memory corruption errors > > [15]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [15]PETSC ERROR: to get more information on the crash. > > [15]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [15]PETSC ERROR: Signal received > > [15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [15]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [15]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 > 05:29:53 2018 > > [15]PETSC ERROR: #2 User provided function() line 0 in unknown file > > [2]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > > [2]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [2]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS > /X to find memory corruption errors > > [2]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [2]PETSC ERROR: to get more information on the crash. > > [2]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [2]PETSC ERROR: Signal received > > [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [2]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [2]PETSC ERROR: ./dtest_pdslin on a named s034 by afarea Mon Apr 30 > 05:29:53 2018 > > [2]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [2]PETSC ERROR: #1 User provided function() line 0 in unknown file > > [3]PETSC ERROR: ------------------------------ > ------------------------------------------ > > [3]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > > [3]PETSC ERROR: Try option -start_in_debugger or > -on_error_attach_debugger > > [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > > [3]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac > OS > /X to find memory corruption errors > > [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, > and run > > [3]PETSC ERROR: to get more information on the crash. > > [3]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [3]PETSC ERROR: Signal received > > [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [3]PETSC ERROR: Petsc Release Version 3.9.0, unknown > > [3]PETSC ERROR: ./dtest_pdslin on a named s034 by afarea Mon Apr 30 > 05:29:53 2018 > > [3]PETSC ERROR: Configure options PETSC_DIR=/X/petsc > prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 > --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc > --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X > --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 > --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 > --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/ > okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/ > X/mkl/lib/intel64 > > [3]PETSC ERROR: #2 User provided function() line 0 in unknown file > > [f002:38724] 17 more processes have sent help message help-mpi-api.t/Xt / > mpi-abort > > [f002:38724] Set MCA parameter "orte_base_help_aggregate" to 0 to see all > help / error messages > > > > > > <Paudik_8_2_16.100234.out> >

1 0

PDSLin hybrid solver performance with PETSc
by Afrah Najib 30 Apr '18

30 Apr '18

Hi all, I am testing PDSLin hybrid solver performance with PETSc krylov method on a SPD matrix called audik_1 matrix(format is ijv full matrix ordered by rows) https://sparse.tamu.edu/GHS_psdef/audikw_1 I am also running on cluster with Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz processor and 28 core , 128 ; 512GB per node but it takes a lot of time for Schur complement factorization. Compiler Subdomains processors Precond TmatrixPartition TmatrixDist Tfactorise TcompApproSchur TprepApproSchur Tfactoschur Tprecond_Total Tsolve gcc 8 128 Sparse 23.49532 0.6009915 6.059546 212.8243 0.04027608 38.74885 257.6796 13.69609 gcc 8 16 Sparse 11.864 0.4385703 27.23499 1390.153 0.5523289 55.20706 1473.207 4.213825 gcc 8 32 Sparse 9.912792 0.2592737 16.79125 934.3571 0.2372452 49.39175 1000.813 37.2029 gcc 8 64 Sparse 12.61929 0.2336042 8.2316 632.839 0.1113939 52.21551 693.4153 13.94034 My PETSc in stallation configuration is like this: export PETSC_DIR=/X/petsc export PETSC_ARCH=arch-real ./configure PETSC_DIR=/X/petsc \ prefix=/X/petsc/petsc-install \ PETSC_ARCH=arch-real \ --with-debugging=0 \ --with-cc=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc \ --with-cxx=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicxx \ --with-fc=/Y/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 \ --with-valgrind=1 \ --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 \ --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 \ --with-blas-lapack-dir=/Y/progs/intel/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64. *My slurm scrpt content is like this:* #!/bin/bash -l #SBATCH -J "Paudik16" #SBATCH -A X #SBATCH -p X #SBATCH --exclusive #SBATCH -o Paudik_8_2_16.%j.out #SBATCH -e Paudik_8_2_16.%j.err #SBATCH -N 8 #SBATCH --ntasks-per-node=2 #SBATCH --cpus-per-task=1 export KMP_AFFINITY=verbose,disabled export MKL_NUM_THREADS=${SLURM_CPUS_PER_TASK} export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} module purge module load gcc/7.1.0 module load mpi/openmpi-3.0.0-gcc-7.1.0 module load intel/mkl_2017.0.098 file=input.in matrixFile=~/dataDir/ijv/audikw_1.ijv mpirun -np $SLURM_NTASKS ./dtest_pdslin $file $matrixFile *I attached an output of one file along with the input configuration of PDSLin.* Any idea of how to enhance these results. Note in PDSLin, I tried with parallel Schur permutation but I got the following error : [1]PETSC ERROR: ------------------------------------------------------------------------ [1]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [1]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [1]PETSC ERROR: to get more information on the crash. [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD with errorcode 59. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on e/Xactly when Open MPI kills them. -------------------------------------------------------------------------- [13]PETSC ERROR: ------------------------------------------------------------------------ [13]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero [13]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [13]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [13]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [13]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [13]PETSC ERROR: to get more information on the crash. [11]PETSC ERROR: ------------------------------------------------------------------------ [11]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero [11]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [11]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [11]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [11]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [11]PETSC ERROR: to get more information on the crash. [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [3]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [3]PETSC ERROR: to get more information on the crash. [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [3]PETSC ERROR: Signal received [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [3]PETSC ERROR: Petsc Release Version 3.9.0, unknown [3]PETSC ERROR: [9]PETSC ERROR: ------------------------------------------------------------------------ [9]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero [9]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [9]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [9]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [9]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [9]PETSC ERROR: to get more information on the crash. [7]PETSC ERROR: ------------------------------------------------------------------------ [7]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero [7]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [7]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [7]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [7]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [7]PETSC ERROR: to get more information on the crash. [5]PETSC ERROR: ------------------------------------------------------------------------ [5]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero [5]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [5]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [5]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [5]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [5]PETSC ERROR: to get more information on the crash. [5]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [5]PETSC ERROR: Signal received [5]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [5]PETSC ERROR: Petsc Release Version 3.9.0, unknown [5]PETSC ERROR: [15]PETSC ERROR: ------------------------------------------------------------------------ [15]PETSC ERROR: Caught signal number 8 FPE: Floating Point E/Xception,probably divide by zero [15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [15]PETSC ERROR: [13]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [13]PETSC ERROR: Signal received [13]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [13]PETSC ERROR: Petsc Release Version 3.9.0, unknown [13]PETSC ERROR: ./dtest_pdslin on a named s060 by afarea Mon Apr 30 05:29:53 2018 [13]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [11]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [11]PETSC ERROR: Signal received [11]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [11]PETSC ERROR: Petsc Release Version 3.9.0, unknown [11]PETSC ERROR: ./dtest_pdslin on a named s054 by afarea Mon Apr 30 05:29:53 2018 [11]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 ./dtest_pdslin on a named s034 by afarea Mon Apr 30 05:29:53 2018 [3]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [3]PETSC ERROR: #1 User provided function() line 0 in unknown file [9]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [9]PETSC ERROR: Signal received [9]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [9]PETSC ERROR: Petsc Release Version 3.9.0, unknown [9]PETSC ERROR: ./dtest_pdslin on a named s044 by afarea Mon Apr 30 05:29:53 2018 [9]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [7]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [7]PETSC ERROR: Signal received [7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [7]PETSC ERROR: Petsc Release Version 3.9.0, unknown [7]PETSC ERROR: ./dtest_pdslin on a named s036 by afarea Mon Apr 30 05:29:53 2018 [7]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [7]PETSC ERROR: #1 User provided function() line 0 in unknown file ./dtest_pdslin on a named s035 by afarea Mon Apr 30 05:29:53 2018 [5]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [5]PETSC ERROR: #1 User provided function() line 0 in unknown file configure using --with-debugging=yes, recompile, link, and run [15]PETSC ERROR: to get more information on the crash. [15]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [15]PETSC ERROR: Signal received [15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [15]PETSC ERROR: Petsc Release Version 3.9.0, unknown [15]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 05:29:53 2018 [13]PETSC ERROR: #1 User provided function() line 0 in unknown file [11]PETSC ERROR: #1 User provided function() line 0 in unknown file [9]PETSC ERROR: #1 User provided function() line 0 in unknown file [15]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [15]PETSC ERROR: #1 User provided function() line 0 in unknown file [1]PETSC ERROR: Signal received [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.9.0, unknown [1]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 05:29:53 2018 [1]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [1]PETSC ERROR: #1 User provided function() line 0 in unknown file [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.9.0, unknown [1]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 05:29:53 2018 [1]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [1]PETSC ERROR: #1 User provided function() line 0 in unknown file [6]PETSC ERROR: ------------------------------------------------------------------------ [6]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [6]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [6]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [6]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [6]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [6]PETSC ERROR: to get more information on the crash. [6]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [6]PETSC ERROR: Signal received [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [6]PETSC ERROR: Petsc Release Version 3.9.0, unknown [6]PETSC ERROR: ./dtest_pdslin on a named s036 by afarea Mon Apr 30 05:29:53 2018 [6]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [6]PETSC ERROR: #1 User provided function() line 0 in unknown file [4]PETSC ERROR: ------------------------------------------------------------------------ [4]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [4]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [4]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [4]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [4]PETSC ERROR: to get more information on the crash. [4]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [4]PETSC ERROR: Signal received [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [4]PETSC ERROR: Petsc Release Version 3.9.0, unknown [4]PETSC ERROR: ./dtest_pdslin on a named s035 by afarea Mon Apr 30 05:29:53 2018 [4]PETSC ERROR: Petsc Release Version 3.9.0, unknown [4]PETSC ERROR: ./dtest_pdslin on a named s035 by afarea Mon Apr 30 05:29:53 2018 [4]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [4]PETSC ERROR: #1 User provided function() line 0 in unknown file [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.9.0, unknown [0]PETSC ERROR: ./dtest_pdslin on a named f002 by afarea Mon Apr 30 05:29:53 2018 [0]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [0]PETSC ERROR: #1 User provided function() line 0 in unknown file [10]PETSC ERROR: ------------------------------------------------------------------------ [10]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [10]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [10]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [10]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [10]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [10]PETSC ERROR: to get more information on the crash. [10]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [10]PETSC ERROR: Signal received [10]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [10]PETSC ERROR: Petsc Release Version 3.9.0, unknown [10]PETSC ERROR: ./dtest_pdslin on a named s054 by afarea Mon Apr 30 05:29:53 2018 [8]PETSC ERROR: ./dtest_pdslin on a named s044 by afarea Mon Apr 30 05:29:53 2018 [8]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [8]PETSC ERROR: #1 User provided function() line 0 in unknown file [14]PETSC ERROR: ------------------------------------------------------------------------ [14]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [14]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [14]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [14]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [14]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [14]PETSC ERROR: to get more information on the crash. [14]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [14]PETSC ERROR: Signal received [14]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [14]PETSC ERROR: Petsc Release Version 3.9.0, unknown [14]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 05:29:53 2018 [14]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [14]PETSC ERROR: #1 User provided function() line 0 in unknown file [15]PETSC ERROR: ------------------------------------------------------------------------ [15]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [15]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [15]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [15]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [15]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [15]PETSC ERROR: to get more information on the crash. [15]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [15]PETSC ERROR: Signal received [15]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [15]PETSC ERROR: Petsc Release Version 3.9.0, unknown [15]PETSC ERROR: ./dtest_pdslin on a named s071 by afarea Mon Apr 30 05:29:53 2018 [15]PETSC ERROR: #2 User provided function() line 0 in unknown file [2]PETSC ERROR: ------------------------------------------------------------------------ [2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [2]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [2]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [2]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [2]PETSC ERROR: to get more information on the crash. [2]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [2]PETSC ERROR: Signal received [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [2]PETSC ERROR: Petsc Release Version 3.9.0, unknown [2]PETSC ERROR: ./dtest_pdslin on a named s034 by afarea Mon Apr 30 05:29:53 2018 [2]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [2]PETSC ERROR: #1 User provided function() line 0 in unknown file [3]PETSC ERROR: ------------------------------------------------------------------------ [3]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [3]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [3]PETSC ERROR: or try http://valgrind.org on GNU/linu/X and Apple Mac OS /X to find memory corruption errors [3]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [3]PETSC ERROR: to get more information on the crash. [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [3]PETSC ERROR: Signal received [3]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [3]PETSC ERROR: Petsc Release Version 3.9.0, unknown [3]PETSC ERROR: ./dtest_pdslin on a named s034 by afarea Mon Apr 30 05:29:53 2018 [3]PETSC ERROR: Configure options PETSC_DIR=/X/petsc prefi/X=/X/petsc/petsc-install PETSC_ARCH=arch-real --with-debugging=0 --with-cc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpicc --with-c/X/X=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpic/X/X --with-fc=/okyanus/progs/openmpi/openmpi-3.0.0-gcc-7.1.0/bin/mpif90 --with-valgrind=1 --with-valgrind-dir=/X/root/usr/local/valgrind-3.13.0 --with-hwloc-dir=/X/root/usr/local/hwloc-1.11.9 --with-blaslapack-dir=/okyanus/progs/intel/compilers_and_libraries_2017.0.098/linu/X/mkl/lib/intel64 [3]PETSC ERROR: #2 User provided function() line 0 in unknown file [f002:38724] 17 more processes have sent help message help-mpi-api.t/Xt / mpi-abort [f002:38724] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

3 3

Finite difference Jacobian in TS context
by Oleksandr Koshkarov 30 Apr '18

30 Apr '18

Hello All, I hope someone can help :) I think I am doing something wrong, but cannot understand what. I have a huge time dependent system with 3d DMDA data structure and I am evolving it with explicit Runge-Kutta by using TS and basically only using "TSSetRHSFunction". Now I want to repeat it with implicit time stepper (for now Crank-Nicolson) and I am trying to provide finite difference Jacobian and I am failing miserably. I also cannot find appropriate example in PETSc tutorial (if you can point me to working example, it would be great). Here is my best attempt (what wrong with it?): DMDACreate3d(PETSC_COMM_WORLD, DM_BOUNDARY_PERIODIC, DM_BOUNDARY_PERIODIC, DM_BOUNDARY_PERIODIC, DMDA_STENCIL_STAR, NX, NY, NZ, PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE, 2*3+NC*NS, 1, NULL, NULL, NULL, &da); DMSetUp(da); DMCreateGlobalVector(da,&x); TSCreate(PETSC_COMM_WORLD,&ts); TSSetProblemType(ts,TS_NONLINEAR); TSSetRHSFunction(ts,NULL,compute_RHS,NULL); TSSetMaxTime(ts,T_FINAL); TSSetExactFinalTime(ts,TS_EXACTFINALTIME_STEPOVER); TSSetDM(ts,da); TSSetType(ts,TSCN); //it works with: TSSetType(ts,TSRK); set_IC(da,x); TSSetTimeStep(ts,DT); TSSetSolution(ts,x); TSGetSNES(ts,&snes); SNESGetKSP(snes,&ksp); KSPGetPC(ksp,&pc); PCSetType(pc,PCNONE); DMSetMatType(da,MATAIJ); DMCreateMatrix(da,&J); ISColoring iscoloring; MatFDColoring matfdcoloring; DMCreateColoring(da,IS_COLORING_GLOBAL,&iscoloring); MatFDColoringCreate(J,iscoloring,&matfdcoloring); MatFDColoringSetType(matfdcoloring,MATMFFD_DS); // I think I do something wrong in the following 3 lines PetscErrorCode (*temp_f)(SNES,Vec,Vec,void*); SNESGetFunction(snes,NULL,&temp_f,NULL); MatFDColoringSetFunction(matfdcoloring,(PetscErrorCode (*)(void))temp_f,NULL); MatFDColoringSetUp(J,iscoloring,matfdcoloring); SNESSetJacobian(snes,J,J,SNESComputeJacobianDefaultColor,matfdcoloring); ISColoringDestroy(&iscoloring); TSSolve(ts,x); Thank you, Oleksandr Koshkarov.

3 8

Re: [petsc-users] config error with hypre on Summit
by Richard Tran Mills 29 Apr '18

29 Apr '18

Mark, I asked OLCF if I could get access to Summit so that I could ensure that the latest PETSc builds and works on there and was told "NO". If you are associated with an allocation on there, though, then I could get on and poke around if the PI is willing to approve me on their project. (I already have an OLCF account, just no access to pre-production Summit as I'm not associated with an allocation on there.) Should we ask the PI of your allocation to get me an account on there? Pat: I see you cc'ed on this message thread. You are doing a terrible job of being "retired"! =) Cheers, Richard On Sun, Apr 29, 2018 at 11:24 AM, Matthew Knepley <knepley(a)gmail.com> wrote: > On Sun, Apr 29, 2018 at 11:38 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > >> I'm getting an error configure hypre on the new IBM at ORNL, Summit. >> > > I can spot your error right away. You are trying to build Hypre ;) > > It looks like autoconf is failing for this system, since the config.guess > script is old. This is > one for the sysadmin I think. > > Thanks, > > Matt > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/> >

1 0