February 2008 - petsc-users - mailman3.cels.anl.gov

Some basic questions about PETSC
by Ben Tay 19 Jan '11

19 Jan '11

Hi, I'm still relatively new to PETSC but I've finally managed to solve my own poisson eqn using it in about a week's time.Thank you for writing such a good solver. However, I have some questions still about PETSC: 1. Are petscscaler the same as real nos.? I am writing in double precision or real(8). So are these 2 interchangeable? (same as petscint and integer). 2. while linking in compaq visual fortran, I get the warning message about conflict library with LIBCMT. Is it important or can I just ignore it? 3. I am writing in f90 free form format. When I modify from fixed f77 format to f90, it gives me a lot of error, which seems to be the include file error. So, is it possible to write in f90 format? 4. I am writing a Navier stokes solver and I'm using PETSC to solve the poisson eqn. If only the RHS changes with each time step, do I need to call VecCreateSeq <insert values in RHS b> VecAssemblyBegin VecAssemblyEnd VecDestroy at each time step? I don't think I can (or I should) change values of b after VecAssemblyBegin,VecAssemblyEnd, or can I? Thank you very much and have a nice day Regards, Ben

4 8

Non repeatability issue
by Aldo Bonfiglioli 02 Apr '08

02 Apr '08

Dear users, in the notes attached here we address an issue concerning nonrepeatability. We know this is a known issue in parallel floating point programs, but we would like to be sure we are interpreting things correctly a not mis-using PETSc. Comments are welcome. Regards, Aldo -- Dr. Aldo Bonfiglioli Dip.to di Ingegneria e Fisica dell'Ambiente (DIFA) Universita' della Basilicata V.le dell'Ateneo lucano, 10 85100 Potenza ITALY tel:+39.0971.205203 fax:+39.0971.205160 Dear all, I have recently observed a non-repeatibility issue with my unstructured CFD code. I solve the steady flow eqns. using pseudo-timestepping. PETSc is used to solve the large sparse linear systems arising at each psudo-time-step. Once close to the steady state solution, the time step size is increased to infinity so that Newton's method is recovered. Here below is a sample 2D calculation performed on 4 procs of a linux Beowulf cluster (but we also tested on a 4 procs ES40). The first two columns respectively show the time-step counter (i.e. the nonlinear iteration counter) and the KSP iterations required to solve the linear system. Cols. 5 - 8 display the nodal residuals (for mass, energy, etc.) In the early stages of the (non-linear) iterative process all residuals are identical among different runs. Once we get close to machine accuracy (iterations 48-50 in the example below), the residuals differ as do the KSP iterations for convergence. We believe to have identified the source of the differences in the following: At each step we solve a linear sistem of the form A x = rhs. Close to steady state, both x and rhs are close to machine eps. This is because rhs is the nodal residual which converges to machine eps as steady state is reached and x is the nodal increment (also converging to machine eps at steady state). As we look at the rhs (for instance) computed on each processor (that is the processor owned elements of a ghosted rhs Vec), this is identical between different runs. Once we call GhostUpdateBegin/End this is not any more true, although differences are of the order of machine eps. Our explanation is the following. The grid is partitioned so that the triangular cells belong only to a processor, but gridpoints (where the variables are stored) are shared among adjacent subdomains/procs. The Rhs is built in a FE manner by assembling contributions from surrounding cells. Given a gridpoint i shared by various subdomains (that is lying on an interdomain boundary), (*) Rhs_i = Rhs_i^p1 + Rhs_i^p2 + Rhs_i^p3 where the various contributions p1,p2, etc. are due to group of cells (triangles) belonging to different processors p1,p2,etc. At steady state Rhs_i must be of order machine eps, however some of the contributions will be of larger order of magnitude but with different sign (in order to cancel out) while other contributions could again be of order machine eps. E.g. Rhs_i^p1 = - Rhs_i^p2 and Rhs_i^p3 order machine eps . In this situation the result of the summation (*) will depend on the order by which the various terms on the right hand side are summed up. This is what we observe in practise on more than 2 procs.. When using two procs. there are only two contributions in the summation (*) and the convergence history is repeatable. Now, although this may look as a minor issue and in fact in the example below (inviscid flow past a 2D profile) as well as in many others all runs converge towards "steady state" we experienced the fact that on larger 3D simulations when we solve the RANS eqns using Newton's algorithm, the convergence history is run-dependent and different runs may behave fairly differently. An example is attached below: here we use ASM + ILU(1) + GMRES(60). Differences start to show up at the fourth non-linear iteration and run1 will eventually "converge" by reducing the non-linear residuals below 1.e-10, while the residuals of run2 stagnate. INVISCID 2D run Newton KSP +------- Non-linear residuals ------------+ step its run1 1 19 0.1516E+02 0.1642E+02 0.9616E-02 0.7013E-01 0.1676E-01 0.1564E-01 0.3000E+02 2 35 0.1186E+01 0.2187E+02 0.3471E-02 0.1860E-01 0.8738E-02 0.7425E-02 0.8312E+02 47 89 0.3674E+01 0.4492E+03 0.4005E-13 0.2490E-12 0.8549E-13 0.7615E-13 0.7203E+13 48 89 0.4874E+01 0.4560E+03 0.2639E-13 0.1710E-12 0.5629E-13 0.4947E-13 0.1093E+14 49 89 0.9156E+01 0.4671E+03 0.1841E-13 0.1334E-12 0.4205E-13 0.3460E-13 0.1567E+14 50 105 0.9106E+01 0.4780E+03 0.1485E-13 0.1138E-12 0.3245E-13 0.2475E-13 0.1942E+14 run2 1 19 0.6034E+01 0.7502E+01 0.9616E-02 0.7013E-01 0.1676E-01 0.1564E-01 0.3000E+02 2 35 0.2107E+01 0.1126E+02 0.3471E-02 0.1860E-01 0.8738E-02 0.7425E-02 0.8312E+02 47 89 0.1152E+02 0.4406E+03 0.3978E-13 0.2469E-12 0.8471E-13 0.7610E-13 0.7253E+13 48 89 0.5312E+01 0.4468E+03 0.2585E-13 0.1728E-12 0.5808E-13 0.5017E-13 0.1116E+14 49 91 0.7624E+01 0.4560E+03 0.1839E-13 0.1333E-12 0.4204E-13 0.3432E-13 0.1569E+14 50 106 0.5025E+01 0.4620E+03 0.1468E-13 0.1074E-12 0.3237E-13 0.2618E-13 0.1965E+14 run3 1 19 0.7433E+01 0.9449E+01 0.9616E-02 0.7013E-01 0.1676E-01 0.1564E-01 0.3000E+02 2 35 0.1460E+01 0.1495E+02 0.3471E-02 0.1860E-01 0.8738E-02 0.7425E-02 0.8312E+02 47 89 0.9440E+01 0.4151E+03 0.3962E-13 0.2455E-12 0.8454E-13 0.7573E-13 0.7280E+13 48 89 0.1344E+02 0.4306E+03 0.2564E-13 0.1704E-12 0.5667E-13 0.5045E-13 0.1125E+14 49 94 0.8316E+01 0.4406E+03 0.1748E-13 0.1274E-12 0.4100E-13 0.3388E-13 0.1651E+14 50 104 0.4968E+01 0.4469E+03 0.1539E-13 0.1098E-12 0.3231E-13 0.2516E-13 0.1874E+14 run4 1 19 0.2243E+01 0.3578E+01 0.9616E-02 0.7013E-01 0.1676E-01 0.1564E-01 0.3000E+02 2 35 0.4621E+01 0.2723E+02 0.3471E-02 0.1860E-01 0.8738E-02 0.7425E-02 0.8312E+02 47 89 0.7702E+01 0.4093E+03 0.3982E-13 0.2487E-12 0.8516E-13 0.7588E-13 0.7244E+13 48 89 0.8366E+01 0.4188E+03 0.2588E-13 0.1739E-12 0.5760E-13 0.4989E-13 0.1114E+14 49 93 0.1078E+02 0.4315E+03 0.1808E-13 0.1248E-12 0.3912E-13 0.3438E-13 0.1596E+14 50 106 0.3265E+01 0.4363E+03 0.1549E-13 0.1135E-12 0.3467E-13 0.2506E-13 0.1863E+14 run5 1 19 0.8703E+01 0.1049E+02 0.9616E-02 0.7013E-01 0.1676E-01 0.1564E-01 0.3000E+02 2 35 0.1402E+01 0.1512E+02 0.3471E-02 0.1860E-01 0.8738E-02 0.7425E-02 0.8312E+02 47 89 0.9737E+01 0.4423E+03 0.3995E-13 0.2498E-12 0.8584E-13 0.7608E-13 0.7221E+13 48 89 0.3896E+01 0.4484E+03 0.2608E-13 0.1717E-12 0.5689E-13 0.4955E-13 0.1106E+14 49 90 0.2708E+01 0.4525E+03 0.1841E-13 0.1318E-12 0.4075E-13 0.3397E-13 0.1567E+14 50 104 0.8198E+01 0.4622E+03 0.1507E-13 0.1137E-12 0.3450E-13 0.2644E-13 0.1914E+14 3D RANS (on 14 procs.) +--------------- residuals -------------------------- + run1 1 300 0.5524E+02 0.7469E+02 0.3694E-07 0.5036E-07 0.2536E-07 0.6433E-07 0.3260E-06 0.5000E+02 2 8 0.2160E+01 0.9547E+02 0.2600E-06 0.3789E-07 0.1731E-06 0.2833E-06 0.3886E-03 0.7102E+01 3 10 0.2455E+01 0.1165E+03 0.1314E-06 0.7531E-07 0.1525E-06 0.2746E-06 0.7369E-04 0.1405E+02 4 16 0.3338E+01 0.1385E+03 0.9384E-07 0.4916E-07 0.1621E-06 0.1764E-06 0.4438E-04 0.1968E+02 5 18 0.3649E+01 0.1608E+03 0.3103E-07 0.3694E-07 0.1028E-06 0.9826E-07 0.2768E-04 0.5952E+02 ... 38 31 0.5803E+01 0.9777E+03 0.7378E-08 0.1099E-07 0.7864E-08 0.8981E-08 0.1958E-04 0.2503E+03 39 53 0.1009E+02 0.1006E+04 0.3250E-08 0.9071E-08 0.2718E-08 0.5060E-08 0.8113E-05 0.5683E+03 40 47 0.8849E+01 0.1034E+04 0.1351E-08 0.6064E-08 0.2013E-08 0.3336E-08 0.4884E-05 0.1367E+04 41 47 0.8856E+01 0.1061E+04 0.3722E-08 0.3737E-08 0.2168E-08 0.4815E-08 0.5811E-05 0.4963E+03 42 31 0.5822E+01 0.1086E+04 0.7197E-08 0.5195E-08 0.1029E-07 0.3716E-08 0.1042E-05 0.2566E+03 43 47 0.8854E+01 0.1113E+04 0.3824E-08 0.1100E-07 0.1788E-08 0.4469E-08 0.5128E-05 0.4830E+03 44 60 0.1162E+02 0.1143E+04 0.2248E-08 0.5162E-08 0.2033E-08 0.2756E-08 0.8732E-06 0.8216E+03 45 115 0.2117E+02 0.1183E+04 0.1570E-08 0.2492E-08 0.9909E-09 0.3213E-08 0.1739E-05 0.1176E+04 46 226 0.4071E+02 0.1243E+04 0.2806E-09 0.1260E-08 0.1742E-09 0.4544E-09 0.5404E-06 0.6582E+04 47 515 0.9198E+02 0.1353E+04 0.4936E-10 0.5056E-09 0.8438E-10 0.3446E-10 0.5972E-07 0.3742E+05 run2 1 300 0.5465E+02 0.7417E+02 0.3694E-07 0.5036E-07 0.2536E-07 0.6433E-07 0.3260E-06 0.5000E+02 2 8 0.2154E+01 0.9502E+02 0.2601E-06 0.3789E-07 0.1731E-06 0.2833E-06 0.3886E-03 0.7102E+01 3 10 0.2449E+01 0.1162E+03 0.1314E-06 0.7530E-07 0.1526E-06 0.2746E-06 0.7369E-04 0.1406E+02 4 18 0.3641E+01 0.1386E+03 0.9381E-07 0.4916E-07 0.1621E-06 0.1764E-06 0.4437E-04 0.1969E+02 5 18 0.3646E+01 0.1610E+03 0.3104E-07 0.3693E-07 0.1028E-06 0.9825E-07 0.2769E-04 0.5950E+02 ... 46 50 0.9383E+01 0.1100E+04 0.4251E-08 0.6184E-07 0.1024E-07 0.9866E-08 0.2737E-04 0.4344E+03 47 43 0.7984E+01 0.1127E+04 0.5129E-08 0.5265E-07 0.5989E-08 0.4442E-08 0.1972E-04 0.3601E+03 48 8 0.2164E+01 0.1148E+04 0.7553E-07 0.5328E-07 0.2634E-07 0.1430E-06 0.9913E-04 0.2445E+02 49 21 0.4108E+01 0.1171E+04 0.1561E-07 0.5173E-07 0.4150E-07 0.6183E-07 0.1828E-04 0.1183E+03 50 35 0.6471E+01 0.1196E+04 0.6973E-08 0.2463E-07 0.1185E-07 0.1466E-07 0.1397E-04 0.2649E+03 51 39 0.7233E+01 0.1222E+04 0.4899E-08 0.1581E-07 0.4077E-08 0.6338E-08 0.1131E-04 0.3770E+03 52 48 0.8969E+01 0.1249E+04 0.3529E-08 0.1495E-07 0.4077E-08 0.3640E-08 0.8881E-05 0.5234E+03 53 42 0.7789E+01 0.1276E+04 0.4898E-08 0.9435E-08 0.2343E-08 0.4393E-08 0.4517E-05 0.3771E+03 54 43 0.7984E+01 0.1303E+04 0.4259E-08 0.8999E-08 0.2747E-08 0.7004E-08 0.3261E-05 0.4337E+03 55 17 0.3496E+01 0.1325E+04 0.5685E-08 0.5167E-08 0.7471E-08 0.7705E-08 0.3686E-05 0.3249E+03

4 14

HYPRE problem
by Gideon Simpson 29 Mar '08

29 Mar '08

While building petsc with MPICH, UMFPACK, and HYPRE, and without fortran, I get the following error: TESTING: configureLibrary from PETSc.packages.hypre(python/PETSc/ packages/hypre.py:120) using: --with- MPI-include="/opt/petsc-2.3.3-p8/externalpackages/mpich2-1.0.5p4/ darwin9.1.0-cxx-debug/include" = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = ========================= Configuring hypre; this may take several minutes = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = ********************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): --------------------------------------------------------------------------------------- Error running configure on HYPRE: Could not execute 'cd /opt/ petsc-2.3.3-p8/externalpackages/hypre-2.0.0/src;make distclean;./ configure --prefix=/opt/petsc-2.3.3-p8/externalpackages/hypre-2.0.0/ darwin9.1.0-cxx-debug CC="/opt/petsc-2.3.3-p8/externalpackages/ mpich2-1.0.5p4/darwin9.1.0-cxx-debug/bin/mpicc -PIC -Wall -Wwrite- strings -Wno-long-double -Wno-strict-aliasing -g3 " CXX="/opt/ petsc-2.3.3-p8/externalpackages/mpich2-1.0.5p4/darwin9.1.0-cxx-debug/ bin/mpicxx -Wall -Wwrite-strings -Wno-long-double -Wno-strict- aliasing -g -PIC " --with-MPI-include="/opt/petsc-2.3.3-p8/ externalpackages/mpich2-1.0.5p4/darwin9.1.0-cxx-debug/include" --with- MPI-lib-dirs="" --with-MPI-libs="" --with-blas-libs= --with-blas-lib- dir= --with-lapack-libs= --with-lapack-lib-dir= --with-blas=yes --with- lapack=yes --without-babel --without-mli --without-fei --without- superlu': Dist-cleaning utilities ... rm -f *.o libHYPRE* f2c.h *blas.h *lapack.h rm -rf pchdir tca.map *inslog* Dist-cleaning krylov ... rm -f *.o libHYPRE* rm -rf pchdir tca.map *inslog* Dist-cleaning struct_mv ... rm -f *.o libHYPRE* rm -rf pchdir tca.map *inslog* Dist-cleaning struct_ls ... rm -f *.o libHYPRE* rm -rf pchdir tca.map *inslog* Dist-cleaning sstruct_mv ... rm -f *.o libHYPRE* rm -rf pchdir tca.map *inslog* Dist-cleaning sstruct_ls ... rm -f *.o libHYPRE* rm -rf pchdir tca.map *inslog* Dist-cleaning seq_mv ... rm -f *.o libHYPRE* rm -rf pchdir tca.map *inslog* Dist-cleaning parcsr_mv ... rm -f *.o libHYPRE* rm -f driver driver_matvec driver_matmul rm -rf pchdir tca.map *inslog* Dist-cleaning parcsr_block_mv ... rm -rf *.o libHYPRE_* rm -f driver driver_matvec driver_matmul Dist-cleaning distributed_matrix ... rm -f *.o libHYPRE* rm -rf pchdir tca.map *inslog* Dist-cleaning matrix_matrix ... rm -f *.o lib* rm -rf pchdir tca.map *inslog* Dist-cleaning IJ_mv ... rm -f *.o libHYPRE* rm -rf pchdir tca.map *inslog* Dist-cleaning distributed_ls ... rm -f *.o libHYPRE* rm -rf pchdir tca.map *inslog* rm -f *.o libHYPRE* rm -rf pchdir tca.map *inslog* rm -f *.o libHYPRE* rm -rf pchdir tca.map *inslog* Dist-cleaning parcsr_ls ... rm -f *.o libHYPRE* rm -rf pchdir tca.map *inslog* Dist-cleaning multivector ... rm -f *.o libHYPRE* rm -rf pchdir tca.map *inslog* Dist-cleaning lib ... rm -f *.o libHYPRE* rm -rf pchdir tca.map *inslog* Dist-cleaning test ... rm -f *.o rm -rf pchdir tca.map *inslog* rm -f ij ij_mv new_ij sstruct sstruct_fac struct maxwell_unscaled ams_driver rm -f ex1_for ex3_for ex5_for ex6_for ex7_for rm -f f77_ij f77_ij_mv f77_struct rm -f struct_b sstruct_b ij_b rm -f rm -f f77_ij_b f77_struct_b rm -f fei rm -f TEST_examples/*.out* rm -f TEST_examples/*.err* rm -f TEST_fac/*.out* rm -f TEST_fac/*.err* rm -f TEST_ij/*.out* rm -f TEST_ij/*.err* rm -f TEST_ij/*.txt* rm -f TEST_ij/vectors.* rm -f TEST_sstruct/*.out* rm -f TEST_sstruct/*.err* rm -f TEST_struct/*.out* rm -f TEST_struct/*.err* rm -f ex1 ex2 ex3 ex4 ex5 ex5b ex5b77 ex6 ex7 ex8 ex9 Dist-cleaning seq_ls ... checking build system type... i686-apple-darwin9.1.0 checking host system type... i686-apple-darwin9.1.0 checking for mpxlf... no checking for mpf77... no checking for mpif77... mpif77 checking for ranlib... ranlib checking for a BSD-compatible install... /usr/bin/install -c checking for gawk... no checking for mawk... no checking for nawk... no checking for awk... awk checking whether ln -s works... yes checking whether make sets $(MAKE)... yes checking for gcc... /opt/petsc-2.3.3-p8/externalpackages/ mpich2-1.0.5p4/darwin9.1.0-cxx-debug/bin/mpicc -PIC -Wall -Wwrite- strings -Wno-long-double -Wno-strict-aliasing -g3 checking for C compiler default output file name... a.out checking whether the C compiler works... yes checking whether we are cross compiling... no checking for suffix of executables... checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether /opt/petsc-2.3.3-p8/externalpackages/mpich2-1.0.5p4/ darwin9.1.0-cxx-debug/bin/mpicc -PIC -Wall -Wwrite-strings -Wno-long- double -Wno-strict-aliasing -g3 accepts -g... yes checking for /opt/petsc-2.3.3-p8/externalpackages/mpich2-1.0.5p4/ darwin9.1.0-cxx-debug/bin/mpicc -PIC -Wall -Wwrite-strings -Wno-long- double -Wno-strict-aliasing -g3 option to accept ANSI C... none needed checking how to run the C preprocessor... /opt/petsc-2.3.3-p8/ externalpackages/mpich2-1.0.5p4/darwin9.1.0-cxx-debug/bin/mpicc -PIC - Wall -Wwrite-strings -Wno-long-double -Wno-strict-aliasing -g3 -E checking for gcc... (cached) /opt/petsc-2.3.3-p8/externalpackages/ mpich2-1.0.5p4/darwin9.1.0-cxx-debug/bin/mpicc -PIC -Wall -Wwrite- strings -Wno-long-double -Wno-strict-aliasing -g3 checking whether we are using the GNU C compiler... (cached) yes checking whether /opt/petsc-2.3.3-p8/externalpackages/mpich2-1.0.5p4/ darwin9.1.0-cxx-debug/bin/mpicc -PIC -Wall -Wwrite-strings -Wno-long- double -Wno-strict-aliasing -g3 accepts -g... (cached) yes checking for /opt/petsc-2.3.3-p8/externalpackages/mpich2-1.0.5p4/ darwin9.1.0-cxx-debug/bin/mpicc -PIC -Wall -Wwrite-strings -Wno-long- double -Wno-strict-aliasing -g3 option to accept ANSI C... (cached) none needed checking whether we are using the GNU C++ compiler... yes checking whether /opt/petsc-2.3.3-p8/externalpackages/mpich2-1.0.5p4/ darwin9.1.0-cxx-debug/bin/mpicxx -Wall -Wwrite-strings -Wno-long- double -Wno-strict-aliasing -g -PIC accepts -g... yes checking how to run the C++ preprocessor... /opt/petsc-2.3.3-p8/ externalpackages/mpich2-1.0.5p4/darwin9.1.0-cxx-debug/bin/mpicxx - Wall -Wwrite-strings -Wno-long-double -Wno-strict-aliasing -g -PIC -E checking whether we are using the GNU Fortran 77 compiler... no checking whether mpif77 accepts -g... no checking how to get verbose linking output from mpif77... checking for Fortran libraries of mpif77... checking for dummy main to link with Fortran libraries... none checking for Fortran name-mangling scheme... make[1]: *** No rule to make target `distclean'. Stop. make: *** [distclean] Error 2 rm: conftest.dSYM: is a directory configure: WARNING: compilation failed rm: conftest.dSYM: is a directory rm: conftest.dSYM: is a directory configure: error: cannot compile a simple Fortran program See `config.log' for more details.

5 15

Direct LU solver
by Amit.Itagi＠seagate.com 03 Mar '08

03 Mar '08

Hi, I need to do direct LU solves (repeatedly, with the same matrix) in one of my MPI applications. I am having trouble implementing the solver. To identify the problem, I wrote a short toy code to run with 2 processes. I can run it with either the spooles parallel matrix or the superlu_dist matrix. I am using C++ and complex matrices. Here is the code listing: #include <iostream> #include <complex> #include <stdlib.h> #include "petsc.h" #include "petscmat.h" #include "petscvec.h" #include "petscksp.h" using namespace std; int main( int argc, char *argv[] ) { int rank, size; Mat A; PetscErrorCode ierr; PetscInt loc; PetscScalar val; Vec x, y; KSP solver; PC prec; MPI_Comm comm; // Number of non-zeros in each row int d_nnz=1, o_nnz=1; ierr=PetscInitialize(&argc,&argv,PETSC_NULL,PETSC_NULL); CHKERRQ(ierr); // Initialization (including MPI) comm=PETSC_COMM_WORLD; ierr=MPI_Comm_size(PETSC_COMM_WORLD,&size); CHKERRQ(ierr); ierr=MPI_Comm_rank(PETSC_COMM_WORLD,&rank); CHKERRQ(ierr); // Assemble matrix A if(rank==0) { ierr=MatCreateMPIAIJ(comm,1,1,2,2,0,&d_nnz,0,&o_nnz,&A); CHKERRQ(ierr); val=complex<double>(1.0,0.0); ierr=MatSetValue(A,0,0,val,INSERT_VALUES);CHKERRQ(ierr); val=complex<double>(0.0,1.0); ierr=MatSetValue(A,0,1,val,INSERT_VALUES);CHKERRQ(ierr); } else { ierr=MatCreateMPIAIJ(PETSC_COMM_WORLD,1,1,2,2,0,&d_nnz,0,&o_nnz,&A); CHKERRQ(ierr); val=complex<double>(1.0,1.0); ierr=MatSetValue(A,1,1,val,INSERT_VALUES);CHKERRQ(ierr); val=complex<double>(0.0,-1.0); ierr=MatSetValue(A,1,0,val,INSERT_VALUES);CHKERRQ(ierr); } ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); cout << "============ Mat A ==================" << endl; ierr=MatView(A,PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); cout << "======================================" << endl; // For spooles //ierr=MatConvert(A,MATMPIAIJSPOOLES,MAT_REUSE_MATRIX,&A); CHKERRQ(ierr); // For superlu_dist ierr=MatConvert(A,MATSUPERLU_DIST,MAT_REUSE_MATRIX,&A); CHKERRQ(ierr); // Direct LU solver ierr=KSPCreate(comm,&solver); CHKERRQ(ierr); ierr=KSPSetType(solver,KSPPREONLY); CHKERRQ(ierr); ierr=KSPSetOperators(solver,A,A,SAME_NONZERO_PATTERN); CHKERRQ(ierr); ierr=KSPGetPC(solver,&prec); CHKERRQ(ierr); ierr=PCSetType(prec,PCLU); CHKERRQ(ierr); ierr=KSPSetFromOptions(solver); CHKERRQ(ierr); //============ Vector assembly ======================== if(rank==0) { ierr=VecCreateMPI(PETSC_COMM_WORLD,1,2,&x); CHKERRQ(ierr); val=complex<double>(1.0,0.0); loc=0; ierr=VecSetValues(x,1,&loc,&val,ADD_VALUES);CHKERRQ(ierr); } else { ierr=VecCreateMPI(PETSC_COMM_WORLD,1,2,&x); CHKERRQ(ierr); val=complex<double>(-1.0,0.0); loc=1; ierr=VecSetValues(x,1,&loc,&val,ADD_VALUES);CHKERRQ(ierr); } ierr=VecAssemblyBegin(x); CHKERRQ(ierr); ierr=VecAssemblyEnd(x); CHKERRQ(ierr); cout << "============== Vec x ==================" << endl; ierr=VecView(x,PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); cout << "======================================" << endl; VecDuplicate(x,&y); // Duplicate the matrix storage // Solve the matrix equation ierr=KSPSolve(solver,x,y); CHKERRQ(ierr); cout << "============== Vec y =================" << endl; ierr=VecView(y,PETSC_VIEWER_STDOUT_WORLD); CHKERRQ(ierr); cout << "======================================" << endl; // Destructors ierr=KSPDestroy(solver); CHKERRQ(ierr); ierr=VecDestroy(x); CHKERRQ(ierr); ierr=VecDestroy(y); CHKERRQ(ierr); ierr=MatDestroy(A); CHKERRQ(ierr); // Finalize ierr=PetscFinalize(); CHKERRQ(ierr); return 0; } When I run the program with spooles, I get the following output. ============ Mat A ================== ============ Mat A ================== ====================================== row 0: (0, 1) (1, 0 + 1 i) row 1: (0, 0 - 1 i) (1, 1 + 1 i) ====================================== ============== Vec x ================== ============== Vec x ================== Process [0] 1 ====================================== Process [1] -1 ====================================== fatal error in InpMtx_MPI_split() firsttag = 0, tagbound = -1 fatal error in InpMtx_MPI_split() firsttag = 0, tagbound = -1 ----------------------------------------------------------------------------- One of the processes started by mpirun has exited with a nonzero exit code. This typically indicates that the process finished in error. If your process did not finish in error, be sure to include a "return 0" or "exit(0)" in your C code before exiting the application. PID 22881 failed on node n0 (127.0.0.1) with exit status 255. ----------------------------------------------------------------------------- mpirun failed with exit status 255 When run in the debugger, there is no stack trace. The error is Program exited with code 0377 With superlu_dist, the output is ============ Mat A ================== ============ Mat A ================== row 0: (0, 1) (1, 0 + 1 i) row 1: (0, 0 - 1 i) (1, 1 + 1 i) ====================================== ====================================== ============== Vec x ================== ============== Vec x ================== Process [0] 1 Process [1] -1 ====================================== ====================================== [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Si… ERROR: or try http://valgrind.org on linux or man libgmalloc on Apple to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Signal received! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 2.3.3, Patch 8, Fri Nov 16 17:03:40 CST 2007 HG revision: 414581156e67e55c761739b0deb119f7590d0f4b [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: lu on a linux-gnu named tabla by amit Thu Feb 28 13:48:54 2008 [0]PETSC ERROR: Libraries linked from /home/amit/programs/ParEM/petsc-2.3.3-p8/lib/linux-gnu-c-debug [0]PETSC ERROR: Configure run at Thu Feb 28 12:19:39 2008 [0]PETSC ERROR: Configure options --with-scalar-type=complex --with-debugging=no --with-fortran-kernels=generic --with-clanguage=cxx --with-metis=1 --download-metis=1 --with-parmetis=1 --download-parmetis=1 --with-superlu_dist=1 --download-superlu_dist=1 --with-spooles=1 --with-spooles-dir=/home/amit/programs/ParEM/spooles-2.2 COPTFLAGS="-O3 -march=p4 -mtune=p4 -ffast-math -malign-double -funroll-loops -pipe -fomit-frame-pointer -finline-functions -msse2" CXXOPTFLAGS="-O3 -march=p4 -mtune=p4 -ffast-math -malign-double -funroll-loops -pipe -fomit-frame-pointer -finline-functions -msse2" FOPTS="-O3 -qarch=p4 -qtune=p4" --with-shared=0 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file ----------------------------------------------------------------------------- One of the processes started by mpirun has exited with a nonzero exit code. This typically indicates that the process finished in error. If your process did not finish in error, be sure to include a "return 0" or "exit(0)" in your C code before exiting the application. PID 22998 failed on node n0 (127.0.0.1) with exit status 1. ----------------------------------------------------------------------------- mpirun failed with exit status 1 The debugger tracks the segmentation violation to main -> KSPSolve -> KSPSetUp -> PCSetUp -> PCSetUp_LU -> MatLUFactorNumeric -> MatLUFactorNumeric_SUperLU_DIST -> pzgssvx Could someone kindly point out what I am missing ? Thanks Rgds, Amit

4 12

--with-clanguage: c and c++
by Yujie 29 Feb '08

29 Feb '08

Hi, everyone When PETSc is compiled with "--with-clanguage=C" or "--with-clanguage=C++", what is the difference between them, the parameters in the functions are adjusted or else? thanks, Yujie

2 1

Re: any good ideas for me from you? thanks a lot.Fwd: any examples to demonstrate how to Spooles package?
by Yujie 27 Feb '08

27 Feb '08

What do you mean about 3)? I am considering to use MatSolve_MPIAIJSpooles with setting b to 1 0 0 . . .0 0 1 0 0 0 0 1 0 . . . . . . . . . 1 for solving Ax=b. After finishing all, I will rearrange X=[x1,x2,x3,x4], which is the inversion of A. whether is it similar with 1) you mentioned? Practically, If I use such method, I may use some iterative methods to solve it, not direct inversion method. What is the time difference or time complexity regarding using spooles (direct inversion method) or other iterative methods? thanks a lot. Regards, Yujie On Wed, Feb 27, 2008 at 2:33 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > This seems like it would involve significant programming time. Therefore, > I suggest > > 1) Solving each vector in a loop > > 2) Taking a look at MatMatSolve_SeqAIJ() and > MatSolve_MPIAIJSpooles() and trying to implement it yourself for > Spooles > > 3) Reformulating your problem so as not use an inverse, but rather just > solves > > Thanks, > > Matt > > On Wed, Feb 27, 2008 at 3:44 PM, Yujie <recrusader(a)gmail.com> wrote: > > > > > > > > ---------- Forwarded message ---------- > > From: Yujie <recrusader(a)gmail.com> > > Date: Wed, Feb 27, 2008 at 11:40 AM > > Subject: Re: any examples to demonstrate how to Spooles package? > > To: petsc-users(a)mcs.anl.gov > > > > > > > > This is why I have recompiled PETSc with spooles. spooles can deal with > > AX=Y(Y is a matrix). However, PETSc only provide the following: > > > > 51: PetscErrorCode MatSolve_MPISpooles(Mat A,Vec b,Vec x) > > I don't set b to a matrix even if I use > > 178: PetscErrorCode MatFactorNumeric_MPISpooles(Mat A,MatFactorInfo > > *info,Mat *F) for LU factorization > > > > Could you give me some advice or examples? thanks a lot. > > > > Regards, > > Yujie > > > > > > > > > > > > On Wed, Feb 27, 2008 at 11:32 AM, Matthew Knepley <knepley(a)gmail.com> > wrote: > > > > > > > > On Wed, Feb 27, 2008 at 1:21 PM, Yujie <recrusader(a)gmail.com> wrote: > > > > Dear Matt: > > > > > > > > I checked the codes about MatMatSolve(). However, currently, PETSc > > didn't > > > > realize its parallel version. Is it right? I want to inverse the > matrix > > > > parallelly. could you give me some examples about it? thanks a lot. > > > > > > Thats right. The parallel version is not implemented. It looks like > this > > would > > > take significant work. > > > > > > > > > > > > > > > Matt > > > > > > > Regards, > > > > Yujie > > > > > > > > > > > > > > > > On 2/27/08, Matthew Knepley <knepley(a)gmail.com> wrote: > > > > > On Wed, Feb 27, 2008 at 11:05 AM, Yujie <recrusader(a)gmail.com> > wrote: > > > > > > Dear Sanjay: > > > > > > > > > > > > Thank you for your reply. I don't understand what you said. Now, > I > > want > > > > to > > > > > > use spooles package to inverse a sparse SPD matrix. I have > further > > > > checked > > > > > > the inferface about spooles in PETSc. I find although spooles > can > > deal > > > > with > > > > > > AX=B (B may be a dense matrix) with parallel LU factorization. > > > > > > However, PETSc only provide the following: > > > > > > 51: PetscErrorCode MatSolve_MPISpooles(Mat A,Vec b,Vec x) > > > > > > I don't set b to a matrix even if I use > > > > > > 178: PetscErrorCode MatFactorNumeric_MPISpooles(Mat > A,MatFactorInfo > > > > > > *info,Mat *F) for LU factorization. > > > > > > > > > > > > Could you have any suggestions about this? thanks a lot. > > > > > > > > > > > > > > > MatMatSolve() > > > > > > > > > > > > > > > > > http://www-unix.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/docs/man… > > > > > > > > > > Matt > > > > > > > > > > > > > > > > Regards, > > > > > > Yujie > > > > > > > > > > > > On 2/26/08, Sanjay Govindjee <sanjay(a)ce.berkeley.edu> wrote: > > > > > > > from my make file > > > > > > > > > > > > > > -@${MPIRUN} -s all -np $(NPROC) $(PROGNAME) -ksp_type preonly > > > > > > > -ksp_monitor -pc_type cholesky -mat_type mpisbaijspooles > > -log_summary > > > > > > > -on_error_attach_debugger -mat_spooles_symmetryflag 0 > > -options_left > > > > > > > > > > > > > > > > > > > > > -sg > > > > > > > > > > > > > > > > > > > > > Yujie wrote: > > > > > > > > Hi, everyone > > > > > > > > > > > > > > > > I have compiled PETSc with spooles. However, I try to find > how > > to > > > > use > > > > > > > > this package in PETSc directory. I can't find any examples > for > > it. > > > > > > > > Could you give me some advice? I want to use spooles to > inverse > > a > > > > > > > > sparse matrix. thanks a lot. > > > > > > > > > > > > > > > > Regards, > > > > > > > > Yujie > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > What most experimenters take for granted before they begin their > > > > > experiments is infinitely more interesting than any results to > which > > > > > their experiments lead. > > > > > -- Norbert Wiener > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > What most experimenters take for granted before they begin their > > > experiments is infinitely more interesting than any results to which > > > their experiments lead. > > > -- Norbert Wiener > > > > > > > > > > > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener >

1 0

MG question
by jens.madsen＠risoe.dk 27 Feb '08

27 Feb '08

Hi I hope that this question is not outside the scope of this mailinglist. As far as I understand PETSc uses preconditioned GMRES(or another KSP method) as pre- and postsmoother on all multigrid levels? I was just wondering why and where in the literature I can read about that method? I thought that a fast method would be to use MG (with Gauss-Seidel RB/zebra smothers) as a preconditioner for GMRES? I have looked at papers written by Oosterlee etc. Kind Regards

3 7

few questions
by amjad ali 27 Feb '08

27 Feb '08

Hello all, Please answer the following, 1) What is the difference between static and dynamic versions of petsc? 2) How to check that which version (static or dynamic) is installed on a system? 3) Plz comment on if there is any effect of static/dynamic version while using/calling petsc from some external package? 4) how to update an already installed petsc version with newerer/latest version of petsc? Thanks to all.

3 3

any examples to demonstrate how to Spooles package?
by Yujie 27 Feb '08

27 Feb '08

Hi, everyone I have compiled PETSc with spooles. However, I try to find how to use this package in PETSc directory. I can't find any examples for it. Could you give me some advice? I want to use spooles to inverse a sparse matrix. thanks a lot. Regards, Yujie

4 8

Error with -log_history
by Timothy Stitt 27 Feb '08

27 Feb '08

Hi PETSc users/developers, I am having some difficulties with the -log_history option on example PETSc codes at my local installation (FYI: Cray XT architecture). When executing the ex2.c code (for example) on multiple processors with the -log_history option I keep getting: Signal number 11 SEGV: Segmentation Violation, probably memory access out of range The log history file is created though but contains the following: Petsc Release Version 2.3.3, Patch 3, Fri Jun 15 16:51:25 CDT 2007 HG revision: f051789beadcd36f77fb6111d20225e26ed7cc0d Wed Feb 27 10:35:55 2008 [8191]PETSC ERROR: [18945200]PETSC ERROR: [118163408]PETSC ERROR: [118163408]PETSC ERROR: [118163408]PETSC ERROR: [118163408]PETSC ERROR: [118163408]PETSC ERROR: [59]PETSC ER ROR: [118163408]PETSC ERROR: [118163408]PETSC ERROR: [118163408]PETSC ERROR: [118163408]PETSC ERROR: [118163408]PETSC ERROR: [118163408]PETSC ERROR: [118163408]PETSC ERROR: [ 23118368]PETSC ERROR: [118163408]PETSC ERROR: [118163408]PETSC ERROR: [118163408]PETSC ERROR: [118163408]PETSC ERROR: [19014780]PETSC ERROR: Without the -log_history option the code runs as expected. Is this a build/architecture issue at my end? Thanks in advance for any advice given. Tim. -- Timothy Stitt HPC Applications Analyst Swiss National Supercomputing Centre (CSCS) Galleria 2 - Via Cantonale CH-6928 Manno, Switzerland +41 (0) 91 610 8233 stitt(a)cscs.ch

2 1