December 2010 - petsc-users - mailman3.cels.anl.gov

Re: Multiple communicators
by irfan.khan＠gatech.edu 02 Apr '12

02 Apr '12

I use seperete communicators for fluid and solid ranks in a fluid-structure interaction code. I use PETSc tools (KSP) to solve for the solid phase (FEA) which is carried out under a different communicator (FEA_Comm). Unless I am doing something wrong, I have found that the command line options: -ksp_type; -mat_partitioning_type, don't work if there are 2 communicators. But for single communicator they work. Please see the attached files containing output of -log_summary for 2 different codes (scroll down to the end). One with single communicator and the other with two communicators. Both the output have been generated with the same PETSc compilation. Futher -log_summary results are not able to detect all the events that have been registered if they are not in the WORLD communicator. Thank you Irfan ----- Original Message ----- From: "Matthew Knepley" <knepley(a)gmail.com> To: "PETSc users list" <petsc-users(a)mcs.anl.gov> Sent: Tuesday, November 24, 2009 8:03:00 PM GMT -05:00 US/Canada Eastern Subject: Re: Multiple communicators On Tue, Nov 24, 2009 at 6:59 PM, < irfan.khan(a)gatech.edu > wrote: Hello Does the procedure for providing command line options and to obtain profiling data through -log_summary change if there are multiple MPI communicators in a petsc code? There is always WORLD, so it should be fine unless I misunderstand your question. Matt If so, can somebody point me to information, references on how to provide command line options if there are multiple communicators? Thank you Irfan -- PhD Candidate G.W. Woodruff School of Mechanical Engineering Georgia Institute of Technology Atlanta, GA (30332) -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- PhD Candidate G.W. Woodruff School of Mechanical Engineering Georgia Institute of Technology Atlanta, GA (30332)

5 14

parmetis weights
by francois pacull 09 Aug '11

09 Aug '11

Dear PETSc team, I just have a small and naive question about graph weights in Parmetis. When looking at mat/partition/impls/pmetis/pmetis.c, it seems that weight arrays (for vertices or edges) can be given to parmetis, however the wgtflag variable is always set to 0 from what I understand, which means that no weights are used in ParMETIS_V3_PartKway. So does this mean that the parmetis routine uses the weight arrays whenever they are not NULL, even if wgtflag is 0? I didn't try to use any weight yet, but will probably try it soon... Thank you for any help you can provide, francois pacull.

2 2

V-cycle multigrid with matrix shells
by Sylvain Barbot 19 May '11

19 May '11

Hi, I am trying to implement the multigrid solver to solve a linear inhomogeneous elliptic equation with non-constant coefficients in 3-D with 3 degrees of freedom with matrix shells. The grid is structured with non-uniform sampling. I insist on using matrix shells because of memory limitations. The direct solver gmres, or alike, gives very accurate solutions even for large problems (say, with a grid of 256x256x256), but computation is lengthy. I am hoping to get significant speedup using a multigrid solver. It looks like the DMMG objects may provide all the tools necessary to do so: smoother, restriction and interpolation. But typically in my problems, I do not have an analytical expression for the right-hand side, just a given forcing term in finest resolution with high-frequency content (not easily subsampled). Also I do not typically have any non-trivial first guess. It looks the the default Petsc multi-grid solver starts from the coarser grid and builds a solution with a finer and finer resolution as levels increase. I would like to do the opposite, with the typical smoothing of the residuals, then restriction, smoothing, restriction, then direct solving at the coarsest level, then as many interpolation-correction-smoothing as necessary to go back to the finest level, the typical V cycle. questions: 1) is it at all possible to specify this mode of operation, from the finest to the coarser level, and back? Any examples out there? 2) is it readily possible to use matrix shells with DMMG? I imagine the Jacobian matrix may simply be provided as a matrix shell. Is there any examples of multi-grid methods with shell matrices online? 3) to deal with non-uniform sampling: can I provide the coordinates of the finest grid with DASetCoordinates, then expect DMMG to provide the subsampled coordinates at the coarser levels? Thanks a lot in advance for any advice. Best, Sylvain Barbot

6 28

Reading matrices into PETSc
by Gaurish Telang 08 Mar '11

08 Mar '11

i have this large text file containing a matrix. This text file contains the non-zero entries of a very large sparse matrix The first two columns indicate the position of the non-zero entry and the last column the actual non-zero value it self for example the matrix 1 0 8 0 0 5 6 0 0 is written in the text file in the form of 1 1 1 1 3 8 2 3 5 3 1 6 This is the standard [(row ,column), non-zero] entry format. i want PETSc to load this matrix from the text file i am not sure how to do that. What commands do I use? I am new to PETSc, so some detail in the explanation will be really helpful. Sincere thanks, Gaurish.

6 8

Some basic questions about PETSC
by Ben Tay 19 Jan '11

19 Jan '11

Hi, I'm still relatively new to PETSC but I've finally managed to solve my own poisson eqn using it in about a week's time.Thank you for writing such a good solver. However, I have some questions still about PETSC: 1. Are petscscaler the same as real nos.? I am writing in double precision or real(8). So are these 2 interchangeable? (same as petscint and integer). 2. while linking in compaq visual fortran, I get the warning message about conflict library with LIBCMT. Is it important or can I just ignore it? 3. I am writing in f90 free form format. When I modify from fixed f77 format to f90, it gives me a lot of error, which seems to be the include file error. So, is it possible to write in f90 format? 4. I am writing a Navier stokes solver and I'm using PETSC to solve the poisson eqn. If only the RHS changes with each time step, do I need to call VecCreateSeq <insert values in RHS b> VecAssemblyBegin VecAssemblyEnd VecDestroy at each time step? I don't think I can (or I should) change values of b after VecAssemblyBegin,VecAssemblyEnd, or can I? Thank you very much and have a nice day Regards, Ben

4 8

Using PETSc from MATLAB code, experimental
by Barry Smith 13 Jan '11

13 Jan '11

PETSc users, It is now possible to write MATLAB programs (sequential) that use PETSc KSP, SNES, and TS solvers directly in MATLAB. The code is still experimental and incomplete. But if you are interested in trying it out, get the development release of PETSc http://www.mcs.anl.gov/petsc/petsc-as/developers/index.html join the development mailing list petsc-dev http://www.mcs.anl.gov/petsc/petsc-as/miscellaneous/mailing-lists.html, read bin/matlab/classes/PetscInitialize.m, configure and make PETSc and join the fun. We are definitely in need of more developers for this code. Barry

4 5

pastix solver break at pastix_checkMatrix
by Gong Ding 03 Jan '11

03 Jan '11

Dear all, I found the pastix solver petsc-3.1-p4 ./configure --with-vendor-compilers=intel --with-blas-lapack-dir=/opt/intel/mkl/10.2.0.013/lib/em64t/ --download-pastix --download-scotch --with-shared=1 --with-debugging=1 For a poisson problem with symmetric matrix, pastix works well. However, for unsymmetric problem, the code break. valgrind reported that: Check : Sort CSC OK ==4959== Invalid read of size 4 ==4959== at 0x1241931: PetscTrFreeDefault (mtr.c:280) ==4959== by 0x150448A: MatConvertToCSC (pastix.c:188) ==4959== by 0x1506638: MatFactorNumeric_PaStiX (pastix.c:390) ==4959== by 0x13980AB: MatLUFactorNumeric (matrix.c:2587) ==4959== by 0x16AB0A6: PCSetUp_LU (lu.c:158) ==4959== by 0x1A9BD42: PCSetUp (precon.c:795) ==4959== by 0x16FA8D0: KSPSetUp (itfunc.c:237) ==4959== by 0x16FBB2A: KSPSolve (itfunc.c:353) ==4959== by 0x17BEC6D: SNES_KSPSolve (snes.c:2944) ==4959== by 0x17CEFEA: SNESSolve_LS (ls.c:191) ==4959== by 0x17B8B78: SNESSolve (snes.c:2255) ==4959== by 0x10B969D: FVM_NonlinearSolver::sens_solve() (fvm_nonlinear_solver.cc:820) ==4959== Address 0x88e2f08 is 8 bytes inside a block of size 40 free'd ==4959== at 0x4A05B16: operator delete(void*) (vg_replace_malloc.c:387) ==4959== by 0xA42A53: __gnu_cxx::new_allocator<std::_Rb_tree_node<CTRI::Triangle*> >::deallocate(std::_Rb_tree_node<CTRI::Triangle*>*, unsigned long) (new_allocator.h:94) ==4959== by 0xA41863: std::_Rb_tree<CTRI::Triangle*, CTRI::Triangle*, std::_Identity<CTRI::Triangle*>, std::less<CTRI::Triangle*>, std::allocator<CTRI::Triangle*> >::_M_put_node(std::_Rb_tree_node<CTRI::Triangle*>*) (stl_tree.h:362) ==4959== by 0xA419C8: std::_Rb_tree<CTRI::Triangle*, CTRI::Triangle*, std::_Identity<CTRI::Triangle*>, std::less<CTRI::Triangle*>, std::allocator<CTRI::Triangle*> >::destroy_node(std::_Rb_tree_node<CTRI::Triangle*>*) (stl_tree.h:392) ==4959== by 0xA42501: std::_Rb_tree<CTRI::Triangle*, CTRI::Triangle*, std::_Identity<CTRI::Triangle*>, std::less<CTRI::Triangle*>, std::allocator<CTRI::Triangle*> >::erase(std::_Rb_tree_iterator<CTRI::Triangle*>) (stl_tree.h:1189) ==4959== by 0xA4264A: std::_Rb_tree<CTRI::Triangle*, CTRI::Triangle*, std::_Identity<CTRI::Triangle*>, std::less<CTRI::Triangle*>, std::allocator<CTRI::Triangle*> >::erase(std::_Rb_tree_iterator<CTRI::Triangle*>, std::_Rb_tree_iterator<CTRI::Triangle*>) (stl_tree.h:1281) ==4959== by 0xA4257B: std::_Rb_tree<CTRI::Triangle*, CTRI::Triangle*, std::_Identity<CTRI::Triangle*>, std::less<CTRI::Triangle*>, std::allocator<CTRI::Triangle*> >::erase(CTRI::Triangle* const&) (stl_tree.h:1215) ==4959== by 0xA415E2: std::set<CTRI::Triangle*, std::less<CTRI::Triangle*>, std::allocator<CTRI::Triangle*> >::erase(CTRI::Triangle* const&) (stl_set.h:387) ==4959== by 0xA3F6BD: CTRI::Triangle::~Triangle() (c_triangle.cc:109) ==4959== by 0xA43F16: CTRI::TriMesh::~TriMesh() (c_trimesh.cc:163) ==4959== by 0xA3F0C6: ctri_triangulate (c_tri_io.cc:35) ==4959== by 0xD53FB9: MeshGeneratorTri3::do_refine(MeshRefinement&) (mesh_generation_tri3.cc:1369) ==4959== [0]PETSC ERROR: PetscTrFreeDefault() called from MatConvertToCSC() line 188 in src/mat/impls/aij/mpi/pastix/pastix.c [0]PETSC ERROR: Block at address 0x88e2ee0 is corrupted; cannot free; may be block not allocated with PetscMalloc() The test problems used to work well under other linear solvers such as MUMPS and superlu. Does any meet this problem before? Yours Gong Ding

3 12

the size of matrix in PETSc solver
by Peter Wang 31 Dec '10

31 Dec '10

I am trying to build a big matrix to be solved by KSP. I am wondering is there any limitation to the matrix size? What is the maximum size of the matrix in KSP? For example, if a 100 M by 100 M matrix can be handled by the KSP? Thanks a lot.

3 2

Meaning of MatStencil
by Raeth, Peter 27 Dec '10

27 Dec '10

Am a new PETSc user trying to make use of the matrix level to create and operate on matrices whose memory exceeds that available on any one node. To populate a distributed dense matrix with results of other matrix calculations we are trying to use MatSetValuesBlockedStencil. Two of the inputs to that function require structures of type MatStencil. After searching the archives, tutorials, examples, and Google, I can not find anything that explains what the values of MatStencil are meant to do relative to where in the target matrix to place the block of values. Would someone please point me in the right direction? Thanks, Peter.

2 2

Very poor speed up performance
by Yongjun Chen 23 Dec '10

23 Dec '10

Hi everyone, I use PETSC (version 3.1-p5) to solve a linear problem Ax=b. The matrix A and right hand vector b are read from files. The dimension of A is 1.2Million*1.2Million. I am pretty sure the matrix A and vector b have been read correctly. I compiled the program with optimized version (--with-debugging=0), tested the speed up performance on two servers, and I have found that the performance is very poor. For the two servers, one is 4 cpus * 4 cores per cpu, i.e., with a total 16 cores. And the other one is 4 cpus * 12 cores per cpu, with a total 48 cores. On each of them, with the increasing of computing cores k from 1 to 8 (mpiexec –n k ./Solver_MPI -pc_type jacobi -ksp-type gmres), the speed up will increase from 1 to 6, but when the computing cores k increase from 9 to 16(for the first server) or 48 (for the second server), the speed up decrease firstly and then remains a constant value 5.0 (for the first server) or 4.5(for the second server). Actually, the program LAMMPS speed up excellently on these two servers. Any comments are very appreciated! Thanks! -------------------------------------------------------------------------------------------------------------------------- PS: the related codes are as following, //firstly read A and b from files ... //then ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY); CHKERRQ(ierr); ierr = VecAssemblyBegin(b); CHKERRQ(ierr); ierr = VecAssemblyEnd(b); CHKERRQ(ierr); ierr = MatSetOption(A,MAT_SYMMETRIC,PETSC_TRUE); CHKERRQ(ierr); ierr = MatGetRowUpperTriangular(A); CHKERRQ(ierr); ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr); ierr = KSPSetOperators(ksp,A,A,DIFFERENT_NONZERO_PATTERN);CHKERRQ(ierr); ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); ierr = KSPSetTolerances(ksp,1.e-7,PETSC_DEFAULT,PETSC_DEFAULT,PETSC_DEFAULT);CHKERRQ(ierr); ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr); ierr = KSPSolve(ksp,b,x);CHKERRQ(ierr); ierr = KSPView(ksp,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); ierr = KSPGetSolution(ksp, &x);CHKERRQ(ierr); ierr = VecAssemblyBegin(x);CHKERRQ(ierr); ierr = VecAssemblyEnd(x);CHKERRQ(ierr); ...

5 23