November 2010 - petsc-users - mailman3.cels.anl.gov

Re: Multiple communicators
by irfan.khan＠gatech.edu 02 Apr '12

02 Apr '12

I use seperete communicators for fluid and solid ranks in a fluid-structure interaction code. I use PETSc tools (KSP) to solve for the solid phase (FEA) which is carried out under a different communicator (FEA_Comm). Unless I am doing something wrong, I have found that the command line options: -ksp_type; -mat_partitioning_type, don't work if there are 2 communicators. But for single communicator they work. Please see the attached files containing output of -log_summary for 2 different codes (scroll down to the end). One with single communicator and the other with two communicators. Both the output have been generated with the same PETSc compilation. Futher -log_summary results are not able to detect all the events that have been registered if they are not in the WORLD communicator. Thank you Irfan ----- Original Message ----- From: "Matthew Knepley" <knepley(a)gmail.com> To: "PETSc users list" <petsc-users(a)mcs.anl.gov> Sent: Tuesday, November 24, 2009 8:03:00 PM GMT -05:00 US/Canada Eastern Subject: Re: Multiple communicators On Tue, Nov 24, 2009 at 6:59 PM, < irfan.khan(a)gatech.edu > wrote: Hello Does the procedure for providing command line options and to obtain profiling data through -log_summary change if there are multiple MPI communicators in a petsc code? There is always WORLD, so it should be fine unless I misunderstand your question. Matt If so, can somebody point me to information, references on how to provide command line options if there are multiple communicators? Thank you Irfan -- PhD Candidate G.W. Woodruff School of Mechanical Engineering Georgia Institute of Technology Atlanta, GA (30332) -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -- PhD Candidate G.W. Woodruff School of Mechanical Engineering Georgia Institute of Technology Atlanta, GA (30332)

5 14

parmetis weights
by francois pacull 09 Aug '11

09 Aug '11

Dear PETSc team, I just have a small and naive question about graph weights in Parmetis. When looking at mat/partition/impls/pmetis/pmetis.c, it seems that weight arrays (for vertices or edges) can be given to parmetis, however the wgtflag variable is always set to 0 from what I understand, which means that no weights are used in ParMETIS_V3_PartKway. So does this mean that the parmetis routine uses the weight arrays whenever they are not NULL, even if wgtflag is 0? I didn't try to use any weight yet, but will probably try it soon... Thank you for any help you can provide, francois pacull.

2 2

V-cycle multigrid with matrix shells
by Sylvain Barbot 19 May '11

19 May '11

Hi, I am trying to implement the multigrid solver to solve a linear inhomogeneous elliptic equation with non-constant coefficients in 3-D with 3 degrees of freedom with matrix shells. The grid is structured with non-uniform sampling. I insist on using matrix shells because of memory limitations. The direct solver gmres, or alike, gives very accurate solutions even for large problems (say, with a grid of 256x256x256), but computation is lengthy. I am hoping to get significant speedup using a multigrid solver. It looks like the DMMG objects may provide all the tools necessary to do so: smoother, restriction and interpolation. But typically in my problems, I do not have an analytical expression for the right-hand side, just a given forcing term in finest resolution with high-frequency content (not easily subsampled). Also I do not typically have any non-trivial first guess. It looks the the default Petsc multi-grid solver starts from the coarser grid and builds a solution with a finer and finer resolution as levels increase. I would like to do the opposite, with the typical smoothing of the residuals, then restriction, smoothing, restriction, then direct solving at the coarsest level, then as many interpolation-correction-smoothing as necessary to go back to the finest level, the typical V cycle. questions: 1) is it at all possible to specify this mode of operation, from the finest to the coarser level, and back? Any examples out there? 2) is it readily possible to use matrix shells with DMMG? I imagine the Jacobian matrix may simply be provided as a matrix shell. Is there any examples of multi-grid methods with shell matrices online? 3) to deal with non-uniform sampling: can I provide the coordinates of the finest grid with DASetCoordinates, then expect DMMG to provide the subsampled coordinates at the coarser levels? Thanks a lot in advance for any advice. Best, Sylvain Barbot

6 28

Some basic questions about PETSC
by Ben Tay 19 Jan '11

19 Jan '11

Hi, I'm still relatively new to PETSC but I've finally managed to solve my own poisson eqn using it in about a week's time.Thank you for writing such a good solver. However, I have some questions still about PETSC: 1. Are petscscaler the same as real nos.? I am writing in double precision or real(8). So are these 2 interchangeable? (same as petscint and integer). 2. while linking in compaq visual fortran, I get the warning message about conflict library with LIBCMT. Is it important or can I just ignore it? 3. I am writing in f90 free form format. When I modify from fixed f77 format to f90, it gives me a lot of error, which seems to be the include file error. So, is it possible to write in f90 format? 4. I am writing a Navier stokes solver and I'm using PETSC to solve the poisson eqn. If only the RHS changes with each time step, do I need to call VecCreateSeq <insert values in RHS b> VecAssemblyBegin VecAssemblyEnd VecDestroy at each time step? I don't think I can (or I should) change values of b after VecAssemblyBegin,VecAssemblyEnd, or can I? Thank you very much and have a nice day Regards, Ben

4 8

Writing PETSc matrices
by Nunion 13 Dec '10

13 Dec '10

Hello, I am new to PETSc and programming. I have a question concerning writing PETSc matrices in binary from binary matrices [compressessed/uncompressed] generated in Matlab. I am attempting to use the files in the /bin/matlab directory, in particular the PetscBinaryWrite.m file. However, the usage; PetscBinaryWrite('matrix.mat','output.ex') does not seem to work. I also tried using the examples in the /mat directory however, matlab does not support the writing of complex matrices in ASCII. Thanks in advance, Tom

5 6

Accessing array indices in 3D
by khalid ashraf 30 Nov '10

30 Nov '10

Thanks Barry. It works fine. Another question, I have a Periodic Boundary condition along x and y. So I want to use the two commented out line below. for (j=ys; j<ys+ym; j++) { for (i=xs; i<xs+xm; i++) { if (i<(mx-1)) { Cx=u_localptr[k][j][i+1]-u_localptr[k][j][i]; } else { // Cx=u_localptr[k][j][i]-u_localptr[k][j][0]; } if (j<(my-1)) { Cy=v_localptr[k][j+1][i]-v_localptr[k][j][i]; } else { // Cy=v_localptr[k][j][i]-v_localptr[k][0][i]; } Is it possible to include these non-local points [k][j][0] or [k][0][i]. Or I have to include the boundary points in the grid ? The same question when forming the A matrix with PBC. How to include the non-neighbour elements within the if statement. Thanks. Khalid

3 2

Accessing array indices in 3D
by khalid ashraf 30 Nov '10

30 Nov '10

Hi, I am writing the following loop for accessing the vectors in a 3D grid. for (k=zs; k<zs+zm; k++) { for (j=ys; j<ys+ym; j++) { for (i=xs; i<xs+xm; i++) { if (i<(mx-1)) { Cx=u_localptr[k][j][i+1]-u_localptr[k][j][i]; } else { Cx=u_localptr[k][j][i]-u_localptr[k][j][i-1]; } if (j<(my-1)) { // Cy=v_localptr[k][j+1][i]-v_localptr[k][j][i]; Cy=v_localptr[k][j][i]-v_localptr[k][j][i]; } else { Cy=v_localptr[k][j][i]-v_localptr[k][j-1][i]; } if (k<(mz-1)) { // Cz=w_localptr[k+1][j][i]-w_localptr[k][j][i]; Cz=w_localptr[k][j][i]-w_localptr[k][j][i]; } else { Cz=w_localptr[k][j][i]-w_localptr[k-1][j][i]; } In this code if I uncomment the two commented lines, then the program gives memory access out of range error when run on multiple processors. Could you please tell me what is the problem with the two commented out lines. Thanks. Khalid

2 1

Why MatMult(A, x, Ax) != A*x?
by Rongliang Chen 30 Nov '10

30 Nov '10

Hi, I am having trouble with using the function MatMult in PETSc. The result of MatMult(A, x, Ax) (*using multiprocessors*) is not equal to the result of A*x computed in Matlab. The format of the matrix A is AIJ and is obtained by function SNESDefaultComputeJacobian. I compute A*x in Matlab with the following A and x and compare it with Ax. ....................... ierr = MatMult(*A, x, Ax);CHKERRQ(ierr); sprintf(filename,"x.m"); ierr = PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&view->viewer);CHKERRQ(ierr); ierr = PetscViewerSetFormat(view->viewer, PETSC_VIEWER_ASCII_MATLAB);CHKERRQ(ierr); ierr = VecView(x, view->viewer);CHKERRQ(ierr); sprintf(filename,"Ax.m"); ierr = PetscViewerASCIIOpen(PETSC_COMM_WORLD,filename,&view->viewer);CHKERRQ(ierr); ierr = PetscViewerSetFormat(view->viewer, PETSC_VIEWER_ASCII_MATLAB);CHKERRQ(ierr); ierr = VecView(Ax, view->viewer);CHKERRQ(ierr); ..................... What kind of bug can cause this problem? Thanks. Best, Rongliang

2 4

Re: [petsc-users] MatPermuteSparsify hanging
by Kelly Fermoyle 30 Nov '10

30 Nov '10

This is what I got on a desktop. It looks a lot more reasonable. I stopped it after a while and it appears to be in MatSetValues. [Thread 0x40200940 (LWP 26776) exited] Program received signal SIGINT, Interrupt. 0x0000000000accb16 in PetscMemcpy (a=0x2aaab7b499b0, b=0x2aaab6de4938, n=1796808) at memc.c:104 104 memcpy((char*)(a),(char*)(b),n); (gdb) backtrace full #0 0x0000000000accb16 in PetscMemcpy (a=0x2aaab7b499b0, b=0x2aaab6de4938, n=1796808) at memc.c:104 al = 46912714873264 bl = 46912700827960 nl = 1796808 #1 0x000000000069b8f9 in MatSetValues_SeqAIJ (A=0x1430490, m=1, im=0x7fffffffd270, n=21, in=0x2aaab2c76200, v=0x2aaab2d9a200, is=INSERT_VALUES) at aij.c:225 len = 224601 new_j = 0x2aaab7d01200 Ain = 0x1431620 new_nz = 1755215 new_i = 0x2aaab83b4200 new_a = 0x2aaab6f9c200 a = 0x1431620 rp = 0x2aaab89c5d48 k = 0 low = 18 high = 21 t = 18 ii = 59410 row = 52376 nrow = 21 i = 21 col = 52922 l = 0 rmax = 21 N = 37 imax = 0x2aaab2e0f200 ai = 0x2aaab8aa2200 ailen = 0x2aaab2e4a200 ierr = 0 aj = 0x2aaab83ef200 nonew = 0 lastcol = 52922 ap = 0x2aaab6de4890 value = 25472.555765646161 aa = 0x2aaab6237200 ignorezeroentries = PETSC_FALSE roworiented = PETSC_TRUE #2 0x000000000061c324 in MatSetValues (mat=0x1430490, m=1, idxm=0x7fffffffd270, n=21, idxn=0x2aaab2c76200, v=0x2aaab2d9a200, addv=INSERT_VALUES) at matrix.c:921 ierr = 0 #3 0x00000000006426d7 in MatPermuteSparsify (A=0x141a1a0, band=-1, frac=0, tol=0.001, rowp=0x13fde70, colp=0x142e630, B=0x7fffffffd678) at matrix.c:4157 irowp = 0x142f050 icolp = 0x142fa70 rows = 0x2aaab2cb1200 cols = 0x2aaab2cec200 M = 59409 N = 59409 locRowStart = 0 locRowEnd = 59409 nz = 21 newNz = 21 cwork = 0x2aaab5271a24 cnew = 0x2aaab2c76200 vwork = 0x2aaab4bdd248 vnew = 0x2aaab2d9a200 bw = 2970 issize = 59409 row = 27126 locRow = 27126 newRow = 52376 col = 21 newCol = 52382 ierr = 0 #4 0x000000000043822f in main (argc=17, argv=0x7fffffffd7a8) at SolveTest.cpp:54 ksp = 0x7fff00000001 colp = 0x142e630 ---Type <return> to continue, or q <return> to quit--- vec_file = "/gpfs/home/kjf198/group/kjf198/SYMMETRIC_MATRICES/20-0.vec\000\000\000\000\000\000\200\325\377\377\377\177\000\000i4\271\255\252*\000\000\250\065\062\001\000\000\000\000\023\000\000\000\252*\000\000\300\065\062\001\000\000\000\000\001\000\000\000\000\000\000\000\250\065\062\001\000\000\000\000\221 \226\256\252*\000\000\303\312\363\000\000\000\000\000\340,\226\256\252*\000\000\250\335\037\001\000\000\000\000\303\312\363\000\000\000\000\000\060\326\377\377\377\177\000\000\000\260\036\001\000\000\000\000\070\330\377\377\377\177\000\000b.\226\256\252*\000\000\000\330\377\377\377\177\000\000\000\260\036\001\000\000\000\000@\326\377\377\377\177\000\000\201\065\343M4\000\000\000\000\260\036\001\000\000\000\000@\326\377\377\377\177\000\000\000\260\036\001\000\000\000\000\t\204C\000\000\000\000" mat_file = "/gpfs/home/kjf198/group/kjf198/SYMMETRIC_MATRICES/20-0.mat\000\377\377\177\000\000\000\020\214\256\252*\000\000>\340\215\256\252*\000\000`\023\214\256\252*\000\000\260\325\377\377\377\177\000\000\000\020\214\256\000\000\000\000\300®\257\252*\000\000`\263\252\252\252*\000\000\320\325\377\377\377\177\000\000\000\260\252\252\252*\000\000\023\246A\000\000\000\000\000`\263\252\252\252*\000\000\360\325\377\377\377\177\000\000\020\224\214\256\252*\000\000\000\020\214\256\252*\000\000\360\233\340M4\000\000\000\340\024\214\256\252*\000\000`\020\273\256\000\000\000\000y:<\a\000\000\000\000\000\260\036\001\000\000\000\000H[/\000\000\000\000\000\b(2\001\000\000\000\000\021\000\000\000\000\000\000\000\250\327\377\377\377\177\000\000\070\330\377\377\377\177\000" reordering = "rcm", '\000' <repeats 252 times> b = 0x13f6e30 rowp = 0x13fde70 ierr = 0 timestep = 20 x = 0x13fb6c0 start = 1.1098831058709664e-312 type = 0xecbb5a "rcm" iters = 15513927 A = 0x141a1a0 P = 0x1430490 residual = 2.1861090613857222e-317 threshold = 0.001 mat_view = 0x13fb6c0 vec_view = 0x13fb6c0 iteration = 0 end = 7.6648711891784312e-317 > On Mon, Nov 29, 2010 at 20:28, Kelly Fermoyle <kjf198(a)cse.psu.edu> wrote: > >> I am using a Cray machine and it requires that I call as aprun -n 1 >> ./prog_name, which does not seem to work with -start_in_debugger. >> > > Can you reproduce with the same code on a local machine (e.g. > desktop/laptop) where you can get a proper debugging session? > > >> Below is the output of the full backtrace: >> > > This trace is not even in your code > >> #2 0x0000000000403be7 in main (argc=20, argv=0x7fffe2814538) at > aprun.c:225 > > Jed >

4 7

Installing Hypre with PETSc [Complex]
by Nunion 30 Nov '10

30 Nov '10

Hello all, when attempting to install Hypre with PETSc (for complex arithmetic), I got a message saying that configuration could not proceed forward because Hypre is not configured for complex arithmetic. Does anyone know of a fix around this issue as I would like to use BoomerAMG interfaced with PETSc. Thanks, Mark

2 1