On Tue, Apr 3, 2012 at 7:40 AM, Barry Smith <[email protected]> wrote:
I think you are mis-understanding the global ordering when using DA vectors in parallel. You might consider using VecSetValuesLocal() or VecSetValuesStencil() instead of VecSetValues(). Read up on the users manual and examples about PETSc global ordering.
More explicitly, you do set the 5th value of the vector each time, but the layout is different when created with the DA since it is splitting the grid somehow. When you call VecView() it reorders the Vec to match the grid before output. Matt
Barry
On Apr 3, 2012, at 3:46 AM, khalid ashraf wrote:
Hi Matt, I ran the program on two processors. I am trying to enter value into a vector associated with DA. I am using VecSetValues since I may need to assign nonlocal values to the vector. However, I find that on 2 procs, a general parallel vector works but the DAVector indexing is not working. Do you see any problem with the DA code ?
Thanks.
Khalid
THIS WORKS: 30 is written at index 5 ------------------------------- ppy=5; ppx=30; VecCreate(PETSC_COMM_WORLD,&testVec); VecSetFromOptions(testVec); VecSetSizes(testVec,PETSC_DECIDE,384);
VecSetValues(testVec,1,&ppy,&ppx,INSERT_VALUES); VecAssemblyBegin(testVec); VecAssemblyEnd(testVec); =====================================
THIS DOES NOT WORK: puts 30 at index 9 instead of 5 when run on two procs. --------------------------------- ppy=5; ppx=30; ierr = DACreate3d(PETSC_COMM_WORLD,DA_YPERIODIC,DA_STENCIL_BOX,8,8,6,
PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL,
&appctx.da);CHKERRQ(ierr); ierr = DACreateGlobalVector(da,&testVec);CHKERRQ(ierr) VecSetValues(testVec,1,&ppy,&ppx,INSERT_VALUES); VecAssemblyBegin(testVec); VecAssemblyEnd(testVec);
From: Matthew Knepley <[email protected]> To: khalid ashraf <[email protected]> Cc: PETSc users list <[email protected]> Sent: Monday, April 2, 2012 3:08 PM Subject: Re: [petsc-users] transfer vector data diagonally on DA
On Mon, Apr 2, 2012 at 4:53 PM, khalid ashraf <[email protected]>
wrote:
Thanks Matt. It works now.
I have another problem. I am writing this program.
1) Always test on 2 procs once
2) I have no idea what you actually want to do. However, it really simple to just print out the indices you are using in row[] and the values in bodyFx[] and check that VecSetValues() is working as you expect.
Matt
for (k=zs; k<zs+zm; k++) { for (j=ys; j<ys+ym; j++) { for (i=xs; i<xs+xm; i++) { if ( i!=(mx-1) || j!=my-1 || k!=mz-1) { bodyFx[0]=1; bodyFx[1]=-0.3; bodyFx[2]=-0.3; bodyFx[3]=-0.3; row[0]=k*mx*my+j*mx+i; row[1]=k*mx*my+j*mx+i+1; row[2]=k*mx*my+(j+1)*mx+i; row[3]=(k+1)*mx*my+j*mx+i; VecSetValues(fx_test,4,row,bodyFx,ADD_VALUES); } VecAssemblyBegin(fx_test); VecAssemblyEnd(fx_test);
Output: Print fx_test
Here fx_test is a global vector. This program gives correct output on 1 proc. but on 4 processors, the program runs but the outputs are wrong.
I also tried for (k=zs; k<zs+zm; k++) { for (j=ys; j<ys+ym; j++) { for (i=xs; i<xs+xm; i++) { if ( i!=(mx-1) || j!=my-1 || k!=mz-1) { bodyFx[0]=1; bodyFx[1]=-0.3; bodyFx[2]=-0.3; bodyFx[3]=-0.3; fx1_localptr[k][j][i+1]+=bodyFx[4]; fx1_localptr[k][j+1][i]+=bodyFx[3]; fx1_localptr[k+1][j][i]+=bodyFx[1]; fx1_localptr[k][j][i]+=bodyFx[0]; } Here, fx1_localptr is an array from a local vector that is communicated from the global vector fx_test. In this case as well, I get similar errors.
Thanks. Khalid
Some final numbers of the vector with 8*8*6 grid, 4 proc, -2.9999999999999999e-01 -2.9999999999999999e-01 -2.9999999999999999e-01 -2.9999999999999999e-01 1.0000000000000009e-01 1.0000000000000009e-01 1.0000000000000009e-01 -2.9999999999999999e-01 -2.9999999999999999e-01 -2.9999999999999999e-01 -2.9999999999999999e-01 0.0000000000000000e+00 4.0000000000000002e-01 1.0000000000000009e-01 1.0000000000000009e-01 1.0000000000000009e-01 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 1.0000000000000009e-01 1.0000000000000009e-01 1.0000000000000009e-01 -2.9999999999999999e-01 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
Some final numbers of the vector with 8*8*6 grid, 1 proc, 0.0000000000000000e+00 -2.9999999999999999e-01 -2.9999999999999999e-01 -2.9999999999999999e-01 -2.9999999999999999e-01 -2.9999999999999999e-01 -2.9999999999999999e-01 -2.9999999999999999e-01 0.0000000000000000e+00 -2.9999999999999999e-01 -2.9999999999999999e-01 -2.9999999999999999e-01 -2.9999999999999999e-01 -2.9999999999999999e-01 -2.9999999999999999e-01 -2.9999999999999999e-01 0.0000000000000000e+00 -2.9999999999999999e-01 -2.9999999999999999e-01 -2.9999999999999999e-01 -2.9999999999999999e-01 -2.9999999999999999e-01 -2.9999999999999999e-01 -2.9999999999999999e-01 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
From: Matthew Knepley <[email protected]> To: khalid ashraf <[email protected]>; PETSc users list < [email protected]> Sent: Monday, April 2, 2012 5:41 AM Subject: Re: [petsc-users] transfer vector data diagonally on DA
On Mon, Apr 2, 2012 at 3:36 AM, khalid ashraf <[email protected]> wrote: Hi Jed,
I am using petsc/3.1 and the include file "petscdmda.h" is not working. I am using the "petscda.h" and DACreate3D. unew_localptr[][][] is from a global vector using VecGetArray u_localptr[][][] is from a local vector communicated from a global vector. I tried unew_localptr[k][j][i]=u_localptr[k-1][j-1][i-1] but it gives Segmentation Violation error.
I would guess (because you provide almost no information about what you are doing), that this is a domain on the edge. If you truly mean to have a periodic domain, you must set that in the creation call. Then the local vector will also ghost regions outside the domain boundary.
Matt
I also tried unew_localptr[k][j][i]=u_localptr[k][j+1][i+1] which works but unew_localptr[k][j][i]=u_localptr[k+1][j+1][i+1] or unew_localptr[k][j][i]=u_localptr[k][j-1][i-1] does not work. I am running the program on4 processors.
Thanks.
ierr = DACreate3d(PETSC_COMM_WORLD,DA_YPERIODIC,DA_STENCIL_BOX,appctx.l,appctx.m,appctx.n,
PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,PETSC_NULL,PETSC_NULL,PETSC_NULL,
&appctx.da);CHKERRQ(ierr); for (k=zs; k<zs+zm; k++) { for (j=ys; j<ys+ym; j++) { for (i=xs; i<xs+xm; i++) {
if(i!=0 || j!=0 || k!=0|| i!=mx-1 || j!=my-1 || k!=mz-1)
unew_localptr[k][j][i]=u_localptr[k+1][j+1][i+1]; }}} From: Jed Brown <[email protected]> To: khalid ashraf <[email protected]>; PETSc users list <
Sent: Sunday, April 1, 2012 10:07 PM Subject: Re: [petsc-users] transfer vector data diagonally on DA
On Sun, Apr 1, 2012 at 22:01, khalid ashraf <[email protected]> wrote: I want to transfer vector data diagonally in the DA grid like for (k=zs; k<zs+zm; k++) { for (j=ys; j<ys+ym; j++) { for (i=xs; i<xs+xm; i++) { if(i!=mx-1 || j!=my-1 || k!=mz-1){ u_new[k+1][j+1][i+1]=u[k][j][i];} }}}
Could you please suggest the best way to do it minimizing interprocessor assignments.
Both are on the same DMDA?
Communicate U to Ulocal (DMGlobalToLocalBegin/End) using a BOX stencil with width at least 1, get the global array u_new[][][] from UGlobalNew and the local arrays u[][][] from Ulocal, then assign u_new[k][j][i] = u[k-1][j-1][i-1].
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener