Ah, the 2D case provides an error message for this case, but not the 3d. [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: Local y-width of domain y 1 is smaller than stencil width s 2! [0]PETSC ERROR: The problem is less about the 1-node-wide domain with 1 proc in that dimension, and more about the case where there are more than one procs in that dimension but less than the stencil size local unknowns. To handle this, the scatter would have to look two proc neighbors away, not just one. This has never been handled by PETSc. Consider the z-case with local size zl and stencil size s... I supposed one could check for the special case where P, the number of procs, in the z-dimension is 1 and s > zl. In that case, DMDA_BOUNDARY_GHOSTED and DMDA_BOUNDARY_NONE both work (as evidenced by the 3D code). Then, it need only error if (s > zl) and ((P > 1) or (zwrap == DMDA_BOUNDARY_PERIODIC)) Alternatively, DMDA_BOUNDARY_PERIODIC could be made to work with more if statements to set up the indices for the scatter, doing the wrap around of the domain multiple times. Not sure how much work this would be. At any rate, I'll fix the error message in 2D and add it to 3D to only error in the cases that actually currently break. Doing DMDA_BOUNDARY_PERIODIC with P=1 and s > zl (the multiple-wraps around the domain case) will take more thought... Ethan On Tue, 2011-04-19 at 21:25 +0200, Jed Brown wrote:
On Tue, 19 Apr 2011 09:18:44 -0600, Ethan Coon <[email protected]> wrote:
Can you shoot me your example? I run 1-node z direction both with and without periodic BCs, and with and without ghost nodes, without a valgrind complaint in DMDACreate3D.
See attached very simple test case. It only crashes for stencil width greater than 1 (the original support request needs stencil width of 2 for TVD FV methods),
mpiexec -n 1 ./a.out # stencil width 2 in code
[0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: Negative index -100 at 0 location! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development HG revision: 335e836c7798747e6242df554ae58fe4db9f803f HG Date: Fri Apr 15 16:04:07 2011 -0500 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./a.out on a ompi named kunyang by jed Tue Apr 19 21:22:07 2011 [0]PETSC ERROR: Libraries linked from /home/jed/petsc/ompi/lib [0]PETSC ERROR: Configure run at Sat Apr 16 13:36:39 2011 [0]PETSC ERROR: Configure options --download-blacs --download-chaco --download-hypre=externalpackages/hypre-2.7.0b.tar.gz --download-ml --download-mum ps --download-parms --download-scalapack --download-scotch --download-spai --download-spooles --download-sundials --download-superlu --download-superl u_dist --with-blas-lapack-dir=/usr --with-c2html --with-cholmod-dir=/usr --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodet ect=0 --with-hdf5-dir=/usr --with-java-dir=/usr --with-lgrind --with-mpi-dir=/usr --with-parmetis-dir=/usr --with-shared-libraries --with-single-libra ry=0 --with-sowing --with-umfpack-dir=/usr -LIBS="-L/usr/lib/openmpi -lmpi_f90 -lmpi_f77 -lmpi_cxx -lstdc++ -lgfortran" PETSC_ARCH=ompi [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: VecScatterCheckIndices_Private() line 26 in /home/jed/petsc/src/vec/vec/utils/vscat.c [0]PETSC ERROR: VecScatterCreate() line 1106 in /home/jed/petsc/src/vec/vec/utils/vscat.c [0]PETSC ERROR: DMSetUp_DA_3D() line 953 in /home/jed/petsc/src/dm/impls/da/da3.c [0]PETSC ERROR: DMSetUp_DA() line 22 in /home/jed/petsc/src/dm/impls/da/dareg.c [0]PETSC ERROR: DMSetUp() line 189 in /home/jed/petsc/src/dm/interface/dm.c [0]PETSC ERROR: DMDACreate3d() line 1366 in /home/jed/petsc/src/dm/impls/da/da3.c [0]PETSC ERROR: main() line 14 in "unknowndirectory/"da3d.c -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 63.
I still see Valgrind noise for stencil width of 1:
mpiexec -n 2 valgind ./a.out # stencil width 1 in code
==26044== Invalid read of size 2 ==26044== at 0x4C26C2C: memcpy (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==26044== by 0x71D8D01: opal_convertor_pack (in /usr/lib/openmpi/libmpi.so.1.0.1) ==26044== by 0x117600C8: ??? (in /usr/lib/openmpi/openmpi/mca_btl_sm.so) ==26044== by 0x10F421B7: ??? (in /usr/lib/openmpi/openmpi/mca_pml_ob1.so) ==26044== by 0x10F30C0C: ??? (in /usr/lib/openmpi/openmpi/mca_pml_ob1.so) ==26044== by 0x7127650: PMPI_Isend (in /usr/lib/openmpi/libmpi.so.1.0.1) ==26044== by 0x6C5A103: PetscGatherMessageLengths (mpimesg.c:127) ==26044== by 0x697A90A: VecScatterCreate_PtoS (vpscat.c:1623) ==26044== by 0x69564FD: VecScatterCreate (vscat.c:1334) ==26044== by 0x7CB339A: DMSetUp_DA_3D (da3.c:953) ==26044== by 0x7D044BF: DMSetUp_DA (dareg.c:22) ==26044== by 0x7C76A6C: DMSetUp (dm.c:189) ==26044== Address 0xdfeda02 is 2 bytes inside a block of size 8 alloc'd ==26044== at 0x4C23918: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==26044== by 0x6CC9DEF: PetscMallocAlign (mal.c:30) ==26044== by 0x697A5EA: VecScatterCreate_PtoS (vpscat.c:1603) ==26044== by 0x69564FD: VecScatterCreate (vscat.c:1334) ==26044== by 0x7CB339A: DMSetUp_DA_3D (da3.c:953) ==26044== by 0x7D044BF: DMSetUp_DA (dareg.c:22) ==26044== by 0x7C76A6C: DMSetUp (dm.c:189) ==26044== by 0x7CB6D8D: DMDACreate3d (da3.c:1366) ==26044== by 0x400938: main (in /home/jed/lang/c/petsc/a.out)
C source code attachment (da3d.c) #include <petscdmda.h>
int main(int argc, char *argv[]) { DM da; PetscErrorCode ierr;
PetscInitialize(&argc,&argv,0,0); ierr = DMDACreate3d(PETSC_COMM_WORLD, DMDA_BOUNDARY_PERIODIC, DMDA_BOUNDARY_PERIODIC, DMDA_BOUNDARY_PERIODIC, DMDA_STENCIL_STAR, 10, 10, 1, PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE, 1, 1, /* Stencil width of 2 in original request */ PETSC_NULL, PETSC_NULL, PETSC_NULL, &da);CHKERRQ(ierr); ierr = DMDestroy(da);CHKERRQ(ierr); PetscFinalize(); return 0; }
-- ------------------------------------ Ethan Coon Post-Doctoral Researcher Applied Mathematics - T-5 Los Alamos National Laboratory 505-665-8289 http://www.ldeo.columbia.edu/~ecoon/ ------------------------------------