February 2012 - petsc-users - mailman3.cels.anl.gov

Re: [petsc-users] access to matnest block (0,1) ?
by Klaij, Christiaan 11 Feb '12

11 Feb '12

> I should add a MatNestGetISs() so you can get out the automatically-created > ISs. They will have the structure described below. If you want to work with > the released version, you should create the ISs yourself. As a user, having MatNestGetISs would be most convenient. > A has a contigous distribution, so the ISs must respect that. Did you try > creating the index sets described above? Please explain "something's wrong". Finally I understand the relation between the nesting and the index sets. After replacing my incorrect ISs with the correct ones, everything's fine (no more [0]PETSC ERROR: Arguments are incompatible! [0]PETSC ERROR: Could not find index set!). Thanks a lot for your help, Jed, I really appreciate it. dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:[email protected] T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl

2 1

Re: [petsc-users] TSSetPostStep
by Konstantinos Kontzialis 11 Feb '12

11 Feb '12

Dear Jed, I use TSSolve() and version 3.2-p5. Kostas

2 1

TSSetPostStep
by Konstantinos Kontzialis 11 Feb '12

11 Feb '12

Dear all, I use the TSSetPostStep function to apply a function at the end of each time step. I code the following: ierr = TSCreate(sys.comm, &sys.ts); CHKERRQ(ierr); ierr = TSSetApplicationContext(sys.ts, &sys); CHKERRQ(ierr); ierr = TSSetProblemType(sys.ts, TS_NONLINEAR); CHKERRQ(ierr); ierr = TSSetSolution(sys.ts, sys.gsv); CHKERRQ(ierr); ierr = TSSetPostStep(sys.ts, limiter_implicit); CHKERRQ(ierr); ierr = TSSetIFunction(sys.ts, sys.gres0, base_residual_implicit, &sys); CHKERRQ(ierr); ierr = TSGetSNES(sys.ts, &sys.snes); CHKERRQ(ierr); ierr = MatCreateSNESMF(sys.snes, &sys.J); CHKERRQ(ierr); and in the function : limiter_implicit set in TSSetPostStep I just put the command: puts("out\n"). But nothing prints on the the output at the end of each step. Any suggestions? Kostas

2 1

PetscViewerASCIISynchronizedPrintf
by Konstantinos Kontzialis 10 Feb '12

10 Feb '12

Dear all, I am trying to print data from all the processes in a single file and I use the following command ierr = PetscViewerASCIIOpen(sys.comm, "cp.dat", &viewer1); CHKERRQ(ierr); ierr = PetscViewerASCIISynchronizedAllow(viewer1, PETSC_TRUE); CHKERRQ(ierr); . . /* Some user operations */ . . . ierr = PetscViewerASCIISynchronizedPrintf(viewer1, "%le\t%le\t%le\n", cp, x[0], x[1]); CHKERRQ(ierr); ierr = PetscViewerFlush(viewer1); CHKERRQ(ierr); but at run time the code hangs. What should I do? Kostas

3 2

VECMPICUSP with ghosted vector
by Fredrik Heffer Valdmanis 10 Feb '12

10 Feb '12

Hi, In FEniCS, we use ghosted vectors for parallel computations, with the functions VecCreateGhost VecGhostGetLocalForm As I am integrating the new GPU-based vectors and matrices in FEniCS, I want the ghosted vectors to be of type VECMPICUSP. I have tried to do this by calling VecSetType after creating the vector, but that makes VecGhostGetLocalForm give an error. Is it possible to set the type to be mpicusp when using ghost vectors, without changing much of the code? If so, how? If not, how would you recommend I proceed to work with mpicusp vectors in this context? Thanks! -- Fredrik

2 2

VecScatter Question
by Mohammad Mirzadeh 09 Feb '12

09 Feb '12

Hi guys, I'm just wondering if I understand how the VecScatter works. Considering (petsc 3.2-p6 manual page 53): VecScatterCreate(Vec x,IS ix,Vec y,IS iy,VecScatter *ctx); VecScatterBegin(VecScatter ctx,Vec x,Vec y,INSERT VALUES,SCATTER FORWARD); VecScatterEnd(VecScatter ctx,Vec x,Vec y,INSERT VALUES,SCATTER FORWARD); VecScatterDestroy(VecScatter *ctx); is the following statement correct? VecScatter looks into "ix" and "iy" index sets and `matches' the global indecies between the two to copy data from vector "x" to vector "y". For example, if "ix" maps local index "1" to global index "10", VecScatter looks inside "iy" to find a local index that is mapped to global index "10" and sends the data accordingly to the correct processor. Thanks, Mohammad

2 5

Is sparsity of Vec exploited in MatMult?
by Hui Zhang 09 Feb '12

09 Feb '12

Hi, Is sparsity of Vec exploited in MatMult? I'm implementing some domain decomposition methods. The residual is non-zero or significant only at very few nodes. Does MatMult knows that most entries of the Vec are zeros? Is there a threshold to regard an value as zero? Thanks, Hui

3 2

code changes from CPU to GPU
by recrusader 08 Feb '12

08 Feb '12

Dear PETSc developers, I have FEM codes. It works very well in CPU computation. Now, I add '-vec_type seqcusp -mat_type aijcusp' when running the codes, I met the following errors. My question that whether there is an example to demonstrate how to revise the codes for the conversion from CPU to GPU. In addition, 'seqcusp' and 'seqaijcusp' are used when vec and mat are saved in one GPU card? Thank you very much. [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Invalid argument! [0]PETSC ERROR: Vector type seqcusp does not have local representation! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Release Version 3.2.0, Patch 3, Fri Sep 30 10:28:33 CDT 2011 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: /work/01820/ylu/libmesh_svn01232012/examples/myproj_sp1/myproj-opt on a westmere- named c300-205.ls4.tacc.utexas.edu by ylu Wed Feb 8 14:34:46 2012 [0]PETSC ERROR: Libraries linked from /opt/apps/intel11_1/mvapich2_1_6/petsc/3.2/westmere-cuda/lib [0]PETSC ERROR: Configure run at Fri Dec 16 11:27:43 2011 [0]PETSC ERROR: Configure options --with-x=0 -with-pic --with-external-packages-dir=/var/tmp/petsc-3.2-buildroot//opt/apps/intel11_1/mvapich2_1_6/petsc/3.2/externalpackages-cuda --with-mpi-compilers=1 --with-mpi-dir=/opt/apps/intel11_1/mvapich2/1.6 --with-scalar-type=real --with-dynamic-loading=0 --with-shared-libraries=0 --with-chaco=1 --download-chaco=/home1/0000/build/rpms/SOURCES/petsc-externalpackages/Chaco-2.2.tar.gz --with-spai=1 --download-spai=/home1/0000/build/rpms/SOURCES/petsc-externalpackages/spai_3.0-mar-06.tar.gz --with-hypre=1 --download-hypre=/home1/0000/build/rpms/SOURCES/petsc-externalpackages/hypre-2.6.0b.tar.gz --with-mumps=1 --download-mumps=/home1/0000/build/rpms/SOURCES/petsc-externalpackages/MUMPS_4.9.2.tar.gz --with-scalapack=1 --download-scalapack=/home1/0000/build/rpms/SOURCES/petsc-externalpackages/scalapack.tgz --with-blacs=1 --download-blacs=/home1/0000/build/rpms/SOURCES/petsc-externalpackages/blacs-dev.tar.gz --with-spooles=1 --download-spooles=/home1/0000/build/rpms/SOURCES/petsc-externalpackages/spooles-2.2-dec-2008.tar.gz --with-superlu=1 --download-superlu=/home1/0000/build/rpms/SOURCES/petsc-externalpackages/SuperLU_4.1-December_20_2010.tar.gz --with-superlu_dist=1 --download-superlu_dist=/home1/0000/build/rpms/SOURCES/petsc-externalpackages/SuperLU_DIST_2.5-December_21_2010.tar.gz --with-parmetis=1 --download-parmetis=/home1/0000/build/rpms/SOURCES/petsc-externalpackages/ParMetis-dev-p3.tar.gz --with-debugging=no --with-blas-lapack-dir=/opt/apps/intel/11.1/mkl/lib/em64t --with-mpiexec=mpirun_rsh --with-cuda=1 --with-cuda-dir=/opt/apps/cuda/4.0/cuda/ --with-cusp-dir=/opt/apps/cuda/4.0/cuda/ --with-thrust-dir=/opt/apps/cuda/4.0/cuda/ --COPTFLAGS=-xW --CXXOPTFLAGS=-xW --FOPTFLAGS=-xW [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: VecGhostGetLocalForm() line 82 in src/vec/vec/impls/mpi/commonmpvec.c [0]PETSC ERROR: zero() line 974 in "unknowndirectory/"/work/01820/ylu/libmesh_svn01232012/include/numerics/petsc_vector.h application called MPI_Abort(comm=0x84000002, 62) - process 0 Best, Yujie

3 4

TS question
by Boyce Griffith 08 Feb '12

08 Feb '12

Hi, Folks -- I was wondering if there was a reference for the time-stepping method implemented by TSARKIMEX2E? Thanks! -- Boyce

2 1

Hang at PetscLayoutSetUp()
by Derek Gaston 08 Feb '12

08 Feb '12

Hello all, I'm running some largish finite element calculations at the moment (50 Million to 400 Million DoFs on up to 10,000 processors) using a code based on PETSc (obviously!) and while most of the simulations are working well, every now again I seem to run into a hang in the setup phase of the simulation. I've attached GDB several times and it seems to alway be hanging in PetscLayoutSetUp() during matrix creation. Here is the top of a stack trace showing what I mean: #0 0x00002aac9d86cef2 in opal_progress () from /apps/local/openmpi/1.4.4/intel-12.1.1/opt/lib/libopen-pal.so.0 #1 0x00002aac9d16a0c4 in ompi_request_default_wait_all () from /apps/local/openmpi/1.4.4/intel-12.1.1/opt/lib/libmpi.so.0 #2 0x00002aac9d1da9ee in ompi_coll_tuned_sendrecv_actual () from /apps/local/openmpi/1.4.4/intel-12.1.1/opt/lib/libmpi.so.0 #3 0x00002aac9d1e2716 in ompi_coll_tuned_allgather_intra_bruck () from /apps/local/openmpi/1.4.4/intel-12.1.1/opt/lib/libmpi.so.0 #4 0x00002aac9d1db439 in ompi_coll_tuned_allgather_intra_dec_fixed () from /apps/local/openmpi/1.4.4/intel-12.1.1/opt/lib/libmpi.so.0 #5 0x00002aac9d1827e6 in PMPI_Allgather () from /apps/local/openmpi/1.4.4/intel-12.1.1/opt/lib/libmpi.so.0 #6 0x0000000000508184 in PetscLayoutSetUp () #7 0x00000000005b9f39 in MatMPIAIJSetPreallocation_MPIAIJ () #8 0x00000000005c1317 in MatCreateMPIAIJ () As you can see, I'm currently using openMPI (even though I do have access to others) along with the intel compiler (this is a mostly C++ code). This problem doesn't exhibit itself on any smaller problems (we run TONS of runs all the time in the 10,000-5,000,000 DoF range on 1-3000 procs) and only seems to come up on these larger runs. I'm starting to suspect that it's an openMPI issue. Has anyone seen anything like this before? Here are some specs for my current environment PETSc 3.1-p8 (I know, I know....) OpenMPI 1.4.4 intel compilers 12.1.1 Modified Redhat with 2.6.18 Kernel QDR Infiniband Thanks for any help! Derek

5 11