March 2012 - petsc-dev - mailman3.cels.anl.gov

cuda 4.1 errors with sm_20
by Satish Balay 27 Mar '12

27 Mar '12

I get errors with building petsc using cuda-4.1 with 'sm_20' on linux. Reverting to cuda-4.0 [with cusp-v0.3.1, thrust-1.6.0] overcomes this issue. >From http://developer.download.nvidia.com/compute/cuda/4_1/rel/toolkit/docs/CUDA… * For target architecture sm_2x, a new compiler component 'cicc' is used instead of 'nvopencc'. So the new compiler used for sm20 - 'cicc' is perhaps buggy. Matt,Sean or Barry, Do you see the same behavior on Mac? Satish --------------------------- balay@bb30:~/petsc-dev/src/mat/impls/aij/seq/seqcusp>nvcc -g -arch=sm_20 -c --compiler-options="-Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -fno-inline -O0 -I/home/balay/petsc-dev/include -I/home/balay/petsc-dev/arch-cuda-double/include -I/usr/local/cuda/include -I/home/balay/petsc-dev/arch-cuda-double/include/txpetscgpu/include -I/usr/local/cusp-v0.3.1/ -D__INSDIR__=src/mat/impls/aij/seq/seqcusp/" aijcusp.cu <built-in>(2): error: "__STDC_HOSTED__" is predefined; attempted redefinition ignored <built-in>(8): error: "__WCHAR_TYPE__" is predefined; attempted redefinition ignored <built-in>(115): error: "__x86_64" is predefined; attempted redefinition ignored <built-in>(116): error: "__x86_64__" is predefined; attempted redefinition ignored <built-in>(126): error: "__linux__" is predefined; attempted redefinition ignored <built-in>(129): error: "__unix__" is predefined; attempted redefinition ignored <built-in>(130): error: "unix" is predefined; attempted redefinition ignored 7 errors detected in the compilation of "/tmp/tmpxft_00003334_00000000-7_aijcusp.cpp3.i".

1 0

Re: [petsc-dev] GASM
by Dmitry Karpeev 27 Mar '12

27 Mar '12

Mark, You should pass your subdomain ISs through the first IS list argument, and leave the second as PETSC_NULL. In that case GASM assumes that the user passed in a nonoverlapping decomposition through the first list. This is the behavior that GASM inherited from ASM and it isn't very well (or at all) documented in either. Let me know if that doesn't fix the problem. Note also that you don't have to explicitly set overlap to 0: if the user sets not overlap PETSc won't try to increase it, and if you set overlap to 0 it will not try to adjust the subdomains to make sure the overlap is 0. Both ASM and GASM simply assume that the user does the right thing and passes in a nonoverlapping decomposition. Incidentally, it may be time to change this API. I'm trying to debug a bad convergence problem with GASM -- inexplicably, it converges slower than ASM with a single process and multiple subdomains -- so now is a good time to mess with the code, while I still remember what's what. Another thing I've considered doing is reordering the subdomains internally (on demand only), to ensure a deadlock-free ordering of the subcomms. Dmitry. On Tue, Mar 27, 2012 at 6:28 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > This is why petsc-dev is better than a random list of email addresses. > No one on your random list uses GASM > > > > Barry > > On Mar 27, 2012, at 4:49 PM, Mark F. Adams wrote: > > > I'm trying to make general non-overlapping domains in GASM. I create a > vector of IS, each with the global indices of the equations in each block. > > > > I use code that looks like: > > > > ierr = PCSetType( subpc, PCGASM ); CHKERRQ(ierr); > > ierr = PCGASMSetLocalSubdomains( subpc, sz, PETSC_NULL, is ); > CHKERRQ(ierr); > > ierr = PetscFree( is ); CHKERRQ(ierr); > > ierr = PCGASMSetOverlap( subpc, 0 ); CHKERRQ(ierr); > > > > In parallel, I get errors like this: > > > > [1]PETSC ERROR: gis_local contains -85 indices outside of gis! > > <snip> > > [1]PETSC ERROR: PCSetUp_GASM() line 332 in > /Users/markadams/Codes/petsc-dev/src/ksp/pc/impls/gasm/gasm.c > > > > Anyone have any ideas on what I'm doing wrong here? I've checked the > input and I think it is fine, > > > > Mark > >

1 0

Re: [petsc-dev] GASM
by Barry Smith 27 Mar '12

27 Mar '12

This is why petsc-dev is better than a random list of email addresses. No one on your random list uses GASM Barry On Mar 27, 2012, at 4:49 PM, Mark F. Adams wrote: > I'm trying to make general non-overlapping domains in GASM. I create a vector of IS, each with the global indices of the equations in each block. > > I use code that looks like: > > ierr = PCSetType( subpc, PCGASM ); CHKERRQ(ierr); > ierr = PCGASMSetLocalSubdomains( subpc, sz, PETSC_NULL, is ); CHKERRQ(ierr); > ierr = PetscFree( is ); CHKERRQ(ierr); > ierr = PCGASMSetOverlap( subpc, 0 ); CHKERRQ(ierr); > > In parallel, I get errors like this: > > [1]PETSC ERROR: gis_local contains -85 indices outside of gis! > <snip> > [1]PETSC ERROR: PCSetUp_GASM() line 332 in /Users/markadams/Codes/petsc-dev/src/ksp/pc/impls/gasm/gasm.c > > Anyone have any ideas on what I'm doing wrong here? I've checked the input and I think it is fine, > > Mark

1 0

Re: [petsc-dev] Who broke the GPU build?
by Satish Balay 27 Mar '12

27 Mar '12

On Tue, 27 Mar 2012, Matthew Knepley wrote: > On Tue, Mar 27, 2012 at 8:31 AM, Matthew Knepley <knepley(a)gmail.com> wrote: > > > On Tue, Mar 27, 2012 at 8:27 AM, Matthew Knepley <knepley(a)gmail.com>wrote: > > > >> I update my GPU machine today by > >> > >> - Pulling BuildSystem, petsc-dev, thrust, cusp > >> - Reconfiguring > >> - Recompiling > >> > >> and none of the Thrust or Cusp things will build because VecType is > >> again messed up. I look in the headers and the #undef VecType seems > >> to be missing. Where did it go? and aren't we testing this? > >> > > > > If I add #undef VecType to seqcusp/cuspvecimpl.h I get a working build. > > What the hell is going on here??? > > > > Okay, this runs on my laptop, but not on this desktop. Is this an > incompatibility with the > Thrust repo, meaning we can only use Thrust embedded in the CUDA distro? No idea about the version of thrust you have - but I've just upgraded the cuda test machine to use cuda-4.1 [with builtin thrust] - and the release cusp-v0.3.1 - and the build went through fine. And I had earlier [a couple of months ago?] noticed that latest thrust-dev [or was it cusp-dev] of that time diverged such that petsc-dev would no longer build - and I had to revert to an earlier working dev snapshots of these packages. Satish

1 0

Re: [petsc-dev] Who broke the GPU build?
by Matthew Knepley 27 Mar '12

27 Mar '12

On Tue, Mar 27, 2012 at 8:31 AM, Matthew Knepley <knepley(a)gmail.com> wrote: > On Tue, Mar 27, 2012 at 8:27 AM, Matthew Knepley <knepley(a)gmail.com>wrote: > >> I update my GPU machine today by >> >> - Pulling BuildSystem, petsc-dev, thrust, cusp >> - Reconfiguring >> - Recompiling >> >> and none of the Thrust or Cusp things will build because VecType is >> again messed up. I look in the headers and the #undef VecType seems >> to be missing. Where did it go? and aren't we testing this? >> > > If I add #undef VecType to seqcusp/cuspvecimpl.h I get a working build. > What the hell is going on here??? > Okay, this runs on my laptop, but not on this desktop. Is this an incompatibility with the Thrust repo, meaning we can only use Thrust embedded in the CUDA distro? Matt > Matt > > >> Matt >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] Who broke the GPU build?
by Matthew Knepley 27 Mar '12

27 Mar '12

On Tue, Mar 27, 2012 at 8:27 AM, Matthew Knepley <knepley(a)gmail.com> wrote: > I update my GPU machine today by > > - Pulling BuildSystem, petsc-dev, thrust, cusp > - Reconfiguring > - Recompiling > > and none of the Thrust or Cusp things will build because VecType is > again messed up. I look in the headers and the #undef VecType seems > to be missing. Where did it go? and aren't we testing this? > If I add #undef VecType to seqcusp/cuspvecimpl.h I get a working build. What the hell is going on here??? Matt > Matt > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Who broke the GPU build?
by Matthew Knepley 27 Mar '12

27 Mar '12

I update my GPU machine today by - Pulling BuildSystem, petsc-dev, thrust, cusp - Reconfiguring - Recompiling and none of the Thrust or Cusp things will build because VecType is again messed up. I look in the headers and the #undef VecType seems to be missing. Where did it go? and aren't we testing this? Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] buildsystem: cusp location
by Barry Smith 26 Mar '12

26 Mar '12

On Mar 25, 2012, at 10:31 PM, Jed Brown wrote: > I'm for strict, precise semantics over fuzzy permissive semantics. I don't think being permissive is likely to cause harm, so I'm fine with checking both. > > I just don't want the extra /include to be a required part because it breaks consistency with other packages (none of which require the extra /include). Given our --with-package-dir model of checking all kinds of possibilities I don't see why that extra include would be required part? Isn't it just one possibility? What is the argument about, just have cusp.py check more possibilities. Barry > > On Mar 25, 2012 9:20 PM, "Barry Smith" <bsmith(a)mcs.anl.gov> wrote: > > So you are arguing we should make life harder for users? > > --with-package-dir=dir currently has the model "look for any way we know about" for installs underneath dir. > > Why is this a bad model? Why do you advocate something much more limited? > > Who agrees with Sean that we should cripple --with-package-dir=dir? Expanding it sounds like a better idea. > > Barry > > > > On Mar 25, 2012, at 4:56 PM, Sean Farley wrote: > > > > > > > Yes, I think it would be far simpler to say: > > > > --with-package-dir = prefix install > > --with-package-{include,lib}dir = non-prefix type of package > > > > And then we'd have less ambiguity. Packages such as MKL are already exceptions to whatever is defined currently, so there'd be no more code to add to handle this, i.e. MKL would remain an exception in the new model. > > > > Sure better errors are useful. This is independent of the > > package-dir=prefix issue. [so you can add this stuff now.] > > > > That's true. > > > > apt-get install should already work. And --with-package-dir is > > generally for manual installs. > > > > apt-get only works because /usr/local is listed as a default search path. Perhaps gcc should have been configured to add its own prefix but that would have to be an issue taken up with each package manager. >

1 0

Re: [petsc-dev] getting norm type or pc side after resetting ksp type
by Matthew Knepley 26 Mar '12

26 Mar '12

It looks like when we change a KSP type, we need to reset the PC_SIDE and KSP_NORM back to the defaults. Is there an argument against this? Matt On Mon, Mar 26, 2012 at 10:02 AM, Lisandro Dalcin <dalcinl(a)gmail.com> wrote: > Please look at the following example. The error is really unexpected > for me, but I'm not sure how to fix it: > > In [1]: from petsc4py import PETSc > > In [2]: ksp = PETSc.KSP().create() > > In [3]: ksp.setType('gmres') > > In [4]: ksp.getNormType() > Out[4]: 1 > > In [5]: ksp.setType('fgmres') # changing ksp type > > In [6]: ksp.getNormType() > --------------------------------------------------------------------------- > Error Traceback (most recent call last) > /Users/dalcinl/Devel/PetIGA/demo/<ipython-input-6-40949252afeb> in > <module>() > ----> 1 ksp.getNormType() > > > /Users/dalcinl/Library/Python/2.7/lib/python/site-packages/petsc4py/lib/arch-darwin-c-debug/PETSc.so > in petsc4py.PETSc.KSP.getNormType (src/petsc4py.PETSc.c:106669)() > > Error: error code 56 > [0] KSPGetNormType() line 317 in > /Users/dalcinl/Devel/petsc-3.2/src/ksp/ksp/interface/itcreate.c > [0] KSPSetUpNorms_Private() line 284 in > /Users/dalcinl/Devel/petsc-3.2/src/ksp/ksp/interface/itcreate.c > [0] No support for this operation for this object type > [0] KSP fgmres does not support PRECONDITIONED with LEFT > > > > -- > Lisandro Dalcin > --------------- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] getting norm type or pc side after resetting ksp type
by Lisandro Dalcin 26 Mar '12

26 Mar '12

Forgot to mention that this is with petsc-3.2 On 26 March 2012 18:02, Lisandro Dalcin <dalcinl(a)gmail.com> wrote: > Please look at the following example. The error is really unexpected > for me, but I'm not sure how to fix it: > > In [1]: from petsc4py import PETSc > > In [2]: ksp = PETSc.KSP().create() > > In [3]: ksp.setType('gmres') > > In [4]: ksp.getNormType() > Out[4]: 1 > > In [5]: ksp.setType('fgmres') # changing ksp type > > In [6]: ksp.getNormType() > --------------------------------------------------------------------------- > Error Traceback (most recent call last) > /Users/dalcinl/Devel/PetIGA/demo/<ipython-input-6-40949252afeb> in <module>() > ----> 1 ksp.getNormType() > > /Users/dalcinl/Library/Python/2.7/lib/python/site-packages/petsc4py/lib/arch-darwin-c-debug/PETSc.so > in petsc4py.PETSc.KSP.getNormType (src/petsc4py.PETSc.c:106669)() > > Error: error code 56 > [0] KSPGetNormType() line 317 in > /Users/dalcinl/Devel/petsc-3.2/src/ksp/ksp/interface/itcreate.c > [0] KSPSetUpNorms_Private() line 284 in > /Users/dalcinl/Devel/petsc-3.2/src/ksp/ksp/interface/itcreate.c > [0] No support for this operation for this object type > [0] KSP fgmres does not support PRECONDITIONED with LEFT > > > > -- > Lisandro Dalcin > --------------- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169 -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

1 0