February 2012 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] petsc-dev on bitbucket
by Stefano Zampini 27 Feb '12

27 Feb '12

I just pushed to bitbucket and my write permissions were revoked. 2012/2/27 Jed Brown <jedbrown(a)mcs.anl.gov> > On Mon, Feb 27, 2012 at 15:48, Satish Balay <balay(a)mcs.anl.gov> wrote: > >> Jed, >> >> Perhaps this is a test case for 'pull request' model? [via petsc.cs] >> > > Sure, Stefano (or anyone) can fork on bitbucket, hack away, then send a > pull request. We can merge locally and push to petsc-dev (which will be > mirrored to bitbucket and should close the pull request). > -- Stefano

1 0

Re: [petsc-dev] PETSc GPU capabilities
by Chetan Jhurani 27 Feb '12

27 Feb '12

John, Paul won't have email access till he is back on March 8th. Meanwhile, I will take a look at this issue. Chetan From: petsc-dev-bounces(a)mcs.anl.gov [mailto:[email protected]] On Behalf Of John Fettig Sent: Monday, February 27, 2012 1:48 PM To: For users of the development version of PETSc Subject: Re: [petsc-dev] PETSc GPU capabilities Hi Paul, This is very interesting. I tried building the code with --download-txpetscgpu and it doesn't work for me. It runs out of memory, no matter how small the problem (this is ex2 from src/ksp/ksp/examples/tutorials): mpirun -np 1 ./ex2 -n 10 -m 10 -ksp_type cg -pc_type sacusp -mat_type aijcusp -vec_type cusp -cusp_storage_format csr -use_cusparse 0 terminate called after throwing an instance of 'thrust::system::detail::bad_alloc' what(): std::bad_alloc: out of memory MPI Application rank 0 killed before MPI_Finalize() with signal 6 This example works fine when I build without your gpu additions (and for much larger problems too). Am I doing something wrong? For reference, I'm using CUDA 4.1, CUSP 0.3, and Thrust 1.5.1 John On Fri, Feb 10, 2012 at 5:04 PM, Paul Mullowney <paulm(a)txcorp.com> wrote: Hi All, I've been developing GPU capabilities for PETSc. The development has focused mostly on (1) An efficient multi-GPU SpMV, i.e. MatMult. This is working well. (2) Triangular Solve used in ILU preconditioners; i.e. MatSolve. The performance of this ... is what it is :| This code is in beta mode. Keep that in mind, if you decide to use it. It supports single and double precision, real numbers only! Complex will be supported at some point in the future, but not any time soon. To build with these capabilities, add the following to your configure line. --download-txpetscgpu=yes The capabilities of the SpMV code are accessed with the following 2 command line flags -cusp_storage_format csr (other options are coo (coordinate), ell (ellpack), dia (diagonal). hyb (hybrid) is not yet supported) -use_cusparse (this is a boolean and at the moment is only supported with csr format matrices. In the future, cusparse will work with ell, coo, and hyb formats). Regarding the number of GPUs to run on: Imagine a system with P nodes, N cores per node, and M GPUs per node. Then, to use only the GPUs, I would run with M ranks per node over P nodes. As an example, I have a system with 2 nodes. Each node has 8 cores, and 4 GPUs attached to each node (P=2, N=8, M=4). In a PBS queue script, one would use 2 nodes at 4 processors per node. Each mpi rank (CPU processor) will be attached to a GPU. You do not need to explicitly manage the GPUs, apart from understanding what type of system you are running on. To learn how many devices are available per node, use the command line flag: -cuda_show_devices -Paul

1 0

Re: [petsc-dev] PETSc GPU capabilities
by Nystrom, William D 27 Feb '12

27 Feb '12

FYI, I think Paul is still on vacation through the end of this week. I have also had problems running with sacusp and sacusppoly and having thrust complain about not being able to allocate memory. I had not connected it with building with the txpetscgpu package. Perhaps I'll try to confirm that as well. Its hard for me to imagine doing without the txpetscgpu package right now. The package allows for some definite performance gains in the matvec using the different matrix storage formats. I've seen my matvecs be anywhere from 2x to 5x faster on some problems using the "dia" format versus the default "csr" format. Also, Paul has added some support for running on multiple gpus that I am using. I'm not sure what is available in that area without his package. Thanks, Dave -- Dave Nystrom LANL HPC-5 Phone: 505-667-7913 Email: wdn(a)lanl.gov Smail: Mail Stop B272 Group HPC-5 Los Alamos National Laboratory Los Alamos, NM 87545 ________________________________ From: petsc-dev-bounces(a)mcs.anl.gov [petsc-dev-bounces(a)mcs.anl.gov] on behalf of John Fettig [john.fettig(a)gmail.com] Sent: Monday, February 27, 2012 2:48 PM To: For users of the development version of PETSc Subject: Re: [petsc-dev] PETSc GPU capabilities Hi Paul, This is very interesting. I tried building the code with --download-txpetscgpu and it doesn't work for me. It runs out of memory, no matter how small the problem (this is ex2 from src/ksp/ksp/examples/tutorials): mpirun -np 1 ./ex2 -n 10 -m 10 -ksp_type cg -pc_type sacusp -mat_type aijcusp -vec_type cusp -cusp_storage_format csr -use_cusparse 0 terminate called after throwing an instance of 'thrust::system::detail::bad_alloc' what(): std::bad_alloc: out of memory MPI Application rank 0 killed before MPI_Finalize() with signal 6 This example works fine when I build without your gpu additions (and for much larger problems too). Am I doing something wrong? For reference, I'm using CUDA 4.1, CUSP 0.3, and Thrust 1.5.1 John On Fri, Feb 10, 2012 at 5:04 PM, Paul Mullowney <paulm(a)txcorp.com<mailto:[email protected]>> wrote: Hi All, I've been developing GPU capabilities for PETSc. The development has focused mostly on (1) An efficient multi-GPU SpMV, i.e. MatMult. This is working well. (2) Triangular Solve used in ILU preconditioners; i.e. MatSolve. The performance of this ... is what it is :| This code is in beta mode. Keep that in mind, if you decide to use it. It supports single and double precision, real numbers only! Complex will be supported at some point in the future, but not any time soon. To build with these capabilities, add the following to your configure line. --download-txpetscgpu=yes The capabilities of the SpMV code are accessed with the following 2 command line flags -cusp_storage_format csr (other options are coo (coordinate), ell (ellpack), dia (diagonal). hyb (hybrid) is not yet supported) -use_cusparse (this is a boolean and at the moment is only supported with csr format matrices. In the future, cusparse will work with ell, coo, and hyb formats). Regarding the number of GPUs to run on: Imagine a system with P nodes, N cores per node, and M GPUs per node. Then, to use only the GPUs, I would run with M ranks per node over P nodes. As an example, I have a system with 2 nodes. Each node has 8 cores, and 4 GPUs attached to each node (P=2, N=8, M=4). In a PBS queue script, one would use 2 nodes at 4 processors per node. Each mpi rank (CPU processor) will be attached to a GPU. You do not need to explicitly manage the GPUs, apart from understanding what type of system you are running on. To learn how many devices are available per node, use the command line flag: -cuda_show_devices -Paul

1 0

Re: [petsc-dev] PETSc GPU capabilities
by John Fettig 27 Feb '12

27 Feb '12

It finally finished running through cuda-gdb. Here's a backtrace. new_size=46912574500784 in the call to thrust::detail::vector_base<double, thrust::device_malloc_allocator<double> >::resize looks suspicious. #0 0x0000003e1c832885 in raise () from /lib64/libc.so.6 #1 0x0000003e1c834065 in abort () from /lib64/libc.so.6 #2 0x0000003e284bea7d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6 #3 0x0000003e284bcc06 in ?? () from /usr/lib64/libstdc++.so.6 #4 0x0000003e284bcc33 in std::terminate() () from /usr/lib64/libstdc++.so.6 #5 0x0000003e284bcd2e in __cxa_throw () from /usr/lib64/libstdc++.so.6 #6 0x00002aaaab45ad71 in thrust::detail::backend::cuda::malloc<0u> (n=375300596006272) at malloc.inl:50 #7 0x00002aaaab454322 in thrust::detail::backend::dispatch::malloc<0u> (n=375300596006272) at malloc.h:56 #8 0x00002aaaab453555 in thrust::device_malloc (n=375300596006272) at device_malloc.inl:32 #9 0x00002aaaab46477d in thrust::device_malloc<double> (n=46912574500784) at device_malloc.inl:38 #10 0x00002aaaab461fce in thrust::device_malloc_allocator<double>::allocate ( this=0x7fffffff9880, cnt=46912574500784) at device_malloc_allocator.h:101 #11 0x00002aaaab45ee91 in thrust::detail::contiguous_storage<double, thrust::device_malloc_allocator<double> >::allocate (this=0x7fffffff9880, n=46912574500784) at contiguous_storage.inl:134 #12 0x00002aaaab46ebba in thrust::detail::contiguous_storage<double, thrust::device_malloc_allocator<double> >::contiguous_storage (this=0x7fffffff9880, n=46912574500784) at contiguous_storage.inl:46 #13 0x00002aaaab46cd1e in thrust::detail::vector_base<double, thrust::device_malloc_allocator<double> >::fill_insert (this=0x13623990, position=..., n=46912574500784, x=@0x7fffffff9f18) at vector_base.inl:792 #14 0x00002aaaab46b058 in thrust::detail::vector_base<double, thrust::device_malloc_allocator<double> >::insert (this=0x13623990, position=..., n=46912574500784, x=@0x7fffffff9f18) at vector_base.inl:561 #15 0x00002aaaab4692a3 in thrust::detail::vector_base<double, thrust::device_malloc_allocator<double> >::resize (this=0x13623990, new_size=46912574500784, x=@0x7fffffff9f18) at vector_base.inl:222 #16 0x00002aaaac2c3d9b in cusp::precond::smoothed_aggregation<int, double, thrust::detail::cuda_device_space_tag>::smoothed_aggregation<cusp::csr_matrix<int, double, thrust::detail::cuda_device_space_tag> > (this=0x136182b0, A=..., theta=0) at smoothed_aggregation.inl:210 #17 0x00002aaaac27cf84 in PCSetUp_SACUSP (pc=0x1360f330) at sacusp.cu:76 #18 0x00002aaaac1f0024 in PCSetUp (pc=0x1360f330) at precon.c:832 #19 0x00002aaaabd02144 in KSPSetUp (ksp=0x135d2a00) at itfunc.c:261 #20 0x00002aaaabd0396e in KSPSolve (ksp=0x135d2a00, b=0x135a0fa0, x=0x135a2b50) at itfunc.c:385 #21 0x0000000000403619 in main (argc=17, args=0x7fffffffc538) at ex2.c:217 On Mon, Feb 27, 2012 at 4:48 PM, John Fettig <john.fettig(a)gmail.com> wrote: > Hi Paul, > > This is very interesting. I tried building the code with > --download-txpetscgpu and it doesn't work for me. It runs out of memory, > no matter how small the problem (this is ex2 from > src/ksp/ksp/examples/tutorials): > > mpirun -np 1 ./ex2 -n 10 -m 10 -ksp_type cg -pc_type sacusp -mat_type > aijcusp -vec_type cusp -cusp_storage_format csr -use_cusparse 0 > > terminate called after throwing an instance of > 'thrust::system::detail::bad_alloc' > what(): std::bad_alloc: out of memory > MPI Application rank 0 killed before MPI_Finalize() with signal 6 > > This example works fine when I build without your gpu additions (and for > much larger problems too). Am I doing something wrong? > > For reference, I'm using CUDA 4.1, CUSP 0.3, and Thrust 1.5.1 > > John > > > On Fri, Feb 10, 2012 at 5:04 PM, Paul Mullowney <paulm(a)txcorp.com> wrote: > >> Hi All, >> >> I've been developing GPU capabilities for PETSc. The development has >> focused mostly on >> (1) An efficient multi-GPU SpMV, i.e. MatMult. This is working well. >> (2) Triangular Solve used in ILU preconditioners; i.e. MatSolve. The >> performance of this ... is what it is :| >> This code is in beta mode. Keep that in mind, if you decide to use it. It >> supports single and double precision, real numbers only! Complex will be >> supported at some point in the future, but not any time soon. >> >> To build with these capabilities, add the following to your configure >> line. >> --download-txpetscgpu=yes >> >> The capabilities of the SpMV code are accessed with the following 2 >> command line flags >> -cusp_storage_format csr (other options are coo (coordinate), ell >> (ellpack), dia (diagonal). hyb (hybrid) is not yet supported) >> -use_cusparse (this is a boolean and at the moment is only supported with >> csr format matrices. In the future, cusparse will work with ell, coo, and >> hyb formats). >> >> Regarding the number of GPUs to run on: >> Imagine a system with P nodes, N cores per node, and M GPUs per node. >> Then, to use only the GPUs, I would run with M ranks per node over P nodes. >> As an example, I have a system with 2 nodes. Each node has 8 cores, and 4 >> GPUs attached to each node (P=2, N=8, M=4). In a PBS queue script, one >> would use 2 nodes at 4 processors per node. Each mpi rank (CPU processor) >> will be attached to a GPU. >> >> You do not need to explicitly manage the GPUs, apart from understanding >> what type of system you are running on. To learn how many devices are >> available per node, use the command line flag: >> -cuda_show_devices >> >> -Paul >> > >

1 0

Re: [petsc-dev] petsc-dev on bitbucket
by Jed Brown 27 Feb '12

27 Feb '12

On Mon, Feb 27, 2012 at 15:48, Satish Balay <balay(a)mcs.anl.gov> wrote: > Jed, > > Perhaps this is a test case for 'pull request' model? [via petsc.cs] > Sure, Stefano (or anyone) can fork on bitbucket, hack away, then send a pull request. We can merge locally and push to petsc-dev (which will be mirrored to bitbucket and should close the pull request).

1 0

Re: [petsc-dev] PETSc GPU capabilities
by Jed Brown 27 Feb '12

27 Feb '12

On Mon, Feb 27, 2012 at 15:48, John Fettig <john.fettig(a)gmail.com> wrote: > -use_cusparse Paul, what is going on with these top-level options? This should all be properly namespaced and use the prefix of the object it applies to.

1 0

Re: [petsc-dev] petsc-dev on bitbucket
by Satish Balay 27 Feb '12

27 Feb '12

Stefano, We haven't moved to bitbucket yet. [it was a temprary host until the primary server could come back up. We've since then checked if we could migrate permanantly - but hit some snags]. Jed, Perhaps this is a test case for 'pull request' model? [via petsc.cs] Satish On Mon, 27 Feb 2012, Stefano Zampini wrote: > Hi, > > I need to push some changes to bddc code. Here is my bitbucket name: > > stefano_zampini

1 0

Re: [petsc-dev] PETSc GPU capabilities
by John Fettig 27 Feb '12

27 Feb '12

Hi Paul, This is very interesting. I tried building the code with --download-txpetscgpu and it doesn't work for me. It runs out of memory, no matter how small the problem (this is ex2 from src/ksp/ksp/examples/tutorials): mpirun -np 1 ./ex2 -n 10 -m 10 -ksp_type cg -pc_type sacusp -mat_type aijcusp -vec_type cusp -cusp_storage_format csr -use_cusparse 0 terminate called after throwing an instance of 'thrust::system::detail::bad_alloc' what(): std::bad_alloc: out of memory MPI Application rank 0 killed before MPI_Finalize() with signal 6 This example works fine when I build without your gpu additions (and for much larger problems too). Am I doing something wrong? For reference, I'm using CUDA 4.1, CUSP 0.3, and Thrust 1.5.1 John On Fri, Feb 10, 2012 at 5:04 PM, Paul Mullowney <paulm(a)txcorp.com> wrote: > Hi All, > > I've been developing GPU capabilities for PETSc. The development has > focused mostly on > (1) An efficient multi-GPU SpMV, i.e. MatMult. This is working well. > (2) Triangular Solve used in ILU preconditioners; i.e. MatSolve. The > performance of this ... is what it is :| > This code is in beta mode. Keep that in mind, if you decide to use it. It > supports single and double precision, real numbers only! Complex will be > supported at some point in the future, but not any time soon. > > To build with these capabilities, add the following to your configure line. > --download-txpetscgpu=yes > > The capabilities of the SpMV code are accessed with the following 2 > command line flags > -cusp_storage_format csr (other options are coo (coordinate), ell > (ellpack), dia (diagonal). hyb (hybrid) is not yet supported) > -use_cusparse (this is a boolean and at the moment is only supported with > csr format matrices. In the future, cusparse will work with ell, coo, and > hyb formats). > > Regarding the number of GPUs to run on: > Imagine a system with P nodes, N cores per node, and M GPUs per node. > Then, to use only the GPUs, I would run with M ranks per node over P nodes. > As an example, I have a system with 2 nodes. Each node has 8 cores, and 4 > GPUs attached to each node (P=2, N=8, M=4). In a PBS queue script, one > would use 2 nodes at 4 processors per node. Each mpi rank (CPU processor) > will be attached to a GPU. > > You do not need to explicitly manage the GPUs, apart from understanding > what type of system you are running on. To learn how many devices are > available per node, use the command line flag: > -cuda_show_devices > > -Paul >

1 0

Re: [petsc-dev] petsc-dev on bitbucket
by Stefano Zampini 27 Feb '12

27 Feb '12

Hi, I need to push some changes to bddc code. Here is my bitbucket name: stefano_zampini Thanks, Stefano 2012/2/8 Sean Farley <sean(a)mcs.anl.gov> > "Yoo, i've switched petsc-dev over to bitbucket since petsc.cs.itt.edu is > down pull/push from https://bitbucket.org/petsc/petsc-dev > > Sean" - Barry impersonating me > > Since petsc.cs is down, I don't have everyone's public SSH key. For now, > if you need push access, just sign up for a free bitbucket account (can > also use OpenID) and reply to this email with your bitbucket username. > Barry, Matt, Jed, and Peter have write access already. Hong (hzhang), > Dmitry (karpeev), and Lois (curfman) should have write access as long as > the username in the parenthesis is your correct bitbucket name. > > Also, BuildSystem is here: > > https://bitbucket.org/petsc/buildsystem > > where the same people as above have write access. > > P.S. - To change your current default push url in mercurial, just edit > $PETSC_DIR/.hg/hgrc > > [paths] > default = https://bitbucket.org/petsc/petsc-dev > > or if you uploaded your public SSH key to bitbucket already > > [paths] > default = ssh://[email protected]/petsc/petsc-dev > > More info here: > > http://confluence.atlassian.com/display/BITBUCKET/Using+the+SSH+protocol+wi… > > P.P.S - While we're at it, here is a nice shortcut in mercurial. Place > this in your ~/.hgrc > > [extensions] > schemes = > > [schemes] > petsc = ssh://[email protected]/petsc > > You can now clone like so: `hg clone petsc://petsc-dev` or `hg clone > petsc://buildsystem` > -- Stefano

1 0

Re: [petsc-dev] SNES, Coloring, DM, and TS
by Jed Brown 27 Feb '12

27 Feb '12

On Mon, Feb 27, 2012 at 11:48, Matthew Knepley <knepley(a)gmail.com> wrote: > ? My signature for FormJacobianLocal(DM dm, Vec X, Mat Jac, AppCtx *user) > definitely has a context, which is > required to have DM as the first member. We could require the solver as > the second member. > "My" is not the one in DM base. We are suffering some fragmentation because different DMs implement the local problem differently. I think it would mostly resolve the issues if the DM functions forwarded the SNES and TS on to the user, but that makes a dependency loop (unless that code does not live in DM; we can hack around it with void*). Alternatively, we can make TS and SNES have PushDM() and PopDM() methods so the callback can be (SNES, Vec, Mat, void *ctx), but SNESGetDM() returns the correct DM. Maybe this is too confusing. In any case, if we fix the DM callbacks, I would prefer to get rid of the ability for SNES and TS to hold their own callbacks. We can keep essentially the same user interface (though perhaps the DM argument would become explicit).

1 0