February 2012 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Understanding Some Parallel Results with PETSc
by recrusader 23 Feb '12

23 Feb '12

Dear Dave, Did you run the codes with double precision? Thanks, Yujie On Thu, Feb 23, 2012 at 11:06 AM, Nystrom, William D <wdn(a)lanl.gov> wrote: > I recently ran a couple of test runs with petsc-dev that I do not > understand. I'm running on a test bed > machine that has 4 nodes with two Tesla 2090 gpus per node. Each node is > dual socket and populated > with Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz processors. These are 8 > core processors and so each > node has 16 cores. On the gpu, I'm running with Paul's latest version of > the txpetscgpu package. I'm > running the src/ksp/ksp/examples/tutorials/ex2.c petsc example with > m=n=10000. My objective was > to compare the performance running on 4 nodes using all 8 gpus to that of > running on the same 4 nodes > with all 64 cores. This problem uses about a third of the memory > available on the gpus. I was using cg > with jacobi preconditioning on both the gpu run and the cpu run. What is > puzzling to me is that the cpu > case ran 44x times slower than the gpu case and the big difference was in > the time spend in functions > like VecTDot, VecNorm and VecAXPY. > > Below is a table that summarizes the performance of the main functions > that were using time in the > two runs. Times are in seconds. > > | GPU | CPU | Ratio > ------------------------------------------------------------------------- > MatMult | 450.64 | 5484.7 | 12.17 > ------------------------------------------------------------------------- > VecTDot | 285.35 | 16688.0 | 58.48 > ------------------------------------------------------------------------- > VecNorm | 19.03 | 9058.8 | 476.03 > ------------------------------------------------------------------------- > VecAXPY | 106.88 | 5636.3 | 52.73 > ------------------------------------------------------------------------- > VecAYPX | 53.69 | 85.1 | 1.58 > ------------------------------------------------------------------------- > KSPSolve | 811.95 | 35930.0 | 44.25 > ------------------------------------------------------------------------- > > The ratio of MatMult for CPU versus GPU is what I typically see when I am > comparing a CPU run on > a single core versus a run on a single GPU. Since both runs are > communicating across node via mpi, > I'm puzzled about why the CPU case is so much slower than the GPU case > especially since there is > communication for the MatMult as well. Both runs compute the same final > error norm using the exact > same number of iterations. Do these results make sense to people who > understand the performance > issues of parallel sparse linear solvers much better than I? Or do these > results look abnormal. I had > wondered if part of the performance issue was related to my running 8 > times as many mpi processes > for the CPU case. However, I ran a smaller problem with m=n=1000 and > using 8 mpi processes and > 2 cores per node and I see the same extreme differences in the times spent > in VecTDot, VecNorm > and VecAXPY. > > Here are the command lines I used for the two runs: > > CPU: > > mpirun -np 64 -mca btl self,sm,openib ex2 -m 10000 -n 10000 -ksp_type cg > -ksp_max_it 100000 -pc_type jacobi -log_summary -options_left > > GPU: > > mpirun -np 8 -npernode 2 -mca btl self,sm,openib ex2 -m 10000 -n 10000 > -ksp_type cg -ksp_max_it 100000 -pc_type jacobi -log_summary -options_left > -mat_type aijcusp -vec_type cusp -cusp_storage_format dia > > Thanks, > > Dave > > -- > Dave Nystrom > LANL HPC-5 > Phone: 505-667-7913 > Email: wdn(a)lanl.gov > Smail: Mail Stop B272 > Group HPC-5 > Los Alamos National Laboratory > Los Alamos, NM 87545 > >

1 0

Re: [petsc-dev] Understanding Some Parallel Results with PETSc
by Matthew Knepley 23 Feb '12

23 Feb '12

On Thu, Feb 23, 2012 at 11:06 AM, Nystrom, William D <wdn(a)lanl.gov> wrote: > I recently ran a couple of test runs with petsc-dev that I do not > understand. I'm running on a test bed > machine that has 4 nodes with two Tesla 2090 gpus per node. Each node is > dual socket and populated > with Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz processors. These are 8 > core processors and so each > node has 16 cores. On the gpu, I'm running with Paul's latest version of > the txpetscgpu package. I'm > running the src/ksp/ksp/examples/tutorials/ex2.c petsc example with > m=n=10000. My objective was > to compare the performance running on 4 nodes using all 8 gpus to that of > running on the same 4 nodes > with all 64 cores. This problem uses about a third of the memory > available on the gpus. I was using cg > with jacobi preconditioning on both the gpu run and the cpu run. What is > puzzling to me is that the cpu > case ran 44x times slower than the gpu case and the big difference was in > the time spend in functions > like VecTDot, VecNorm and VecAXPY. > > Below is a table that summarizes the performance of the main functions > that were using time in the > two runs. Times are in seconds. > > | GPU | CPU | Ratio > ------------------------------------------------------------------------- > MatMult | 450.64 | 5484.7 | 12.17 > ------------------------------------------------------------------------- > VecTDot | 285.35 | 16688.0 | 58.48 > ------------------------------------------------------------------------- > VecNorm | 19.03 | 9058.8 | 476.03 > ------------------------------------------------------------------------- > VecAXPY | 106.88 | 5636.3 | 52.73 > ------------------------------------------------------------------------- > VecAYPX | 53.69 | 85.1 | 1.58 > ------------------------------------------------------------------------- > KSPSolve | 811.95 | 35930.0 | 44.25 > ------------------------------------------------------------------------- > > The ratio of MatMult for CPU versus GPU is what I typically see when I am > comparing a CPU run on > a single core versus a run on a single GPU. Since both runs are > communicating across node via mpi, > I'm puzzled about why the CPU case is so much slower than the GPU case > especially since there is > communication for the MatMult as well. Both runs compute the same final > error norm using the exact > same number of iterations. Do these results make sense to people who > understand the performance > issues of parallel sparse linear solvers much better than I? Or do these > results look abnormal. I had > wondered if part of the performance issue was related to my running 8 > times as many mpi processes > for the CPU case. However, I ran a smaller problem with m=n=1000 and > using 8 mpi processes and > 2 cores per node and I see the same extreme differences in the times spent > in VecTDot, VecNorm > and VecAXPY. > > Here are the command lines I used for the two runs: > > CPU: > > mpirun -np 64 -mca btl self,sm,openib ex2 -m 10000 -n 10000 -ksp_type cg > -ksp_max_it 100000 -pc_type jacobi -log_summary -options_left > > GPU: > > mpirun -np 8 -npernode 2 -mca btl self,sm,openib ex2 -m 10000 -n 10000 > -ksp_type cg -ksp_max_it 100000 -pc_type jacobi -log_summary -options_left > -mat_type aijcusp -vec_type cusp -cusp_storage_format dia > 1) Always send -log_summary with performance questions 2) Comparing two things will not make any sense beyond "one ran faster" without a model for execution time 3) In order to make sense of my model, I need flop rates for those events Matt > Thanks, > > Dave > > -- > Dave Nystrom > LANL HPC-5 > Phone: 505-667-7913 > Email: wdn(a)lanl.gov > Smail: Mail Stop B272 > Group HPC-5 > Los Alamos National Laboratory > Los Alamos, NM 87545 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Understanding Some Parallel Results with PETSc
by Nystrom, William D 23 Feb '12

23 Feb '12

I recently ran a couple of test runs with petsc-dev that I do not understand. I'm running on a test bed machine that has 4 nodes with two Tesla 2090 gpus per node. Each node is dual socket and populated with Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz processors. These are 8 core processors and so each node has 16 cores. On the gpu, I'm running with Paul's latest version of the txpetscgpu package. I'm running the src/ksp/ksp/examples/tutorials/ex2.c petsc example with m=n=10000. My objective was to compare the performance running on 4 nodes using all 8 gpus to that of running on the same 4 nodes with all 64 cores. This problem uses about a third of the memory available on the gpus. I was using cg with jacobi preconditioning on both the gpu run and the cpu run. What is puzzling to me is that the cpu case ran 44x times slower than the gpu case and the big difference was in the time spend in functions like VecTDot, VecNorm and VecAXPY. Below is a table that summarizes the performance of the main functions that were using time in the two runs. Times are in seconds. | GPU | CPU | Ratio ------------------------------------------------------------------------- MatMult | 450.64 | 5484.7 | 12.17 ------------------------------------------------------------------------- VecTDot | 285.35 | 16688.0 | 58.48 ------------------------------------------------------------------------- VecNorm | 19.03 | 9058.8 | 476.03 ------------------------------------------------------------------------- VecAXPY | 106.88 | 5636.3 | 52.73 ------------------------------------------------------------------------- VecAYPX | 53.69 | 85.1 | 1.58 ------------------------------------------------------------------------- KSPSolve | 811.95 | 35930.0 | 44.25 ------------------------------------------------------------------------- The ratio of MatMult for CPU versus GPU is what I typically see when I am comparing a CPU run on a single core versus a run on a single GPU. Since both runs are communicating across node via mpi, I'm puzzled about why the CPU case is so much slower than the GPU case especially since there is communication for the MatMult as well. Both runs compute the same final error norm using the exact same number of iterations. Do these results make sense to people who understand the performance issues of parallel sparse linear solvers much better than I? Or do these results look abnormal. I had wondered if part of the performance issue was related to my running 8 times as many mpi processes for the CPU case. However, I ran a smaller problem with m=n=1000 and using 8 mpi processes and 2 cores per node and I see the same extreme differences in the times spent in VecTDot, VecNorm and VecAXPY. Here are the command lines I used for the two runs: CPU: mpirun -np 64 -mca btl self,sm,openib ex2 -m 10000 -n 10000 -ksp_type cg -ksp_max_it 100000 -pc_type jacobi -log_summary -options_left GPU: mpirun -np 8 -npernode 2 -mca btl self,sm,openib ex2 -m 10000 -n 10000 -ksp_type cg -ksp_max_it 100000 -pc_type jacobi -log_summary -options_left -mat_type aijcusp -vec_type cusp -cusp_storage_format dia Thanks, Dave -- Dave Nystrom LANL HPC-5 Phone: 505-667-7913 Email: wdn(a)lanl.gov Smail: Mail Stop B272 Group HPC-5 Los Alamos National Laboratory Los Alamos, NM 87545

1 0

Re: [petsc-dev] maxIndSetAgg in gamg
by Jed Brown 22 Feb '12

22 Feb '12

On Wed, Feb 22, 2012 at 11:55, Verena Kuhlemann <vkuhlem(a)gmail.com> wrote: > Hi, > > I was planning to use the maxIndSetAgg function in gamg.c but it is not > available in the newest > nightly build. Was it thrown out or renamed? Any change it could be made > available again? > This code is in flux and that function was never public. petsc-dev has a MatCoarsen object now, of which one implementation is MIS-based aggregation. The new design should be much better, but it will likely still evolve some from its current form.

1 0

maxIndSetAgg in gamg
by Verena Kuhlemann 22 Feb '12

22 Feb '12

Hi, I was planning to use the maxIndSetAgg function in gamg.c but it is not available in the newest nightly build. Was it thrown out or renamed? Any change it could be made available again? Thanks, Verena

1 0

Re: [petsc-dev] Crash in MatDestroy_MUMPS
by John Fettig 22 Feb '12

22 Feb '12

On Tue, Feb 21, 2012 at 4:55 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Tue, Feb 21, 2012 at 15:38, John Fettig <john.fettig(a)gmail.com> wrote: > >> Is this likely memory corruption on my side or on the PETSc side, or >> something else entirely? I seem unable to reproduce it with an example, >> unfortunately. Running it through valgrind may take a week. > > > I don't see a reason why that specific line would naturally end up with an > invalid value, so I suspect memory corruption. Make a smaller problem so > that valgrind runs fast. > You were right. I was able to run it through valgrind overnight and it was definitely memory corruption on my end. After cleaning that up, the segfault in KSPDestroy goes away. Thanks, John

1 0

Re: [petsc-dev] GAMG with matrix free multiple
by Mark F. Adams 22 Feb '12

22 Feb '12

On Feb 21, 2012, at 9:56 PM, Barry Smith wrote: > > Mark, > > In constructing your coarser levels it appears you are using the A operator from KSPSetOperators() not the B operator. When one is using a matrix free form to apply A I think it makes sense to use the B to construct the coarser grid operators? Yes, of course. I'm a PC, I should always look at B. Thanks, Mark > Or should you always use B or should there be an option for PCGAMG to use either the A or the B for constructing the coarser and if A is matrix free that options gets automatically flicked to use the B? > > Barry > > src/snes/examples/tutorials barrysmith$ ./ex5 -pc_type gamg -snes_mf_operator -ksp_monitor > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: No support for this operation for this object type! > [0]PETSC ERROR: Matrix of type <mffd> does not support PtAP! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Development HG revision: c132289977191163d4de2df6e7df77d9dd0b072f HG Date: Tue Feb 21 16:29:59 2012 -0600 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: ./ex5 on a arch-gnu named barry-smiths-macbook-pro.local by barrysmith Tue Feb 21 20:50:59 2012 > [0]PETSC ERROR: Libraries linked from /Users/barrysmith/Src/petsc-dev/arch-gnu/lib > [0]PETSC ERROR: Configure run at Tue Feb 21 16:36:50 2012 > [0]PETSC ERROR: Configure options --download-blacs --download-fftw --download-hypre --download-metis --download-mpich --download-mumps --download-parmetis --download-ptscotch --download-scalapack --download-superlu --download-superlu_dist --download-yaml --with-shared-libraries PETSC_ARCH=arch-gnu > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: MatPtAP() line 8263 in /Users/barrysmith/Src/petsc-dev/src/mat/interface/matrix.c > [0]PETSC ERROR: createLevel() line 88 in /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/gamg/gamg.c > [0]PETSC ERROR: PCSetUp_GAMG() line 562 in /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/gamg/gamg.c > [0]PETSC ERROR: PCSetUp() line 832 in /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: KSPSetUp() line 261 in /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: KSPSolve() line 385 in /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: SNES_KSPSolve() line 3939 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c > [0]PETSC ERROR: SNESSolve_LS() line 641 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/ls/ls.c > [0]PETSC ERROR: SNESSolve() line 3163 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c > [0]PETSC ERROR: main() line 152 in src/snes/examples/tutorials/ex5.c > application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 > [unset]: aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 > >

1 0

GAMG with matrix free multiple
by Barry Smith 21 Feb '12

21 Feb '12

Mark, In constructing your coarser levels it appears you are using the A operator from KSPSetOperators() not the B operator. When one is using a matrix free form to apply A I think it makes sense to use the B to construct the coarser grid operators? Or should you always use B or should there be an option for PCGAMG to use either the A or the B for constructing the coarser and if A is matrix free that options gets automatically flicked to use the B? Barry src/snes/examples/tutorials barrysmith$ ./ex5 -pc_type gamg -snes_mf_operator -ksp_monitor [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: No support for this operation for this object type! [0]PETSC ERROR: Matrix of type <mffd> does not support PtAP! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development HG revision: c132289977191163d4de2df6e7df77d9dd0b072f HG Date: Tue Feb 21 16:29:59 2012 -0600 [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./ex5 on a arch-gnu named barry-smiths-macbook-pro.local by barrysmith Tue Feb 21 20:50:59 2012 [0]PETSC ERROR: Libraries linked from /Users/barrysmith/Src/petsc-dev/arch-gnu/lib [0]PETSC ERROR: Configure run at Tue Feb 21 16:36:50 2012 [0]PETSC ERROR: Configure options --download-blacs --download-fftw --download-hypre --download-metis --download-mpich --download-mumps --download-parmetis --download-ptscotch --download-scalapack --download-superlu --download-superlu_dist --download-yaml --with-shared-libraries PETSC_ARCH=arch-gnu [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: MatPtAP() line 8263 in /Users/barrysmith/Src/petsc-dev/src/mat/interface/matrix.c [0]PETSC ERROR: createLevel() line 88 in /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/gamg/gamg.c [0]PETSC ERROR: PCSetUp_GAMG() line 562 in /Users/barrysmith/Src/petsc-dev/src/ksp/pc/impls/gamg/gamg.c [0]PETSC ERROR: PCSetUp() line 832 in /Users/barrysmith/Src/petsc-dev/src/ksp/pc/interface/precon.c [0]PETSC ERROR: KSPSetUp() line 261 in /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: KSPSolve() line 385 in /Users/barrysmith/Src/petsc-dev/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: SNES_KSPSolve() line 3939 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: SNESSolve_LS() line 641 in /Users/barrysmith/Src/petsc-dev/src/snes/impls/ls/ls.c [0]PETSC ERROR: SNESSolve() line 3163 in /Users/barrysmith/Src/petsc-dev/src/snes/interface/snes.c [0]PETSC ERROR: main() line 152 in src/snes/examples/tutorials/ex5.c application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0 [unset]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 56) - process 0

1 0

Re: [petsc-dev] [petsc-users] Add a matrix to MatNest twice
by Jed Brown 21 Feb '12

21 Feb '12

On Tue, Feb 21, 2012 at 16:42, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Tue, Feb 21, 2012 at 16:37, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > >> Oh, you mean using pthreads to spawn a thread that waits at blocking >> collective? :-) >> > The thread should (eventually) be handled by the MPI implementation (because I don't want every library spawning their own thread). > >> BTW; do you really think MPI-3 will exist? Or it should exist? >> > > I can call MPIX_Iallreduce() (and lots of other non-blocking collectives) > today. > An interface by which libraries could make progress on "user-defined non-blocking collectives" (like matrix assembly, collective semantics, but requiring multiple rounds of communication) is something I would like to see eventually, preferably in MPI. I chatted with Bill and others at the ICERM meeting last month and wrote a paragraph on it for the PUF proposal.

1 0

Re: [petsc-dev] [petsc-users] Add a matrix to MatNest twice
by Jed Brown 21 Feb '12

21 Feb '12

On Tue, Feb 21, 2012 at 16:37, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > Oh, you mean using pthreads to spawn a thread that waits at blocking > collective? :-) > > BTW; do you really think MPI-3 will exist? Or it should exist? > I can call MPIX_Iallreduce() (and lots of other non-blocking collectives) today.

1 0