December 2021 - petsc-dev - mailman3.cels.anl.gov

Kokkos build fail
by Mark Adams 08 Dec '21

08 Dec '21

I am failing on OSX with openmpi. Kokkos is failing to build. I seem to be using: (conda_env) 08:46 1 adams/fix_mat_ex5k *= ~/Codes/petsc2$ /usr/local/Cellar/open-mpi/4.1.1_2/bin/mpicxx --version Apple clang version 13.0.0 (clang-1300.0.29.3) Target: x86_64-apple-darwin21.1.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin Any ideas? Thanks, Mark

1 0

Re: [petsc-dev] PETSc GPU example
by Jed Brown 07 Dec '21

07 Dec '21

Fande, let me know if you'd like to run the libCEED hyperelasticity solver on GPUs. It's matrix-free p-multigrid (assembling only the coarse problem). It uses DMPlex for mesh management. Mark Adams <mfadams(a)lbl.gov> writes: > Also Fande, > > If you are _not_ using NVIDIA with the MPS system, then you should run with > the default -cells 1,1,1, and use just one MPI process and one GPU. > This will be fine for evaluating the GPU. > > If you want to use more than one MPI process per GPU (because you want to > use the CPUs in the rest of your app) then the MPS system is important (I > see 3x speedup) and I would use NVIDIA+MPS unless you talk to > someone/vendor knowledgeable about using more than one MPI/GPU. > > Now if you can use NVIDIA+MPS it would be interesting to compare GPU solver > performance with single vs multiple MPI/GPU. It should be faster to use one > MPI/GPU (running the same problem of course), but it would be interesting > to quantify this. > If you want to do this then I can explain how to do it. > > Thanks, > Mark > > On Mon, Dec 6, 2021 at 10:03 PM Mark Adams <mfadams(a)lbl.gov> wrote: > >> * snes/ex56 runs a convergence study and confusingly sets the options >> manually, thus erasing your -ex56_dm_refine. >> >> * To refine, use -max_conv_its N <3>, this sets the number of steps of >> refinement. That is, the length of the convergence study >> >> * You can adjust where it starts from with -cells i,j,k <1,1,1> >> You do want to set this if you have multiple MPI processes so that the >> size of this mesh is the number of processes. That way it starts with one >> cell per process and refines from there. >> >> * GPU speedup is all about subdomain size. AMG has lots of kernel launches >> and you need to overcome this before you get net gain. >> Very rough numbers: I see a speedup of about 5-10x with a few million >> equations per GPU. >> As Matt said the assembly is on the CPU and ex56 gets really slow on >> larger problems. Be prepared to run the largest case for close to an hour. >> This setup is not measured in KSP[SNES]Solve in the -log_view output so >> look at that. >> When you do this convergence study there will be a new stage created for >> each refinement, so one run will give you a range of problem size data. >> Each refinement step increases the problem size by 8x so when the solve >> times increase by ~8x then that tells you you are past the >> latency dominated regime. You want to get into that to see gain. >> >> * The end of the source file has example parameters that you should use >> (the gamg one) >> >> * src/snes/tests/ex13.c is designed to be a benchmark test and it >> partitions the problem better in parallel and has modern Plex usage. If you >> are doing large scale parallelism then you should use this. >> (It is a little hard to understand. Not well documented.) >> >> Hope that helps, >> Mark >> >> On Mon, Dec 6, 2021 at 9:05 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: >> >>> >>> >>> On Mon, Dec 6, 2021 at 5:59 PM Matthew Knepley <knepley(a)gmail.com> wrote: >>> >>>> On Mon, Dec 6, 2021 at 7:54 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: >>>> >>>>> Thanks, Matt, >>>>> >>>>> Sorry, I still have more questions on this example. How to refine mesh >>>>> to make the problem larger? >>>>> >>>>> I tried the following options, and none of them worked. I might do >>>>> something wrong. >>>>> >>>>> -ex56_dm_refine 9 >>>>> >>>>> and >>>>> >>>>> -dm_refine 4 >>>>> >>>> >>>> The mesh handling in this example does not conform to the others, but it >>>> appears that >>>> >>>> -ex56_dm_refine <k> >>>> >>>> should take effect at >>>> >>>> >>>> https://gitlab.com/petsc/petsc/-/blob/main/src/snes/tutorials/ex56.c#L381 >>>> >>>> >>> I was puzzled about this because DMSetFromOptions does not seem to >>> trigger -ex56_dm_refine. >>> >>> I did a search, and could not find where we call " -ex56_dm_refine" in >>> PETSc. >>> >>> I got the same result by running the following two combinations: >>> >>> 1) ./ex56 -log_view -snes_view -max_conv_its 3 -ex56_dm_refine 10 >>> >>> 2) ./ex56 -log_view -snes_view -max_conv_its 3 -ex56_dm_refine 0 >>> >>> Thanks, >>> >>> Fande >>> >>> >>> unless you are setting max_conv_its to 0 somehow. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thanks, >>>>> >>>>> Fande >>>>> >>>>> On Mon, Dec 6, 2021 at 5:04 PM Matthew Knepley <knepley(a)gmail.com> >>>>> wrote: >>>>> >>>>>> On Mon, Dec 6, 2021 at 7:02 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: >>>>>> >>>>>>> Thanks, Matt >>>>>>> >>>>>>> On Mon, Dec 6, 2021 at 4:47 PM Matthew Knepley <knepley(a)gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> On Mon, Dec 6, 2021 at 6:40 PM Fande Kong <fdkong.jd(a)gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Dear PETSc team, >>>>>>>>> >>>>>>>>> I am interested in a careful evaluation of PETSc GPU performance in >>>>>>>>> our INL cluster. >>>>>>>>> >>>>>>>>> Any example in PETSc that can show GPU speedup with solving a >>>>>>>>> nonlinear equation? >>>>>>>>> >>>>>>>>> I talked to Junchao; he suggested that I try SNES/tutorial/ex56. I >>>>>>>>> tried that, but I could not find any speedup using the GPU. I could attach >>>>>>>>> some results of "log_view" later if we would like to see that. >>>>>>>>> >>>>>>>> >>>>>>>> We should note that you will only see speedup in the solver, so that >>>>>>>> problem has to be pretty large. I believe Mark has good results with it. >>>>>>>> The assembly is still all on the CPU. I am working on this over >>>>>>>> break, and hope to have a CEED version of it by the new year. >>>>>>>> >>>>>>> >>>>>>> Are both function and matrix assmelies on CPU? Or just the matrix >>>>>>> assembly? >>>>>>> >>>>>> >>>>>> There is no GPU assembly right now. >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> OK, I will try to check the solver part >>>>>>> >>>>>>> Thanks, again >>>>>>> >>>>>>> Fande >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>>> Appreciate any instructions/comments about running a simple PETSc >>>>>>>>> GPU example to get a speedup. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Fande >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> <http://www.cse.buffalo.edu/~knepley/> >>>> >>>

1 0

*****SPAM*****petsc-3.16.2 now available
by Satish Balay 07 Dec '21

07 Dec '21

Dear PETSc users, The patch release petsc-3.16.2 is now available for download. https://petsc.org/release/download/ Satish

1 0

Re: [petsc-dev] PETSc GPU example
by Mark Adams 07 Dec '21

07 Dec '21

Also Fande, If you are _not_ using NVIDIA with the MPS system, then you should run with the default -cells 1,1,1, and use just one MPI process and one GPU. This will be fine for evaluating the GPU. If you want to use more than one MPI process per GPU (because you want to use the CPUs in the rest of your app) then the MPS system is important (I see 3x speedup) and I would use NVIDIA+MPS unless you talk to someone/vendor knowledgeable about using more than one MPI/GPU. Now if you can use NVIDIA+MPS it would be interesting to compare GPU solver performance with single vs multiple MPI/GPU. It should be faster to use one MPI/GPU (running the same problem of course), but it would be interesting to quantify this. If you want to do this then I can explain how to do it. Thanks, Mark On Mon, Dec 6, 2021 at 10:03 PM Mark Adams <mfadams(a)lbl.gov> wrote: > * snes/ex56 runs a convergence study and confusingly sets the options > manually, thus erasing your -ex56_dm_refine. > > * To refine, use -max_conv_its N <3>, this sets the number of steps of > refinement. That is, the length of the convergence study > > * You can adjust where it starts from with -cells i,j,k <1,1,1> > You do want to set this if you have multiple MPI processes so that the > size of this mesh is the number of processes. That way it starts with one > cell per process and refines from there. > > * GPU speedup is all about subdomain size. AMG has lots of kernel launches > and you need to overcome this before you get net gain. > Very rough numbers: I see a speedup of about 5-10x with a few million > equations per GPU. > As Matt said the assembly is on the CPU and ex56 gets really slow on > larger problems. Be prepared to run the largest case for close to an hour. > This setup is not measured in KSP[SNES]Solve in the -log_view output so > look at that. > When you do this convergence study there will be a new stage created for > each refinement, so one run will give you a range of problem size data. > Each refinement step increases the problem size by 8x so when the solve > times increase by ~8x then that tells you you are past the > latency dominated regime. You want to get into that to see gain. > > * The end of the source file has example parameters that you should use > (the gamg one) > > * src/snes/tests/ex13.c is designed to be a benchmark test and it > partitions the problem better in parallel and has modern Plex usage. If you > are doing large scale parallelism then you should use this. > (It is a little hard to understand. Not well documented.) > > Hope that helps, > Mark > > On Mon, Dec 6, 2021 at 9:05 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: > >> >> >> On Mon, Dec 6, 2021 at 5:59 PM Matthew Knepley <knepley(a)gmail.com> wrote: >> >>> On Mon, Dec 6, 2021 at 7:54 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: >>> >>>> Thanks, Matt, >>>> >>>> Sorry, I still have more questions on this example. How to refine mesh >>>> to make the problem larger? >>>> >>>> I tried the following options, and none of them worked. I might do >>>> something wrong. >>>> >>>> -ex56_dm_refine 9 >>>> >>>> and >>>> >>>> -dm_refine 4 >>>> >>> >>> The mesh handling in this example does not conform to the others, but it >>> appears that >>> >>> -ex56_dm_refine <k> >>> >>> should take effect at >>> >>> >>> https://gitlab.com/petsc/petsc/-/blob/main/src/snes/tutorials/ex56.c#L381 >>> >>> >> I was puzzled about this because DMSetFromOptions does not seem to >> trigger -ex56_dm_refine. >> >> I did a search, and could not find where we call " -ex56_dm_refine" in >> PETSc. >> >> I got the same result by running the following two combinations: >> >> 1) ./ex56 -log_view -snes_view -max_conv_its 3 -ex56_dm_refine 10 >> >> 2) ./ex56 -log_view -snes_view -max_conv_its 3 -ex56_dm_refine 0 >> >> Thanks, >> >> Fande >> >> >> unless you are setting max_conv_its to 0 somehow. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks, >>>> >>>> Fande >>>> >>>> On Mon, Dec 6, 2021 at 5:04 PM Matthew Knepley <knepley(a)gmail.com> >>>> wrote: >>>> >>>>> On Mon, Dec 6, 2021 at 7:02 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: >>>>> >>>>>> Thanks, Matt >>>>>> >>>>>> On Mon, Dec 6, 2021 at 4:47 PM Matthew Knepley <knepley(a)gmail.com> >>>>>> wrote: >>>>>> >>>>>>> On Mon, Dec 6, 2021 at 6:40 PM Fande Kong <fdkong.jd(a)gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Dear PETSc team, >>>>>>>> >>>>>>>> I am interested in a careful evaluation of PETSc GPU performance in >>>>>>>> our INL cluster. >>>>>>>> >>>>>>>> Any example in PETSc that can show GPU speedup with solving a >>>>>>>> nonlinear equation? >>>>>>>> >>>>>>>> I talked to Junchao; he suggested that I try SNES/tutorial/ex56. I >>>>>>>> tried that, but I could not find any speedup using the GPU. I could attach >>>>>>>> some results of "log_view" later if we would like to see that. >>>>>>>> >>>>>>> >>>>>>> We should note that you will only see speedup in the solver, so that >>>>>>> problem has to be pretty large. I believe Mark has good results with it. >>>>>>> The assembly is still all on the CPU. I am working on this over >>>>>>> break, and hope to have a CEED version of it by the new year. >>>>>>> >>>>>> >>>>>> Are both function and matrix assmelies on CPU? Or just the matrix >>>>>> assembly? >>>>>> >>>>> >>>>> There is no GPU assembly right now. >>>>> >>>>> Matt >>>>> >>>>> >>>>>> OK, I will try to check the solver part >>>>>> >>>>>> Thanks, again >>>>>> >>>>>> Fande >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> Appreciate any instructions/comments about running a simple PETSc >>>>>>>> GPU example to get a speedup. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Fande >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> <http://www.cse.buffalo.edu/~knepley/> >>> >>

1 0

Re: [petsc-dev] PETSc GPU example
by Mark Adams 06 Dec '21

06 Dec '21

* snes/ex56 runs a convergence study and confusingly sets the options manually, thus erasing your -ex56_dm_refine. * To refine, use -max_conv_its N <3>, this sets the number of steps of refinement. That is, the length of the convergence study * You can adjust where it starts from with -cells i,j,k <1,1,1> You do want to set this if you have multiple MPI processes so that the size of this mesh is the number of processes. That way it starts with one cell per process and refines from there. * GPU speedup is all about subdomain size. AMG has lots of kernel launches and you need to overcome this before you get net gain. Very rough numbers: I see a speedup of about 5-10x with a few million equations per GPU. As Matt said the assembly is on the CPU and ex56 gets really slow on larger problems. Be prepared to run the largest case for close to an hour. This setup is not measured in KSP[SNES]Solve in the -log_view output so look at that. When you do this convergence study there will be a new stage created for each refinement, so one run will give you a range of problem size data. Each refinement step increases the problem size by 8x so when the solve times increase by ~8x then that tells you you are past the latency dominated regime. You want to get into that to see gain. * The end of the source file has example parameters that you should use (the gamg one) * src/snes/tests/ex13.c is designed to be a benchmark test and it partitions the problem better in parallel and has modern Plex usage. If you are doing large scale parallelism then you should use this. (It is a little hard to understand. Not well documented.) Hope that helps, Mark On Mon, Dec 6, 2021 at 9:05 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: > > > On Mon, Dec 6, 2021 at 5:59 PM Matthew Knepley <knepley(a)gmail.com> wrote: > >> On Mon, Dec 6, 2021 at 7:54 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: >> >>> Thanks, Matt, >>> >>> Sorry, I still have more questions on this example. How to refine mesh >>> to make the problem larger? >>> >>> I tried the following options, and none of them worked. I might do >>> something wrong. >>> >>> -ex56_dm_refine 9 >>> >>> and >>> >>> -dm_refine 4 >>> >> >> The mesh handling in this example does not conform to the others, but it >> appears that >> >> -ex56_dm_refine <k> >> >> should take effect at >> >> >> https://gitlab.com/petsc/petsc/-/blob/main/src/snes/tutorials/ex56.c#L381 >> >> > I was puzzled about this because DMSetFromOptions does not seem to trigger > -ex56_dm_refine. > > I did a search, and could not find where we call " -ex56_dm_refine" in > PETSc. > > I got the same result by running the following two combinations: > > 1) ./ex56 -log_view -snes_view -max_conv_its 3 -ex56_dm_refine 10 > > 2) ./ex56 -log_view -snes_view -max_conv_its 3 -ex56_dm_refine 0 > > Thanks, > > Fande > > > unless you are setting max_conv_its to 0 somehow. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> >>> Fande >>> >>> On Mon, Dec 6, 2021 at 5:04 PM Matthew Knepley <knepley(a)gmail.com> >>> wrote: >>> >>>> On Mon, Dec 6, 2021 at 7:02 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: >>>> >>>>> Thanks, Matt >>>>> >>>>> On Mon, Dec 6, 2021 at 4:47 PM Matthew Knepley <knepley(a)gmail.com> >>>>> wrote: >>>>> >>>>>> On Mon, Dec 6, 2021 at 6:40 PM Fande Kong <fdkong.jd(a)gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Dear PETSc team, >>>>>>> >>>>>>> I am interested in a careful evaluation of PETSc GPU performance in >>>>>>> our INL cluster. >>>>>>> >>>>>>> Any example in PETSc that can show GPU speedup with solving a >>>>>>> nonlinear equation? >>>>>>> >>>>>>> I talked to Junchao; he suggested that I try SNES/tutorial/ex56. I >>>>>>> tried that, but I could not find any speedup using the GPU. I could attach >>>>>>> some results of "log_view" later if we would like to see that. >>>>>>> >>>>>> >>>>>> We should note that you will only see speedup in the solver, so that >>>>>> problem has to be pretty large. I believe Mark has good results with it. >>>>>> The assembly is still all on the CPU. I am working on this over >>>>>> break, and hope to have a CEED version of it by the new year. >>>>>> >>>>> >>>>> Are both function and matrix assmelies on CPU? Or just the matrix >>>>> assembly? >>>>> >>>> >>>> There is no GPU assembly right now. >>>> >>>> Matt >>>> >>>> >>>>> OK, I will try to check the solver part >>>>> >>>>> Thanks, again >>>>> >>>>> Fande >>>>> >>>>> >>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Appreciate any instructions/comments about running a simple PETSc >>>>>>> GPU example to get a speedup. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Fande >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> <http://www.cse.buffalo.edu/~knepley/> >>>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> >

1 0

Re: [petsc-dev] PETSc GPU example
by Matthew Knepley 06 Dec '21

06 Dec '21

On Mon, Dec 6, 2021 at 9:04 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: > On Mon, Dec 6, 2021 at 5:59 PM Matthew Knepley <knepley(a)gmail.com> wrote: > >> On Mon, Dec 6, 2021 at 7:54 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: >> >>> Thanks, Matt, >>> >>> Sorry, I still have more questions on this example. How to refine mesh >>> to make the problem larger? >>> >>> I tried the following options, and none of them worked. I might do >>> something wrong. >>> >>> -ex56_dm_refine 9 >>> >>> and >>> >>> -dm_refine 4 >>> >> >> The mesh handling in this example does not conform to the others, but it >> appears that >> >> -ex56_dm_refine <k> >> >> should take effect at >> >> >> https://gitlab.com/petsc/petsc/-/blob/main/src/snes/tutorials/ex56.c#L381 >> >> > I was puzzled about this because DMSetFromOptions does not seem to trigger > -ex56_dm_refine. > > I did a search, and could not find where we call " -ex56_dm_refine" in > PETSc. > > I got the same result by running the following two combinations: > > 1) ./ex56 -log_view -snes_view -max_conv_its 3 -ex56_dm_refine 10 > > 2) ./ex56 -log_view -snes_view -max_conv_its 3 -ex56_dm_refine 0 > It is refining: nathawani/feature-1D-droplet-model $:/PETSc3/petsc/petsc-dev/src/snes/tutorials$ ./ex56 -ex56_dm_view -max_conv_its 1 DM Object: Mesh (ex56_) 1 MPI processes type: plex Mesh in 3 dimensions: Number of 0-cells per rank: 8 Number of 1-cells per rank: 12 Number of 2-cells per rank: 6 Number of 3-cells per rank: 1 Labels: marker: 1 strata with value/size (1 (18)) Face Sets: 6 strata with value/size (6 (1), 5 (1), 3 (1), 4 (1), 1 (1), 2 (1)) depth: 4 strata with value/size (0 (8), 1 (12), 2 (6), 3 (1)) boundary: 1 strata with value/size (1 (26)) celltype: 4 strata with value/size (7 (1), 0 (8), 4 (6), 1 (12)) Field deformation: adjacency FEM nathawani/feature-1D-droplet-model $:/PETSc3/petsc/petsc-dev/src/snes/tutorials$ ./ex56 -ex56_dm_view -max_conv_its 1 -ex56_dm_refine 3 DM Object: Mesh (ex56_) 1 MPI processes type: plex Mesh in 3 dimensions: Number of 0-cells per rank: 729 Number of 1-cells per rank: 1944 Number of 2-cells per rank: 1728 Number of 3-cells per rank: 512 Labels: celltype: 4 strata with value/size (0 (729), 1 (1944), 4 (1728), 7 (512)) depth: 4 strata with value/size (0 (729), 1 (1944), 2 (1728), 3 (512)) marker: 1 strata with value/size (1 (1530)) Face Sets: 6 strata with value/size (6 (225), 5 (225), 3 (225), 4 (225), 1 (225), 2 (225)) boundary: 1 strata with value/size (1 (1538)) Field deformation: adjacency FEM Matt > Thanks, > > Fande > > > unless you are setting max_conv_its to 0 somehow. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> >>> Fande >>> >>> On Mon, Dec 6, 2021 at 5:04 PM Matthew Knepley <knepley(a)gmail.com> >>> wrote: >>> >>>> On Mon, Dec 6, 2021 at 7:02 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: >>>> >>>>> Thanks, Matt >>>>> >>>>> On Mon, Dec 6, 2021 at 4:47 PM Matthew Knepley <knepley(a)gmail.com> >>>>> wrote: >>>>> >>>>>> On Mon, Dec 6, 2021 at 6:40 PM Fande Kong <fdkong.jd(a)gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Dear PETSc team, >>>>>>> >>>>>>> I am interested in a careful evaluation of PETSc GPU performance in >>>>>>> our INL cluster. >>>>>>> >>>>>>> Any example in PETSc that can show GPU speedup with solving a >>>>>>> nonlinear equation? >>>>>>> >>>>>>> I talked to Junchao; he suggested that I try SNES/tutorial/ex56. I >>>>>>> tried that, but I could not find any speedup using the GPU. I could attach >>>>>>> some results of "log_view" later if we would like to see that. >>>>>>> >>>>>> >>>>>> We should note that you will only see speedup in the solver, so that >>>>>> problem has to be pretty large. I believe Mark has good results with it. >>>>>> The assembly is still all on the CPU. I am working on this over >>>>>> break, and hope to have a CEED version of it by the new year. >>>>>> >>>>> >>>>> Are both function and matrix assmelies on CPU? Or just the matrix >>>>> assembly? >>>>> >>>> >>>> There is no GPU assembly right now. >>>> >>>> Matt >>>> >>>> >>>>> OK, I will try to check the solver part >>>>> >>>>> Thanks, again >>>>> >>>>> Fande >>>>> >>>>> >>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Appreciate any instructions/comments about running a simple PETSc >>>>>>> GPU example to get a speedup. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Fande >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> <http://www.cse.buffalo.edu/~knepley/> >>>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] PETSc GPU example
by Fande Kong 06 Dec '21

06 Dec '21

On Mon, Dec 6, 2021 at 5:59 PM Matthew Knepley <knepley(a)gmail.com> wrote: > On Mon, Dec 6, 2021 at 7:54 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: > >> Thanks, Matt, >> >> Sorry, I still have more questions on this example. How to refine mesh to >> make the problem larger? >> >> I tried the following options, and none of them worked. I might do >> something wrong. >> >> -ex56_dm_refine 9 >> >> and >> >> -dm_refine 4 >> > > The mesh handling in this example does not conform to the others, but it > appears that > > -ex56_dm_refine <k> > > should take effect at > > > https://gitlab.com/petsc/petsc/-/blob/main/src/snes/tutorials/ex56.c#L381 > > I was puzzled about this because DMSetFromOptions does not seem to trigger -ex56_dm_refine. I did a search, and could not find where we call " -ex56_dm_refine" in PETSc. I got the same result by running the following two combinations: 1) ./ex56 -log_view -snes_view -max_conv_its 3 -ex56_dm_refine 10 2) ./ex56 -log_view -snes_view -max_conv_its 3 -ex56_dm_refine 0 Thanks, Fande unless you are setting max_conv_its to 0 somehow. > > Thanks, > > Matt > > >> Thanks, >> >> Fande >> >> On Mon, Dec 6, 2021 at 5:04 PM Matthew Knepley <knepley(a)gmail.com> wrote: >> >>> On Mon, Dec 6, 2021 at 7:02 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: >>> >>>> Thanks, Matt >>>> >>>> On Mon, Dec 6, 2021 at 4:47 PM Matthew Knepley <knepley(a)gmail.com> >>>> wrote: >>>> >>>>> On Mon, Dec 6, 2021 at 6:40 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: >>>>> >>>>>> Dear PETSc team, >>>>>> >>>>>> I am interested in a careful evaluation of PETSc GPU performance in >>>>>> our INL cluster. >>>>>> >>>>>> Any example in PETSc that can show GPU speedup with solving a >>>>>> nonlinear equation? >>>>>> >>>>>> I talked to Junchao; he suggested that I try SNES/tutorial/ex56. I >>>>>> tried that, but I could not find any speedup using the GPU. I could attach >>>>>> some results of "log_view" later if we would like to see that. >>>>>> >>>>> >>>>> We should note that you will only see speedup in the solver, so that >>>>> problem has to be pretty large. I believe Mark has good results with it. >>>>> The assembly is still all on the CPU. I am working on this over break, >>>>> and hope to have a CEED version of it by the new year. >>>>> >>>> >>>> Are both function and matrix assmelies on CPU? Or just the matrix >>>> assembly? >>>> >>> >>> There is no GPU assembly right now. >>> >>> Matt >>> >>> >>>> OK, I will try to check the solver part >>>> >>>> Thanks, again >>>> >>>> Fande >>>> >>>> >>>> >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Appreciate any instructions/comments about running a simple PETSc GPU >>>>>> example to get a speedup. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Fande >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> <http://www.cse.buffalo.edu/~knepley/> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> >

1 0

Re: [petsc-dev] PETSc GPU example
by Matthew Knepley 06 Dec '21

06 Dec '21

On Mon, Dec 6, 2021 at 7:54 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: > Thanks, Matt, > > Sorry, I still have more questions on this example. How to refine mesh to > make the problem larger? > > I tried the following options, and none of them worked. I might do > something wrong. > > -ex56_dm_refine 9 > > and > > -dm_refine 4 > The mesh handling in this example does not conform to the others, but it appears that -ex56_dm_refine <k> should take effect at https://gitlab.com/petsc/petsc/-/blob/main/src/snes/tutorials/ex56.c#L381 unless you are setting max_conv_its to 0 somehow. Thanks, Matt > Thanks, > > Fande > > On Mon, Dec 6, 2021 at 5:04 PM Matthew Knepley <knepley(a)gmail.com> wrote: > >> On Mon, Dec 6, 2021 at 7:02 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: >> >>> Thanks, Matt >>> >>> On Mon, Dec 6, 2021 at 4:47 PM Matthew Knepley <knepley(a)gmail.com> >>> wrote: >>> >>>> On Mon, Dec 6, 2021 at 6:40 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: >>>> >>>>> Dear PETSc team, >>>>> >>>>> I am interested in a careful evaluation of PETSc GPU performance in >>>>> our INL cluster. >>>>> >>>>> Any example in PETSc that can show GPU speedup with solving a >>>>> nonlinear equation? >>>>> >>>>> I talked to Junchao; he suggested that I try SNES/tutorial/ex56. I >>>>> tried that, but I could not find any speedup using the GPU. I could attach >>>>> some results of "log_view" later if we would like to see that. >>>>> >>>> >>>> We should note that you will only see speedup in the solver, so that >>>> problem has to be pretty large. I believe Mark has good results with it. >>>> The assembly is still all on the CPU. I am working on this over break, >>>> and hope to have a CEED version of it by the new year. >>>> >>> >>> Are both function and matrix assmelies on CPU? Or just the matrix >>> assembly? >>> >> >> There is no GPU assembly right now. >> >> Matt >> >> >>> OK, I will try to check the solver part >>> >>> Thanks, again >>> >>> Fande >>> >>> >>> >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Appreciate any instructions/comments about running a simple PETSc GPU >>>>> example to get a speedup. >>>>> >>>>> Thanks, >>>>> >>>>> Fande >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> <http://www.cse.buffalo.edu/~knepley/> >>>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] PETSc GPU example
by Fande Kong 06 Dec '21

06 Dec '21

Thanks, Matt, Sorry, I still have more questions on this example. How to refine mesh to make the problem larger? I tried the following options, and none of them worked. I might do something wrong. -ex56_dm_refine 9 and -dm_refine 4 Thanks, Fande On Mon, Dec 6, 2021 at 5:04 PM Matthew Knepley <knepley(a)gmail.com> wrote: > On Mon, Dec 6, 2021 at 7:02 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: > >> Thanks, Matt >> >> On Mon, Dec 6, 2021 at 4:47 PM Matthew Knepley <knepley(a)gmail.com> wrote: >> >>> On Mon, Dec 6, 2021 at 6:40 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: >>> >>>> Dear PETSc team, >>>> >>>> I am interested in a careful evaluation of PETSc GPU performance in our >>>> INL cluster. >>>> >>>> Any example in PETSc that can show GPU speedup with solving a nonlinear >>>> equation? >>>> >>>> I talked to Junchao; he suggested that I try SNES/tutorial/ex56. I >>>> tried that, but I could not find any speedup using the GPU. I could attach >>>> some results of "log_view" later if we would like to see that. >>>> >>> >>> We should note that you will only see speedup in the solver, so that >>> problem has to be pretty large. I believe Mark has good results with it. >>> The assembly is still all on the CPU. I am working on this over break, >>> and hope to have a CEED version of it by the new year. >>> >> >> Are both function and matrix assmelies on CPU? Or just the matrix >> assembly? >> > > There is no GPU assembly right now. > > Matt > > >> OK, I will try to check the solver part >> >> Thanks, again >> >> Fande >> >> >> >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Appreciate any instructions/comments about running a simple PETSc GPU >>>> example to get a speedup. >>>> >>>> Thanks, >>>> >>>> Fande >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> <http://www.cse.buffalo.edu/~knepley/> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> >

1 0

Re: [petsc-dev] PETSc GPU example
by Matthew Knepley 06 Dec '21

06 Dec '21

On Mon, Dec 6, 2021 at 7:02 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: > Thanks, Matt > > On Mon, Dec 6, 2021 at 4:47 PM Matthew Knepley <knepley(a)gmail.com> wrote: > >> On Mon, Dec 6, 2021 at 6:40 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: >> >>> Dear PETSc team, >>> >>> I am interested in a careful evaluation of PETSc GPU performance in our >>> INL cluster. >>> >>> Any example in PETSc that can show GPU speedup with solving a nonlinear >>> equation? >>> >>> I talked to Junchao; he suggested that I try SNES/tutorial/ex56. I tried >>> that, but I could not find any speedup using the GPU. I could attach some >>> results of "log_view" later if we would like to see that. >>> >> >> We should note that you will only see speedup in the solver, so that >> problem has to be pretty large. I believe Mark has good results with it. >> The assembly is still all on the CPU. I am working on this over break, >> and hope to have a CEED version of it by the new year. >> > > Are both function and matrix assmelies on CPU? Or just the matrix assembly? > There is no GPU assembly right now. Matt > OK, I will try to check the solver part > > Thanks, again > > Fande > > > >> >> Thanks, >> >> Matt >> >> >>> Appreciate any instructions/comments about running a simple PETSc GPU >>> example to get a speedup. >>> >>> Thanks, >>> >>> Fande >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0