July 2020 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Question about MPICH device we use
by Jeff Hammond 23 Jul '20

23 Jul '20

Open-MPI refuses to let users over subscribe without an extra flag to mpirun. I think Intel MPI has an option for blocking poll that supports oversubscription “nicely”. MPICH might have a “no local” option that disables shared memory, in which case nemesis over libfabric with the sockets or TCP provider _might_ do the right thing. But you should ask MPICH people for details. Jeff On Thu, Jul 23, 2020 at 12:40 PM Jed Brown <jed(a)jedbrown.org> wrote: > I think we should default to ch3:nemesis when --download-mpich, and only > do ch3:sock when requested (which we would do in CI). > > Satish Balay via petsc-dev <petsc-dev(a)mcs.anl.gov> writes: > > > Primarily because ch3:sock performance does not degrade in oversubscribe > mode - which is developer friendly - i.e on your laptop. > > > > And folks doing optimized runs should use a properly tuned MPI for their > setup anyway. > > > > In this case --download-mpich-device=ch3:nemesis is likely appropriate > if using --download-mpich [and not using a separate/optimized MPI] > > > > Having defaults that satisfy all use cases is not practical. > > > > Satish > > > > On Wed, 22 Jul 2020, Matthew Knepley wrote: > > > >> We default to ch3:sock. Scott MacLachlan just had a long thread on the > >> Firedrake list where it ended up that reconfiguring using ch3:nemesis > had a > >> 2x performance boost on his 16-core proc, and noticeable effect on the 4 > >> core speedup. > >> > >> Why do we default to sock? > >> > >> Thanks, > >> > >> Matt > >> > >> > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [petsc-dev] Question about MPICH device we use
by Jed Brown 23 Jul '20

23 Jul '20

I think we should default to ch3:nemesis when --download-mpich, and only do ch3:sock when requested (which we would do in CI). Satish Balay via petsc-dev <petsc-dev(a)mcs.anl.gov> writes: > Primarily because ch3:sock performance does not degrade in oversubscribe mode - which is developer friendly - i.e on your laptop. > > And folks doing optimized runs should use a properly tuned MPI for their setup anyway. > > In this case --download-mpich-device=ch3:nemesis is likely appropriate if using --download-mpich [and not using a separate/optimized MPI] > > Having defaults that satisfy all use cases is not practical. > > Satish > > On Wed, 22 Jul 2020, Matthew Knepley wrote: > >> We default to ch3:sock. Scott MacLachlan just had a long thread on the >> Firedrake list where it ended up that reconfiguring using ch3:nemesis had a >> 2x performance boost on his 16-core proc, and noticeable effect on the 4 >> core speedup. >> >> Why do we default to sock? >> >> Thanks, >> >> Matt >> >>

1 0

Re: [petsc-dev] Question about MPICH device we use
by Matthew Knepley 22 Jul '20

22 Jul '20

On Wed, Jul 22, 2020 at 2:34 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > BTW: Last we compared performance [many years ago] the difference was not > a factor of 2 - but a few percentage points. > > Which petsc example can we re-run to compare? > Not sure. Scott I think was running his new MHD stuff. His post is on the #general channel on the Firedrake slack. Thanks, Matt > Satish > > On Wed, 22 Jul 2020, Satish Balay via petsc-dev wrote: > > > Primarily because ch3:sock performance does not degrade in oversubscribe > mode - which is developer friendly - i.e on your laptop. > > > > And folks doing optimized runs should use a properly tuned MPI for their > setup anyway. > > > > In this case --download-mpich-device=ch3:nemesis is likely appropriate > if using --download-mpich [and not using a separate/optimized MPI] > > > > Having defaults that satisfy all use cases is not practical. > > > > Satish > > > > On Wed, 22 Jul 2020, Matthew Knepley wrote: > > > > > We default to ch3:sock. Scott MacLachlan just had a long thread on the > > > Firedrake list where it ended up that reconfiguring using ch3:nemesis > had a > > > 2x performance boost on his 16-core proc, and noticeable effect on the > 4 > > > core speedup. > > > > > > Why do we default to sock? > > > > > > Thanks, > > > > > > Matt > > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] Question about MPICH device we use
by Satish Balay 22 Jul '20

22 Jul '20

BTW: Last we compared performance [many years ago] the difference was not a factor of 2 - but a few percentage points. Which petsc example can we re-run to compare? Satish On Wed, 22 Jul 2020, Satish Balay via petsc-dev wrote: > Primarily because ch3:sock performance does not degrade in oversubscribe mode - which is developer friendly - i.e on your laptop. > > And folks doing optimized runs should use a properly tuned MPI for their setup anyway. > > In this case --download-mpich-device=ch3:nemesis is likely appropriate if using --download-mpich [and not using a separate/optimized MPI] > > Having defaults that satisfy all use cases is not practical. > > Satish > > On Wed, 22 Jul 2020, Matthew Knepley wrote: > > > We default to ch3:sock. Scott MacLachlan just had a long thread on the > > Firedrake list where it ended up that reconfiguring using ch3:nemesis had a > > 2x performance boost on his 16-core proc, and noticeable effect on the 4 > > core speedup. > > > > Why do we default to sock? > > > > Thanks, > > > > Matt > > > > >

1 0

Re: [petsc-dev] Question about MPICH device we use
by Satish Balay 22 Jul '20

22 Jul '20

Primarily because ch3:sock performance does not degrade in oversubscribe mode - which is developer friendly - i.e on your laptop. And folks doing optimized runs should use a properly tuned MPI for their setup anyway. In this case --download-mpich-device=ch3:nemesis is likely appropriate if using --download-mpich [and not using a separate/optimized MPI] Having defaults that satisfy all use cases is not practical. Satish On Wed, 22 Jul 2020, Matthew Knepley wrote: > We default to ch3:sock. Scott MacLachlan just had a long thread on the > Firedrake list where it ended up that reconfiguring using ch3:nemesis had a > 2x performance boost on his 16-core proc, and noticeable effect on the 4 > core speedup. > > Why do we default to sock? > > Thanks, > > Matt > >

1 0

Re: [petsc-dev] Question about MPICH device we use
by Barry Smith 22 Jul '20

22 Jul '20

Because nemesis does function in a reasonable way when the system is oversubscribed, that is there are more MPI ranks than cores. Many users getting started have whatever number of cores (4?) but want to run tests with a few more ranks, this is not feasible with nemesis. In addition with the test harness we don't need to worry about the oversubscription issue and can run any number of ranks on any system. A possible alternative which means digging in the guts slightly is to default to nemesis BUT at PetscInitialize() if the user has selected more ranks than cores, stop and tell them they can't do that with nemesis and they must use sock if they want to do that. For the CI we would always use sock. Barry > On Jul 22, 2020, at 1:10 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > > We default to ch3:sock. Scott MacLachlan just had a long thread on the Firedrake list where it ended up that reconfiguring using ch3:nemesis had a 2x performance boost on his 16-core proc, and noticeable effect on the 4 core speedup. > > Why do we default to sock? > > Thanks, > > Matt > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Question about MPICH device we use
by Matthew Knepley 22 Jul '20

22 Jul '20

We default to ch3:sock. Scott MacLachlan just had a long thread on the Firedrake list where it ended up that reconfiguring using ch3:nemesis had a 2x performance boost on his 16-core proc, and noticeable effect on the 4 core speedup. Why do we default to sock? Thanks, Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] linking C++ mangling problem + Kokkos
by Mark Adams 18 Jul '20

18 Jul '20

Thanks, I looked for this but couldn't find it. Now I see it. On Sat, Jul 18, 2020 at 6:34 PM Barry Smith <bsmith(a)petsc.dev> wrote: > > My branch https://gitlab.com/petsc/petsc/-/merge_requests/2984 is in > review, I have modified it with a kinder interface for users based on Jed's > suggestion. Undergoing testing now. > > Once all the tests pass I will rebase and it will be ready for review > again. > > Barry > > > > On Jul 18, 2020, at 3:19 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > I am trying to experiment with Kokkos and have a PETSc method resolved > in a C++ main file, with a Kokkos kernel. The linker is not seeing it > because of name mangling. Is there a simple way to make this work? > > > > On a related topic. Should I work on getting --download-kokkos working? > There has been some talk about this and I'm not sure how hard this would be > for me to do. > > > > Thanks, > > Mark > >

1 0

Re: [petsc-dev] linking C++ mangling problem + Kokkos
by Barry Smith 18 Jul '20

18 Jul '20

My branch https://gitlab.com/petsc/petsc/-/merge_requests/2984 is in review, I have modified it with a kinder interface for users based on Jed's suggestion. Undergoing testing now. Once all the tests pass I will rebase and it will be ready for review again. Barry > On Jul 18, 2020, at 3:19 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > I am trying to experiment with Kokkos and have a PETSc method resolved in a C++ main file, with a Kokkos kernel. The linker is not seeing it because of name mangling. Is there a simple way to make this work? > > On a related topic. Should I work on getting --download-kokkos working? There has been some talk about this and I'm not sure how hard this would be for me to do. > > Thanks, > Mark

1 0

linking C++ mangling problem + Kokkos
by Mark Adams 18 Jul '20

18 Jul '20

I am trying to experiment with Kokkos and have a PETSc method resolved in a C++ main file, with a Kokkos kernel. The linker is not seeing it because of name mangling. Is there a simple way to make this work? On a related topic. Should I work on getting --download-kokkos working? There has been some talk about this and I'm not sure how hard this would be for me to do. Thanks, Mark

1 0