December 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] PETSc GPU example
by Fande Kong 06 Dec '21

06 Dec '21

Thanks, Matt On Mon, Dec 6, 2021 at 4:47 PM Matthew Knepley <knepley(a)gmail.com> wrote: > On Mon, Dec 6, 2021 at 6:40 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: > >> Dear PETSc team, >> >> I am interested in a careful evaluation of PETSc GPU performance in our >> INL cluster. >> >> Any example in PETSc that can show GPU speedup with solving a nonlinear >> equation? >> >> I talked to Junchao; he suggested that I try SNES/tutorial/ex56. I tried >> that, but I could not find any speedup using the GPU. I could attach some >> results of "log_view" later if we would like to see that. >> > > We should note that you will only see speedup in the solver, so that > problem has to be pretty large. I believe Mark has good results with it. > The assembly is still all on the CPU. I am working on this over break, and > hope to have a CEED version of it by the new year. > Are both function and matrix assmelies on CPU? Or just the matrix assembly? OK, I will try to check the solver part Thanks, again Fande > > Thanks, > > Matt > > >> Appreciate any instructions/comments about running a simple PETSc GPU >> example to get a speedup. >> >> Thanks, >> >> Fande >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> >

1 0

Re: [petsc-dev] PETSc GPU example
by Matthew Knepley 06 Dec '21

06 Dec '21

On Mon, Dec 6, 2021 at 6:40 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: > Dear PETSc team, > > I am interested in a careful evaluation of PETSc GPU performance in our > INL cluster. > > Any example in PETSc that can show GPU speedup with solving a nonlinear > equation? > > I talked to Junchao; he suggested that I try SNES/tutorial/ex56. I tried > that, but I could not find any speedup using the GPU. I could attach some > results of "log_view" later if we would like to see that. > We should note that you will only see speedup in the solver, so that problem has to be pretty large. I believe Mark has good results with it. The assembly is still all on the CPU. I am working on this over break, and hope to have a CEED version of it by the new year. Thanks, Matt > Appreciate any instructions/comments about running a simple PETSc GPU > example to get a speedup. > > Thanks, > > Fande > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

PETSc GPU example
by Fande Kong 06 Dec '21

06 Dec '21

Dear PETSc team, I am interested in a careful evaluation of PETSc GPU performance in our INL cluster. Any example in PETSc that can show GPU speedup with solving a nonlinear equation? I talked to Junchao; he suggested that I try SNES/tutorial/ex56. I tried that, but I could not find any speedup using the GPU. I could attach some results of "log_view" later if we would like to see that. Appreciate any instructions/comments about running a simple PETSc GPU example to get a speedup. Thanks, Fande

1 0

Re: [petsc-dev] Help tracking down unexpected Fortran behavior
by Barry Smith 06 Dec '21

06 Dec '21

Interfaces are generated automatically by sowing if the Fortran stub is generated automatically. For manual stubs one should always write a manual interface, but since we only started providing interfaces recently many manual stubs are still missing their corresponding interface. One can add any they notice are missing. Barry > On Dec 6, 2021, at 6:22 AM, Jose E. Roman <jroman(a)dsic.upv.es> wrote: > > PCSetType() has an interface in src/ksp/f90-mod/petscpc.h90 while PCFactorSetMatOrderingType() does not. > > I don't know if there is a clear criterion for when to add an interface in the corresponding h90 file. One criterion is that we need a F90 interface in case one of the arguments is allowed to be NULL. > > Jose > > >> El 6 dic 2021, a las 11:20, Patrick Sanan <patrick.sanan(a)gmail.com> escribió: >> >> I ran into an unexpected seg fault, which took me too long to realize was because of the old-school "you forgot the ierr" mistake! I was expecting the compiler to complain, since we've had better checking for a while. E.g. as in the attached code to reproduce, my compiler indeed errors on this >> >> call PCSetType(pc, PCLU) >> >> but not this >> >> call PCFactorSetMatOrderingType(pc, MATORDERINGEXTERNAL) >> >> I'm not yet seeing what the difference is, but there is still plenty I don't understand about how the custom fortran interfaces work. E.g. both of those functions have custom interfaces in ftn-custom directories, accepting an additional "len" argument to be used with FIXCHAR(), but I'm not sure how that argument is ultimately populated. >> >> >> <ex999f.F90> >

1 0

Re: [petsc-dev] Help tracking down unexpected Fortran behavior
by Jose E. Roman 06 Dec '21

06 Dec '21

PCSetType() has an interface in src/ksp/f90-mod/petscpc.h90 while PCFactorSetMatOrderingType() does not. I don't know if there is a clear criterion for when to add an interface in the corresponding h90 file. One criterion is that we need a F90 interface in case one of the arguments is allowed to be NULL. Jose > El 6 dic 2021, a las 11:20, Patrick Sanan <patrick.sanan(a)gmail.com> escribió: > > I ran into an unexpected seg fault, which took me too long to realize was because of the old-school "you forgot the ierr" mistake! I was expecting the compiler to complain, since we've had better checking for a while. E.g. as in the attached code to reproduce, my compiler indeed errors on this > > call PCSetType(pc, PCLU) > > but not this > > call PCFactorSetMatOrderingType(pc, MATORDERINGEXTERNAL) > > I'm not yet seeing what the difference is, but there is still plenty I don't understand about how the custom fortran interfaces work. E.g. both of those functions have custom interfaces in ftn-custom directories, accepting an additional "len" argument to be used with FIXCHAR(), but I'm not sure how that argument is ultimately populated. > > > <ex999f.F90>

1 0

Help tracking down unexpected Fortran behavior
by Patrick Sanan 06 Dec '21

06 Dec '21

I ran into an unexpected seg fault, which took me too long to realize was because of the old-school "you forgot the ierr" mistake! I was expecting the compiler to complain, since we've had better checking for a while. E.g. as in the attached code to reproduce, my compiler indeed errors on this call PCSetType(pc, PCLU) but not this call PCFactorSetMatOrderingType(pc, MATORDERINGEXTERNAL) I'm not yet seeing what the difference is, but there is still plenty I don't understand about how the custom fortran interfaces work. E.g. both of those functions have custom interfaces in ftn-custom directories, accepting an additional "len" argument to be used with FIXCHAR(), but I'm not sure how that argument is ultimately populated.

1 0

Re: [petsc-dev] mpi_comm_create_group()
by Adrian Croucher 05 Dec '21

05 Dec '21

On 12/4/21 2:27 AM, Matthew Knepley wrote: > This seems like hyperbole to me. At worst you need one collective > operation to figure out who is in. > This is no more expensive than one dot product, and probably less. > Unless you plan to do this millions > of times in the simulation, I do not see the dire problem. OK, I'll stop worrying and learn to love the bomb :-) - Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher(a)auckland.ac.nz tel: +64 (0)9 923 4611

1 0

Re: [petsc-dev] mpi_comm_create_group()
by Matthew Knepley 03 Dec '21

03 Dec '21

On Thu, Dec 2, 2021 at 11:56 PM Adrian Croucher <a.croucher(a)auckland.ac.nz> wrote: > > On 12/3/21 5:53 PM, Jed Brown wrote: > >> Ah, interesting. I guess mpi_comm_split() might also have the advantage > >> that each rank only needs to know if it is in the group or not, rather > >> than needing an array of all participating ranks, as mpi_comm_create() > >> and mpi_comm_create_group() do. > > That's exactly why it's often more convenient. But it is collective on > the parent communicator. > > Yes, I was a bit concerned about that after reading this (from > https://cvw.cac.cornell.edu/MPIAdvTopics/subdividing) > > " ...the MPI specification states that the call to MPI_Comm_create must > be executed by all processes in the input communicator (in our case, > MPI_COMM_WORLD), and that all processes must pass the same value for the > group argument (grp), even if they do not belong to the new group. This > can be a dire problem with a very large number of processes, such as are > found in petascale systems, so MPI_COMM_CREATE_GROUP was introduced in > MPI-3 to alleviate this problem." > This seems like hyperbole to me. At worst you need one collective operation to figure out who is in. This is no more expensive than one dot product, and probably less. Unless you plan to do this millions of times in the simulation, I do not see the dire problem. Thanks, Matt > - Adrian > > -- > Dr Adrian Croucher > Senior Research Fellow > Department of Engineering Science > University of Auckland, New Zealand > email: a.croucher(a)auckland.ac.nz > tel: +64 (0)9 923 4611 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] mpi_comm_create_group()
by Adrian Croucher 02 Dec '21

02 Dec '21

On 12/3/21 5:53 PM, Jed Brown wrote: >> Ah, interesting. I guess mpi_comm_split() might also have the advantage >> that each rank only needs to know if it is in the group or not, rather >> than needing an array of all participating ranks, as mpi_comm_create() >> and mpi_comm_create_group() do. > That's exactly why it's often more convenient. But it is collective on the parent communicator. Yes, I was a bit concerned about that after reading this (from https://cvw.cac.cornell.edu/MPIAdvTopics/subdividing) " ...the MPI specification states that the call to MPI_Comm_create must be executed by all processes in the input communicator (in our case, MPI_COMM_WORLD), and that all processes must pass the same value for the group argument (grp), even if they do not belong to the new group. This can be a dire problem with a very large number of processes, such as are found in petascale systems, so MPI_COMM_CREATE_GROUP was introduced in MPI-3 to alleviate this problem." - Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher(a)auckland.ac.nz tel: +64 (0)9 923 4611

1 0

Re: [petsc-dev] mpi_comm_create_group()
by Jed Brown 02 Dec '21

02 Dec '21

Adrian Croucher <a.croucher(a)auckland.ac.nz> writes: > On 12/3/21 5:31 PM, Jed Brown wrote: >> You'd need more calls in case of overlap (so MPI_Comm_create_group is more natural), but non-participating ranks is easy. >> >> | A process may supply the color value MPI_UNDEFINED, in which case newcomm returns MPI_COMM_NULL. > > Ah, interesting. I guess mpi_comm_split() might also have the advantage > that each rank only needs to know if it is in the group or not, rather > than needing an array of all participating ranks, as mpi_comm_create() > and mpi_comm_create_group() do. That's exactly why it's often more convenient. But it is collective on the parent communicator.

1 0