February 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Understanding Vecscatter with Kokkos Vecs
by Stefano Zampini 19 Feb '21

19 Feb '21

I dont understand the issue. I assume your vecscatter is not a throw away object but you will reuse it many times. Once the setup is done, the indices used internally by the implementation should be on gpu , or not? Il Ven 19 Feb 2021, 13:33 Patrick Sanan <patrick.sanan(a)gmail.com> ha scritto: > Thanks! That helps a lot. > > I assume "no," but is ISCUDA simple to add? > > More on what I'm trying to do, in case I'm missing an obvious approach: > > I'm working on a demo code that uses an external library, based on Kokkos, > as a solver - I create a Vec of type KOKKOS and populate it with the > solution data from the library, by getting access to the raw Kokkos view > with VecKokkosGetDeviceView() * . > > I then want to reorder that solution data into PETSc-native ordering (for > a velocity-pressure DMStag), so I create a pair of ISs and a VecScatter to > do that. > > The issue is that to create this scatter, I need to use information > (essentially, an element-to-index map) from the external library's > mesh-management object, which lives on the device. This doesn't work (when > host != device), because of course the ISs live on the host and to create > them I need to provide host arrays of indices. > > Am I stuck, for now, with sending the index information information from > the device to the host, using it to create the IS, and then having > essentially the same information go back to the device when I use the > scatter? > > * As an aside, it looks like some of these Kokkos-related functions and > types are missing man pages - if you have time to add them, even as stubs, > that'd be great (if not let me know and I'll just try to formally do it, so > that at least the existence of the functions in the API is reflected on the > website). > > Am 18.02.2021 um 23:17 schrieb Junchao Zhang <junchao.zhang(a)gmail.com>: > > > On Thu, Feb 18, 2021 at 4:04 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: > >> >> >> On Thu, Feb 18, 2021 at 1:55 PM Junchao Zhang <junchao.zhang(a)gmail.com> >> wrote: >> >>> VecScatter (i.e., SF, the two are the same thing) setup (building >>> various index lists, rank lists) is done on the CPU. is1, is2 must be host >>> data. >>> >> >> Just out of curiosity, is1 and is2 can not be created on a GPU device in >> the first place? That being said, it is technically impossible? Or we just >> did not implement them yet? >> > Simply because we do not have an ISCUDA class. > > >> >> Fande, >> >> >>> When the SF is used to communicate device data, indices are copied to >>> the device.. >>> >>> --Junchao Zhang >>> >>> >>> On Thu, Feb 18, 2021 at 11:50 AM Patrick Sanan <patrick.sanan(a)gmail.com> >>> wrote: >>> >>>> I'm trying to understand how VecScatters work with GPU-native Kokkos >>>> Vecs. >>>> >>>> Specifically, I'm interested in what will happen in code like in >>>> src/vec/vec/tests/ex22.c, >>>> >>>> ierr = VecScatterCreate(x,is1,y,is2,&ctx);CHKERRQ(ierr); >>>> >>>> (from >>>> https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vec/tests/ex22.c#L44 >>>> ) >>>> >>>> Here, x and y can be set to type KOKKOS using -vec_type kokkos at the >>>> command line. But is1 and is2 are (I think), always >>>> CPU/host data. Assuming that the scatter itself can happen on the GPU, >>>> the indices must make it to the device somehow - are they copied there when >>>> the scatter is created? Is there a way to create the scatter using indices >>>> already on the GPU (Maybe using SF more directly)? >>>> >>>> >

1 0

Re: [petsc-dev] Understanding Vecscatter with Kokkos Vecs
by Patrick Sanan 19 Feb '21

19 Feb '21

Thanks! That helps a lot. I assume "no," but is ISCUDA simple to add? More on what I'm trying to do, in case I'm missing an obvious approach: I'm working on a demo code that uses an external library, based on Kokkos, as a solver - I create a Vec of type KOKKOS and populate it with the solution data from the library, by getting access to the raw Kokkos view with VecKokkosGetDeviceView() * . I then want to reorder that solution data into PETSc-native ordering (for a velocity-pressure DMStag), so I create a pair of ISs and a VecScatter to do that. The issue is that to create this scatter, I need to use information (essentially, an element-to-index map) from the external library's mesh-management object, which lives on the device. This doesn't work (when host != device), because of course the ISs live on the host and to create them I need to provide host arrays of indices. Am I stuck, for now, with sending the index information information from the device to the host, using it to create the IS, and then having essentially the same information go back to the device when I use the scatter? * As an aside, it looks like some of these Kokkos-related functions and types are missing man pages - if you have time to add them, even as stubs, that'd be great (if not let me know and I'll just try to formally do it, so that at least the existence of the functions in the API is reflected on the website). > Am 18.02.2021 um 23:17 schrieb Junchao Zhang <junchao.zhang(a)gmail.com>: > > > On Thu, Feb 18, 2021 at 4:04 PM Fande Kong <fdkong.jd(a)gmail.com <mailto:[email protected]>> wrote: > > > On Thu, Feb 18, 2021 at 1:55 PM Junchao Zhang <junchao.zhang(a)gmail.com <mailto:[email protected]>> wrote: > VecScatter (i.e., SF, the two are the same thing) setup (building various index lists, rank lists) is done on the CPU. is1, is2 must be host data. > > Just out of curiosity, is1 and is2 can not be created on a GPU device in the first place? That being said, it is technically impossible? Or we just did not implement them yet? > Simply because we do not have an ISCUDA class. > > > Fande, > > When the SF is used to communicate device data, indices are copied to the device.. > > --Junchao Zhang > > > On Thu, Feb 18, 2021 at 11:50 AM Patrick Sanan <patrick.sanan(a)gmail.com <mailto:[email protected]>> wrote: > I'm trying to understand how VecScatters work with GPU-native Kokkos Vecs. > > Specifically, I'm interested in what will happen in code like in src/vec/vec/tests/ex22.c, > > ierr = VecScatterCreate(x,is1,y,is2,&ctx);CHKERRQ(ierr); > > (from https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vec/tests/ex22.c#L44 <https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vec/tests/ex22.c#L44>) > > Here, x and y can be set to type KOKKOS using -vec_type kokkos at the command line. But is1 and is2 are (I think), always > CPU/host data. Assuming that the scatter itself can happen on the GPU, the indices must make it to the device somehow - are they copied there when the scatter is created? Is there a way to create the scatter using indices already on the GPU (Maybe using SF more directly)? >

1 0

Comparing binary output files in test harness
by Blaise A Bourdin 18 Feb '21

18 Feb '21

Hi, I would like to write better tests for my exodus I/O functions and compare the binary files written to the drive instead of the output of the examples. For instance, would it be possible to do the following: ex26 -i <input mesh> -o output1.exo; mpirun -np 2 ex26 -i <input mesh> -o output2.exo; exodiff output1.exo output2.exo And check the result of exodiff, or run exodiff between output1.exo or output2.exo and a stored binary result? Regards, Blaise -- A.K. & Shirley Barton Professor of Mathematics Adjunct Professor of Mechanical Engineering Adjunct of the Center for Computation & Technology Louisiana State University, Lockett Hall Room 344, Baton Rouge, LA 70803, USA Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 Web http://www.math.lsu.edu/~bourdin

1 0

Re: [petsc-dev] Understanding Vecscatter with Kokkos Vecs
by Junchao Zhang 18 Feb '21

18 Feb '21

On Thu, Feb 18, 2021 at 4:04 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: > > > On Thu, Feb 18, 2021 at 1:55 PM Junchao Zhang <junchao.zhang(a)gmail.com> > wrote: > >> VecScatter (i.e., SF, the two are the same thing) setup (building various >> index lists, rank lists) is done on the CPU. is1, is2 must be host data. >> > > Just out of curiosity, is1 and is2 can not be created on a GPU device in > the first place? That being said, it is technically impossible? Or we just > did not implement them yet? > Simply because we do not have an ISCUDA class. > > Fande, > > >> When the SF is used to communicate device data, indices are copied to the >> device.. >> >> --Junchao Zhang >> >> >> On Thu, Feb 18, 2021 at 11:50 AM Patrick Sanan <patrick.sanan(a)gmail.com> >> wrote: >> >>> I'm trying to understand how VecScatters work with GPU-native Kokkos >>> Vecs. >>> >>> Specifically, I'm interested in what will happen in code like in >>> src/vec/vec/tests/ex22.c, >>> >>> ierr = VecScatterCreate(x,is1,y,is2,&ctx);CHKERRQ(ierr); >>> >>> (from >>> https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vec/tests/ex22.c#L44 >>> ) >>> >>> Here, x and y can be set to type KOKKOS using -vec_type kokkos at the >>> command line. But is1 and is2 are (I think), always >>> CPU/host data. Assuming that the scatter itself can happen on the GPU, >>> the indices must make it to the device somehow - are they copied there when >>> the scatter is created? Is there a way to create the scatter using indices >>> already on the GPU (Maybe using SF more directly)? >>> >>>

1 0

Re: [petsc-dev] Understanding Vecscatter with Kokkos Vecs
by Fande Kong 18 Feb '21

18 Feb '21

On Thu, Feb 18, 2021 at 1:55 PM Junchao Zhang <junchao.zhang(a)gmail.com> wrote: > VecScatter (i.e., SF, the two are the same thing) setup (building various > index lists, rank lists) is done on the CPU. is1, is2 must be host data. > Just out of curiosity, is1 and is2 can not be created on a GPU device in the first place? That being said, it is technically impossible? Or we just did not implement them yet? Fande, > When the SF is used to communicate device data, indices are copied to the > device.. > > --Junchao Zhang > > > On Thu, Feb 18, 2021 at 11:50 AM Patrick Sanan <patrick.sanan(a)gmail.com> > wrote: > >> I'm trying to understand how VecScatters work with GPU-native Kokkos >> Vecs. >> >> Specifically, I'm interested in what will happen in code like in >> src/vec/vec/tests/ex22.c, >> >> ierr = VecScatterCreate(x,is1,y,is2,&ctx);CHKERRQ(ierr); >> >> (from >> https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vec/tests/ex22.c#L44 >> ) >> >> Here, x and y can be set to type KOKKOS using -vec_type kokkos at the >> command line. But is1 and is2 are (I think), always >> CPU/host data. Assuming that the scatter itself can happen on the GPU, >> the indices must make it to the device somehow - are they copied there when >> the scatter is created? Is there a way to create the scatter using indices >> already on the GPU (Maybe using SF more directly)? >> >>

1 0

Re: [petsc-dev] Understanding Vecscatter with Kokkos Vecs
by Junchao Zhang 18 Feb '21

18 Feb '21

VecScatter (i.e., SF, the two are the same thing) setup (building various index lists, rank lists) is done on the CPU. is1, is2 must be host data. When the SF is used to communicate device data, indices are copied to the device.. --Junchao Zhang On Thu, Feb 18, 2021 at 11:50 AM Patrick Sanan <patrick.sanan(a)gmail.com> wrote: > I'm trying to understand how VecScatters work with GPU-native Kokkos Vecs. > > Specifically, I'm interested in what will happen in code like in > src/vec/vec/tests/ex22.c, > > ierr = VecScatterCreate(x,is1,y,is2,&ctx);CHKERRQ(ierr); > > (from > https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vec/tests/ex22.c#L44) > > Here, x and y can be set to type KOKKOS using -vec_type kokkos at the > command line. But is1 and is2 are (I think), always > CPU/host data. Assuming that the scatter itself can happen on the GPU, the > indices must make it to the device somehow - are they copied there when the > scatter is created? Is there a way to create the scatter using indices > already on the GPU (Maybe using SF more directly)? > >

1 0

Understanding Vecscatter with Kokkos Vecs
by Patrick Sanan 18 Feb '21

18 Feb '21

I'm trying to understand how VecScatters work with GPU-native Kokkos Vecs. Specifically, I'm interested in what will happen in code like in src/vec/vec/tests/ex22.c, ierr = VecScatterCreate(x,is1,y,is2,&ctx);CHKERRQ(ierr); (from https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vec/tests/ex22.c#L44 <https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vec/tests/ex22.c#L44>) Here, x and y can be set to type KOKKOS using -vec_type kokkos at the command line. But is1 and is2 are (I think), always CPU/host data. Assuming that the scatter itself can happen on the GPU, the indices must make it to the device somehow - are they copied there when the scatter is created? Is there a way to create the scatter using indices already on the GPU (Maybe using SF more directly)?

1 0

RSE and Postdoc openings at CU Boulder
by Jed Brown 18 Feb '21

18 Feb '21

My research group has openings for a Research Software Engineer and a Postdoc. Details and application links below; feel free to email me with questions. ## Research Software Engineer CU Boulder’s PSAAP Multidisciplinary Simulation Center for Micromorphic Multiphysics Porous and Particulate Materials Simulations Within Exascale Computing Workflows has an opening for a *Research Software Engineer* to co-lead development of robust, extensible open source software for extreme-scale simulation of large-deformation composite poro-elasto-visco-plastic media across a broad range of regimes with experimental validation and coordination with micromorphic multiscale models. Successful applicants will have strong written and verbal communication skills, and interest in working with an interdisciplinary team to apply the following to real-world problems: * collaborative software development and devops (Git, continuous integration, etc.); * maintainable, high-performance programming techniques for CPUs and GPUs; * finite element and material-point discretizations; * computational mechanics/inelasticity; * parallel algebraic solvers such as PETSc; and * scalable data-intensive computing. This position can start immediately and is remote-friendly, especially during the pandemic. https://jobs.colorado.edu/jobs/JobDetail/Research-Associate/28703 ## Postdoc We also have an immediate opening for a *Postdoc* to conduct research in collaboration with the DOE Exascale Computing Project’s co-design Center for Efficient Exascale Discretization (CEED) on the development of robust and efficient methods for high order/compatible PDE discretization and multilevel solvers, including deployment in open source libraries. The project is especially interested in strategies to provide performance portability on emerging architectures and novel parallelization techniques to improve time to solution in the strong scaling limit. The methods will be applied in a variety of applications areas including sustainable energy and geophysics. Successful applicants will have strong written and verbal communication skills to collaborate with a distributed inter-disciplinary team and disseminate results via publications and presentations, as well as an interest in research and development of high-quality community software infrastructure in areas including, but not limited to: * element-based PDE discretization; * high-performance computing on emerging architectures, including CPU and GPUs; * scalable algebraic solvers; * applications in fluid and solid mechanics; and * data-intensive PDE workflows. This position can start immediately and is remote-friendly, especially during the pandemic. https://jobs.colorado.edu/jobs/JobDetail/PostDoctoral-Associate/28691 The University of Colorado Boulder is committed to building a culturally diverse community of faculty, staff, and students dedicated to contributing to an inclusive campus environment. We are an Equal Opportunity employer. We offer a competitive salary and a comprehensive benefits package.

1 0

Re: [petsc-dev] TSSetConvergedReason(ts,TS_CONVERGED_USER);
by Mark Adams 18 Feb '21

18 Feb '21

OK, makes sense. I didn't know monitors were called before the step. Thanks, On Thu, Feb 18, 2021 at 7:25 AM Matthew Knepley <knepley(a)gmail.com> wrote: > Stefano is right. Changes in convergence should probably go in TSPostStep. > > Thanks, > > Matt > > On Thu, Feb 18, 2021 at 7:22 AM Stefano Zampini <stefano.zampini(a)gmail.com> > wrote: > >> Mark >> >> monitors are not supposed to change the TS. You can think at monitors >> being 'const' methods of the TS. Also, TSMonitor is called at the beginning >> of each step , see here >> https://gitlab.com/petsc/petsc/-/blob/master/src/ts/interface/ts.c#L4169 >> >> >> Il giorno gio 18 feb 2021 alle ore 15:16 Mark Adams <mfadams(a)lbl.gov> ha >> scritto: >> >>> TSSetConvergedReason(ts,TS_CONVERGED_USER); >>> >>> does not seem to stop the iteration in a user monitor function. I have >>> verified that it works from a post step method. Is this intentional? >>> >>> Mark >>> >> >> >> -- >> Stefano >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> >

1 0

Re: [petsc-dev] TSSetConvergedReason(ts,TS_CONVERGED_USER);
by Matthew Knepley 18 Feb '21

18 Feb '21

Stefano is right. Changes in convergence should probably go in TSPostStep. Thanks, Matt On Thu, Feb 18, 2021 at 7:22 AM Stefano Zampini <stefano.zampini(a)gmail.com> wrote: > Mark > > monitors are not supposed to change the TS. You can think at monitors > being 'const' methods of the TS. Also, TSMonitor is called at the beginning > of each step , see here > https://gitlab.com/petsc/petsc/-/blob/master/src/ts/interface/ts.c#L4169 > > > Il giorno gio 18 feb 2021 alle ore 15:16 Mark Adams <mfadams(a)lbl.gov> ha > scritto: > >> TSSetConvergedReason(ts,TS_CONVERGED_USER); >> >> does not seem to stop the iteration in a user monitor function. I have >> verified that it works from a post step method. Is this intentional? >> >> Mark >> > > > -- > Stefano > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0