VecScatter (i.e., SF, the two are the same thing) setup (building various index lists, rank lists) is done on the CPU. is1, is2 must be host data. When the SF is used to communicate device data, indices are copied to the device.. --Junchao Zhang On Thu, Feb 18, 2021 at 11:50 AM Patrick Sanan <[email protected]> wrote:

I'm trying to understand how VecScatters work with GPU-native Kokkos Vecs.

Specifically, I'm interested in what will happen in code like in src/vec/vec/tests/ex22.c,

ierr = VecScatterCreate(x,is1,y,is2,&ctx);CHKERRQ(ierr);

(from https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vec/tests/ex22.c#L44)

Here, x and y can be set to type KOKKOS using -vec_type kokkos at the command line. But is1 and is2 are (I think), always CPU/host data. Assuming that the scatter itself can happen on the GPU, the indices must make it to the device somehow - are they copied there when the scatter is created? Is there a way to create the scatter using indices already on the GPU (Maybe using SF more directly)?