On Thu, Feb 18, 2021 at 1:55 PM Junchao Zhang <[email protected]> wrote:

VecScatter (i.e., SF, the two are the same thing) setup (building various index lists, rank lists) is done on the CPU. is1, is2 must be host data.

Just out of curiosity, is1 and is2 can not be created on a GPU device in the first place? That being said, it is technically impossible? Or we just did not implement them yet? Fande,

When the SF is used to communicate device data, indices are copied to the device..

--Junchao Zhang

On Thu, Feb 18, 2021 at 11:50 AM Patrick Sanan <[email protected]> wrote:

...

I'm trying to understand how VecScatters work with GPU-native Kokkos Vecs.

Specifically, I'm interested in what will happen in code like in src/vec/vec/tests/ex22.c,

ierr = VecScatterCreate(x,is1,y,is2,&ctx);CHKERRQ(ierr);

(from https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vec/tests/ex22.c#L44 )

Here, x and y can be set to type KOKKOS using -vec_type kokkos at the command line. But is1 and is2 are (I think), always CPU/host data. Assuming that the scatter itself can happen on the GPU, the indices must make it to the device somehow - are they copied there when the scatter is created? Is there a way to create the scatter using indices already on the GPU (Maybe using SF more directly)?