Can you paste -log_view result so I can see what functions are used? --Junchao Zhang On Tue, Nov 15, 2022 at 10:24 AM Fackler, Philip <[email protected]> wrote:
Yes, most (but not all) of our system test cases fail with the kokkos/cuda or cuda backends. All of them pass with the CPU-only kokkos backend.
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory* ------------------------------ *From:* Junchao Zhang <[email protected]> *Sent:* Monday, November 14, 2022 19:34 *To:* Fackler, Philip <[email protected]> *Cc:* [email protected] < [email protected]>; [email protected] < [email protected]>; Blondel, Sophie <[email protected]>; Zhang, Junchao <[email protected]>; Roth, Philip <[email protected]> *Subject:* [EXTERNAL] Re: [petsc-users] Kokkos backend for Mat and Vec diverging when running on CUDA device.
Hi, Philip, Sorry to hear that. It seems you could run the same code on CPUs but not no GPUs (with either petsc/Kokkos backend or petsc/cuda backend, is it right?
--Junchao Zhang
On Mon, Nov 14, 2022 at 12:13 PM Fackler, Philip via petsc-users < [email protected]> wrote:
This is an issue I've brought up before (and discussed in-person with Richard). I wanted to bring it up again because I'm hitting the limits of what I know to do, and I need help figuring this out.
The problem can be reproduced using Xolotl's "develop" branch built against a petsc build with kokkos and kokkos-kernels enabled. Then, either add the relevant kokkos options to the "petscArgs=" line in the system test parameter file(s), or just replace the system test parameter files with the ones from the "feature-petsc-kokkos" branch. See here the files that begin with "params_system_".
Note that those files use the "kokkos" options, but the problem is similar using the corresponding cuda/cusparse options. I've already tried building kokkos-kernels with no TPLs and got slightly different results, but the same problem.
Any help would be appreciated.
Thanks,
*Philip Fackler * Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division *Oak Ridge National Laboratory*