February 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] valgrind now a point of failure?
by Satish Balay 22 Feb '21

22 Feb '21

I get the following with a debug build. >>>>>>>> balay@petsc-02:/scratch/balay/petsc/src/mat/tests$ make ex238 gcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g3 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g3 -I/scratch/balay/petsc/include -I/scratch/balay/petsc/arch-linux-c-debug/include ex238.c -Wl,-rpath,/scratch/balay/petsc/arch-linux-c-debug/lib -L/scratch/balay/petsc/arch-linux-c-debug/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/7 -L/usr/lib/gcc/x86_64-linux-gnu/7 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lpetsc -llapack -lblas -lpthread -lm -lX11 -lstdc++ -ldl -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -ldl -o ex238 balay@petsc-02:/scratch/balay/petsc/src/mat/tests$ valgrind --tool=memcheck ./ex238 -mat_block_size 12 ==34355== Memcheck, a memory error detector ==34355== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==34355== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info ==34355== Command: ./ex238 -mat_block_size 12 ==34355== ==34355== Warning: set address range perms: large range [0x59e43040, 0xb696a840) (undefined) <<<<<<<< Hang? takes a long time. try a different example >>>>>> balay@petsc-02:/scratch/balay/petsc/src/mat/tests$ make ex237 gcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g3 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g3 -I/scratch/balay/petsc/include -I/scratch/balay/petsc/arch-linux-c-debug/include ex237.c -Wl,-rpath,/scratch/balay/petsc/arch-linux-c-debug/lib -L/scratch/balay/petsc/arch-linux-c-debug/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/7 -L/usr/lib/gcc/x86_64-linux-gnu/7 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lpetsc -llapack -lblas -lpthread -lm -lX11 -lstdc++ -ldl -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -ldl -o ex237 balay@petsc-02:/scratch/balay/petsc/src/mat/tests$ valgrind --tool=memcheck -q ./ex237 -f /scratch/balay/petsc/share/petsc/datafiles/matrices/spd-real-int32-float64 Benchmarking MatMult: with A seqaij 12x12 Benchmarking MatProduct AB: with A seqaij 12x12 and B seqdense 12x2 Benchmarking MatProduct AB: with A seqaij 12x12 and B seqdense 12x4 Benchmarking MatProduct AB: with A seqaij 12x12 and B seqdense 12x8 Benchmarking MatProduct AB: with A seqaij 12x12 and B seqdense 12x16 Benchmarking MatProduct AB: with A seqaij 12x12 and B seqdense 12x32 Benchmarking MatProduct AB: with A seqaij 12x12 and B seqdense 12x64 Benchmarking MatProduct AB: with A seqaij 12x12 and B seqdense 12x128 balay@petsc-02:/scratch/balay/petsc/src/mat/tests$ <<<<<<<<< So the likely issue is - this opt build with '-march=native' [perhaps this valgrind version is older than the cpu]. Ok try an optimized build on an older CPU - aka es [@gce] >>>>> balay@es:/scratch/balay/petsc/src/mat/tests$ make ex237 gcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -march=native -O3 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -march=native -O3 -I/scratch/balay/petsc/include -I/scratch/balay/petsc/arch-linux-c-opt/include ex237.c -Wl,-rpath,/scratch/balay/petsc/arch-linux-c-opt/lib -L/scratch/balay/petsc/arch-linux-c-opt/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/7 -L/usr/lib/gcc/x86_64-linux-gnu/7 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lpetsc -llapack -lblas -lpthread -lm -lX11 -lstdc++ -ldl -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -ldl -o ex237 balay@es:/scratch/balay/petsc/src/mat/tests$ valgrind --tool=memcheck -q ./ex237 -f /scratch/balay/petsc/share/petsc/datafiles/matrices/spd-real-int32-float64 Benchmarking MatMult: with A seqaij 12x12 Benchmarking MatProduct AB: with A seqaij 12x12 and B seqdense 12x2 Benchmarking MatProduct AB: with A seqaij 12x12 and B seqdense 12x4 Benchmarking MatProduct AB: with A seqaij 12x12 and B seqdense 12x8 Benchmarking MatProduct AB: with A seqaij 12x12 and B seqdense 12x16 Benchmarking MatProduct AB: with A seqaij 12x12 and B seqdense 12x32 Benchmarking MatProduct AB: with A seqaij 12x12 and B seqdense 12x64 Benchmarking MatProduct AB: with A seqaij 12x12 and B seqdense 12x128 balay@es:/scratch/balay/petsc/src/mat/tests$ <<<<<<< Satish On Mon, 22 Feb 2021, Barry Smith wrote: > > I knew they hate Macs but now Linux? Any trustworthy machines to run valgrind? > > > $ petscmpiexec -valgrind -n 1 ./ex238 -mat_block_size 12 > ==14144== > ==14144== Process terminating with default action of signal 4 (SIGILL) > ==14144== Illegal opcode at address 0x4F808A9 > ==14144== at 0x4F808A9: PetscSetDisplay (in /scratch/bsmith/petsc/arch-add-baij-12/lib/libpetsc.so.3.014.4) > ==14144== by 0x4F086BD: PetscOptionsCheckInitial_Private (in /scratch/bsmith/petsc/arch-add-baij-12/lib/libpetsc.so.3.014.4) > ==14144== by 0x4F0D5BC: PetscInitialize (in /scratch/bsmith/petsc/arch-add-baij-12/lib/libpetsc.so.3.014.4) > ==14144== by 0x108D0E: main (in /scratch/bsmith/petsc/src/mat/tests/ex238) > Illegal instruction (core dumped) > /scratch/bsmith/petsc/src/mat/tests (barry/2021-02-12/add-baij-12=) arch-add-baij-12 > $ echo $PETSC_OPTIONS > > /scratch/bsmith/petsc/src/mat/tests (barry/2021-02-12/add-baij-12=) arch-add-baij-12 > $ hostname > petsc-02 > /scratch/bsmith/petsc/src/mat/tests (barry/2021-02-12/add-baij-12=) arch-add-baij-12 > $ uname -a > Linux petsc-02 4.15.0-135-generic #139-Ubuntu SMP Mon Jan 18 17:38:24 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux > /scratch/bsmith/petsc/src/mat/tests (barry/2021-02-12/add-baij-12=) arch-add-baij-12 > $ which valgrind > /usr/bin/valgrind > > $ make ex237 > gcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -march=native -O3 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -march=native -O3 -I/scratch/bsmith/petsc/include -I/scratch/bsmith/petsc/arch-add-baij-12/include ex237.c -Wl,-rpath,/scratch/bsmith/petsc/arch-add-baij-12/lib -L/scratch/bsmith/petsc/arch-add-baij-12/lib -lpetsc -llapack -lblas -lpthread -lm -lX11 -lquadmath -ldl -o ex237 > /scratch/bsmith/petsc/src/mat/tests (barry/2021-02-12/add-baij-12=) arch-add-baij-12 > $ petscmpiexec -valgrind -n 1 ./ex237 > ==14841== > ==14841== Process terminating with default action of signal 4 (SIGILL) > ==14841== Illegal opcode at address 0x4F808A9 > ==14841== at 0x4F808A9: PetscSetDisplay (in /scratch/bsmith/petsc/arch-add-baij-12/lib/libpetsc.so.3.014.4) > ==14841== by 0x4F086BD: PetscOptionsCheckInitial_Private (in /scratch/bsmith/petsc/arch-add-baij-12/lib/libpetsc.so.3.014.4) > ==14841== by 0x4F0D5BC: PetscInitialize (in /scratch/bsmith/petsc/arch-add-baij-12/lib/libpetsc.so.3.014.4) > ==14841== by 0x109DE0: main (in /scratch/bsmith/petsc/src/mat/tests/ex237) > Illegal instruction (core dumped) > /scratch/bsmith/petsc/src/mat/tests (barry/2021-02-12/add-baij-12=) arch-add-baij-12 > >

1 0

Re: [petsc-dev] valgrind now a point of failure?
by Stefano Zampini 22 Feb '21

22 Feb '21

Valgrind does not play well with optimization flags. It's either 03 or march=native. Il Mar 23 Feb 2021, 06:38 Barry Smith <bsmith(a)petsc.dev> ha scritto: > > I knew they hate Macs but now Linux? Any trustworthy machines to run > valgrind? > > > $ petscmpiexec -valgrind -n 1 ./ex238 -mat_block_size 12 > ==14144== > ==14144== Process terminating with default action of signal 4 (SIGILL) > ==14144== Illegal opcode at address 0x4F808A9 > ==14144== at 0x4F808A9: PetscSetDisplay (in > /scratch/bsmith/petsc/arch-add-baij-12/lib/libpetsc.so.3.014.4) > ==14144== by 0x4F086BD: PetscOptionsCheckInitial_Private (in > /scratch/bsmith/petsc/arch-add-baij-12/lib/libpetsc.so.3.014.4) > ==14144== by 0x4F0D5BC: PetscInitialize (in > /scratch/bsmith/petsc/arch-add-baij-12/lib/libpetsc.so.3.014.4) > ==14144== by 0x108D0E: main (in > /scratch/bsmith/petsc/src/mat/tests/ex238) > Illegal instruction (core dumped) > /scratch/bsmith/petsc/src/mat/tests* (barry/2021-02-12/add-baij-12=)* > arch-add-baij-12 > $ echo $PETSC_OPTIONS > > /scratch/bsmith/petsc/src/mat/tests* (barry/2021-02-12/add-baij-12=)* > arch-add-baij-12 > $ hostname > petsc-02 > /scratch/bsmith/petsc/src/mat/tests* (barry/2021-02-12/add-baij-12=)* > arch-add-baij-12 > $ uname -a > Linux petsc-02 4.15.0-135-generic #139-Ubuntu SMP Mon Jan 18 17:38:24 UTC > 2021 x86_64 x86_64 x86_64 GNU/Linux > /scratch/bsmith/petsc/src/mat/tests* (barry/2021-02-12/add-baij-12=)* > arch-add-baij-12 > $ which valgrind > /usr/bin/valgrind > > $ make ex237 > gcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas > -fstack-protector -fvisibility=hidden -march=native -O3 -fPIC -Wall > -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector > -fvisibility=hidden -march=native -O3 -I/scratch/bsmith/petsc/include > -I/scratch/bsmith/petsc/arch-add-baij-12/include ex237.c > -Wl,-rpath,/scratch/bsmith/petsc/arch-add-baij-12/lib > -L/scratch/bsmith/petsc/arch-add-baij-12/lib -lpetsc -llapack -lblas > -lpthread -lm -lX11 -lquadmath -ldl -o ex237 > /scratch/bsmith/petsc/src/mat/tests* (barry/2021-02-12/add-baij-12=)* > arch-add-baij-12 > $ petscmpiexec -valgrind -n 1 ./ex237 > ==14841== > ==14841== Process terminating with default action of signal 4 (SIGILL) > ==14841== Illegal opcode at address 0x4F808A9 > ==14841== at 0x4F808A9: PetscSetDisplay (in > /scratch/bsmith/petsc/arch-add-baij-12/lib/libpetsc.so.3.014.4) > ==14841== by 0x4F086BD: PetscOptionsCheckInitial_Private (in > /scratch/bsmith/petsc/arch-add-baij-12/lib/libpetsc.so.3.014.4) > ==14841== by 0x4F0D5BC: PetscInitialize (in > /scratch/bsmith/petsc/arch-add-baij-12/lib/libpetsc.so.3.014.4) > ==14841== by 0x109DE0: main (in > /scratch/bsmith/petsc/src/mat/tests/ex237) > Illegal instruction (core dumped) > /scratch/bsmith/petsc/src/mat/tests* (barry/2021-02-12/add-baij-12=)* > arch-add-baij-12 > >

1 0

valgrind now a point of failure?
by Barry Smith 22 Feb '21

22 Feb '21

I knew they hate Macs but now Linux? Any trustworthy machines to run valgrind? $ petscmpiexec -valgrind -n 1 ./ex238 -mat_block_size 12 ==14144== ==14144== Process terminating with default action of signal 4 (SIGILL) ==14144== Illegal opcode at address 0x4F808A9 ==14144== at 0x4F808A9: PetscSetDisplay (in /scratch/bsmith/petsc/arch-add-baij-12/lib/libpetsc.so.3.014.4) ==14144== by 0x4F086BD: PetscOptionsCheckInitial_Private (in /scratch/bsmith/petsc/arch-add-baij-12/lib/libpetsc.so.3.014.4) ==14144== by 0x4F0D5BC: PetscInitialize (in /scratch/bsmith/petsc/arch-add-baij-12/lib/libpetsc.so.3.014.4) ==14144== by 0x108D0E: main (in /scratch/bsmith/petsc/src/mat/tests/ex238) Illegal instruction (core dumped) /scratch/bsmith/petsc/src/mat/tests (barry/2021-02-12/add-baij-12=) arch-add-baij-12 $ echo $PETSC_OPTIONS /scratch/bsmith/petsc/src/mat/tests (barry/2021-02-12/add-baij-12=) arch-add-baij-12 $ hostname petsc-02 /scratch/bsmith/petsc/src/mat/tests (barry/2021-02-12/add-baij-12=) arch-add-baij-12 $ uname -a Linux petsc-02 4.15.0-135-generic #139-Ubuntu SMP Mon Jan 18 17:38:24 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux /scratch/bsmith/petsc/src/mat/tests (barry/2021-02-12/add-baij-12=) arch-add-baij-12 $ which valgrind /usr/bin/valgrind $ make ex237 gcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -march=native -O3 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -march=native -O3 -I/scratch/bsmith/petsc/include -I/scratch/bsmith/petsc/arch-add-baij-12/include ex237.c -Wl,-rpath,/scratch/bsmith/petsc/arch-add-baij-12/lib -L/scratch/bsmith/petsc/arch-add-baij-12/lib -lpetsc -llapack -lblas -lpthread -lm -lX11 -lquadmath -ldl -o ex237 /scratch/bsmith/petsc/src/mat/tests (barry/2021-02-12/add-baij-12=) arch-add-baij-12 $ petscmpiexec -valgrind -n 1 ./ex237 ==14841== ==14841== Process terminating with default action of signal 4 (SIGILL) ==14841== Illegal opcode at address 0x4F808A9 ==14841== at 0x4F808A9: PetscSetDisplay (in /scratch/bsmith/petsc/arch-add-baij-12/lib/libpetsc.so.3.014.4) ==14841== by 0x4F086BD: PetscOptionsCheckInitial_Private (in /scratch/bsmith/petsc/arch-add-baij-12/lib/libpetsc.so.3.014.4) ==14841== by 0x4F0D5BC: PetscInitialize (in /scratch/bsmith/petsc/arch-add-baij-12/lib/libpetsc.so.3.014.4) ==14841== by 0x109DE0: main (in /scratch/bsmith/petsc/src/mat/tests/ex237) Illegal instruction (core dumped) /scratch/bsmith/petsc/src/mat/tests (barry/2021-02-12/add-baij-12=) arch-add-baij-12

1 0

Re: [petsc-dev] Understanding Vecscatter with Kokkos Vecs
by Barry Smith 20 Feb '21

20 Feb '21

Using the git history one can presumably get the version a function was introduced (appeared in the source). Changes are a bit trickier but I guess one could take the current function, march back in time in git until there is a diff appear. Of course many diffs are trivial. In theory one can list each set of changes for each function. Except when people mess up the history cutting and pasting and changing the cut code instead of the paste code. (Is there a way we can detect this and reject such MR?). Barry > On Feb 19, 2021, at 1:44 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > Patrick Sanan <patrick.sanan(a)gmail.com> writes: > >> Aside: it's been on the list of good things to do, docs-wise, to be able to label parts of the API as more or less stable, so I'm hoping we'll get to that (though I think it makes sense to wait until we've finished some of the current migrations tasks). > > Yes, I would love to have version introduced, version changed, and stability level as structured information for each function. Ideally, with tooling to alert when changing "stable" interfaces and to make clear to users when they are using interfaces that are not stable. > > Rust is a strict demonstration of this, with features limited to "nightly" (inaccessible with any flags when building with stable Rust) until they stabilize (if ever).

1 0

Re: [petsc-dev] Comparing binary output files in test harness
by Scott Kruger 19 Feb '21

19 Feb '21

Could you file a gitlab issue on this? Offhand, I think it'd be best to modify petscdiff to check the suffix and then pass it off to exodiff. This would be more easily extended to other file types (e.g., h5diff). Scott On 2021-02-19 04:59, Blaise A Bourdin did write: > Hi, > > I would like to write better tests for my exodus I/O functions and compare the binary files written to the drive instead of the output of the examples. > For instance, would it be possible to do the following: > ex26 -i <input mesh> -o output1.exo; mpirun -np 2 ex26 -i <input mesh> -o output2.exo; exodiff output1.exo output2.exo > And check the result of exodiff, or run exodiff between output1.exo or output2.exo and a stored binary result? > > Regards, > Blaise > > -- > A.K. & Shirley Barton Professor of Mathematics > Adjunct Professor of Mechanical Engineering > Adjunct of the Center for Computation & Technology > Louisiana State University, Lockett Hall Room 344, Baton Rouge, LA 70803, USA > Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 Web http://www.math.lsu.edu/~bourdin > -- Scott Kruger Tech-X Corporation kruger(a)txcorp.com 5621 Arapahoe Ave, Suite A Phone: (720) 466-3196 Boulder, CO 80303 Fax: (303) 448-7756

1 0

Re: [petsc-dev] Understanding Vecscatter with Kokkos Vecs
by Jed Brown 19 Feb '21

19 Feb '21

Patrick Sanan <patrick.sanan(a)gmail.com> writes: > Aside: it's been on the list of good things to do, docs-wise, to be able to label parts of the API as more or less stable, so I'm hoping we'll get to that (though I think it makes sense to wait until we've finished some of the current migrations tasks). Yes, I would love to have version introduced, version changed, and stability level as structured information for each function. Ideally, with tooling to alert when changing "stable" interfaces and to make clear to users when they are using interfaces that are not stable. Rust is a strict demonstration of this, with features limited to "nightly" (inaccessible with any flags when building with stable Rust) until they stabilize (if ever).

1 0

Re: [petsc-dev] Understanding Vecscatter with Kokkos Vecs
by Patrick Sanan 19 Feb '21

19 Feb '21

We ended up doing it just as you say - copy the data to the host, use it to build the IS, and build the scatter. Would be fun to optimize further, maybe, but as you say that might be premature since there's ongoing work. Happy to get to play with it a bit, though! Aside: it's been on the list of good things to do, docs-wise, to be able to label parts of the API as more or less stable, so I'm hoping we'll get to that (though I think it makes sense to wait until we've finished some of the current migrations tasks). > Am 19.02.2021 um 16:18 schrieb Jed Brown <jed(a)jedbrown.org>: > > ISCUDA isn't even right (perhaps ISGENERALCUDA, ISBLOCKCUDA). I agree that this isn't a priority, but I could see it being needed in the next few years to avoid bottlenecks in adaptive mesh refinement or other adaptive algorithms. It's not a small amount of work, but I think all the index coordination can be done efficiently on a GPU. > > Junchao Zhang <junchao.zhang(a)gmail.com> writes: > >> Even ISCUDA is simple to add, the PetscSFSetUp algorithm and many functions >> involved are done on host (and are not simple to be parallelized on GPU) >> The indices passed to VecScatter are analyzed and re-grouped. Even they are >> copied to device eventually, they are likely not in their original form. >> So, copying the indices from device to host and build a VecScatter there >> seems the easiest approach. >> >> The Kokkos-related functions are experimental. We need to decide whether >> they are good or not. >> >> --Junchao Zhang >> >> >> On Fri, Feb 19, 2021 at 4:32 AM Patrick Sanan <patrick.sanan(a)gmail.com> >> wrote: >> >>> Thanks! That helps a lot. >>> >>> I assume "no," but is ISCUDA simple to add? >>> >>> More on what I'm trying to do, in case I'm missing an obvious approach: >>> >>> I'm working on a demo code that uses an external library, based on Kokkos, >>> as a solver - I create a Vec of type KOKKOS and populate it with the >>> solution data from the library, by getting access to the raw Kokkos view >>> with VecKokkosGetDeviceView() * . >>> >>> I then want to reorder that solution data into PETSc-native ordering (for >>> a velocity-pressure DMStag), so I create a pair of ISs and a VecScatter to >>> do that. >>> >>> The issue is that to create this scatter, I need to use information >>> (essentially, an element-to-index map) from the external library's >>> mesh-management object, which lives on the device. This doesn't work (when >>> host != device), because of course the ISs live on the host and to create >>> them I need to provide host arrays of indices. >>> >>> Am I stuck, for now, with sending the index information information from >>> the device to the host, using it to create the IS, and then having >>> essentially the same information go back to the device when I use the >>> scatter? >>> >>> * As an aside, it looks like some of these Kokkos-related functions and >>> types are missing man pages - if you have time to add them, even as stubs, >>> that'd be great (if not let me know and I'll just try to formally do it, so >>> that at least the existence of the functions in the API is reflected on the >>> website). >>> >>> Am 18.02.2021 um 23:17 schrieb Junchao Zhang <junchao.zhang(a)gmail.com>: >>> >>> >>> On Thu, Feb 18, 2021 at 4:04 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: >>> >>>> >>>> >>>> On Thu, Feb 18, 2021 at 1:55 PM Junchao Zhang <junchao.zhang(a)gmail.com> >>>> wrote: >>>> >>>>> VecScatter (i.e., SF, the two are the same thing) setup (building >>>>> various index lists, rank lists) is done on the CPU. is1, is2 must be host >>>>> data. >>>>> >>>> >>>> Just out of curiosity, is1 and is2 can not be created on a GPU device in >>>> the first place? That being said, it is technically impossible? Or we just >>>> did not implement them yet? >>>> >>> Simply because we do not have an ISCUDA class. >>> >>> >>>> >>>> Fande, >>>> >>>> >>>>> When the SF is used to communicate device data, indices are copied to >>>>> the device.. >>>>> >>>>> --Junchao Zhang >>>>> >>>>> >>>>> On Thu, Feb 18, 2021 at 11:50 AM Patrick Sanan <patrick.sanan(a)gmail.com> >>>>> wrote: >>>>> >>>>>> I'm trying to understand how VecScatters work with GPU-native Kokkos >>>>>> Vecs. >>>>>> >>>>>> Specifically, I'm interested in what will happen in code like in >>>>>> src/vec/vec/tests/ex22.c, >>>>>> >>>>>> ierr = VecScatterCreate(x,is1,y,is2,&ctx);CHKERRQ(ierr); >>>>>> >>>>>> (from >>>>>> https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vec/tests/ex22.c#L44 >>>>>> ) >>>>>> >>>>>> Here, x and y can be set to type KOKKOS using -vec_type kokkos at the >>>>>> command line. But is1 and is2 are (I think), always >>>>>> CPU/host data. Assuming that the scatter itself can happen on the GPU, >>>>>> the indices must make it to the device somehow - are they copied there when >>>>>> the scatter is created? Is there a way to create the scatter using indices >>>>>> already on the GPU (Maybe using SF more directly)? >>>>>> >>>>>> >>>

1 0

Re: [petsc-dev] Understanding Vecscatter with Kokkos Vecs
by Jed Brown 19 Feb '21

19 Feb '21

ISCUDA isn't even right (perhaps ISGENERALCUDA, ISBLOCKCUDA). I agree that this isn't a priority, but I could see it being needed in the next few years to avoid bottlenecks in adaptive mesh refinement or other adaptive algorithms. It's not a small amount of work, but I think all the index coordination can be done efficiently on a GPU. Junchao Zhang <junchao.zhang(a)gmail.com> writes: > Even ISCUDA is simple to add, the PetscSFSetUp algorithm and many functions > involved are done on host (and are not simple to be parallelized on GPU) > The indices passed to VecScatter are analyzed and re-grouped. Even they are > copied to device eventually, they are likely not in their original form. > So, copying the indices from device to host and build a VecScatter there > seems the easiest approach. > > The Kokkos-related functions are experimental. We need to decide whether > they are good or not. > > --Junchao Zhang > > > On Fri, Feb 19, 2021 at 4:32 AM Patrick Sanan <patrick.sanan(a)gmail.com> > wrote: > >> Thanks! That helps a lot. >> >> I assume "no," but is ISCUDA simple to add? >> >> More on what I'm trying to do, in case I'm missing an obvious approach: >> >> I'm working on a demo code that uses an external library, based on Kokkos, >> as a solver - I create a Vec of type KOKKOS and populate it with the >> solution data from the library, by getting access to the raw Kokkos view >> with VecKokkosGetDeviceView() * . >> >> I then want to reorder that solution data into PETSc-native ordering (for >> a velocity-pressure DMStag), so I create a pair of ISs and a VecScatter to >> do that. >> >> The issue is that to create this scatter, I need to use information >> (essentially, an element-to-index map) from the external library's >> mesh-management object, which lives on the device. This doesn't work (when >> host != device), because of course the ISs live on the host and to create >> them I need to provide host arrays of indices. >> >> Am I stuck, for now, with sending the index information information from >> the device to the host, using it to create the IS, and then having >> essentially the same information go back to the device when I use the >> scatter? >> >> * As an aside, it looks like some of these Kokkos-related functions and >> types are missing man pages - if you have time to add them, even as stubs, >> that'd be great (if not let me know and I'll just try to formally do it, so >> that at least the existence of the functions in the API is reflected on the >> website). >> >> Am 18.02.2021 um 23:17 schrieb Junchao Zhang <junchao.zhang(a)gmail.com>: >> >> >> On Thu, Feb 18, 2021 at 4:04 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: >> >>> >>> >>> On Thu, Feb 18, 2021 at 1:55 PM Junchao Zhang <junchao.zhang(a)gmail.com> >>> wrote: >>> >>>> VecScatter (i.e., SF, the two are the same thing) setup (building >>>> various index lists, rank lists) is done on the CPU. is1, is2 must be host >>>> data. >>>> >>> >>> Just out of curiosity, is1 and is2 can not be created on a GPU device in >>> the first place? That being said, it is technically impossible? Or we just >>> did not implement them yet? >>> >> Simply because we do not have an ISCUDA class. >> >> >>> >>> Fande, >>> >>> >>>> When the SF is used to communicate device data, indices are copied to >>>> the device.. >>>> >>>> --Junchao Zhang >>>> >>>> >>>> On Thu, Feb 18, 2021 at 11:50 AM Patrick Sanan <patrick.sanan(a)gmail.com> >>>> wrote: >>>> >>>>> I'm trying to understand how VecScatters work with GPU-native Kokkos >>>>> Vecs. >>>>> >>>>> Specifically, I'm interested in what will happen in code like in >>>>> src/vec/vec/tests/ex22.c, >>>>> >>>>> ierr = VecScatterCreate(x,is1,y,is2,&ctx);CHKERRQ(ierr); >>>>> >>>>> (from >>>>> https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vec/tests/ex22.c#L44 >>>>> ) >>>>> >>>>> Here, x and y can be set to type KOKKOS using -vec_type kokkos at the >>>>> command line. But is1 and is2 are (I think), always >>>>> CPU/host data. Assuming that the scatter itself can happen on the GPU, >>>>> the indices must make it to the device somehow - are they copied there when >>>>> the scatter is created? Is there a way to create the scatter using indices >>>>> already on the GPU (Maybe using SF more directly)? >>>>> >>>>> >>

1 0

Re: [petsc-dev] Understanding Vecscatter with Kokkos Vecs
by Junchao Zhang 19 Feb '21

19 Feb '21

Even ISCUDA is simple to add, the PetscSFSetUp algorithm and many functions involved are done on host (and are not simple to be parallelized on GPU) The indices passed to VecScatter are analyzed and re-grouped. Even they are copied to device eventually, they are likely not in their original form. So, copying the indices from device to host and build a VecScatter there seems the easiest approach. The Kokkos-related functions are experimental. We need to decide whether they are good or not. --Junchao Zhang On Fri, Feb 19, 2021 at 4:32 AM Patrick Sanan <patrick.sanan(a)gmail.com> wrote: > Thanks! That helps a lot. > > I assume "no," but is ISCUDA simple to add? > > More on what I'm trying to do, in case I'm missing an obvious approach: > > I'm working on a demo code that uses an external library, based on Kokkos, > as a solver - I create a Vec of type KOKKOS and populate it with the > solution data from the library, by getting access to the raw Kokkos view > with VecKokkosGetDeviceView() * . > > I then want to reorder that solution data into PETSc-native ordering (for > a velocity-pressure DMStag), so I create a pair of ISs and a VecScatter to > do that. > > The issue is that to create this scatter, I need to use information > (essentially, an element-to-index map) from the external library's > mesh-management object, which lives on the device. This doesn't work (when > host != device), because of course the ISs live on the host and to create > them I need to provide host arrays of indices. > > Am I stuck, for now, with sending the index information information from > the device to the host, using it to create the IS, and then having > essentially the same information go back to the device when I use the > scatter? > > * As an aside, it looks like some of these Kokkos-related functions and > types are missing man pages - if you have time to add them, even as stubs, > that'd be great (if not let me know and I'll just try to formally do it, so > that at least the existence of the functions in the API is reflected on the > website). > > Am 18.02.2021 um 23:17 schrieb Junchao Zhang <junchao.zhang(a)gmail.com>: > > > On Thu, Feb 18, 2021 at 4:04 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: > >> >> >> On Thu, Feb 18, 2021 at 1:55 PM Junchao Zhang <junchao.zhang(a)gmail.com> >> wrote: >> >>> VecScatter (i.e., SF, the two are the same thing) setup (building >>> various index lists, rank lists) is done on the CPU. is1, is2 must be host >>> data. >>> >> >> Just out of curiosity, is1 and is2 can not be created on a GPU device in >> the first place? That being said, it is technically impossible? Or we just >> did not implement them yet? >> > Simply because we do not have an ISCUDA class. > > >> >> Fande, >> >> >>> When the SF is used to communicate device data, indices are copied to >>> the device.. >>> >>> --Junchao Zhang >>> >>> >>> On Thu, Feb 18, 2021 at 11:50 AM Patrick Sanan <patrick.sanan(a)gmail.com> >>> wrote: >>> >>>> I'm trying to understand how VecScatters work with GPU-native Kokkos >>>> Vecs. >>>> >>>> Specifically, I'm interested in what will happen in code like in >>>> src/vec/vec/tests/ex22.c, >>>> >>>> ierr = VecScatterCreate(x,is1,y,is2,&ctx);CHKERRQ(ierr); >>>> >>>> (from >>>> https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vec/tests/ex22.c#L44 >>>> ) >>>> >>>> Here, x and y can be set to type KOKKOS using -vec_type kokkos at the >>>> command line. But is1 and is2 are (I think), always >>>> CPU/host data. Assuming that the scatter itself can happen on the GPU, >>>> the indices must make it to the device somehow - are they copied there when >>>> the scatter is created? Is there a way to create the scatter using indices >>>> already on the GPU (Maybe using SF more directly)? >>>> >>>> >

1 0

Re: [petsc-dev] Understanding Vecscatter with Kokkos Vecs
by Patrick Sanan 19 Feb '21

19 Feb '21

True that it's not a huge efficiency concern, as this only affects the setup stage. This is more wondering if there's a simpler way to do the setup (sounds like there isn't at the moment). > Am 19.02.2021 um 11:41 schrieb Stefano Zampini <stefano.zampini(a)gmail.com>: > > I dont understand the issue. I assume your vecscatter is not a throw away object but you will reuse it many times. Once the setup is done, the indices used internally by the implementation should be on gpu , or not? > > Il Ven 19 Feb 2021, 13:33 Patrick Sanan <patrick.sanan(a)gmail.com <mailto:[email protected]>> ha scritto: > Thanks! That helps a lot. > > I assume "no," but is ISCUDA simple to add? > > More on what I'm trying to do, in case I'm missing an obvious approach: > > I'm working on a demo code that uses an external library, based on Kokkos, as a solver - I create a Vec of type KOKKOS and populate it with the solution data from the library, by getting access to the raw Kokkos view with VecKokkosGetDeviceView() * . > > I then want to reorder that solution data into PETSc-native ordering (for a velocity-pressure DMStag), so I create a pair of ISs and a VecScatter to do that. > > The issue is that to create this scatter, I need to use information (essentially, an element-to-index map) from the external library's mesh-management object, which lives on the device. This doesn't work (when host != device), because of course the ISs live on the host and to create them I need to provide host arrays of indices. > > Am I stuck, for now, with sending the index information information from the device to the host, using it to create the IS, and then having essentially the same information go back to the device when I use the scatter? > > * As an aside, it looks like some of these Kokkos-related functions and types are missing man pages - if you have time to add them, even as stubs, that'd be great (if not let me know and I'll just try to formally do it, so that at least the existence of the functions in the API is reflected on the website). > >> Am 18.02.2021 um 23:17 schrieb Junchao Zhang <junchao.zhang(a)gmail.com <mailto:[email protected]>>: >> >> >> On Thu, Feb 18, 2021 at 4:04 PM Fande Kong <fdkong.jd(a)gmail.com <mailto:[email protected]>> wrote: >> >> >> On Thu, Feb 18, 2021 at 1:55 PM Junchao Zhang <junchao.zhang(a)gmail.com <mailto:[email protected]>> wrote: >> VecScatter (i.e., SF, the two are the same thing) setup (building various index lists, rank lists) is done on the CPU. is1, is2 must be host data. >> >> Just out of curiosity, is1 and is2 can not be created on a GPU device in the first place? That being said, it is technically impossible? Or we just did not implement them yet? >> Simply because we do not have an ISCUDA class. >> >> >> Fande, >> >> When the SF is used to communicate device data, indices are copied to the device.. >> >> --Junchao Zhang >> >> >> On Thu, Feb 18, 2021 at 11:50 AM Patrick Sanan <patrick.sanan(a)gmail.com <mailto:[email protected]>> wrote: >> I'm trying to understand how VecScatters work with GPU-native Kokkos Vecs. >> >> Specifically, I'm interested in what will happen in code like in src/vec/vec/tests/ex22.c, >> >> ierr = VecScatterCreate(x,is1,y,is2,&ctx);CHKERRQ(ierr); >> >> (from https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vec/tests/ex22.c#L44 <https://gitlab.com/petsc/petsc/-/blob/master/src/vec/vec/tests/ex22.c#L44>) >> >> Here, x and y can be set to type KOKKOS using -vec_type kokkos at the command line. But is1 and is2 are (I think), always >> CPU/host data. Assuming that the scatter itself can happen on the GPU, the indices must make it to the device somehow - are they copied there when the scatter is created? Is there a way to create the scatter using indices already on the GPU (Maybe using SF more directly)? >> >

1 0