If you are referring to your device set values, I guess it is not currently tested See the discussions here https://gitlab.com/petsc/petsc/-/merge_requests/3411 I started cleaning up the code to prepare for testing but we never finished it https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-d... <https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-device/>
On May 28, 2021, at 6:53 PM, Mark Adams <[email protected]> wrote:
Is there a test with MatSetValues and CUDA?
On Fri, May 28, 2021 at 11:57 AM Stefano Zampini <[email protected]> wrote:
If you are referring to your device set values, I guess it is not currently tested
No. There is a test for that (ex5cu). I have a user that is getting a segv in MatSetValues with aijcusparse. I suspect there is memory corruption but I'm trying to cover all the bases. I have added a cuda test to ksp/ex56 that works. I can do an MR for it if such a test does not exist.
See the discussions here https://gitlab.com/petsc/petsc/-/merge_requests/3411 I started cleaning up the code to prepare for testing but we never finished it https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-d...
On May 28, 2021, at 6:53 PM, Mark Adams <[email protected]> wrote:
Is there a test with MatSetValues and CUDA?
That test is not run in the testsuite Il Ven 28 Mag 2021, 19:13 Mark Adams <[email protected]> ha scritto:
On Fri, May 28, 2021 at 11:57 AM Stefano Zampini < [email protected]> wrote:
If you are referring to your device set values, I guess it is not currently tested
No. There is a test for that (ex5cu). I have a user that is getting a segv in MatSetValues with aijcusparse. I suspect there is memory corruption but I'm trying to cover all the bases. I have added a cuda test to ksp/ex56 that works. I can do an MR for it if such a test does not exist.
See the discussions here https://gitlab.com/petsc/petsc/-/merge_requests/3411 I started cleaning up the code to prepare for testing but we never finished it https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-d...
On May 28, 2021, at 6:53 PM, Mark Adams <[email protected]> wrote:
Is there a test with MatSetValues and CUDA?
~/petsc/src/mat/tutorials (barry/2021-05-28/robustify-cuda-gencodearch-check=) arch-robustify-cuda-gencodearch-check $ ./ex5cu terminate called after throwing an instance of 'thrust::system::system_error' what(): fill_n: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered Aborted (core dumped) requires: cuda !define(PETSC_USE_CTABLE) CI does not test with CUDA and no ctable. The code is still broken as it was six months ago in the discussion Stefano pointed to. It is clear why just no one has had the time to clean things up. Barry
On May 28, 2021, at 11:13 AM, Mark Adams <[email protected]> wrote:
On Fri, May 28, 2021 at 11:57 AM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: If you are referring to your device set values, I guess it is not currently tested
No. There is a test for that (ex5cu). I have a user that is getting a segv in MatSetValues with aijcusparse. I suspect there is memory corruption but I'm trying to cover all the bases. I have added a cuda test to ksp/ex56 that works. I can do an MR for it if such a test does not exist.
See the discussions here https://gitlab.com/petsc/petsc/-/merge_requests/3411 <https://gitlab.com/petsc/petsc/-/merge_requests/3411> I started cleaning up the code to prepare for testing but we never finished it https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-d... <https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-device/>
On May 28, 2021, at 6:53 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
Is there a test with MatSetValues and CUDA?
That branch provides a fix for MatSetValuesDevice but it never got merged because of the CI issues with the —download-openmpi. We can probably try to skip the test in that specific configuration?
On May 28, 2021, at 7:45 PM, Barry Smith <[email protected]> wrote:
~/petsc/src/mat/tutorials (barry/2021-05-28/robustify-cuda-gencodearch-check=) arch-robustify-cuda-gencodearch-check $ ./ex5cu terminate called after throwing an instance of 'thrust::system::system_error' what(): fill_n: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered Aborted (core dumped)
requires: cuda !define(PETSC_USE_CTABLE)
CI does not test with CUDA and no ctable. The code is still broken as it was six months ago in the discussion Stefano pointed to. It is clear why just no one has had the time to clean things up.
Barry
On May 28, 2021, at 11:13 AM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
On Fri, May 28, 2021 at 11:57 AM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: If you are referring to your device set values, I guess it is not currently tested
No. There is a test for that (ex5cu). I have a user that is getting a segv in MatSetValues with aijcusparse. I suspect there is memory corruption but I'm trying to cover all the bases. I have added a cuda test to ksp/ex56 that works. I can do an MR for it if such a test does not exist.
See the discussions here https://gitlab.com/petsc/petsc/-/merge_requests/3411 <https://gitlab.com/petsc/petsc/-/merge_requests/3411> I started cleaning up the code to prepare for testing but we never finished it https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-d... <https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-device/>
On May 28, 2021, at 6:53 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
Is there a test with MatSetValues and CUDA?
Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
On May 28, 2021, at 8:12 PM, Stefano Zampini <[email protected]> wrote:
That branch provides a fix for MatSetValuesDevice but it never got merged because of the CI issues with the —download-openmpi. We can probably try to skip the test in that specific configuration?
On May 28, 2021, at 7:45 PM, Barry Smith <[email protected] <mailto:[email protected]>> wrote:
~/petsc/src/mat/tutorials (barry/2021-05-28/robustify-cuda-gencodearch-check=) arch-robustify-cuda-gencodearch-check $ ./ex5cu terminate called after throwing an instance of 'thrust::system::system_error' what(): fill_n: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered Aborted (core dumped)
requires: cuda !define(PETSC_USE_CTABLE)
CI does not test with CUDA and no ctable. The code is still broken as it was six months ago in the discussion Stefano pointed to. It is clear why just no one has had the time to clean things up.
Barry
On May 28, 2021, at 11:13 AM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
On Fri, May 28, 2021 at 11:57 AM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: If you are referring to your device set values, I guess it is not currently tested
No. There is a test for that (ex5cu). I have a user that is getting a segv in MatSetValues with aijcusparse. I suspect there is memory corruption but I'm trying to cover all the bases. I have added a cuda test to ksp/ex56 that works. I can do an MR for it if such a test does not exist.
See the discussions here https://gitlab.com/petsc/petsc/-/merge_requests/3411 <https://gitlab.com/petsc/petsc/-/merge_requests/3411> I started cleaning up the code to prepare for testing but we never finished it https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-d... <https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-device/>
On May 28, 2021, at 6:53 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
Is there a test with MatSetValues and CUDA?
I am fixing rebasing this branch over main. On Fri, May 28, 2021 at 1:16 PM Stefano Zampini <[email protected]> wrote:
Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
On May 28, 2021, at 8:12 PM, Stefano Zampini <[email protected]> wrote:
That branch provides a fix for MatSetValuesDevice but it never got merged because of the CI issues with the —download-openmpi. We can probably try to skip the test in that specific configuration?
On May 28, 2021, at 7:45 PM, Barry Smith <[email protected]> wrote:
~/petsc/src/mat/tutorials* (barry/2021-05-28/robustify-cuda-gencodearch-check=)* arch-robustify-cuda-gencodearch-check $ ./ex5cu terminate called after throwing an instance of 'thrust::system::system_error' what(): fill_n: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered Aborted (core dumped)
requires: cuda !define(PETSC_USE_CTABLE)
CI does not test with CUDA and no ctable. The code is still broken as it was six months ago in the discussion Stefano pointed to. It is clear why just no one has had the time to clean things up.
Barry
On May 28, 2021, at 11:13 AM, Mark Adams <[email protected]> wrote:
On Fri, May 28, 2021 at 11:57 AM Stefano Zampini < [email protected]> wrote:
If you are referring to your device set values, I guess it is not currently tested
No. There is a test for that (ex5cu). I have a user that is getting a segv in MatSetValues with aijcusparse. I suspect there is memory corruption but I'm trying to cover all the bases. I have added a cuda test to ksp/ex56 that works. I can do an MR for it if such a test does not exist.
See the discussions here https://gitlab.com/petsc/petsc/-/merge_requests/3411 I started cleaning up the code to prepare for testing but we never finished it https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-d...
On May 28, 2021, at 6:53 PM, Mark Adams <[email protected]> wrote:
Is there a test with MatSetValues and CUDA?
Is this the correct branch? It conflicted with ex5cu so I assume it is. stefanozampini/simplify-setvalues-device <https://gitlab.com/petsc/petsc/-/tree/stefanozampini/simplify-setvalues-device> On Fri, May 28, 2021 at 1:24 PM Mark Adams <[email protected]> wrote:
I am fixing rebasing this branch over main.
On Fri, May 28, 2021 at 1:16 PM Stefano Zampini <[email protected]> wrote:
Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
On May 28, 2021, at 8:12 PM, Stefano Zampini <[email protected]> wrote:
That branch provides a fix for MatSetValuesDevice but it never got merged because of the CI issues with the —download-openmpi. We can probably try to skip the test in that specific configuration?
On May 28, 2021, at 7:45 PM, Barry Smith <[email protected]> wrote:
~/petsc/src/mat/tutorials* (barry/2021-05-28/robustify-cuda-gencodearch-check=)* arch-robustify-cuda-gencodearch-check $ ./ex5cu terminate called after throwing an instance of 'thrust::system::system_error' what(): fill_n: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered Aborted (core dumped)
requires: cuda !define(PETSC_USE_CTABLE)
CI does not test with CUDA and no ctable. The code is still broken as it was six months ago in the discussion Stefano pointed to. It is clear why just no one has had the time to clean things up.
Barry
On May 28, 2021, at 11:13 AM, Mark Adams <[email protected]> wrote:
On Fri, May 28, 2021 at 11:57 AM Stefano Zampini < [email protected]> wrote:
If you are referring to your device set values, I guess it is not currently tested
No. There is a test for that (ex5cu). I have a user that is getting a segv in MatSetValues with aijcusparse. I suspect there is memory corruption but I'm trying to cover all the bases. I have added a cuda test to ksp/ex56 that works. I can do an MR for it if such a test does not exist.
See the discussions here https://gitlab.com/petsc/petsc/-/merge_requests/3411 I started cleaning up the code to prepare for testing but we never finished it https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-d...
On May 28, 2021, at 6:53 PM, Mark Adams <[email protected]> wrote:
Is there a test with MatSetValues and CUDA?
Yes, it is the branch I was using before force pushing to Barry’s barry/2020-11-11/cleanup-matsetvaluesdevice You can use both I guess
On May 28, 2021, at 8:25 PM, Mark Adams <[email protected]> wrote:
Is this the correct branch? It conflicted with ex5cu so I assume it is.
stefanozampini/simplify-setvalues-device <https://gitlab.com/petsc/petsc/-/tree/stefanozampini/simplify-setvalues-device>
On Fri, May 28, 2021 at 1:24 PM Mark Adams <[email protected] <mailto:[email protected]>> wrote: I am fixing rebasing this branch over main.
On Fri, May 28, 2021 at 1:16 PM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
On May 28, 2021, at 8:12 PM, Stefano Zampini <[email protected] <mailto:[email protected]>> wrote:
That branch provides a fix for MatSetValuesDevice but it never got merged because of the CI issues with the —download-openmpi. We can probably try to skip the test in that specific configuration?
On May 28, 2021, at 7:45 PM, Barry Smith <[email protected] <mailto:[email protected]>> wrote:
~/petsc/src/mat/tutorials (barry/2021-05-28/robustify-cuda-gencodearch-check=) arch-robustify-cuda-gencodearch-check $ ./ex5cu terminate called after throwing an instance of 'thrust::system::system_error' what(): fill_n: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered Aborted (core dumped)
requires: cuda !define(PETSC_USE_CTABLE)
CI does not test with CUDA and no ctable. The code is still broken as it was six months ago in the discussion Stefano pointed to. It is clear why just no one has had the time to clean things up.
Barry
On May 28, 2021, at 11:13 AM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
On Fri, May 28, 2021 at 11:57 AM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: If you are referring to your device set values, I guess it is not currently tested
No. There is a test for that (ex5cu). I have a user that is getting a segv in MatSetValues with aijcusparse. I suspect there is memory corruption but I'm trying to cover all the bases. I have added a cuda test to ksp/ex56 that works. I can do an MR for it if such a test does not exist.
See the discussions here https://gitlab.com/petsc/petsc/-/merge_requests/3411 <https://gitlab.com/petsc/petsc/-/merge_requests/3411> I started cleaning up the code to prepare for testing but we never finished it https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-d... <https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-device/>
On May 28, 2021, at 6:53 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
Is there a test with MatSetValues and CUDA?
OK, I will try to rebase and test Barry's branch. On Fri, May 28, 2021 at 1:26 PM Stefano Zampini <[email protected]> wrote:
Yes, it is the branch I was using before force pushing to Barry’s barry/2020-11-11/cleanup-matsetvaluesdevice You can use both I guess
On May 28, 2021, at 8:25 PM, Mark Adams <[email protected]> wrote:
Is this the correct branch? It conflicted with ex5cu so I assume it is.
stefanozampini/simplify-setvalues-device <https://gitlab.com/petsc/petsc/-/tree/stefanozampini/simplify-setvalues-device>
On Fri, May 28, 2021 at 1:24 PM Mark Adams <[email protected]> wrote:
I am fixing rebasing this branch over main.
On Fri, May 28, 2021 at 1:16 PM Stefano Zampini < [email protected]> wrote:
Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
On May 28, 2021, at 8:12 PM, Stefano Zampini <[email protected]> wrote:
That branch provides a fix for MatSetValuesDevice but it never got merged because of the CI issues with the —download-openmpi. We can probably try to skip the test in that specific configuration?
On May 28, 2021, at 7:45 PM, Barry Smith <[email protected]> wrote:
~/petsc/src/mat/tutorials* (barry/2021-05-28/robustify-cuda-gencodearch-check=)* arch-robustify-cuda-gencodearch-check $ ./ex5cu terminate called after throwing an instance of 'thrust::system::system_error' what(): fill_n: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered Aborted (core dumped)
requires: cuda !define(PETSC_USE_CTABLE)
CI does not test with CUDA and no ctable. The code is still broken as it was six months ago in the discussion Stefano pointed to. It is clear why just no one has had the time to clean things up.
Barry
On May 28, 2021, at 11:13 AM, Mark Adams <[email protected]> wrote:
On Fri, May 28, 2021 at 11:57 AM Stefano Zampini < [email protected]> wrote:
If you are referring to your device set values, I guess it is not currently tested
No. There is a test for that (ex5cu). I have a user that is getting a segv in MatSetValues with aijcusparse. I suspect there is memory corruption but I'm trying to cover all the bases. I have added a cuda test to ksp/ex56 that works. I can do an MR for it if such a test does not exist.
See the discussions here https://gitlab.com/petsc/petsc/-/merge_requests/3411 I started cleaning up the code to prepare for testing but we never finished it https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-d...
On May 28, 2021, at 6:53 PM, Mark Adams <[email protected]> wrote:
Is there a test with MatSetValues and CUDA?
Stefano, who has a far better memory than me, wrote
Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
MPI.py depends on cuda.py so we cannot also have cuda.py depend on MPI.py using the generic dependencies of configure/packages but perhaps we can just hardwire the rerunning of cuda.py when the MPI compilers are reset. I will try that now and if I can get it to work we should be able to move those old fix branches along as MR. Barry
On May 28, 2021, at 12:41 PM, Mark Adams <[email protected]> wrote:
OK, I will try to rebase and test Barry's branch.
On Fri, May 28, 2021 at 1:26 PM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: Yes, it is the branch I was using before force pushing to Barry’s barry/2020-11-11/cleanup-matsetvaluesdevice You can use both I guess
On May 28, 2021, at 8:25 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
Is this the correct branch? It conflicted with ex5cu so I assume it is.
stefanozampini/simplify-setvalues-device <https://gitlab.com/petsc/petsc/-/tree/stefanozampini/simplify-setvalues-device>
On Fri, May 28, 2021 at 1:24 PM Mark Adams <[email protected] <mailto:[email protected]>> wrote: I am fixing rebasing this branch over main.
On Fri, May 28, 2021 at 1:16 PM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
On May 28, 2021, at 8:12 PM, Stefano Zampini <[email protected] <mailto:[email protected]>> wrote:
That branch provides a fix for MatSetValuesDevice but it never got merged because of the CI issues with the —download-openmpi. We can probably try to skip the test in that specific configuration?
On May 28, 2021, at 7:45 PM, Barry Smith <[email protected] <mailto:[email protected]>> wrote:
~/petsc/src/mat/tutorials (barry/2021-05-28/robustify-cuda-gencodearch-check=) arch-robustify-cuda-gencodearch-check $ ./ex5cu terminate called after throwing an instance of 'thrust::system::system_error' what(): fill_n: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered Aborted (core dumped)
requires: cuda !define(PETSC_USE_CTABLE)
CI does not test with CUDA and no ctable. The code is still broken as it was six months ago in the discussion Stefano pointed to. It is clear why just no one has had the time to clean things up.
Barry
On May 28, 2021, at 11:13 AM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
On Fri, May 28, 2021 at 11:57 AM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: If you are referring to your device set values, I guess it is not currently tested
No. There is a test for that (ex5cu). I have a user that is getting a segv in MatSetValues with aijcusparse. I suspect there is memory corruption but I'm trying to cover all the bases. I have added a cuda test to ksp/ex56 that works. I can do an MR for it if such a test does not exist.
See the discussions here https://gitlab.com/petsc/petsc/-/merge_requests/3411 <https://gitlab.com/petsc/petsc/-/merge_requests/3411> I started cleaning up the code to prepare for testing but we never finished it https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-d... <https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-device/>
On May 28, 2021, at 6:53 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
Is there a test with MatSetValues and CUDA?
OpenMPI.py depends on cuda.py in that, if cuda is present, configures using cuda. MPI.py or MPICH.py do not depend on cuda.py (MPICH, only weakly, it adds a print if cuda is present) Since eventually the MPI distro will only need a hint to be configured with CUDA, why not removing the dependency at all and add only a flag —download-openmpi-use-cuda?
On May 28, 2021, at 8:44 PM, Barry Smith <[email protected]> wrote:
Stefano, who has a far better memory than me, wrote
Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
MPI.py depends on cuda.py so we cannot also have cuda.py depend on MPI.py using the generic dependencies of configure/packages
but perhaps we can just hardwire the rerunning of cuda.py when the MPI compilers are reset. I will try that now and if I can get it to work we should be able to move those old fix branches along as MR.
Barry
On May 28, 2021, at 12:41 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
OK, I will try to rebase and test Barry's branch.
On Fri, May 28, 2021 at 1:26 PM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: Yes, it is the branch I was using before force pushing to Barry’s barry/2020-11-11/cleanup-matsetvaluesdevice You can use both I guess
On May 28, 2021, at 8:25 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
Is this the correct branch? It conflicted with ex5cu so I assume it is.
stefanozampini/simplify-setvalues-device <https://gitlab.com/petsc/petsc/-/tree/stefanozampini/simplify-setvalues-device>
On Fri, May 28, 2021 at 1:24 PM Mark Adams <[email protected] <mailto:[email protected]>> wrote: I am fixing rebasing this branch over main.
On Fri, May 28, 2021 at 1:16 PM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
On May 28, 2021, at 8:12 PM, Stefano Zampini <[email protected] <mailto:[email protected]>> wrote:
That branch provides a fix for MatSetValuesDevice but it never got merged because of the CI issues with the —download-openmpi. We can probably try to skip the test in that specific configuration?
On May 28, 2021, at 7:45 PM, Barry Smith <[email protected] <mailto:[email protected]>> wrote:
~/petsc/src/mat/tutorials (barry/2021-05-28/robustify-cuda-gencodearch-check=) arch-robustify-cuda-gencodearch-check $ ./ex5cu terminate called after throwing an instance of 'thrust::system::system_error' what(): fill_n: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered Aborted (core dumped)
requires: cuda !define(PETSC_USE_CTABLE)
CI does not test with CUDA and no ctable. The code is still broken as it was six months ago in the discussion Stefano pointed to. It is clear why just no one has had the time to clean things up.
Barry
On May 28, 2021, at 11:13 AM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
On Fri, May 28, 2021 at 11:57 AM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: If you are referring to your device set values, I guess it is not currently tested
No. There is a test for that (ex5cu). I have a user that is getting a segv in MatSetValues with aijcusparse. I suspect there is memory corruption but I'm trying to cover all the bases. I have added a cuda test to ksp/ex56 that works. I can do an MR for it if such a test does not exist.
See the discussions here https://gitlab.com/petsc/petsc/-/merge_requests/3411 <https://gitlab.com/petsc/petsc/-/merge_requests/3411> I started cleaning up the code to prepare for testing but we never finished it https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-d... <https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-device/>
On May 28, 2021, at 6:53 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
Is there a test with MatSetValues and CUDA?
I am rebasing over main and its a bit of a mess. I must have missed something. I get this. I think the _n_SplitCSRMat must be wrong. In file included from /autofs/nccs-svm1_home1/adams/petsc/src/vec/is/sf/impls/basic/sfbasic.c:128:0: /ccs/home/adams/petsc/include/petscmat.h:1976:32: error: conflicting types for 'PetscSplitCSRDataStructure' typedef struct _n_SplitCSRMat *PetscSplitCSRDataStructure; ^~~~~~~~~~~~~~~~~~~~~~~~~~ /ccs/home/adams/petsc/include/petscmat.h:1922:31: note: previous declaration of 'PetscSplitCSRDataStructure' was here typedef struct _p_SplitCSRMat PetscSplitCSRDataStructure; ^~~~~~~~~~~~~~~~~~~~~~~~~~ CC arch-summit-opt-gnu-cuda/obj/vec/vec/impls/seq/dvec2.o On Fri, May 28, 2021 at 1:50 PM Stefano Zampini <[email protected]> wrote:
OpenMPI.py depends on cuda.py in that, if cuda is present, configures using cuda. MPI.py or MPICH.py do not depend on cuda.py (MPICH, only weakly, it adds a print if cuda is present) Since eventually the MPI distro will only need a hint to be configured with CUDA, why not removing the dependency at all and add only a flag —download-openmpi-use-cuda?
On May 28, 2021, at 8:44 PM, Barry Smith <[email protected]> wrote:
Stefano, who has a far better memory than me, wrote
Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
MPI.py depends on cuda.py so we cannot also have cuda.py depend on MPI.py using the generic dependencies of configure/packages
but perhaps we can just hardwire the rerunning of cuda.py when the MPI compilers are reset. I will try that now and if I can get it to work we should be able to move those old fix branches along as MR.
Barry
On May 28, 2021, at 12:41 PM, Mark Adams <[email protected]> wrote:
OK, I will try to rebase and test Barry's branch.
On Fri, May 28, 2021 at 1:26 PM Stefano Zampini <[email protected]> wrote:
Yes, it is the branch I was using before force pushing to Barry’s barry/2020-11-11/cleanup-matsetvaluesdevice You can use both I guess
On May 28, 2021, at 8:25 PM, Mark Adams <[email protected]> wrote:
Is this the correct branch? It conflicted with ex5cu so I assume it is.
stefanozampini/simplify-setvalues-device <https://gitlab.com/petsc/petsc/-/tree/stefanozampini/simplify-setvalues-device>
On Fri, May 28, 2021 at 1:24 PM Mark Adams <[email protected]> wrote:
I am fixing rebasing this branch over main.
On Fri, May 28, 2021 at 1:16 PM Stefano Zampini < [email protected]> wrote:
Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
On May 28, 2021, at 8:12 PM, Stefano Zampini <[email protected]> wrote:
That branch provides a fix for MatSetValuesDevice but it never got merged because of the CI issues with the —download-openmpi. We can probably try to skip the test in that specific configuration?
On May 28, 2021, at 7:45 PM, Barry Smith <[email protected]> wrote:
~/petsc/src/mat/tutorials* (barry/2021-05-28/robustify-cuda-gencodearch-check=)* arch-robustify-cuda-gencodearch-check $ ./ex5cu terminate called after throwing an instance of 'thrust::system::system_error' what(): fill_n: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered Aborted (core dumped)
requires: cuda !define(PETSC_USE_CTABLE)
CI does not test with CUDA and no ctable. The code is still broken as it was six months ago in the discussion Stefano pointed to. It is clear why just no one has had the time to clean things up.
Barry
On May 28, 2021, at 11:13 AM, Mark Adams <[email protected]> wrote:
On Fri, May 28, 2021 at 11:57 AM Stefano Zampini < [email protected]> wrote:
If you are referring to your device set values, I guess it is not currently tested
No. There is a test for that (ex5cu). I have a user that is getting a segv in MatSetValues with aijcusparse. I suspect there is memory corruption but I'm trying to cover all the bases. I have added a cuda test to ksp/ex56 that works. I can do an MR for it if such a test does not exist.
See the discussions here https://gitlab.com/petsc/petsc/-/merge_requests/3411 I started cleaning up the code to prepare for testing but we never finished it https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-d...
On May 28, 2021, at 6:53 PM, Mark Adams <[email protected]> wrote:
Is there a test with MatSetValues and CUDA?
I am getting messed up in trying to resolve conflicts in rebasing over main. Is there a better way of doing this? Can I just tell git to use Barry's version and then test it? Or should I just try it again? On Fri, May 28, 2021 at 2:15 PM Mark Adams <[email protected]> wrote:
I am rebasing over main and its a bit of a mess. I must have missed something. I get this. I think the _n_SplitCSRMat must be wrong.
In file included from /autofs/nccs-svm1_home1/adams/petsc/src/vec/is/sf/impls/basic/sfbasic.c:128:0: /ccs/home/adams/petsc/include/petscmat.h:1976:32: error: conflicting types for 'PetscSplitCSRDataStructure' typedef struct _n_SplitCSRMat *PetscSplitCSRDataStructure; ^~~~~~~~~~~~~~~~~~~~~~~~~~ /ccs/home/adams/petsc/include/petscmat.h:1922:31: note: previous declaration of 'PetscSplitCSRDataStructure' was here typedef struct _p_SplitCSRMat PetscSplitCSRDataStructure; ^~~~~~~~~~~~~~~~~~~~~~~~~~ CC arch-summit-opt-gnu-cuda/obj/vec/vec/impls/seq/dvec2.o
On Fri, May 28, 2021 at 1:50 PM Stefano Zampini <[email protected]> wrote:
OpenMPI.py depends on cuda.py in that, if cuda is present, configures using cuda. MPI.py or MPICH.py do not depend on cuda.py (MPICH, only weakly, it adds a print if cuda is present) Since eventually the MPI distro will only need a hint to be configured with CUDA, why not removing the dependency at all and add only a flag —download-openmpi-use-cuda?
On May 28, 2021, at 8:44 PM, Barry Smith <[email protected]> wrote:
Stefano, who has a far better memory than me, wrote
Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
MPI.py depends on cuda.py so we cannot also have cuda.py depend on MPI.py using the generic dependencies of configure/packages
but perhaps we can just hardwire the rerunning of cuda.py when the MPI compilers are reset. I will try that now and if I can get it to work we should be able to move those old fix branches along as MR.
Barry
On May 28, 2021, at 12:41 PM, Mark Adams <[email protected]> wrote:
OK, I will try to rebase and test Barry's branch.
On Fri, May 28, 2021 at 1:26 PM Stefano Zampini < [email protected]> wrote:
Yes, it is the branch I was using before force pushing to Barry’s barry/2020-11-11/cleanup-matsetvaluesdevice You can use both I guess
On May 28, 2021, at 8:25 PM, Mark Adams <[email protected]> wrote:
Is this the correct branch? It conflicted with ex5cu so I assume it is.
stefanozampini/simplify-setvalues-device <https://gitlab.com/petsc/petsc/-/tree/stefanozampini/simplify-setvalues-device>
On Fri, May 28, 2021 at 1:24 PM Mark Adams <[email protected]> wrote:
I am fixing rebasing this branch over main.
On Fri, May 28, 2021 at 1:16 PM Stefano Zampini < [email protected]> wrote:
Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
On May 28, 2021, at 8:12 PM, Stefano Zampini < [email protected]> wrote:
That branch provides a fix for MatSetValuesDevice but it never got merged because of the CI issues with the —download-openmpi. We can probably try to skip the test in that specific configuration?
On May 28, 2021, at 7:45 PM, Barry Smith <[email protected]> wrote:
~/petsc/src/mat/tutorials* (barry/2021-05-28/robustify-cuda-gencodearch-check=)* arch-robustify-cuda-gencodearch-check $ ./ex5cu terminate called after throwing an instance of 'thrust::system::system_error' what(): fill_n: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered Aborted (core dumped)
requires: cuda !define(PETSC_USE_CTABLE)
CI does not test with CUDA and no ctable. The code is still broken as it was six months ago in the discussion Stefano pointed to. It is clear why just no one has had the time to clean things up.
Barry
On May 28, 2021, at 11:13 AM, Mark Adams <[email protected]> wrote:
On Fri, May 28, 2021 at 11:57 AM Stefano Zampini < [email protected]> wrote:
If you are referring to your device set values, I guess it is not currently tested
No. There is a test for that (ex5cu). I have a user that is getting a segv in MatSetValues with aijcusparse. I suspect there is memory corruption but I'm trying to cover all the bases. I have added a cuda test to ksp/ex56 that works. I can do an MR for it if such a test does not exist.
See the discussions here https://gitlab.com/petsc/petsc/-/merge_requests/3411 I started cleaning up the code to prepare for testing but we never finished it https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-d...
On May 28, 2021, at 6:53 PM, Mark Adams <[email protected]> wrote:
Is there a test with MatSetValues and CUDA?
I can take a quick look at it tomorrow, what are the main changes you made since then?
On May 28, 2021, at 9:51 PM, Mark Adams <[email protected]> wrote:
I am getting messed up in trying to resolve conflicts in rebasing over main. Is there a better way of doing this? Can I just tell git to use Barry's version and then test it? Or should I just try it again?
On Fri, May 28, 2021 at 2:15 PM Mark Adams <[email protected] <mailto:[email protected]>> wrote: I am rebasing over main and its a bit of a mess. I must have missed something. I get this. I think the _n_SplitCSRMat must be wrong.
In file included from /autofs/nccs-svm1_home1/adams/petsc/src/vec/is/sf/impls/basic/sfbasic.c:128:0: /ccs/home/adams/petsc/include/petscmat.h:1976:32: error: conflicting types for 'PetscSplitCSRDataStructure' typedef struct _n_SplitCSRMat *PetscSplitCSRDataStructure; ^~~~~~~~~~~~~~~~~~~~~~~~~~ /ccs/home/adams/petsc/include/petscmat.h:1922:31: note: previous declaration of 'PetscSplitCSRDataStructure' was here typedef struct _p_SplitCSRMat PetscSplitCSRDataStructure; ^~~~~~~~~~~~~~~~~~~~~~~~~~ CC arch-summit-opt-gnu-cuda/obj/vec/vec/impls/seq/dvec2.o
On Fri, May 28, 2021 at 1:50 PM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: OpenMPI.py depends on cuda.py in that, if cuda is present, configures using cuda. MPI.py or MPICH.py do not depend on cuda.py (MPICH, only weakly, it adds a print if cuda is present) Since eventually the MPI distro will only need a hint to be configured with CUDA, why not removing the dependency at all and add only a flag —download-openmpi-use-cuda?
On May 28, 2021, at 8:44 PM, Barry Smith <[email protected] <mailto:[email protected]>> wrote:
Stefano, who has a far better memory than me, wrote
Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
MPI.py depends on cuda.py so we cannot also have cuda.py depend on MPI.py using the generic dependencies of configure/packages
but perhaps we can just hardwire the rerunning of cuda.py when the MPI compilers are reset. I will try that now and if I can get it to work we should be able to move those old fix branches along as MR.
Barry
On May 28, 2021, at 12:41 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
OK, I will try to rebase and test Barry's branch.
On Fri, May 28, 2021 at 1:26 PM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: Yes, it is the branch I was using before force pushing to Barry’s barry/2020-11-11/cleanup-matsetvaluesdevice You can use both I guess
On May 28, 2021, at 8:25 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
Is this the correct branch? It conflicted with ex5cu so I assume it is.
stefanozampini/simplify-setvalues-device <https://gitlab.com/petsc/petsc/-/tree/stefanozampini/simplify-setvalues-device>
On Fri, May 28, 2021 at 1:24 PM Mark Adams <[email protected] <mailto:[email protected]>> wrote: I am fixing rebasing this branch over main.
On Fri, May 28, 2021 at 1:16 PM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
On May 28, 2021, at 8:12 PM, Stefano Zampini <[email protected] <mailto:[email protected]>> wrote:
That branch provides a fix for MatSetValuesDevice but it never got merged because of the CI issues with the —download-openmpi. We can probably try to skip the test in that specific configuration?
On May 28, 2021, at 7:45 PM, Barry Smith <[email protected] <mailto:[email protected]>> wrote:
~/petsc/src/mat/tutorials (barry/2021-05-28/robustify-cuda-gencodearch-check=) arch-robustify-cuda-gencodearch-check $ ./ex5cu terminate called after throwing an instance of 'thrust::system::system_error' what(): fill_n: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered Aborted (core dumped)
requires: cuda !define(PETSC_USE_CTABLE)
CI does not test with CUDA and no ctable. The code is still broken as it was six months ago in the discussion Stefano pointed to. It is clear why just no one has had the time to clean things up.
Barry
On May 28, 2021, at 11:13 AM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
On Fri, May 28, 2021 at 11:57 AM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: If you are referring to your device set values, I guess it is not currently tested
No. There is a test for that (ex5cu). I have a user that is getting a segv in MatSetValues with aijcusparse. I suspect there is memory corruption but I'm trying to cover all the bases. I have added a cuda test to ksp/ex56 that works. I can do an MR for it if such a test does not exist.
See the discussions here https://gitlab.com/petsc/petsc/-/merge_requests/3411 <https://gitlab.com/petsc/petsc/-/merge_requests/3411> I started cleaning up the code to prepare for testing but we never finished it https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-d... <https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-device/>
On May 28, 2021, at 6:53 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
Is there a test with MatSetValues and CUDA?
Thanks, I did not intend to make any (real) changes. The only thing that I did not intend to use from Barry's branch, that conflicted, was the help and comment block at the top of ex5cu.cu * I ended up with two declarations of PetscSplitCSRDataStructure * I added some includes to fix errors like this: /ccs/home/adams/petsc/include/../src/mat/impls/aij/seq/seqcusparse/cusparsematimpl.h(263): error: incomplete type is not allowed * I end ended not having csr2csc_i in Mat_SeqAIJCUSPARSE so I get: /autofs/nccs-svm1_home1/adams/petsc/src/mat/impls/aij/seq/seqcusparse/ aijcusparse.cu(1348): error: class "Mat_SeqAIJCUSPARSE" has no member "csr2csc_i" On Fri, May 28, 2021 at 3:13 PM Stefano Zampini <[email protected]> wrote:
I can take a quick look at it tomorrow, what are the main changes you made since then?
On May 28, 2021, at 9:51 PM, Mark Adams <[email protected]> wrote:
I am getting messed up in trying to resolve conflicts in rebasing over main. Is there a better way of doing this? Can I just tell git to use Barry's version and then test it? Or should I just try it again?
On Fri, May 28, 2021 at 2:15 PM Mark Adams <[email protected]> wrote:
I am rebasing over main and its a bit of a mess. I must have missed something. I get this. I think the _n_SplitCSRMat must be wrong.
In file included from /autofs/nccs-svm1_home1/adams/petsc/src/vec/is/sf/impls/basic/sfbasic.c:128:0: /ccs/home/adams/petsc/include/petscmat.h:1976:32: error: conflicting types for 'PetscSplitCSRDataStructure' typedef struct _n_SplitCSRMat *PetscSplitCSRDataStructure; ^~~~~~~~~~~~~~~~~~~~~~~~~~ /ccs/home/adams/petsc/include/petscmat.h:1922:31: note: previous declaration of 'PetscSplitCSRDataStructure' was here typedef struct _p_SplitCSRMat PetscSplitCSRDataStructure; ^~~~~~~~~~~~~~~~~~~~~~~~~~ CC arch-summit-opt-gnu-cuda/obj/vec/vec/impls/seq/dvec2.o
On Fri, May 28, 2021 at 1:50 PM Stefano Zampini < [email protected]> wrote:
OpenMPI.py depends on cuda.py in that, if cuda is present, configures using cuda. MPI.py or MPICH.py do not depend on cuda.py (MPICH, only weakly, it adds a print if cuda is present) Since eventually the MPI distro will only need a hint to be configured with CUDA, why not removing the dependency at all and add only a flag —download-openmpi-use-cuda?
On May 28, 2021, at 8:44 PM, Barry Smith <[email protected]> wrote:
Stefano, who has a far better memory than me, wrote
Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
MPI.py depends on cuda.py so we cannot also have cuda.py depend on MPI.py using the generic dependencies of configure/packages
but perhaps we can just hardwire the rerunning of cuda.py when the MPI compilers are reset. I will try that now and if I can get it to work we should be able to move those old fix branches along as MR.
Barry
On May 28, 2021, at 12:41 PM, Mark Adams <[email protected]> wrote:
OK, I will try to rebase and test Barry's branch.
On Fri, May 28, 2021 at 1:26 PM Stefano Zampini < [email protected]> wrote:
Yes, it is the branch I was using before force pushing to Barry’s barry/2020-11-11/cleanup-matsetvaluesdevice You can use both I guess
On May 28, 2021, at 8:25 PM, Mark Adams <[email protected]> wrote:
Is this the correct branch? It conflicted with ex5cu so I assume it is.
stefanozampini/simplify-setvalues-device <https://gitlab.com/petsc/petsc/-/tree/stefanozampini/simplify-setvalues-device>
On Fri, May 28, 2021 at 1:24 PM Mark Adams <[email protected]> wrote:
I am fixing rebasing this branch over main.
On Fri, May 28, 2021 at 1:16 PM Stefano Zampini < [email protected]> wrote:
Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
On May 28, 2021, at 8:12 PM, Stefano Zampini < [email protected]> wrote:
That branch provides a fix for MatSetValuesDevice but it never got merged because of the CI issues with the —download-openmpi. We can probably try to skip the test in that specific configuration?
On May 28, 2021, at 7:45 PM, Barry Smith <[email protected]> wrote:
~/petsc/src/mat/tutorials* (barry/2021-05-28/robustify-cuda-gencodearch-check=)* arch-robustify-cuda-gencodearch-check $ ./ex5cu terminate called after throwing an instance of 'thrust::system::system_error' what(): fill_n: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered Aborted (core dumped)
requires: cuda !define(PETSC_USE_CTABLE)
CI does not test with CUDA and no ctable. The code is still broken as it was six months ago in the discussion Stefano pointed to. It is clear why just no one has had the time to clean things up.
Barry
On May 28, 2021, at 11:13 AM, Mark Adams <[email protected]> wrote:
On Fri, May 28, 2021 at 11:57 AM Stefano Zampini < [email protected]> wrote:
If you are referring to your device set values, I guess it is not currently tested
No. There is a test for that (ex5cu). I have a user that is getting a segv in MatSetValues with aijcusparse. I suspect there is memory corruption but I'm trying to cover all the bases. I have added a cuda test to ksp/ex56 that works. I can do an MR for it if such a test does not exist.
See the discussions here https://gitlab.com/petsc/petsc/-/merge_requests/3411 I started cleaning up the code to prepare for testing but we never finished it https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-d...
On May 28, 2021, at 6:53 PM, Mark Adams <[email protected]> wrote:
Is there a test with MatSetValues and CUDA?
I have rebased and tried to fix everything. I am now fixing the issues of --download-openmpi and cuda, once that is done I will test, rebase with main again if needed and restart the MR and get it into main. Barry I was stupid to let the MR lay fallow, I should have figured out a solution to the openmpi and cuda issue instead of punting and waiting for a dream fix.
On May 28, 2021, at 2:39 PM, Mark Adams <[email protected]> wrote:
Thanks,
I did not intend to make any (real) changes. The only thing that I did not intend to use from Barry's branch, that conflicted, was the help and comment block at the top of ex5cu.cu <http://ex5cu.cu/>
* I ended up with two declarations of PetscSplitCSRDataStructure * I added some includes to fix errors like this: /ccs/home/adams/petsc/include/../src/mat/impls/aij/seq/seqcusparse/cusparsematimpl.h(263): error: incomplete type is not allowed * I end ended not having csr2csc_i in Mat_SeqAIJCUSPARSE so I get: /autofs/nccs-svm1_home1/adams/petsc/src/mat/impls/aij/seq/seqcusparse/aijcusparse.cu <http://aijcusparse.cu/>(1348): error: class "Mat_SeqAIJCUSPARSE" has no member "csr2csc_i"
On Fri, May 28, 2021 at 3:13 PM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: I can take a quick look at it tomorrow, what are the main changes you made since then?
On May 28, 2021, at 9:51 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
I am getting messed up in trying to resolve conflicts in rebasing over main. Is there a better way of doing this? Can I just tell git to use Barry's version and then test it? Or should I just try it again?
On Fri, May 28, 2021 at 2:15 PM Mark Adams <[email protected] <mailto:[email protected]>> wrote: I am rebasing over main and its a bit of a mess. I must have missed something. I get this. I think the _n_SplitCSRMat must be wrong.
In file included from /autofs/nccs-svm1_home1/adams/petsc/src/vec/is/sf/impls/basic/sfbasic.c:128:0: /ccs/home/adams/petsc/include/petscmat.h:1976:32: error: conflicting types for 'PetscSplitCSRDataStructure' typedef struct _n_SplitCSRMat *PetscSplitCSRDataStructure; ^~~~~~~~~~~~~~~~~~~~~~~~~~ /ccs/home/adams/petsc/include/petscmat.h:1922:31: note: previous declaration of 'PetscSplitCSRDataStructure' was here typedef struct _p_SplitCSRMat PetscSplitCSRDataStructure; ^~~~~~~~~~~~~~~~~~~~~~~~~~ CC arch-summit-opt-gnu-cuda/obj/vec/vec/impls/seq/dvec2.o
On Fri, May 28, 2021 at 1:50 PM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: OpenMPI.py depends on cuda.py in that, if cuda is present, configures using cuda. MPI.py or MPICH.py do not depend on cuda.py (MPICH, only weakly, it adds a print if cuda is present) Since eventually the MPI distro will only need a hint to be configured with CUDA, why not removing the dependency at all and add only a flag —download-openmpi-use-cuda?
On May 28, 2021, at 8:44 PM, Barry Smith <[email protected] <mailto:[email protected]>> wrote:
Stefano, who has a far better memory than me, wrote
Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
MPI.py depends on cuda.py so we cannot also have cuda.py depend on MPI.py using the generic dependencies of configure/packages
but perhaps we can just hardwire the rerunning of cuda.py when the MPI compilers are reset. I will try that now and if I can get it to work we should be able to move those old fix branches along as MR.
Barry
On May 28, 2021, at 12:41 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
OK, I will try to rebase and test Barry's branch.
On Fri, May 28, 2021 at 1:26 PM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: Yes, it is the branch I was using before force pushing to Barry’s barry/2020-11-11/cleanup-matsetvaluesdevice You can use both I guess
On May 28, 2021, at 8:25 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
Is this the correct branch? It conflicted with ex5cu so I assume it is.
stefanozampini/simplify-setvalues-device <https://gitlab.com/petsc/petsc/-/tree/stefanozampini/simplify-setvalues-device>
On Fri, May 28, 2021 at 1:24 PM Mark Adams <[email protected] <mailto:[email protected]>> wrote: I am fixing rebasing this branch over main.
On Fri, May 28, 2021 at 1:16 PM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
On May 28, 2021, at 8:12 PM, Stefano Zampini <[email protected] <mailto:[email protected]>> wrote:
That branch provides a fix for MatSetValuesDevice but it never got merged because of the CI issues with the —download-openmpi. We can probably try to skip the test in that specific configuration?
On May 28, 2021, at 7:45 PM, Barry Smith <[email protected] <mailto:[email protected]>> wrote:
~/petsc/src/mat/tutorials (barry/2021-05-28/robustify-cuda-gencodearch-check=) arch-robustify-cuda-gencodearch-check $ ./ex5cu terminate called after throwing an instance of 'thrust::system::system_error' what(): fill_n: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered Aborted (core dumped)
requires: cuda !define(PETSC_USE_CTABLE)
CI does not test with CUDA and no ctable. The code is still broken as it was six months ago in the discussion Stefano pointed to. It is clear why just no one has had the time to clean things up.
Barry
On May 28, 2021, at 11:13 AM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
On Fri, May 28, 2021 at 11:57 AM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: If you are referring to your device set values, I guess it is not currently tested
No. There is a test for that (ex5cu). I have a user that is getting a segv in MatSetValues with aijcusparse. I suspect there is memory corruption but I'm trying to cover all the bases. I have added a cuda test to ksp/ex56 that works. I can do an MR for it if such a test does not exist.
See the discussions here https://gitlab.com/petsc/petsc/-/merge_requests/3411 <https://gitlab.com/petsc/petsc/-/merge_requests/3411> I started cleaning up the code to prepare for testing but we never finished it https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-d... <https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-device/>
On May 28, 2021, at 6:53 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
Is there a test with MatSetValues and CUDA?
I am working away on this branch, making some progress, also cleaning things up with some small simplifications. Hope I can succeed, a bunch of stuff got moved around and some structs had changes, the merge could not handle some of these so I have to do a good amount of code wrangling to fix it. I'll let you know as I progress. Barry
On May 28, 2021, at 10:53 PM, Barry Smith <[email protected]> wrote:
I have rebased and tried to fix everything. I am now fixing the issues of --download-openmpi and cuda, once that is done I will test, rebase with main again if needed and restart the MR and get it into main.
Barry
I was stupid to let the MR lay fallow, I should have figured out a solution to the openmpi and cuda issue instead of punting and waiting for a dream fix.
On May 28, 2021, at 2:39 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
Thanks,
I did not intend to make any (real) changes. The only thing that I did not intend to use from Barry's branch, that conflicted, was the help and comment block at the top of ex5cu.cu <http://ex5cu.cu/>
* I ended up with two declarations of PetscSplitCSRDataStructure * I added some includes to fix errors like this: /ccs/home/adams/petsc/include/../src/mat/impls/aij/seq/seqcusparse/cusparsematimpl.h(263): error: incomplete type is not allowed * I end ended not having csr2csc_i in Mat_SeqAIJCUSPARSE so I get: /autofs/nccs-svm1_home1/adams/petsc/src/mat/impls/aij/seq/seqcusparse/aijcusparse.cu <http://aijcusparse.cu/>(1348): error: class "Mat_SeqAIJCUSPARSE" has no member "csr2csc_i"
On Fri, May 28, 2021 at 3:13 PM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: I can take a quick look at it tomorrow, what are the main changes you made since then?
On May 28, 2021, at 9:51 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
I am getting messed up in trying to resolve conflicts in rebasing over main. Is there a better way of doing this? Can I just tell git to use Barry's version and then test it? Or should I just try it again?
On Fri, May 28, 2021 at 2:15 PM Mark Adams <[email protected] <mailto:[email protected]>> wrote: I am rebasing over main and its a bit of a mess. I must have missed something. I get this. I think the _n_SplitCSRMat must be wrong.
In file included from /autofs/nccs-svm1_home1/adams/petsc/src/vec/is/sf/impls/basic/sfbasic.c:128:0: /ccs/home/adams/petsc/include/petscmat.h:1976:32: error: conflicting types for 'PetscSplitCSRDataStructure' typedef struct _n_SplitCSRMat *PetscSplitCSRDataStructure; ^~~~~~~~~~~~~~~~~~~~~~~~~~ /ccs/home/adams/petsc/include/petscmat.h:1922:31: note: previous declaration of 'PetscSplitCSRDataStructure' was here typedef struct _p_SplitCSRMat PetscSplitCSRDataStructure; ^~~~~~~~~~~~~~~~~~~~~~~~~~ CC arch-summit-opt-gnu-cuda/obj/vec/vec/impls/seq/dvec2.o
On Fri, May 28, 2021 at 1:50 PM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: OpenMPI.py depends on cuda.py in that, if cuda is present, configures using cuda. MPI.py or MPICH.py do not depend on cuda.py (MPICH, only weakly, it adds a print if cuda is present) Since eventually the MPI distro will only need a hint to be configured with CUDA, why not removing the dependency at all and add only a flag —download-openmpi-use-cuda?
On May 28, 2021, at 8:44 PM, Barry Smith <[email protected] <mailto:[email protected]>> wrote:
Stefano, who has a far better memory than me, wrote
Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
MPI.py depends on cuda.py so we cannot also have cuda.py depend on MPI.py using the generic dependencies of configure/packages
but perhaps we can just hardwire the rerunning of cuda.py when the MPI compilers are reset. I will try that now and if I can get it to work we should be able to move those old fix branches along as MR.
Barry
On May 28, 2021, at 12:41 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
OK, I will try to rebase and test Barry's branch.
On Fri, May 28, 2021 at 1:26 PM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: Yes, it is the branch I was using before force pushing to Barry’s barry/2020-11-11/cleanup-matsetvaluesdevice You can use both I guess
On May 28, 2021, at 8:25 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
Is this the correct branch? It conflicted with ex5cu so I assume it is.
stefanozampini/simplify-setvalues-device <https://gitlab.com/petsc/petsc/-/tree/stefanozampini/simplify-setvalues-device>
On Fri, May 28, 2021 at 1:24 PM Mark Adams <[email protected] <mailto:[email protected]>> wrote: I am fixing rebasing this branch over main.
On Fri, May 28, 2021 at 1:16 PM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
On May 28, 2021, at 8:12 PM, Stefano Zampini <[email protected] <mailto:[email protected]>> wrote:
That branch provides a fix for MatSetValuesDevice but it never got merged because of the CI issues with the —download-openmpi. We can probably try to skip the test in that specific configuration?
On May 28, 2021, at 7:45 PM, Barry Smith <[email protected] <mailto:[email protected]>> wrote:
~/petsc/src/mat/tutorials (barry/2021-05-28/robustify-cuda-gencodearch-check=) arch-robustify-cuda-gencodearch-check $ ./ex5cu terminate called after throwing an instance of 'thrust::system::system_error' what(): fill_n: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered Aborted (core dumped)
requires: cuda !define(PETSC_USE_CTABLE)
CI does not test with CUDA and no ctable. The code is still broken as it was six months ago in the discussion Stefano pointed to. It is clear why just no one has had the time to clean things up.
Barry
On May 28, 2021, at 11:13 AM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
On Fri, May 28, 2021 at 11:57 AM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: If you are referring to your device set values, I guess it is not currently tested
No. There is a test for that (ex5cu). I have a user that is getting a segv in MatSetValues with aijcusparse. I suspect there is memory corruption but I'm trying to cover all the bases. I have added a cuda test to ksp/ex56 that works. I can do an MR for it if such a test does not exist.
See the discussions here https://gitlab.com/petsc/petsc/-/merge_requests/3411 <https://gitlab.com/petsc/petsc/-/merge_requests/3411> I started cleaning up the code to prepare for testing but we never finished it https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-d... <https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-device/>
On May 28, 2021, at 6:53 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
Is there a test with MatSetValues and CUDA?
I believe I have finally successfully rebased the branch barry/2020-11-11/cleanup-matsetvaluesdevice against main and cleaned up all the issues. Please read the commit message. I have submitted a CI pipeline with ctables turned off temporarily for testing of the MatSetValuesDevice(). If it works hopefully Mark can maybe run a few additional tests of his Landau code that are not in the usual testing to verify and we can finally get the branch into main. Mark, Since this change is involved, it is likely your Landau mass matrix branch may not rebase cleanly. Let me know if you would like me to do the rebase and testing of your Landau mass matrix branch. I can get it ready to work with the results of barry/2020-11-11/cleanup-matsetvaluesdevice and then hand it back to you for further development. Barry
On May 29, 2021, at 11:32 AM, Barry Smith <[email protected]> wrote:
I am working away on this branch, making some progress, also cleaning things up with some small simplifications. Hope I can succeed, a bunch of stuff got moved around and some structs had changes, the merge could not handle some of these so I have to do a good amount of code wrangling to fix it.
I'll let you know as I progress.
Barry
On May 28, 2021, at 10:53 PM, Barry Smith <[email protected] <mailto:[email protected]>> wrote:
I have rebased and tried to fix everything. I am now fixing the issues of --download-openmpi and cuda, once that is done I will test, rebase with main again if needed and restart the MR and get it into main.
Barry
I was stupid to let the MR lay fallow, I should have figured out a solution to the openmpi and cuda issue instead of punting and waiting for a dream fix.
On May 28, 2021, at 2:39 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
Thanks,
I did not intend to make any (real) changes. The only thing that I did not intend to use from Barry's branch, that conflicted, was the help and comment block at the top of ex5cu.cu <http://ex5cu.cu/>
* I ended up with two declarations of PetscSplitCSRDataStructure * I added some includes to fix errors like this: /ccs/home/adams/petsc/include/../src/mat/impls/aij/seq/seqcusparse/cusparsematimpl.h(263): error: incomplete type is not allowed * I end ended not having csr2csc_i in Mat_SeqAIJCUSPARSE so I get: /autofs/nccs-svm1_home1/adams/petsc/src/mat/impls/aij/seq/seqcusparse/aijcusparse.cu <http://aijcusparse.cu/>(1348): error: class "Mat_SeqAIJCUSPARSE" has no member "csr2csc_i"
On Fri, May 28, 2021 at 3:13 PM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: I can take a quick look at it tomorrow, what are the main changes you made since then?
On May 28, 2021, at 9:51 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
I am getting messed up in trying to resolve conflicts in rebasing over main. Is there a better way of doing this? Can I just tell git to use Barry's version and then test it? Or should I just try it again?
On Fri, May 28, 2021 at 2:15 PM Mark Adams <[email protected] <mailto:[email protected]>> wrote: I am rebasing over main and its a bit of a mess. I must have missed something. I get this. I think the _n_SplitCSRMat must be wrong.
In file included from /autofs/nccs-svm1_home1/adams/petsc/src/vec/is/sf/impls/basic/sfbasic.c:128:0: /ccs/home/adams/petsc/include/petscmat.h:1976:32: error: conflicting types for 'PetscSplitCSRDataStructure' typedef struct _n_SplitCSRMat *PetscSplitCSRDataStructure; ^~~~~~~~~~~~~~~~~~~~~~~~~~ /ccs/home/adams/petsc/include/petscmat.h:1922:31: note: previous declaration of 'PetscSplitCSRDataStructure' was here typedef struct _p_SplitCSRMat PetscSplitCSRDataStructure; ^~~~~~~~~~~~~~~~~~~~~~~~~~ CC arch-summit-opt-gnu-cuda/obj/vec/vec/impls/seq/dvec2.o
On Fri, May 28, 2021 at 1:50 PM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: OpenMPI.py depends on cuda.py in that, if cuda is present, configures using cuda. MPI.py or MPICH.py do not depend on cuda.py (MPICH, only weakly, it adds a print if cuda is present) Since eventually the MPI distro will only need a hint to be configured with CUDA, why not removing the dependency at all and add only a flag —download-openmpi-use-cuda?
On May 28, 2021, at 8:44 PM, Barry Smith <[email protected] <mailto:[email protected]>> wrote:
Stefano, who has a far better memory than me, wrote
Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
MPI.py depends on cuda.py so we cannot also have cuda.py depend on MPI.py using the generic dependencies of configure/packages
but perhaps we can just hardwire the rerunning of cuda.py when the MPI compilers are reset. I will try that now and if I can get it to work we should be able to move those old fix branches along as MR.
Barry
On May 28, 2021, at 12:41 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
OK, I will try to rebase and test Barry's branch.
On Fri, May 28, 2021 at 1:26 PM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: Yes, it is the branch I was using before force pushing to Barry’s barry/2020-11-11/cleanup-matsetvaluesdevice You can use both I guess
On May 28, 2021, at 8:25 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
Is this the correct branch? It conflicted with ex5cu so I assume it is.
stefanozampini/simplify-setvalues-device <https://gitlab.com/petsc/petsc/-/tree/stefanozampini/simplify-setvalues-device>
On Fri, May 28, 2021 at 1:24 PM Mark Adams <[email protected] <mailto:[email protected]>> wrote: I am fixing rebasing this branch over main.
On Fri, May 28, 2021 at 1:16 PM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
On May 28, 2021, at 8:12 PM, Stefano Zampini <[email protected] <mailto:[email protected]>> wrote:
That branch provides a fix for MatSetValuesDevice but it never got merged because of the CI issues with the —download-openmpi. We can probably try to skip the test in that specific configuration?
On May 28, 2021, at 7:45 PM, Barry Smith <[email protected] <mailto:[email protected]>> wrote:
~/petsc/src/mat/tutorials (barry/2021-05-28/robustify-cuda-gencodearch-check=) arch-robustify-cuda-gencodearch-check $ ./ex5cu terminate called after throwing an instance of 'thrust::system::system_error' what(): fill_n: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered Aborted (core dumped)
requires: cuda !define(PETSC_USE_CTABLE)
CI does not test with CUDA and no ctable. The code is still broken as it was six months ago in the discussion Stefano pointed to. It is clear why just no one has had the time to clean things up.
Barry
On May 28, 2021, at 11:13 AM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
On Fri, May 28, 2021 at 11:57 AM Stefano Zampini <[email protected] <mailto:[email protected]>> wrote: If you are referring to your device set values, I guess it is not currently tested
No. There is a test for that (ex5cu). I have a user that is getting a segv in MatSetValues with aijcusparse. I suspect there is memory corruption but I'm trying to cover all the bases. I have added a cuda test to ksp/ex56 that works. I can do an MR for it if such a test does not exist.
See the discussions here https://gitlab.com/petsc/petsc/-/merge_requests/3411 <https://gitlab.com/petsc/petsc/-/merge_requests/3411> I started cleaning up the code to prepare for testing but we never finished it https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-d... <https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-device/>
On May 28, 2021, at 6:53 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
Is there a test with MatSetValues and CUDA?
On Sun, May 30, 2021 at 12:55 PM Barry Smith <[email protected]> wrote:
I believe I have finally successfully rebased the branch barry/2020-11-11/cleanup-matsetvaluesdevice against main and cleaned up all the issues. Please read the commit message.
I have submitted a CI pipeline with ctables turned off temporarily for testing of the MatSetValuesDevice(). If it works hopefully Mark can maybe run a few additional tests of his Landau code that are not in the usual testing to verify and we can finally get the branch into main.
If ex2_[cuda|kokkos] pass then you are fine. Thanks for doing this, and Stefano, this needed to be looked at by someone that knows what they are doing.
Mark,
Since this change is involved, it is likely your Landau mass matrix branch may not rebase cleanly.
Oh ya, you touched a lot of landau code, but it does not look hard. You removed * from PetscSplitCSRDataStructure *d_mat=NULL; and the rest look like it will be easy to pick your version.
Let me know if you would like me to do the rebase and testing of your Landau mass matrix branch. I can get it ready to work with the results of barry/2020-11-11/cleanup-matsetvaluesdevice and then hand it back to you for further development.
I will remove my changes to petscaijdevice.h. All I (Peng) did was protect printf statements with DEBUG, because printf takes up register(s) and registers are the limiting resource in Landau. So I guess you can merge and I will rebase over main. Mark
Barry
On May 29, 2021, at 11:32 AM, Barry Smith <[email protected]> wrote:
I am working away on this branch, making some progress, also cleaning things up with some small simplifications. Hope I can succeed, a bunch of stuff got moved around and some structs had changes, the merge could not handle some of these so I have to do a good amount of code wrangling to fix it.
I'll let you know as I progress.
Barry
On May 28, 2021, at 10:53 PM, Barry Smith <[email protected]> wrote:
I have rebased and tried to fix everything. I am now fixing the issues of --download-openmpi and cuda, once that is done I will test, rebase with main again if needed and restart the MR and get it into main.
Barry
I was stupid to let the MR lay fallow, I should have figured out a solution to the openmpi and cuda issue instead of punting and waiting for a dream fix.
On May 28, 2021, at 2:39 PM, Mark Adams <[email protected]> wrote:
Thanks,
I did not intend to make any (real) changes. The only thing that I did not intend to use from Barry's branch, that conflicted, was the help and comment block at the top of ex5cu.cu
* I ended up with two declarations of PetscSplitCSRDataStructure * I added some includes to fix errors like this: /ccs/home/adams/petsc/include/../src/mat/impls/aij/seq/seqcusparse/cusparsematimpl.h(263): error: incomplete type is not allowed * I end ended not having csr2csc_i in Mat_SeqAIJCUSPARSE so I get: /autofs/nccs-svm1_home1/adams/petsc/src/mat/impls/aij/seq/seqcusparse/ aijcusparse.cu(1348): error: class "Mat_SeqAIJCUSPARSE" has no member "csr2csc_i"
On Fri, May 28, 2021 at 3:13 PM Stefano Zampini <[email protected]> wrote:
I can take a quick look at it tomorrow, what are the main changes you made since then?
On May 28, 2021, at 9:51 PM, Mark Adams <[email protected]> wrote:
I am getting messed up in trying to resolve conflicts in rebasing over main. Is there a better way of doing this? Can I just tell git to use Barry's version and then test it? Or should I just try it again?
On Fri, May 28, 2021 at 2:15 PM Mark Adams <[email protected]> wrote:
I am rebasing over main and its a bit of a mess. I must have missed something. I get this. I think the _n_SplitCSRMat must be wrong.
In file included from /autofs/nccs-svm1_home1/adams/petsc/src/vec/is/sf/impls/basic/sfbasic.c:128:0: /ccs/home/adams/petsc/include/petscmat.h:1976:32: error: conflicting types for 'PetscSplitCSRDataStructure' typedef struct _n_SplitCSRMat *PetscSplitCSRDataStructure; ^~~~~~~~~~~~~~~~~~~~~~~~~~ /ccs/home/adams/petsc/include/petscmat.h:1922:31: note: previous declaration of 'PetscSplitCSRDataStructure' was here typedef struct _p_SplitCSRMat PetscSplitCSRDataStructure; ^~~~~~~~~~~~~~~~~~~~~~~~~~ CC arch-summit-opt-gnu-cuda/obj/vec/vec/impls/seq/dvec2.o
On Fri, May 28, 2021 at 1:50 PM Stefano Zampini < [email protected]> wrote:
OpenMPI.py depends on cuda.py in that, if cuda is present, configures using cuda. MPI.py or MPICH.py do not depend on cuda.py (MPICH, only weakly, it adds a print if cuda is present) Since eventually the MPI distro will only need a hint to be configured with CUDA, why not removing the dependency at all and add only a flag —download-openmpi-use-cuda?
On May 28, 2021, at 8:44 PM, Barry Smith <[email protected]> wrote:
Stefano, who has a far better memory than me, wrote
Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
MPI.py depends on cuda.py so we cannot also have cuda.py depend on MPI.py using the generic dependencies of configure/packages
but perhaps we can just hardwire the rerunning of cuda.py when the MPI compilers are reset. I will try that now and if I can get it to work we should be able to move those old fix branches along as MR.
Barry
On May 28, 2021, at 12:41 PM, Mark Adams <[email protected]> wrote:
OK, I will try to rebase and test Barry's branch.
On Fri, May 28, 2021 at 1:26 PM Stefano Zampini < [email protected]> wrote:
Yes, it is the branch I was using before force pushing to Barry’s barry/2020-11-11/cleanup-matsetvaluesdevice You can use both I guess
On May 28, 2021, at 8:25 PM, Mark Adams <[email protected]> wrote:
Is this the correct branch? It conflicted with ex5cu so I assume it is.
stefanozampini/simplify-setvalues-device <https://gitlab.com/petsc/petsc/-/tree/stefanozampini/simplify-setvalues-device>
On Fri, May 28, 2021 at 1:24 PM Mark Adams <[email protected]> wrote:
I am fixing rebasing this branch over main.
On Fri, May 28, 2021 at 1:16 PM Stefano Zampini < [email protected]> wrote:
Or probably remove —download-openmpi ? Or, just for the moment, why can’t we just tell configure that mpi is a weak dependence of cuda.py, so that it will be forced to be configured later?
On May 28, 2021, at 8:12 PM, Stefano Zampini < [email protected]> wrote:
That branch provides a fix for MatSetValuesDevice but it never got merged because of the CI issues with the —download-openmpi. We can probably try to skip the test in that specific configuration?
On May 28, 2021, at 7:45 PM, Barry Smith <[email protected]> wrote:
~/petsc/src/mat/tutorials* (barry/2021-05-28/robustify-cuda-gencodearch-check=)* arch-robustify-cuda-gencodearch-check $ ./ex5cu terminate called after throwing an instance of 'thrust::system::system_error' what(): fill_n: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered Aborted (core dumped)
requires: cuda !define(PETSC_USE_CTABLE)
CI does not test with CUDA and no ctable. The code is still broken as it was six months ago in the discussion Stefano pointed to. It is clear why just no one has had the time to clean things up.
Barry
On May 28, 2021, at 11:13 AM, Mark Adams <[email protected]> wrote:
On Fri, May 28, 2021 at 11:57 AM Stefano Zampini < [email protected]> wrote:
If you are referring to your device set values, I guess it is not currently tested
No. There is a test for that (ex5cu). I have a user that is getting a segv in MatSetValues with aijcusparse. I suspect there is memory corruption but I'm trying to cover all the bases. I have added a cuda test to ksp/ex56 that works. I can do an MR for it if such a test does not exist.
See the discussions here https://gitlab.com/petsc/petsc/-/merge_requests/3411 I started cleaning up the code to prepare for testing but we never finished it https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-d...
On May 28, 2021, at 6:53 PM, Mark Adams <[email protected]> wrote:
Is there a test with MatSetValues and CUDA?
It looks like MatAssemblyEnd is not setting up correctly in parallel . Segvv here. I'll take a look at what Stefano did. #18 main () (at 0x00000000100019a8) #17 MatMult (mat=0x2155a750, x=0x56dc29c0, y=0x5937b190) at /autofs/nccs-svm1_home1/adams/petsc/src/mat/interface/matrix.c:2448 (at 0x00002000005f4858) #16 MatMult_MPIAIJCUSPARSE(_p_Mat*, _p_Vec*, _p_Vec*) () from /ccs/home/adams/petsc/arch-summit-opt64-gnu-cuda/lib/libpetsc.so.3.15 (at 0x000020000095e298) #15 VecScatterBegin (sf=0x5937fbf0, x=0x56dc29c0, y=0x5937cd20, addv=<optimized out>, mode=<optimized out>) at /autofs/nccs-svm1_home1/adams/petsc/src/vec/is/sf/interface/vscat.c:1345 (at 0x00002000003a44fc) #14 VecScatterBegin_Internal (sf=0x5937fbf0, x=0x56dc29c0, y=0x5937cd20, addv=INSERT_VALUES, mode=SCATTER_FORWARD) at /autofs/nccs-svm1_home1/adams/petsc/src/vec/is/sf/interface/vscat.c:72 (at 0x000020000039e9cc) #13 PetscSFBcastWithMemTypeBegin (sf=0x5937fbf0, unit=0x200024529ed0, rootmtype=<optimized out>, rootdata=0x200076ea1e00, leafmtype=<optimized out>, leafdata=0x200076ea2200, op=0x200024539c70) at /autofs/nccs-svm1_home1/adams/petsc/src/vec/is/sf/interface/sf.c:1493 (at 0x0000200000396f04) #12 PetscSFBcastBegin_Basic (sf=0x5937fbf0, unit=<optimized out>, rootmtype=<optimized out>, rootdata=0x200076ea1e00, leafmtype=<optimized out>, leafdata=0x200076ea2200, op=0x200024539c70) at /autofs/nccs-svm1_home1/adams/petsc/src/vec/is/sf/impls/basic/sfbasic.c:191 (at 0x00002000002de188) #11 PetscSFLinkStartCommunication (direction=PETSCSF_ROOT2LEAF, link=<optimized out>, sf=0x5937fbf0) at /ccs/home/adams/petsc/include/../src/vec/is/sf/impls/basic/sfpack.h:267 (at 0x00002000002de188) #10 PetscSFLinkStartRequests_MPI (sf=<optimized out>, link=0x5937f080, direction=<optimized out>) at /autofs/nccs-svm1_home1/adams/petsc/src/vec/is/sf/impls/basic/sfmpi.c:41 (at 0x00002000003850dc) #9 PMPI_Startall () from /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/libmpi_ibm.so.3 (at 0x0000200024493d98) #8 mca_pml_pami_start () from /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/spectrum_mpi/mca_pml_pami.so (at 0x00002000301ce6e0) #7 pml_pami_persis_send_start () from /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/spectrum_mpi/mca_pml_pami.so (at 0x00002000301ce29c) #6 pml_pami_send () from /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/spectrum_mpi/mca_pml_pami.so (at 0x00002000301cf69c) #5 PAMI_Send_immediate () from /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 (at 0x0000200030395814) #4 PAMI::Protocol::Send::Eager<PAMI::Device::Shmem::PacketModel<PAMI::Device::ShmemDevice<PAMI::Fifo::WrapFifo<PAMI::Fifo::FifoPacket<64u, 4096u>, PAMI::Counter::IndirectBounded<PAMI::Atomic::NativeAtomic>, 256u>, PAMI::Counter::Indirect<PAMI::Counter::Native>, PAMI::Device::Shmem::CMAShaddr, 256u, 512u> >, PAMI::Device::IBV::PacketModel<PAMI::Device::IBV::Device, true>
::EagerImpl<(PAMI::Protocol::Send::configuration_t)5, true>::immediate(pami_send_immediate_t*) () from /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 (at 0x0000200030457bac) #3 PAMI::Protocol::Send::EagerSimple<PAMI::Device::Shmem::PacketModel<PAMI::Device::ShmemDevice<PAMI::Fifo::WrapFifo<PAMI::Fifo::FifoPacket<64u, 4096u>, PAMI::Counter::IndirectBounded<PAMI::Atomic::NativeAtomic>, 256u>, PAMI::Counter::Indirect<PAMI::Counter::Native>, PAMI::Device::Shmem::CMAShaddr, 256u, 512u> >, (PAMI::Protocol::Send::configuration_t)5>::immediate_impl(pami_send_immediate_t*) () from /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 (at 0x0000200030457824) #2 bool PAMI::Device::Interface::PacketModel<PAMI::Device::Shmem::PacketModel<PAMI::Device::ShmemDevice<PAMI::Fifo::WrapFifo<PAMI::Fifo::FifoPacket<64u, 4096u>, PAMI::Counter::IndirectBounded<PAMI::Atomic::NativeAtomic>, 256u>, PAMI::Counter::Indirect<PAMI::Counter::Native>, PAMI::Device::Shmem::CMAShaddr, 256u, 512u> > >::postPacket<2u>(unsigned long, unsigned long, void*, unsigned long, iovec (&) [2u]) () from /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 (at 0x0000200030456c18) #1 PAMI::Device::Shmem::Packet<PAMI::Fifo::FifoPacket<64u, 4096u> ::writePayload(PAMI::Fifo::FifoPacket<64u, 4096u>&, iovec*, unsigned long) () from /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/spectrum-mpi-10.3.1.2-20200121-awz2q5brde7wgdqqw4ugalrkukeub4eb/container/../lib/pami_port/libpami.so.3 (at 0x0000200030435a7c) #0 __memcpy_power7 () from /lib64/libc.so.6 (at 0x000020002463b804)
On Fri, May 28, 2021 at 12:45 PM Barry Smith <[email protected]> wrote:
~/petsc/src/mat/tutorials* (barry/2021-05-28/robustify-cuda-gencodearch-check=)* arch-robustify-cuda-gencodearch-check $ ./ex5cu terminate called after throwing an instance of 'thrust::system::system_error' what(): fill_n: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered Aborted (core dumped)
requires: cuda !define(PETSC_USE_CTABLE)
CI does not test with CUDA and no ctable. The code is still broken as it was six months ago in the discussion Stefano pointed to. It is clear why just no one has had the time to clean things up.
Barry
On May 28, 2021, at 11:13 AM, Mark Adams <[email protected]> wrote:
On Fri, May 28, 2021 at 11:57 AM Stefano Zampini < [email protected]> wrote:
If you are referring to your device set values, I guess it is not currently tested
No. There is a test for that (ex5cu). I have a user that is getting a segv in MatSetValues with aijcusparse. I suspect there is memory corruption but I'm trying to cover all the bases. I have added a cuda test to ksp/ex56 that works. I can do an MR for it if such a test does not exist.
See the discussions here https://gitlab.com/petsc/petsc/-/merge_requests/3411 I started cleaning up the code to prepare for testing but we never finished it https://gitlab.com/petsc/petsc/-/commits/stefanozampini/simplify-setvalues-d...
On May 28, 2021, at 6:53 PM, Mark Adams <[email protected]> wrote:
Is there a test with MatSetValues and CUDA?
participants (3)
-
Barry Smith -
Mark Adams -
Stefano Zampini