Re: [petsc-dev] Regression in Vec assembly
Jed, Will you be working on this? I don't have a clue how your "new stuff" works. Thanks Barry
On Oct 19, 2015, at 3:20 AM, Lisandro Dalcin <[email protected]> wrote:
Barry, you merged a branch from Jed with changes to the Vec assembly routines. I've found a serious regression using PetIGA, however I could not managed to reproduce with a basic, PETSc-only example. I cannot figure out where the issue is, so I'll need your help.
Unfortunately, as I said, you have to use PetIGA to run the tests. Building PetIGA should be trivial, just "hg clone bitbucket/dalcinl/petiga", cd petiga, export PETIGA_DIR=$PWD, "make".
Next put the attached makefile and C source somewhere and run "make".
You should see three runs (see the `test` target in the makefile).
* The first one is sequential, you should see the output of a VecView, all entries in the vector equal.
* The next run uses two MPI processes, and pass the option -vec_assembly_bts 0 to use the legacy assembly routines. The output is the expected one, and equal to the sequential run.
* In the last run, I use again two processes but now I pass -vec_assembly_bts 1 (not really needed, now in petsc/master the new assembly routines are the default). Then you should see that the output is different, the first two entries in process one are missing some contributions.
Please note that in these tests the vectors have block size 2 and PetIGA then uses VecSetValuesBlockedLocal(..., ADD_VALUES). When running with block size 1 (-iga_dof 1 in the makefile) everything is OK, but any bs>1 shows the issue.
The example uses a small periodic grid to make the output easy to parse at first sight, you can however try larger grids (-iga_element) and remove the periodic option, pipe the output to files, then diff, and you should still see the issue.
-- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/
4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa
Office Phone: +966 12 808-0459 <makefile><IGAVecAssembly.c>
Barry Smith <[email protected]> writes:
Jed,
Will you be working on this? I don't have a clue how your "new stuff" works.
Yeah, I'll debug it.
Any news about this regression? My PetIGA tests are still failing with petsc/master. On 21 October 2015 at 06:23, Jed Brown <[email protected]> wrote:
Barry Smith <[email protected]> writes:
Jed,
Will you be working on this? I don't have a clue how your "new stuff" works.
Yeah, I'll debug it.
-- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459
Do you remember about this regression? My PetIGA tests are still failing with petsc/master. On 15 February 2016 at 13:56, Lisandro Dalcin <[email protected]> wrote:
Any news about this regression? My PetIGA tests are still failing with petsc/master.
On 21 October 2015 at 06:23, Jed Brown <[email protected]> wrote:
Barry Smith <[email protected]> writes:
Jed,
Will you be working on this? I don't have a clue how your "new stuff" works.
Yeah, I'll debug it.
-- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/
4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa
Office Phone: +966 12 808-0459
-- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 0109 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459
Lisandro Dalcin <[email protected]> writes:
Do you remember about this regression? My PetIGA tests are still failing with petsc/master.
Yes, I have most of a fix in a branch that I'm working to finish.
Jed, I’m resurrecting this thread since I believe this bug has not yet been fixed. I have spent a couple of hours today tracking down a bug in my code, that I finally confined to be related with VecSetValuesBlocked. I have pushed an example that shows the problem here https://bitbucket.org/petsc/petsc/commits/1cb7fb90b416ce1d4e4767a0d5a2ef5043... <https://bitbucket.org/petsc/petsc/commits/1cb7fb90b416ce1d4e4767a0d5a2ef5043656d53?at=stefano_zampini/feature-pcbddc-enablelor> Could you please have a look? Thanks, Stefano
On Apr 24, 2016, at 10:52 PM, Jed Brown <[email protected]> wrote:
Lisandro Dalcin <[email protected]> writes:
Do you remember about this regression? My PetIGA tests are still failing with petsc/master.
Yes, I have most of a fix in a branch that I'm working to finish.
Is this what I'm supposed to see? What should I be looking at? $ mpiexec.hydra -n 3 mpich-clang/tests/vec/vec/examples/tests/ex29 -n 126 -info | grep malloc [0] VecAssemblyBegin_MPI_BTS(): Stash has 252 entries, uses 2 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 252 entries, uses 2 mallocs. $ mpiexec.hydra -n 3 mpich-clang/tests/vec/vec/examples/tests/ex29 -n 126 -vec_assembly_legacy -info | grep malloc [0] VecAssemblyBegin_MPI(): Stash has 252 entries, uses 2 mallocs. [0] VecAssemblyBegin_MPI(): Block-Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI(): Block-Stash has 252 entries, uses 2 mallocs. Stefano Zampini <[email protected]> writes:
Jed,
I’m resurrecting this thread since I believe this bug has not yet been fixed. I have spent a couple of hours today tracking down a bug in my code, that I finally confined to be related with VecSetValuesBlocked.
I have pushed an example that shows the problem here https://bitbucket.org/petsc/petsc/commits/1cb7fb90b416ce1d4e4767a0d5a2ef5043... <https://bitbucket.org/petsc/petsc/commits/1cb7fb90b416ce1d4e4767a0d5a2ef5043656d53?at=stefano_zampini/feature-pcbddc-enablelor>
Could you please have a look?
Thanks, Stefano
On Apr 24, 2016, at 10:52 PM, Jed Brown <[email protected]> wrote:
Lisandro Dalcin <[email protected]> writes:
Do you remember about this regression? My PetIGA tests are still failing with petsc/master.
Yes, I have most of a fix in a branch that I'm working to finish.
In the link I have sent you, ex29 was modified to invert the order of values set. There’s also a command line that shows the error mpiexec -n 3 ./ex29 -n 4 -invert -bs 2 I have added the correct output, but if you run with bts you’ll see some of the entries are scrambled up.
On Feb 20, 2019, at 1:27 AM, Jed Brown <[email protected]> wrote:
Is this what I'm supposed to see? What should I be looking at?
$ mpiexec.hydra -n 3 mpich-clang/tests/vec/vec/examples/tests/ex29 -n 126 -info | grep malloc [0] VecAssemblyBegin_MPI_BTS(): Stash has 252 entries, uses 2 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 252 entries, uses 2 mallocs.
$ mpiexec.hydra -n 3 mpich-clang/tests/vec/vec/examples/tests/ex29 -n 126 -vec_assembly_legacy -info | grep malloc [0] VecAssemblyBegin_MPI(): Stash has 252 entries, uses 2 mallocs. [0] VecAssemblyBegin_MPI(): Block-Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI(): Block-Stash has 252 entries, uses 2 mallocs.
Stefano Zampini <[email protected]> writes:
Jed,
I’m resurrecting this thread since I believe this bug has not yet been fixed. I have spent a couple of hours today tracking down a bug in my code, that I finally confined to be related with VecSetValuesBlocked.
I have pushed an example that shows the problem here https://bitbucket.org/petsc/petsc/commits/1cb7fb90b416ce1d4e4767a0d5a2ef5043... <https://bitbucket.org/petsc/petsc/commits/1cb7fb90b416ce1d4e4767a0d5a2ef5043656d53?at=stefano_zampini/feature-pcbddc-enablelor>
Could you please have a look?
Thanks, Stefano
On Apr 24, 2016, at 10:52 PM, Jed Brown <[email protected]> wrote:
Lisandro Dalcin <[email protected]> writes:
Do you remember about this regression? My PetIGA tests are still failing with petsc/master.
Yes, I have most of a fix in a branch that I'm working to finish.
Yuck, bad bug. I pushed a one-line fix to your branch. Can you let me know if there are any outstanding issues? If it solves the problem for you, I'll cherry-pick it to 'maint'. Thanks. Stefano Zampini <[email protected]> writes:
In the link I have sent you, ex29 was modified to invert the order of values set. There’s also a command line that shows the error
mpiexec -n 3 ./ex29 -n 4 -invert -bs 2
I have added the correct output, but if you run with bts you’ll see some of the entries are scrambled up.
On Feb 20, 2019, at 1:27 AM, Jed Brown <[email protected]> wrote:
Is this what I'm supposed to see? What should I be looking at?
$ mpiexec.hydra -n 3 mpich-clang/tests/vec/vec/examples/tests/ex29 -n 126 -info | grep malloc [0] VecAssemblyBegin_MPI_BTS(): Stash has 252 entries, uses 2 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 252 entries, uses 2 mallocs.
$ mpiexec.hydra -n 3 mpich-clang/tests/vec/vec/examples/tests/ex29 -n 126 -vec_assembly_legacy -info | grep malloc [0] VecAssemblyBegin_MPI(): Stash has 252 entries, uses 2 mallocs. [0] VecAssemblyBegin_MPI(): Block-Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI(): Block-Stash has 252 entries, uses 2 mallocs.
Stefano Zampini <[email protected]> writes:
Jed,
I’m resurrecting this thread since I believe this bug has not yet been fixed. I have spent a couple of hours today tracking down a bug in my code, that I finally confined to be related with VecSetValuesBlocked.
I have pushed an example that shows the problem here https://bitbucket.org/petsc/petsc/commits/1cb7fb90b416ce1d4e4767a0d5a2ef5043... <https://bitbucket.org/petsc/petsc/commits/1cb7fb90b416ce1d4e4767a0d5a2ef5043656d53?at=stefano_zampini/feature-pcbddc-enablelor>
Could you please have a look?
Thanks, Stefano
On Apr 24, 2016, at 10:52 PM, Jed Brown <[email protected]> wrote:
Lisandro Dalcin <[email protected]> writes:
Do you remember about this regression? My PetIGA tests are still failing with petsc/master.
Yes, I have most of a fix in a branch that I'm working to finish.
I tested your fix (on top of master) with my original PetIGA reproducer, and the issue seems to be fixed. On Wed, 20 Feb 2019 at 08:32, Jed Brown <[email protected]> wrote:
Yuck, bad bug. I pushed a one-line fix to your branch. Can you let me know if there are any outstanding issues? If it solves the problem for you, I'll cherry-pick it to 'maint'. Thanks.
Stefano Zampini <[email protected]> writes:
In the link I have sent you, ex29 was modified to invert the order of values set. There’s also a command line that shows the error
mpiexec -n 3 ./ex29 -n 4 -invert -bs 2
I have added the correct output, but if you run with bts you’ll see some of the entries are scrambled up.
On Feb 20, 2019, at 1:27 AM, Jed Brown <[email protected]> wrote:
Is this what I'm supposed to see? What should I be looking at?
$ mpiexec.hydra -n 3 mpich-clang/tests/vec/vec/examples/tests/ex29 -n 126 -info | grep malloc [0] VecAssemblyBegin_MPI_BTS(): Stash has 252 entries, uses 2 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 252 entries, uses 2 mallocs.
$ mpiexec.hydra -n 3 mpich-clang/tests/vec/vec/examples/tests/ex29 -n 126 -vec_assembly_legacy -info | grep malloc [0] VecAssemblyBegin_MPI(): Stash has 252 entries, uses 2 mallocs. [0] VecAssemblyBegin_MPI(): Block-Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI(): Block-Stash has 252 entries, uses 2 mallocs.
Stefano Zampini <[email protected]> writes:
Jed,
I’m resurrecting this thread since I believe this bug has not yet been fixed. I have spent a couple of hours today tracking down a bug in my code, that I finally confined to be related with VecSetValuesBlocked.
I have pushed an example that shows the problem here https://bitbucket.org/petsc/petsc/commits/1cb7fb90b416ce1d4e4767a0d5a2ef5043... < https://bitbucket.org/petsc/petsc/commits/1cb7fb90b416ce1d4e4767a0d5a2ef5043...
Could you please have a look?
Thanks, Stefano
On Apr 24, 2016, at 10:52 PM, Jed Brown <[email protected]> wrote:
Lisandro Dalcin <[email protected]> writes:
Do you remember about this regression? My PetIGA tests are still failing with petsc/master.
Yes, I have most of a fix in a branch that I'm working to finish.
-- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 0109 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459
Jed, thanks. It fixed my issue too. Il giorno mer 20 feb 2019 alle ore 09:33 Lisandro Dalcin <[email protected]> ha scritto:
I tested your fix (on top of master) with my original PetIGA reproducer, and the issue seems to be fixed.
On Wed, 20 Feb 2019 at 08:32, Jed Brown <[email protected]> wrote:
Yuck, bad bug. I pushed a one-line fix to your branch. Can you let me know if there are any outstanding issues? If it solves the problem for you, I'll cherry-pick it to 'maint'. Thanks.
Stefano Zampini <[email protected]> writes:
In the link I have sent you, ex29 was modified to invert the order of values set. There’s also a command line that shows the error
mpiexec -n 3 ./ex29 -n 4 -invert -bs 2
I have added the correct output, but if you run with bts you’ll see some of the entries are scrambled up.
On Feb 20, 2019, at 1:27 AM, Jed Brown <[email protected]> wrote:
Is this what I'm supposed to see? What should I be looking at?
$ mpiexec.hydra -n 3 mpich-clang/tests/vec/vec/examples/tests/ex29 -n 126 -info | grep malloc [0] VecAssemblyBegin_MPI_BTS(): Stash has 252 entries, uses 2 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 252 entries, uses 2 mallocs.
$ mpiexec.hydra -n 3 mpich-clang/tests/vec/vec/examples/tests/ex29 -n 126 -vec_assembly_legacy -info | grep malloc [0] VecAssemblyBegin_MPI(): Stash has 252 entries, uses 2 mallocs. [0] VecAssemblyBegin_MPI(): Block-Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI(): Block-Stash has 252 entries, uses 2 mallocs.
Stefano Zampini <[email protected]> writes:
Jed,
I’m resurrecting this thread since I believe this bug has not yet been fixed. I have spent a couple of hours today tracking down a bug in my code, that I finally confined to be related with VecSetValuesBlocked.
I have pushed an example that shows the problem here https://bitbucket.org/petsc/petsc/commits/1cb7fb90b416ce1d4e4767a0d5a2ef5043... < https://bitbucket.org/petsc/petsc/commits/1cb7fb90b416ce1d4e4767a0d5a2ef5043...
Could you please have a look?
Thanks, Stefano
On Apr 24, 2016, at 10:52 PM, Jed Brown <[email protected]> wrote:
Lisandro Dalcin <[email protected]> writes:
Do you remember about this regression? My PetIGA tests are still failing with petsc/master.
Yes, I have most of a fix in a branch that I'm working to finish.
-- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/
4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 0109 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa
Office Phone: +966 12 808-0459
-- Stefano
https://bitbucket.org/petsc/petsc/pull-requests/1389/vecstashsortcompress_pr... Stefano Zampini <[email protected]> writes:
Jed,
thanks. It fixed my issue too.
Il giorno mer 20 feb 2019 alle ore 09:33 Lisandro Dalcin <[email protected]> ha scritto:
I tested your fix (on top of master) with my original PetIGA reproducer, and the issue seems to be fixed.
On Wed, 20 Feb 2019 at 08:32, Jed Brown <[email protected]> wrote:
Yuck, bad bug. I pushed a one-line fix to your branch. Can you let me know if there are any outstanding issues? If it solves the problem for you, I'll cherry-pick it to 'maint'. Thanks.
Stefano Zampini <[email protected]> writes:
In the link I have sent you, ex29 was modified to invert the order of values set. There’s also a command line that shows the error
mpiexec -n 3 ./ex29 -n 4 -invert -bs 2
I have added the correct output, but if you run with bts you’ll see some of the entries are scrambled up.
On Feb 20, 2019, at 1:27 AM, Jed Brown <[email protected]> wrote:
Is this what I'm supposed to see? What should I be looking at?
$ mpiexec.hydra -n 3 mpich-clang/tests/vec/vec/examples/tests/ex29 -n 126 -info | grep malloc [0] VecAssemblyBegin_MPI_BTS(): Stash has 252 entries, uses 2 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI_BTS(): Block-Stash has 252 entries, uses 2 mallocs.
$ mpiexec.hydra -n 3 mpich-clang/tests/vec/vec/examples/tests/ex29 -n 126 -vec_assembly_legacy -info | grep malloc [0] VecAssemblyBegin_MPI(): Stash has 252 entries, uses 2 mallocs. [0] VecAssemblyBegin_MPI(): Block-Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI(): Stash has 0 entries, uses 0 mallocs. [0] VecAssemblyBegin_MPI(): Block-Stash has 252 entries, uses 2 mallocs.
Stefano Zampini <[email protected]> writes:
Jed,
I’m resurrecting this thread since I believe this bug has not yet been fixed. I have spent a couple of hours today tracking down a bug in my code, that I finally confined to be related with VecSetValuesBlocked.
I have pushed an example that shows the problem here https://bitbucket.org/petsc/petsc/commits/1cb7fb90b416ce1d4e4767a0d5a2ef5043... < https://bitbucket.org/petsc/petsc/commits/1cb7fb90b416ce1d4e4767a0d5a2ef5043...
Could you please have a look?
Thanks, Stefano
On Apr 24, 2016, at 10:52 PM, Jed Brown <[email protected]> wrote:
Lisandro Dalcin <[email protected]> writes:
Do you remember about this regression? My PetIGA tests are still failing with petsc/master.
Yes, I have most of a fix in a branch that I'm working to finish.
-- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/
4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 0109 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa
Office Phone: +966 12 808-0459
-- Stefano
participants (4)
-
Barry Smith -
Jed Brown -
Lisandro Dalcin -
Stefano Zampini