Is this just the 3rd problem?

Regarding how you can end up with changes not being caught:
The default (going all the way back to the old harness) is to not check numbers to avoid round-off errors giving false negatives (failures).
Of course, sometimes you *want* to check the numbers; e.g., for indices.� The solution for this is to add:

���� diff_args: -j

Scott
On 12/31/20 4:40 PM, Barry Smith wrote:

� I think I have it "fixed" now in the branch, once it passes the pipeline I will shepard it through the MR quickly. Sorry about this, even all our CI testing can miss a great deal.

� Barry



On Dec 31, 2020, at 2:44 PM, Barry Smith <bsmith@petsc.dev> wrote:


� This is a different (3rd) problem. Funny it didn't bother anyone for two months.�

� Fix is in�barry/2020-12-29/fix-petscdiff-bracket but the pipeline keeps failing�ts_tutorials_advection-diffusion-reaction-ex3_2 fails on different machines with slightly different counts. I don't see how this change could cause that! But gets old results on my machine. Very frustrating.
Barry

On Dec 31, 2020, at 1:02 PM, Matthew Knepley <knepley@gmail.com> wrote:

On Thu, Dec 31, 2020 at 1:48 PM Barry Smith <bsmith@petsc.dev> wrote:

� So the programs output changes and should no longer match that in the output/* file yet the test harness does not error with a statement that the two outputs do not match?�

� �I noticed the gmakegentest.py is not being run before it runs the test? Does this mean it is just running all the old stuff which does match fine?�

� �Then either how petscdiff is called by the test harness has changed or petscdiff has changed and does not detect changes anymore�

� �BTW: I always use -f ./gmakefile.test test not just the gmakefile

� �All the PETSc changes are trivial and can be seen with a simple diff, it is hard to believe they would cause this behavior but I guess they must.

� �You can go to PETSC_ARCH/tests/snes/tests and run the ex13 shell script directly.

It is the sed problem:

master *$:/PETSc3/petsc/petsc-dev$ /PETSc3/petsc/petsc-dev/lib/petsc/bin/petscdiff � /PETSc3/petsc/petsc-dev/src/snes/tests/output/ex13_bench.out ex13_bench.tmp

sed: 1: "s/\033[1;31m//g": unbalanced brackets ([])
sed: 1: "s/\033[0;39m\033[0;49m//g": unbalanced brackets ([])
sed: 1: "s/\033[1;31m//g": unbalanced brackets ([])
sed: 1: "s/\033[0;39m\033[0;49m//g": unbalanced brackets ([])

The error was getting eaten.

This is in current master. Is it fixed in a branch?�

� �Matt
�
� Barry


On Dec 31, 2020, at 12:38 PM, Matthew Knepley <knepley@gmail.com> wrote:

I just�pulled master, and simple alterations to tests do not produce a failure:

master *$:/PETSc3/petsc/petsc-dev$ PETSC_ARCH=arch-master-debug make -f ./gmakefile test search="snes_tests-ex13_bench" TIMEOUT=5000 EXTRA_OPTIONS="-dm_
refine 0"
Using MAKEFLAGS: EXTRA_OPTIONS=-dm_refine 0 TIMEOUT=5000 search=snes_tests-ex13_bench
� � � � TEST arch-master-debug/tests/counts/snes_tests-ex13_bench.counts
�ok snes_tests-ex13_bench
�ok diff-snes_tests-ex13_bench

I check that the runs produce different output when done manually.

Scott and Barry, could this be related to changed to testing?

� Thanks,

� � �Matt

--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener




--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener




-- 
Tech-X Corporation               kruger@txcorp.com
5621 Arapahoe Ave, Suite A       Phone: (720) 466-3196
Boulder, CO 80303                Fax:   (303) 448-7756