On Aug 29, 2015, at 6:13 PM, Tobin Isaac <[email protected]> wrote:

On Sat, Aug 29, 2015 at 05:53:22PM -0500, Barry Smith wrote:

...

...

On Aug 29, 2015, at 5:47 PM, Jed Brown <[email protected]> wrote:

Tobin Isaac <[email protected]> writes:

...

How am I supposed to add a test to the intersection of a package (testexamples_TRIANGLE) and a scalar type (testexamples_C_NoComplex). It's pretty frustrating, given how obvious this would be with something like sharness [1], that I couldn't easily figure this out.

I've used sharness a reasonable amount and generally like it, especially when paired with "prove". A couple years ago, I did some exploratory work to parse PETSc makefile tests and generate sharness-style tests. I still don't think it's a bad idea and might yet be able to get back to

It seems to require a lot of boilerplate for each test. I sure as heck hope we don't have to write that by hand for each test.

Really? Compare sharness in hpgmpg [1] to PETSc, keeping in mind that the .out files in PETSc-parlance are incorporated into the test script. Every test in PETSc (a) writes out the command line in full, (b) diff's against the expected output, (c) writes out the "Possible problem" message, and (d) removes the temporary files. If sharness can meet PETSc's portability needs, that would strike me as a good reduction in boiler plate.

I never said the current system doesn't have boilerplate! It absolutely does, all I am saying is that a replacement should not have any. A replacement has to be fundamentally better than the current model, not just a bit better. For example the MORONs who switch from GNU autoconf to cmake (out of the frying pan into the fire). Barry

Toby

[1] https://bitbucket.org/hpgmg/hpgmg/src/26f74a2120cefc078b9b855297b77bf8d62ffb...

...

I'm not a fan of the "parse the PETSc makefiles approach"; I think we need to define a standard format for declaring tests that is very concise (only exactly what is needed; I have one try in a branch somewhere). Then we can have code that translates from the standard format to anything people want to try with testing scripts. I am even willing to manually convert makefile crap to this format when necessary

Barry

...

it.