Re: [PATCH 3/3] t-ctype: do one test per class and char

["René Scharfe" <l.s.r@web.de>]

Am 27.02.24 um 11:04 schrieb Christian Couder:
> On Mon, Feb 26, 2024 at 7:58 PM Josh Steadmon <steadmon@google.com> wrote:
>>
>> On 2024.02.26 18:26, René Scharfe wrote:
>
>>> The output is clean as well, but there's a lot of it.  Perhaps too much.
>>> The success messages are boring, though, and if all checks pass then the
>>> only useful information is the status code.  A TAP harness like prove
>>> summarizes that nicely:
>>>
>>>    $ prove t/unit-tests/bin/t-ctype
>>>    t/unit-tests/bin/t-ctype .. ok
>>>    All tests successful.
>>>    Files=1, Tests=3598,  0 wallclock secs ( 0.08 usr +  0.00 sys =  0.08 CPU)
>>>    Result: PASS
>>>
>>> Filtering out passing checks e.g. with "| grep -v ^ok" would help when
>>> debugging a test failure. I vaguely miss the --immediate switch from the
>>> regular test library, however.
>>
>> Yeah, I agree here. It's a lot of output but it's almost always going to
>> be consumed by a test harness rather than a human, and it's easy to
>> filter out the noise if someone does need to do some manual debugging.
>
> Yeah, I know about TAP harnesses like prove, but the most
> straightforward way to run the unit tests is still `make unit-tests`
> in the t/ directory. Also when you add or change some tests, it's a
> good idea to run `make unit-tests` to see what the output is, so you
> still have to see that output quite often when you work on tests and
> going through 3598 of mostly useless output instead of just 14 isn't
> nice.

I was starting the programs from t/unit-tests/bin/ individually because
I didn't know 'make unit-tests' exists.  This is much nicer, thank you!
Especially after adding 'DEFAULT_UNIT_TEST_TARGET = unit-tests-prove' to
config.mak to complement the 'DEFAULT_TEST_TARGET = prove' I added long
ago.  It would be even nicer if the former was the default when the
latter is set.

As unit tests are added, their output is surely going to grow to
multiple screens with or without prove, no?  So someone writing or
debugging tests will still go back to starting then individually
eventually.

The size of the output in itself is not a problem, I assume, but that
most of it is useless -- details of successful tests are uninteresting.
A test harness can aggregate the output, but prove annoyed me when used
with the regular tests by also aggregating error output and only showing
the numbers of failed tests.  Finding their names involved running the
test script again without prove.  Turns out it has an option for that.
Added 'GIT_PROVE_OPTS = --failures' to config.mak as well, will see if
it helps.

Is it too much to ask developers to use a test harness?  Perhaps: It's
yet another dependency and not enabled by default.

What's the right level of aggregation and how do we achieve it?
Grouping by class is natural and follows the test definition.  We
could stop after patch 2.  Dunno.

René