Re: [RFC/PATCH] WIP: Report intra-test progress with TAP subtests

[Jonathan Nieder <jrnieder@gmail.com>]

Hi Ævar,

Ævar Arnfjörð Bjarmason wrote:

>    ok 1 - A git test
>        ok 1 - doing test -f file
>        ok 2 - git commit ...
>        ok 3 - test_tick...
>        1..3
>    ok 2 - A git test
>    1..2
> 
> Here's an attempt at that, I've convented test_commit, test_merge and
> test_cmp to report intra-test progress.

What if a test uses none of those commands?

The ultimate conclusion of such an approach would be to use
replacements for common commands like “git” and “cd”, making tests a
lot harder to read and write for the uninitiated.

If we get peeks where the infrastructure has a chance to provide
them, maybe it would be nice to allow explicitly inserting such
checkpoints, too?  Like:

	mv init init2 &&
	checkpoint &&
	git config -f .gitmodules submodule.example.url "$(pwd)/init2" &&
	checkpoint &&
	git config --remove-section submodule.example &&
	checkpoint &&
	[...]

while debugging.

However, I would rather see those inserted automatically.
Two ideas without code to back them up:

1. Script parsing.

Yes, parsing shell command language is hard, but tests use only
a subset of it.  In particular the &&-chaining means it might
be relatively easy to find where each command starts and ends.

The hard bits: “if”, “while”, and “for” flow control constructs.

The payoff of even partial progress in this would be very high,
since it should make static analysis of &&-chaining possible.

2. Letting the shell trace for us.

When tests fail, I generally use

	sh -x ./failing-test.sh -v -i

and scroll back through the log to the important part.

Maybe the test harness could automate that a little, by using
set -x at the beginning and set +x at the end of each test.

Hope that helps,
Jonathan