Re: [PATCH v4 1/5] test-lib: Adjust output to be valid TAP format

[Jakub Narebski <jnareb@gmail.com>]

Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:

> TAP, the Test Anything Protocol, is a simple text-based interface
> between testing modules in a test harness. test-lib.sh's output was
> already very close to being valid TAP. This change brings it all the
> way there. Before:
> 
>    $ ./t0005-signals.sh
>    *   ok 1: sigchain works
>    * passed all 1 test(s)
> 
> And after:
> 
>    $ ./t0005-signals.sh
>    ok 1 - sigchain works
>    # passed all 1 test(s)
>    1..1

This is incomplete information.  It only tells us how the output is
changed for the case when all test_expect_success tests passes.  This
commit message doesn't tell us how failing tests looks like, and how
test_expect_failure results gets translated.

Take for example a following test script:
-- >8 --
#!/bin/sh

test_description='this is a sample test.

This test is here to see various test outputs.'

. ./test-lib.sh

say 'diagnostic message'

test_expect_success 'true  test' 'true'
test_expect_success 'false test' 'false'

test_expect_failure 'true  test (todo)' 'true'
test_expect_failure 'false test (todo)' 'false'

test_debug 'echo "debug message"'

test_done
-- 8< --

This test script output looks like the following (the comments are not
part of output, but denote color of given line of test output):
----
* diagnostic message                      # yellow
*   ok 1: true  test
* FAIL 2: false test                      # bold red
        false
*   FIXED 3: true  test (todo)
*   still broken 4: false test (todo)     # bold green
* fixed 1 known breakage(s)               # green
* still have 1 known breakage(s)          # bold red
* failed 1 among remaining 3 test(s)      # bold red
----

How would the output of this test look like after TAP-ification?
Would it still provide color output when run on terminal?

This test also generates summary of test run in a file in
test-results/ subdirectory.  Currently such file can be e.g. named
test-results/test_test-8030, and consist of
-- >8 --
total 4
success 1
fixed 1
broken 1
failed 1

-- 8< --

Would TAP-ification change that?  Even if there is no change, this is
worth a sentence or a few words in a commit message.

P.S. With current output pass / fail results of test_expect_failure
are justified on ':' separator.  This is not the case of TAP output.
-- 
Jakub Narebski
Poland
ShadeHawk on #git