Re: [PATCH v8 2/3] unit tests: add TAP unit test framework

[Josh Steadmon <steadmon@google.com>]

On 2023.10.27 22:15, Christian Couder wrote:
> On Tue, Oct 10, 2023 at 12:22 AM Josh Steadmon <steadmon@google.com> wrote:
> >
> > From: Phillip Wood <phillip.wood@dunelm.org.uk>
> >
> > This patch contains an implementation for writing unit tests with TAP
> > output. Each test is a function that contains one or more checks. The
> > test is run with the TEST() macro and if any of the checks fail then the
> > test will fail. A complete program that tests STRBUF_INIT would look
> > like
> >
> >      #include "test-lib.h"
> >      #include "strbuf.h"
> >
> >      static void t_static_init(void)
> >      {
> >              struct strbuf buf = STRBUF_INIT;
> >
> >              check_uint(buf.len, ==, 0);
> >              check_uint(buf.alloc, ==, 0);
> >              check_char(buf.buf[0], ==, '\0');
> >      }
> >
> >      int main(void)
> >      {
> >              TEST(t_static_init(), "static initialization works);
> >
> >              return test_done();
> >      }
> >
> > The output of this program would be
> >
> >      ok 1 - static initialization works
> >      1..1
> >
> > If any of the checks in a test fail then they print a diagnostic message
> > to aid debugging and the test will be reported as failing. For example a
> > failing integer check would look like
> >
> >      # check "x >= 3" failed at my-test.c:102
> 
> I wonder if it would be a bit better to say that the test was an
> integer test for example with "check_int(x >= 3) failed ..."
> 
> >      #    left: 2
> >      #   right: 3
> 
> I like "expected" and "actual" better than "left" and "right", not
> sure how it's possible to have that in a way consistent with the shell
> tests though.

I also prefer expected/actual, but I don't think it's possible where we
accept arbitrary operators, and I don't want to plumb a flag through to
specify whether to display left/right vs expected/actual.


> >      not ok 1 - x is greater than or equal to three
> >
> > There are a number of check functions implemented so far. check() checks
> > a boolean condition, check_int(), check_uint() and check_char() take two
> > values to compare and a comparison operator. check_str() will check if
> > two strings are equal. Custom checks are simple to implement as shown in
> > the comments above test_assert() in test-lib.h.
> 
> Yeah, nice.
> 
> > Tests can be skipped with test_skip() which can be supplied with a
> > reason for skipping which it will print. Tests can print diagnostic
> > messages with test_msg().  Checks that are known to fail can be wrapped
> > in TEST_TODO().
> 
> Maybe TEST_TOFIX() would be a bit more clear, but "TODO" is something
> that is more likely to be searched for than "TOFIX", so Ok.
> 
> > There are a couple of example test programs included in this
> > patch. t-basic.c implements some self-tests and demonstrates the
> > diagnostic output for failing test. The output of this program is
> > checked by t0080-unit-test-output.sh. t-strbuf.c shows some example
> > unit tests for strbuf.c
> >
> > The unit tests will be built as part of the default "make all" target,
> > to avoid bitrot. If you wish to build just the unit tests, you can run
> > "make build-unit-tests". To run the tests, you can use "make unit-tests"
> > or run the test binaries directly, as in "./t/unit-tests/bin/t-strbuf".
> 
> Nice!
> 
> > +unit-tests-prove:
> > +       @echo "*** prove - unit tests ***"; $(PROVE) $(GIT_PROVE_OPTS) $(UNIT_TESTS)
> 
> Nice, but DEFAULT_TEST_TARGET=prove isn't used. So not sure how
> important or relevant the 'prove' related sections are in the
> Documentation/technical/unit-tests.txt file introduced by the previous
> patch.

The "unit-tests" target runs DEFAULT_UNIT_TEST_TARGET, which can be
overridden to "unit-tests-prove".


> > +int test_assert(const char *location, const char *check, int ok)
> > +{
> > +       assert(ctx.running);
> > +
> > +       if (ctx.result == RESULT_SKIP) {
> > +               test_msg("skipping check '%s' at %s", check, location);
> > +               return 1;
> > +       } else if (!ctx.todo) {
> 
> I think it would be a bit clearer without the "else" above and with
> the "if (!ctx.todo) {" starting on a new line.

Fixed in v9.


> > +               if (ok) {
> > +                       test_pass();
> > +               } else {
> > +                       test_msg("check \"%s\" failed at %s", check, location);
> > +                       test_fail();
> > +               }
> > +       }
> > +
> > +       return !!ok;
> > +}
> 
> Otherwise it looks good to me.

Thanks for the review!