Re: [PATCH] test-lib: save test counts across invocations

[Jeff King <peff@peff.net>]

On Thu, Sep 01, 2011 at 03:08:45PM +0200, Thomas Rast wrote:

> Save the number of tests run ($test_count) in a file under
> test-counts/.  Then when sourcing test-lib.sh the next time, compare
> the timestamps.  If the counts file is older than the test, discard.
> Otherwise use the count that we saved and give prove the test plan
> ("1..N") up front.

Hmm. What happens when we're wrong? Does our eye-candy just print
something non-sensical like "13/12", or does prove actually care that we
run the right number of tests?

> Sparked by a discussion on G+.  I think this is the "simple" approach.
> The "cute" approach would be to let test-lib.sh define test_* as
> test-counting dummies once, source the test script itself (avoiding
> the sourcing loop with test-lib) to count what it does, then do the
> real work.

I don't think the "cute" approach will ever be accurate. Deciding
whether to run later tests sometimes depends on the results of earlier
tests, in at least two cases:

  1. Some tests find out which capabilities the system has, and set
     prerequisites. You need to actually run those tests, not make them
     counting dummies.

  2. Some tests create state that we then iterate on. For example, I
     think the mailinfo tests do something like:

        test_expect_success 'split' '
                git mailsplit -o patches mbox
        '
        for i in patches/*; do
          test_expect_success "check patch $i" '
                  git mailinfo $i >output
                  ...
          '
        done

      You'd get an inaccurate count if you didn't actually run the
      mailsplit command.

Anyway, this whole thing is a cute idea, and I do love eye candy, but I
wonder if it's worth the complexity. All this is telling us is how far
into each of the scripts it is. But we have literally hundreds of test
scripts, all with varying numbers of tests of varying speeds, and you're
probably running 16 or more at one time. So it doesn't tell you what you
really want to know, which is: how soon will the test suite probably be
done running.

-Peff