Re: [PATCH 1/2] t0303: set reason for skipping tests

["Zbigniew Jędrzejewski-Szmek" <zbyszek@in.waw.pl>]

On Tue, Mar 13, 2012 at 05:53:32PM -0400, Jeff King wrote:
> The reason is that the individual tests do not verify all of the
> preconditions themselves, but rather build on each other.
Right. I added a note based on this sentence in the test description.

> In an ideal world, each test snippet would be totally independent and
> check its preconditions. That would give us an accurate count of how
> many tests actually passed or failed. But fundamentally we only care
> about "did they all succeed or not?", which the current script does tell
> us (either test 2 fails, or if it succeeds, then we have checked the
> precondition for test 4). And the tests end up way shorter, because we
> don't repeat the preconditions over and over.
> 
> If you want to try to make the tests more robust, you can (for example,
> you can tighten the precondition on 4 to check "does it give the right
> answer with the right protocol" instead of just "does it ever give us
> the right answer"). But personally, I'm not sure it's worth that much
> effort.
Yeah.

> > >Should they actually say "# SKIP ..." to tell prove what's going on? I
> > >don't know very much about TAP.
> > # SKIP is used when skipping individual tests (IIUC), but when we
> > skip a group of tests, we simply jump over them and this message is
> > purely informative output that is not interpreted by the harness.
> 
> Just looking at test-lib.sh, it seems like we output "# SKIP" when we do
> skip_all. But I think you would have to give a count of which tests you
> skipped (e.g., try "./t5541-http-push.sh" to see its TAP output). Which
> means when skipping a subset, you'd have to deal with test numbering,
> which is a pain. So it's probably not worth worrying about.
Skipped test numbering could done automatically by using test prereqs,
but (after actually doing that and discarding) I agree that it isn't
worth the trouble.


Jonathan Nieder wrote:
> Nice idea, but shouldn't this description be in test_description so I
> can view it by running "sh t0303-credential-external.sh --help"?
Done.

Updated patches follow.

(This time I tested with GIT_TEST_CREDENTIAL_HELPER=cache
GIT_TEST_CREDENTIAL_HELPER_TIMEOUT="cache --timeout=1,3" and things
seem to work as expected.)

Zbyszek