Re: [PATCH] t/README: clarify test_must_fail description

["Ævar Arnfjörð Bjarmason" <avarab@gmail.com>]

On Tue, Jul 20, 2010 at 18:34, Junio C Hamano <gitster@pobox.com> wrote:

> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>
>> On Tue, Jul 20, 2010 at 18:00, Junio C Hamano <gitster@pobox.com> wrote:
>>
>>>    Run a git command and ensure it fails in a controlled way.  Use
>>>    this instead of "! <git-command>".  When git-command dies due to a
>>>    segfault, test_must_fail diagnoses it as an error; "! <git-command>"
>>>    treats it as just another expected failure. letting such a bug go
>>>    unnoticed.
>>
>> To add to that:
>>
>>     Don't use test_must_fail to negate the return values of commands
>>     on the system like grep, sed etc. If we can't trust that the core
>>     utilities won't randomly segfault we might as well die horribly.
>
> I think you are being incoherent.  If we can't trust system "grep" and it
> randomly segfaults, then a test:
>
>    git some-command >actual &&
>    ! grep string-that-should-not-be-in-the-output actual
>
> would _pass_ when the command segfaults.  I do agree with you that "We
> might as well die horribly", and the way you do so is by protecting the
> test with test_must_fail, like this:
>
>    git some-command >actual &&
>    test_must_fail grep string-that-should-not-be-in-the-output actual
>
> Having said that, as we _do_ trust system tools to a certain degree, we do
> not care very deeply about this.  IOW, I wouldn't want to see a patch that
> rewrites "! grep" to "test_must_fail grep".

An individual test would pass, yes. But if test or grep are
segfaulting we're going to bail out horribly eventually anyway, so I
don't think it's worth the effort to guard them with test_must_fail,
and I wouldn't write tests to do that. I'd just use !.

That's what we seem to be doing in the tests so far, i.e. test_must_fail
is reserved for git commands only.