Re: [PATCH 2/3] t: teach 'test_must_fail' to save the command's stderr to a file

[Jeff King <peff@peff.net>]

On Fri, Feb 09, 2018 at 03:42:34AM +0100, SZEDER Gábor wrote:

> To check that a git command fails and to inspect its error message we
> usually execute a command like this throughout our test suite:
> 
>   test_must_fail git command --option 2>output.err
> 
> Note that this command doesn't limit the redirection to the git
> command, but it redirects the standard error of the 'test_must_fail'
> helper function as well.  This is wrong for several reasons:
> 
>   - If that git command were to succeed or die for an unexpected
>     reason e.g. signal, then 'test_must_fail's helpful error message
>     would end up in the given file instead of on the terminal or in
>     't/test-results/$T.out', when the test is run with '-v' or
>     '--verbose-log', respectively.
> 
>   - If the test is run with '-x', the trace of executed commands in
>     'test_must_fail' will go to stderr as well (except for more recent
>     Bash versions supporting $BASH_XTRACEFD), and then in turn will
>     end up in the given file.

I have to admit that I'm slightly negative on this approach, just
because:

  1. It requires every caller to do something special, rather than just
     using normal redirection. And the feature isn't as powerful as
     redirection. E.g., some callers do:

       test_must_fail foo >output 2>&1

     But:

       test_must_fail stderr=output foo >output

     is not quite right (stdout and stderr outputs may clobber each
     other, because they have independent position pointers).

  2. The "-x" problems aren't specific to test_must_fail at all. They're
     a general issue with shell functions.

I'm not entirely happy with saying "if you want to use -x, please use
bash". But given that it actually solves the problems everywhere with no
further effort, is it really that bad a solution?

For the error messages from test_must_fail, could we go in the same
direction, and send them to descriptor 4 rather than 2? We've already
staked out descriptor 4 as something magical that must be left alone
(see 9be795fb). If we can rely on that, then it becomes a convenient way
for functions to make sure their output is going to the script's stderr.

-Peff