Re: [PATCH 0/7] [RFC] advice: refuse to output if stderr not TTY

[Patrick Steinhardt <ps@pks.im>]

On Wed, Aug 21, 2024 at 09:36:56AM -0700, Junio C Hamano wrote:
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
> 
> > Advice is supposed to be for humans, not machines. Why do we output it when
> > stderr is not a terminal? Let's stop doing that.
> 
> Last night while skimming the series on my phone (read: not a real
> review at all), I found it very annoying that GIT_ADVICE=1 had to be
> sprinkled all over the place.  I wonder if we want to instead set
> and export it in t/test-lib.sh and turn it off as needed?
> 
> The end-to-end tests we have are primarily to guarantee the
> continuity of the end-user experience by humans, and ensuring that
> an advice message is given when appropriate and it does not get
> shown otherwise is very much inherent part of them.  An alternative
> workaround to counteract the breakage this series causes of course
> is to run everything under test_terminal and it probably is much
> more kosher philosophically ;-), but compared to that, globally
> disabling the "if (!isatty(2))" while running the tests, and
> temporarily lifting that disabling during tests of the new feature
> added by this series would be easier to reason about, I would
> suspect.
> 
> > This series is motivated by an internal tool breaking due to the advice
> > message added to Git 2.46.0 by 9479a31d603 (advice: warn when sparse index
> > expands, 2024-07-08). This tool is assuming that any output to stderr is an
> > error, and in this case is attempting to parse it to determine what kind of
> > error (warning, error, or failure).
> 
> The "anything on stderr is an error" attitude needs to be fixed
> regardless of where it comes from (tcl/tk scripts have, or at least
> used to have, the tendency, which I found annoying), but regardless,
> I thought we added a mechanism to squelch all advice messages for
> this exact purpose at f0e21837 (Merge branch 'jl/git-no-advice',
> 2024-05-16).  Why isn't the tool using the mechanism that already
> exists?
> 
> I would have supported the behaviour proposed by this series 100% if
> it were on the table when we were introducing the advise mechanism,
> but unfortunately nobody seemed have suggested it back then.  I am
> willing to go with an "experiment" to change the behaviour,
> deliberately breaking "backward compatibility", if we have a wide
> support here during the review period.  FWIW, I think any scripts
> that scrape the advice messages are already broken.

I continue to believe that the biggest issue in this context is that
there is no proper interface between Git and its caller that would allow
the caller to learn about errors in a machine-parseable way. Matching
error messages against regular expressions is bad, and can easily be
broken by the output changing in whatever way. This may be because the
error message itself was changed, or it may be because we have started
to show advice messages. It's extremely fragile, and from my point of
view there is no good way to classify errors right now.

I won't argue that checking whether stderr is empty or not is good -- it
almost certainly feels wrong to me. But that's only one small part of a
more widespread issue. Having structured error handling in Git, e.g. via
a new structure that represents errors as discussed a couple of months
ago [1] would go a long way. I didn't quite like the approach chosen by
that patch series, but think that the idea certainly has merit.

The other question is why advice is being shown in the first place. In
theory, all one should ever use in scripted usecases are plumbing tools.
And as plumbing tools are explicitly not designed for users, they should
never show advice in the first place. I guess chances are high though
that the scripts in question used porcelain. That is also understandable
though: our plumbing tools are often not as powerful as the porcelain
ones, which has been lamented on the mailing list several times.

So I certainly get the sentiment of this patch series, but feel like we
continue to work around the underlying problems. Those are rooted rather
deep though, so fixing them is nothing we can do in a release or two,
but rather on the order of years. Meanwhile I guess we have to find
short-term solutions.

Patrick

[1]: https://lore.kernel.org/git/pull.1666.git.git.1708241612.gitgitgadget@gmail.com/