Re: [PATCH 0/4] meson: parse TAP output generated by our tests

[Patrick Steinhardt <ps@pks.im>]

On Mon, May 26, 2025 at 09:54:40AM -0400, Eli Schwartz wrote:
> On 5/26/25 8:44 AM, Patrick Steinhardt wrote:
> > On Fri, May 23, 2025 at 12:33:23PM -0700, Junio C Hamano wrote:
> >> Ramsay Jones <ramsay@ramsayjones.plus.com> writes:
> >>
> >>> Question: should meson (or indeed prove) fail the test because of an
> >>> unexpected _pass_?
> >>
> >> Yes, it is a very good question.  I do not mind if the answer is "it
> >> should, and the make and prove shouldn't let unexpected pass go
> >> unnoticed".  The difference between the build systems bothers me
> > 
> > Indeed, a good question. The TAP specification [1] has this to say:
> > 
> >     Should a todo test point begin succeeding, the harness may report it
> >     in some way that indicates that whatever was supposed to be done has
> >     been, and it should be promoted to a normal Test Point.
> > 
> >     Harnesses must not treat failing TODO test points as a test failure.
> > 
> >     Harneses should report TODO test points found as a list of items
> >     needing work, if that is appropriate for their use case.
> > 
> > So if my reading of this is correct then Meson isn't wrong in reporting
> > this as an error -- "in some way" basically gives it full permission to
> > do so. So this is plain old undefined behaviour we rely on :/
> 
> 
> I don't see how you can possibly compare this to UB. It is a documented
> variant behavior and thus "implementation-defined", not "undefined".

Fair enough, "implementation-defined" is the better way to put it.

> Also,
> 
> https://www.gnu.org/software/automake/manual/automake.html#Generalities-about-Testing
> 
> """
> It’s not uncommon, especially during early development stages, that some
> tests fail for known reasons, and that the developer doesn’t want to
> tackle these failures immediately (this is especially true when the
> failing tests deal with corner cases). In this situation, the better
> policy is to declare that each of those failures is an expected failure
> (or xfail). In case a test that is expected to fail ends up passing
> instead, many testing environments will flag the result as a special
> kind of failure called unexpected pass (or xpass).
> """
> 
> > I don't think it's inherently a bad thing to fail on unexpected passes.
> > After all, it shows that our assumption that the test fails is broken,
> > and that we should have a look why that is. But I can see arguments both
> > ways.
> 
> As Phillip noted, treating them as ordinary passes undermines the reason
> for having them.

Yup, and I tend to agree.

Patrick