Re: [PATCH 0/4] meson: parse TAP output generated by our tests

[Patrick Steinhardt <ps@pks.im>]

On Fri, May 23, 2025 at 12:33:23PM -0700, Junio C Hamano wrote:
> Ramsay Jones <ramsay@ramsayjones.plus.com> writes:
> 
> > Question: should meson (or indeed prove) fail the test because of an
> > unexpected _pass_?
> 
> Yes, it is a very good question.  I do not mind if the answer is "it
> should, and the make and prove shouldn't let unexpected pass go
> unnoticed".  The difference between the build systems bothers me

Indeed, a good question. The TAP specification [1] has this to say:

    Should a todo test point begin succeeding, the harness may report it
    in some way that indicates that whatever was supposed to be done has
    been, and it should be promoted to a normal Test Point.

    Harnesses must not treat failing TODO test points as a test failure.

    Harneses should report TODO test points found as a list of items
    needing work, if that is appropriate for their use case.

So if my reading of this is correct then Meson isn't wrong in reporting
this as an error -- "in some way" basically gives it full permission to
do so. So this is plain old undefined behaviour we rely on :/

I don't think it's inherently a bad thing to fail on unexpected passes.
After all, it shows that our assumption that the test fails is broken,
and that we should have a look why that is. But I can see arguments both
ways.

The pragmatic solution would be to just fix the unexpected pass and then
proceed with wiring up TAP support.

Patrick

https://testanything.org/tap-version-14-specification.html