Re: ab/cmake-nix-and-ci, was Re: [PATCH] test-lib.sh: discover "git" in subdirs of "contrib/buildsystems/out"

[Junio C Hamano <gitster@pobox.com>]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> Junio, maybe you could clarify your take on this? As project lead, it is
> your decision to define how Git uses Continuous Builds, and how the
> project handles failed CI runs.

I have pretty much been with what Peff and Taylor said in the thread
already ever since we added CMake support to help Windows/VS folks.
I agree with you that we do not need to run it for Linux or macOS,
and if the promised/hoped-for benefit, i.e. that running them on
non-Windows build would uncover issues that are common across the
platform and help Windows, is not something that is likely to
materialize, I'd prefer to see our resources (CI time and developer
attention) not spent on that.

I do not think "how the project handles filed CI runs" is a very big
issue.  I often ignore partial failures (e.g. "winVS(n) test job
triggered rate limit") and the only annoyance I feel is that such a
temporary failure contribute one more message to my trash mailbox,
and I can learn to do the same for a test that marked as failed due
to linux-cmake-ctest job.  I expect that regular contributors are
doing the same pretty much.

How blocking is a CI failure for drive-by contributors who use GGG?
While I do not necessarily value drive-by contributions as much as
you do, if such "an unimportant failure we can ignore" discourages
those coming from GGG route, that would be unfortunate, exactly
because they may not have contributed anything to the failures.
This is not just cmake-ctest, but the leak checking job where a new
use of a tool that is known to be leaky in a test can turn a test
that has been passing to fail.  If we can mark failures in selected
jobs as non-blocking, we definitely should do so.

Between keeping and marking linux-cmake-ctest as non-blocking, and
removing it altogether, I am inclined to say that I'd favor the
latter for the reasons I explained earlier in this message.  But to
help casual contributors coming via GGG, we would anyway need to (1)
allow submitting even with failing tests, and (2) tell them that it
is OK to do so.  Which means it is not the end of the world, from
the point of view of helping casual developers, if we had kept these
brittle CI jobs like linux-cmake-ctest and linux-leaks.