Re: [PATCH] t7300: mark test with SANITY

[Jeff King <peff@peff.net>]

On Tue, May 03, 2016 at 12:28:12PM -0700, Junio C Hamano wrote:

> By the way, it is easy to make a mistake like this, not to notice it
> during a review, and to leave it unnoticed for a long time,
> especially because I do not think anybody active in the development
> community runs tests as 'root'.

I think the advice for running the tests as root has long been: don't.
But I guess some people are in situations where it's hard to do
otherwise.

I'm torn on whether it really matters or not. On the one hand, if a
breakage that is purely in a test and not in git itself goes for a long
time without anyone noticing, it's not really hurting anyone. On the
other, "noticing" and "reporting" are two different things; there may be
people who run "make test" and give up.

FWIW, I just did a "sudo make test" and the only other failures were
around HTTP (which correctly punts with "you can't run http tests as
root", but I set GIT_TEST_HTTPD=1, which turns skipping them into an
error).

> Perhaps in a future update, Travis should learn a step to catch
> breakages like like this.  Or perhaps it is not worth the effort?

Maybe. I admit to not really using the Travis tests myself, as they are
way too slow and cumbersome to debug compared to just running "make
test".  The primary value to me of centralized CI is:

  1. _If_ people are looking at PRs on GitHub, the test status is shown
     right there in the PR, without a reviewer having to wonder whether
     the submitter ran "make test". But since I don't ever look at PRs
     for Git, that's not helpful.

  2. Quicker testing on a variety of platforms that I don't have.

     We've known for years that the tip of master passes your "make
     test" under Linux, but other-platform breakage slips through, and
     we rely on people who use those platforms to report. But that may
     not happen in a timely way, and CI can let us know sooner.

     Right now we've taken a baby step there. CI auto-builds for various
     setups, but we still rely on people who care about those platforms
     to look at the results and surface the data to the list. If we
     tested individual topics picked up from the mailing list and sent
     out an email saying "by the way, this is broken on platform X",
     that would go a long way (and would help point 1, too). I think
     there was some discussion in the 0day thread elsewhere on the list,
     but I didn't follow it too closely.

Anyway, back to the original question. I do think "test as root" can be
considered another platform, which makes it a good match for CI. But at
the same time, I don't know that it has ever surfaced an actual bug in
_Git_, and not just the test suite. So seeing those bugs quickly is a
lot less interesting.

-Peff