Re: [PATCH] t1410: fix breakage on case-insensitive filesystems

[Jeff King <peff@peff.net>]

On Sun, Nov 09, 2014 at 10:47:52PM -0800, Junio C Hamano wrote:

> On Sun, Nov 9, 2014 at 10:30 PM, Jeff King <peff@peff.net> wrote:
> >
> > I know you "make test" before pushing out the results of any integration
> > you do. And I recall that for a while (and maybe still?) you even did so
> > on VMs of a few common platforms. OS X is notoriously irritating to run
> > in a VM, but would you be interested in a network-accessible install
> > that you could push to and "make test" on as part of your routine?
> >
> > If what Michael is offering cannot do that, I am sure I can get GitHub
> > to set something up.
> 
> Even if it were network-accessible, the latency for doing an integration
> cycle and pushing there and waiting for the tests to come back may
> make it impractical to use it as "part of pre-pushout test".

I had the impression you were already doing "for i in my_vms; do git
push $i && ssh $i make test; done", so in theory this could be done in
parallel. But yeah, I definitely wouldn't want to add latency to your
workflow.

> But I would think that the project certainly would benefit if a
> post-receive hook at Github did an automated test on the tip of
> 'pu' and bisected a breakage, if found, to pinpoint the patch
> that breaks. It could even send a notice to the author of the
> non-portable patch ;-).

Yeah, you're right. Testing on "pu" is probably enough. Coupled with
automated bisection, the "pinpointing" part is not that important, and
pu is often "early enough" to catch problems before people try to build
on top of them. So I think if Michael is still willing to set up his
build box, just pulling and building your "pu" automatically would be a
good start.

GitHub supports trigger hooks on pushes, but I do not think that is even
necessary here. Just polling once or twice a day would probably give us
good enough turn-around time.

I think somebody just needs to write the auto-bisect script. I suspect
the simplest form is something like:

  # fetch a new version; exit early if we already checked this one
  git fetch
  test "$(git rev-parse pu-ok)" = "$(git rev-parse origin/pu)" &&exit 0
  git reset --hard origin/pu

  # quick path: all tests succeed
  if make test; then
    git tag -f pu-ok HEAD
    exit 0
  fi

  # slow path: bisect the actual commit. We do not need worry about
  # cutting each bisection step down to the minimal size (e.g., finding
  # the actual failing test), because this should run only infrequently.
  git bisect start HEAD pu-ok
  git bisect run make test

The real work would be in wrapping it up in a cron-job or similar so
that it produces no output on success, and sends an email (or whatever)
with the bisect results when it fails.

-Peff