Re: [PATCH] travis-ci: run previously failed tests first, then slowest to fastest

[Jeff King <peff@peff.net>]

On Tue, Jan 19, 2016 at 02:53:00PM -0800, Junio C Hamano wrote:

> Junio C Hamano <gitster@pobox.com> writes:
> 
> > This is cute, but isn't it useful even outside Travis's context?  I
> > am not suggesting to touch anything other than .travis.yml file in
> > this patch, but if I wanted to get the benefit from the idea in this
> > patch when I run my tests manually, I can just tell prove to use the
> > cached states, no?
> 
> It seems that exporting something like
> 
>     GIT_PROVE_OPTS="--timer --state=slow,save -j8" 
> 
> when running "make DEFAULT_TEST_TARGET=prove test" does give me the
> same benefit by leaving the stats from the previous run in t/.prove
> when making the test scheduling decisions.

Yes, I've been using this on my local machine for years (which is why I
suggested it to Lars for the Travis build). I have also noticed that my
test runs take about as much time as the longest-running test, and do
not fully utilize all of my processors. I suspect we could drop the
run-time of the test suite substantially by splitting a few of the
longer tests.

You also wrote earlier:

> IOW, I am confused by the beginning of the log message that says
> this is taking advantage of "the Travis-CI cache feature".  This
> improvement looks to me like using the feature of "prove" that
> allows us to run slower tests first, and does not have much to do
> with Travis.

The interesting Travis feature we are using is that we are allowed to
store some data from run-to-run. So we use the Travis feature that lets
us use the prove feature. :)

> One thing I noticed but didn't dig further to fix was that this
> "prove --state" business did not seem to work well together with
> 
>     make T="...list of tests..." test
> 
> that limits the set of tests to perform.

Right, it does not do what you want.  The "prove --state" feature is not
just about ordering, but also about selecting. When run via "make", we
always give prove the full list of tests. But you can also do:

  prove --state=failed

manually to just run whatever failed on the last run.

I don't know if there is a way to tell prove "use the state for
ordering, but don't otherwise select from it", which would do what you
want above.

You can also note that if we ever delete a test script, it will still be
mentioned in prove's state file. I think prove is smart enough to
realize it went away and not bother you.

-Peff