Re: [PATCH] travis-ci: run previously failed tests first, then slowest to fastest

[Junio C Hamano <gitster@pobox.com>]

larsxschneider@gmail.com writes:

> From: Lars Schneider <larsxschneider@gmail.com>
>
> Use the Travis-CI cache feature to store prove test results and make them
> available in subsequent builds. This allows to run previously failed tests
> first and run remaining tests in slowest to fastest order. As a result it
> is less likely that Travis-CI needs to wait for a single test at the end
> which speeds up the test suite execution by ~2 min.
>
> Unfortunately the cache feature is only available (for free) on the
> Travis-CI Linux environment.
>
> Suggested-by: Jeff King <peff@peff.net>
> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
> ---
>  .travis.yml | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)

This is cute, but isn't it useful even outside Travis's context?  I
am not suggesting to touch anything other than .travis.yml file in
this patch, but if I wanted to get the benefit from the idea in this
patch when I run my tests manually, I can just tell prove to use the
cached states, no?

IOW, I am confused by the beginning of the log message that says
this is taking advantage of "the Travis-CI cache feature".  This
improvement looks to me like using the feature of "prove" that
allows us to run slower tests first, and does not have much to do
with Travis.

You are relying on the assumption that things under $HOME/ is stable
while things under t/ (or in our source tree in general) are not,
and I think that is a sensible thing to take advantage of, but are
we sure that they are running in an environment where "ln -s" would
work?  Otherwise, it may be more robust to copy $HOME/.prove to
t/.prove before starting to test and then copy it back once the
tests are done.

Thanks.

>
> diff --git a/.travis.yml b/.travis.yml
> index c3bf9c6..f34726b 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -1,5 +1,9 @@
>  language: c
>
> +cache:
> +  directories:
> +    - $HOME/.prove-cache
> +
>  os:
>    - linux
>    - osx
> @@ -18,7 +22,7 @@ env:
>      - P4_VERSION="15.2"
>      - GIT_LFS_VERSION="1.1.0"
>      - DEFAULT_TEST_TARGET=prove
> -    - GIT_PROVE_OPTS="--timer --jobs 3"
> +    - GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"
>      - GIT_TEST_OPTS="--verbose --tee"
>      - CFLAGS="-g -O2 -Wall -Werror"
>      - GIT_TEST_CLONE_2GB=YesPlease
> @@ -67,6 +71,8 @@ before_install:
>      p4 -V | grep Rev.;
>      echo "$(tput setaf 6)Git-LFS Version$(tput sgr0)";
>      git-lfs version;
> +    mkdir -p $HOME/.prove-cache;
> +    ln -s $HOME/.prove-cache/.prove t/.prove;
>
>  before_script: make --jobs=2
>
> --
> 2.5.1