Re: [igt] Making the test-suite easier to run

[Daniel Vetter <daniel@ffwll.ch>]

On Fri, Nov 15, 2013 at 04:33:17PM +0000, Damien Lespiau wrote:
> The objective of this series is to make the test-suite easier to run by
> embedding a copy a piglit and providing porcelain on top of it in the form of a
> makefile target. Beside python, there's no external dependency to run the test
> suite after this series.
> 
> The provided makefile target runs the full test suite. I'm still interested in
> providing, as a follow-up of this work, shorcuts for subsets of the testsuite
> that can be useful for developpers, subset what we would maintain in tree. For
> instance:
>   - tests selected by topic: running gem_.* Vs kms_.* Vs pm_.* (or maybe when
>     running, say, gt tests, exlude ^kms_.* ^pm_.* so we still still a lot of
>     the other tests (core_, drm_, ...)
>   - quick subtests (where we disable long running stress, race, ... tests)
> 
> patches 21-23 are something a bit different, try to pave the way for quick
> runs (by really the first tiny step).

I'm ok with doing the slow/quick thing if people like it.o

On the real deal (i.e. patches 1-20) a few overall comments:

- I think we need to make it very clear that piglit should still be
  developed upstream. So local changes shouldn't be allowed at all imo.
  Afaics that would only affect tests/igt.tests - can't we fix that by
  replicating the symlink, too?

- The makefile target looks like a script. I think it'd be better to
  extract it as a real script.

- I'm not too terribly sold on the convenience script. Imo it shouldn't be
  more than executable documentation, since I'm a bit afraid that we'll
  add neat features (like fancy resume with auto-blocking of crashing
  tests) to it that would better be done in upstream piglit to benefit
  everyone.

But overall I agree that we need a local copy of piglit since apparently
way too few people on our team us it to run igts. And running igts without
piglit is just nuts.

Cheers, Daniel
> 
> The README has been updated and I copy/paste it here the documentation for what
> would be the new way to run tests:
> 
>         After having compiled the tests, one can run the test-suite with:
> 
>         $ sudo make run-tests
> 
>         "make run-tests" create a $date-piglit-results.$n directory with the
>         results of the run. More specifically:
>           - $date-piglit-results.$n/main JSON file with the test results
>           - $date-piglit-results.$n/html/index.html HTML summary of the run
> 
>         Where $date is the date formated with `date +%Y%m%d` and $n the nth run
>         of the day.
> 
>         PIGLIT_FLAGS can be used to give options to the underlying piglit
>         runner. For instance, to exclude test matching '^kms_':
> 
>         $ sudo make run-tests PIGLIT_FLAGS="-x ^kms_"
> 
>         For the list of piglit options, run:
> 
>         $ ./piglit/piglit-run.py -h
> 
>         Another useful feature is to be able to resume an interrupted run. To
>         do that, make run-tests needs to know which run we are talking about:
> 
>         $ sudo make run-tests RESUME=$date-piglit-results.$n
> 
>         or, more succinctly:
> 
>         $ sudo make run-tests R=$date-piglit-results.$n
> 
>         It's possible to combine PIGLIT_FLAGS and RESUME. This is useful to
>         resume runs where a specific test deterministically hang the machine:
> 
>         $ sudo make run-tests PIGLIT_FLAGS="-x drv_module_reload" R=$date-piglit-results.$n
> 
> HTH,
> 
> -- 
> Damien
> 
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/intel-gfx

-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch