Re: [igt] Making the test-suite easier to run

[Damien Lespiau <damien.lespiau@intel.com>]

On Fri, Nov 15, 2013 at 06:23:18PM +0100, Daniel Vetter wrote:
> - I think we need to make it very clear that piglit should still be
>   developed upstream. So local changes shouldn't be allowed at all imo.

Sure, I'll wait until my patches are upstreamed to resend the series
without the local patches.

>   Afaics that would only affect tests/igt.tests - can't we fix that by
>   replicating the symlink, too?

I was thinking to completely remove igt.tests from upstream piglit.
There isn't anything special about the location of igt.tests, you just
need to give a (absolute) path to it when launching piglit-run.py and
not being in the piglit-run.py directory. As igt.test is really coupled
with igt, it makes sense to carry it in our tree, IMO.

> - The makefile target looks like a script. I think it'd be better to
>   extract it as a real script.

Can do.

> - I'm not too terribly sold on the convenience script. Imo it shouldn't be
>   more than executable documentation, since I'm a bit afraid that we'll
>   add neat features (like fancy resume with auto-blocking of crashing
>   tests) to it that would better be done in upstream piglit to benefit
>   everyone.

Point taken. We can judge when adding a feature if it'd be better to do
it upstream of if it's just a convenience wrapper for us.

But what I really want is the convenience and well defined runs that
everyone can replicate (a lof of details to remember, concurrent Vs
not-concurrent, how to generate the HTML report, ...). And I'd like to
add more: eg. give the 10 tests that take the longest time in the run so
people can look at them and try to make them faster, ...

Also I really want quicker subsets of the test-suite, people don't run
the full test-suite everytime they work on a series and this just step
1/ towards that goal. Running a subset is better than running nothing at
all :)

-- 
Damien