Re: [igt] Making the test-suite easier to run

[Daniel Vetter <daniel@ffwll.ch>]

On Fri, Nov 15, 2013 at 05:41:28PM +0000, Damien Lespiau wrote:
> On Fri, Nov 15, 2013 at 06:23:18PM +0100, Daniel Vetter wrote:
> > - I think we need to make it very clear that piglit should still be
> >   developed upstream. So local changes shouldn't be allowed at all imo.
> 
> Sure, I'll wait until my patches are upstreamed to resend the series
> without the local patches.
> 
> >   Afaics that would only affect tests/igt.tests - can't we fix that by
> >   replicating the symlink, too?
> 
> I was thinking to completely remove igt.tests from upstream piglit.
> There isn't anything special about the location of igt.tests, you just
> need to give a (absolute) path to it when launching piglit-run.py and
> not being in the piglit-run.py directory. As igt.test is really coupled
> with igt, it makes sense to carry it in our tree, IMO.

It's also coupled with piglit. If we move it completely then enabling new
piglit features will be a bit a pain. The only reason I can see to keep it
in igt only is if we want to change the way we generate the test list (the
current way with Makefile target will probably not amuse Oscar). But
that's about the only thing.

> > - The makefile target looks like a script. I think it'd be better to
> >   extract it as a real script.
> 
> Can do.
> 
> > - I'm not too terribly sold on the convenience script. Imo it shouldn't be
> >   more than executable documentation, since I'm a bit afraid that we'll
> >   add neat features (like fancy resume with auto-blocking of crashing
> >   tests) to it that would better be done in upstream piglit to benefit
> >   everyone.
> 
> Point taken. We can judge when adding a feature if it'd be better to do
> it upstream of if it's just a convenience wrapper for us.
> 
> But what I really want is the convenience and well defined runs that
> everyone can replicate (a lof of details to remember, concurrent Vs
> not-concurrent, how to generate the HTML report, ...). And I'd like to
> add more: eg. give the 10 tests that take the longest time in the run so
> people can look at them and try to make them faster, ...
> 
> Also I really want quicker subsets of the test-suite, people don't run
> the full test-suite everytime they work on a series and this just step
> 1/ towards that goal. Running a subset is better than running nothing at
> all :)

Yeah, like I've said it makes sense. I just have the long-term dream of
pimping piglit into a generally useful gfx testrunner, maybe even with a
bit of code to support centrally stored test results. Would help a lot for
my own little lab here ;-)

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch