Re: [PATCH i-g-t 3/5] tests: Use IGT_SIMULATION to tune the list of tests to run

[Ben Widawsky <ben@bwidawsk.net>]

On Mon, Apr 01, 2013 at 07:21:59AM +0000, Sun, Yi wrote:
> > On Tue, Mar 26, 2013 at 10:00:23AM -0700, Ben Widawsky wrote:
> > > On Tue, Mar 26, 2013 at 04:38:58PM +0100, Daniel Vetter wrote:
> > > > On Tue, Mar 26, 2013 at 03:24:37PM +0000, Damien Lespiau wrote:
> > > > > There are two ways to run tests, directly with make check/test or
> > > > > through piglit.
> > > > >
> > > > > When IGT_SIMULATION is set to '1', we substitute the list of tests
> > > > > in those two code paths with carefully selected tests. The stress
> > > > > tests and other horrors are left to torture the real hardware as
> > > > > they don't make too much sense in simulation.
> > > > >
> > > > > Signed-off-by: Damien Lespiau <damien.lespiau@intel.com>
> > > >
> > > > This feels a bit fragile since even just now I often fail to put a
> > > > test newly converted to the subtest stuff into the right
> > > > single/multi make target list. Also, this way we exclude a test by
> > > > default, which also feels like the wrong way round (most of the
> > > > tests we're adding aren't heavy stress-tests, but more excercise some
> > corner-case).
> > > >
> > > > So what about a sprinkling
> > > >
> > > > drmtest_skip_on_simulation();
> > > >
> > > > over the remaining testcases instead, which just calls exit(77);?
> > > > Only caveat is that for subtests we need to put it into the right
> > > > spot to not break subtest enumeration in piglit (it's a bit fragile).
> > > >
> > > > Cheers, Daniel
> > >
> > > I prefer Damien's explicit definitions in a file. It makes it much
> > > easier to see exactly what's run, and add a test as needed. Since
> > > simulation testing will always be a massive subset of the whole i-g-t
> > > suite, and we'll probably only rarely add or remove a test, I think
> > > it's not a big concern that we might miss a test.
> > 
> > Imo that's the wrong approach, since most of the tests we've recently added
> > exercise corner-cases of our code, and in a rather deterministic way. If we
> > currently have too many tests to get through all of the useful ones in a day, we
> > need more machines, not fewer tests.
> > 
> > In the end we want to be able to run full piglit on all this stuff in simulation after
> > all. Cc'ing Yi so he knows where I'm aiming at ;-) -Daniel
> > 
> [Sun, Yi] So what's the conclusion, how can I do to solve the too long time issue?
> I noticed Damien's patches aren't on the branch yet.
> If we want to split all i-g-t test cases to different HAS, how can we split it?
> 

Daniel, can you please advise Sun Yi since the solution I like has been
shot down?

-- 
Ben Widawsky, Intel Open Source Technology Center