Re: [PATCH 2/8] lib: Introduce drmtest_skip_on_simulation()

[Daniel Vetter <daniel@ffwll.ch>]

On Thu, Jul 18, 2013 at 05:55:00PM +0100, Damien Lespiau wrote:
> On Thu, Jul 18, 2013 at 09:31:15AM -0700, Ben Widawsky wrote:
> > On Thu, Jul 18, 2013 at 09:18:34AM -0700, Jesse Barnes wrote:
> > > On Thu, 18 Jul 2013 16:19:07 +0100
> > > Damien Lespiau <damien.lespiau@intel.com> wrote:
> > > 
> > > > This will allow us to explicitely blacklist tests we don't want to run
> > > > on simulation.
> > 
> > I agree with Jesse on this. Mostly what I've wanted is an "opt-in"
> > approach as opposed to an "opt-out" one. Such a thing is better
> > controlled via a Makefile target, or separate script.
> 
> Well, for this one I'd like a slightly different approach. Actually have
> a drmtest structure with metadata about the test. Something like:
> 
> #define DECLARE_SUBTEST(n, func, d)    \
>        {                               \
>                .name = n,              \
>                .run = func,            \
>                .data = d,              \
>        }
> 
> enum drmtest_category {
>        TEST_DISPLAY    = (1 << 0),
>        TEST_GT         = (1 << 1),
>        TEST_CONTEXT     = (1 << 2),
>        TEST_GEM        = (1 << 3),
>        TEST_TORTURE    = (1 << 31),
> };
> 
> 
> struct drm_subtest {
>        const char *name;
>        int (*run)(struct drm_test *test);
>        void *data;
> };
> 
> struct drm_test {
>        const char *name;
>        int fd;
>        enum drmtest_category categories;
>        int subtests_nr;
>        struct drm_subtest *subtests;
> };
> 
> and in the test:
> 
> struct drm_subtest subtests[] = {
>        DECLARE_SUBTEST("bad-close", test_bad_close, &test_data),
>        DECLARE_SUBTEST("create-close", test_create_close, &test_data),
>        DECLARE_SUBTEST("create-fd-close", test_create_fd_close, &test_data),
> };
> 
> struct drm_test test = {
>        .name = "gem-basic",
>        .subtests = subtests,
>        .subtests_nr = ARRAY_SIZE(subtests),
> };
> 
> (better macros possible)
> 
> and:
>        drmtest_init(&test, argc, argv);
>        drmtest_run(&test);
> 
> That'll allow you to do things like "please run all the context tests" for
> instance.
> 
> But well, still quite far away, and it maybe quite hard/time consuming to
> convert some tests to that.

More metaprogramming for subtest would certainly be nice, currently we use
a lot of add-hoc macro infrastructure. For a group of tests that just
share a bunch of helper functions but not the overall logic like kms_flip
approach used in prime quite a bit. Maybe we need a few different variants
to declare subtests ...

Not sure how useful the grouping of tests is, since the big usecase for
igt is to check for unrelated breakage imo. In any way we have lots of
room for improvement ;-) I've added a few points to our internal wiki for
infrastructure ideas around igt.

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch