[igt-dev] Intel CI blacklist suggestion for tests that are skipped on all shards

[igt-dev] Intel CI blacklist suggestion for tests that are skipped on all shards

[Lofstedt, Marta]

After summarizing the skip time for our shards:
GLK: 428s, APL: 371s, SNB:325s, KBL:283, HSW:182s

The idea came up to put tests that are skipped on all shards to the blacklist.
Here is a list of patterns that should match all tests that are currently skipped on all shards.

Some random or very small patterns
----------------------------------
igt@gem_busy@extended-semaphore-bsd
igt@gem_busy@extended-semaphore-bsd1
igt@gem_create@stolen-invalid-flag
igt@gem_exec_params@no-blt
igt@gem_exec_params@no-bsd
igt@gem_exec_params@rsvd2-dirt

igt@gem_softpin@evict-snoop
igt@gem_softpin@evict-snoop-interruptible

igt@gvt_basic@invalid-placeholder-test

igt@kms_flip@2x-flip-vs-bad-tiling
igt@kms_flip@2x-flip-vs-bad-tiling-interruptible

igt@kms_flip@flip-vs-bad-tiling
igt@kms_flip@flip-vs-bad-tiling-interruptible

igt@kms_tv_load_detect@load-detect

igt@pm_lpsp@edp-native
igt@pm_lpsp@edp-panel-fitter

igt@pm_rpm@gem-execbuf-stress-pc8
igt@pm_rpm@modeset-pc8-residency-stress
igt@pm_rpm@pc8-residency

igt@prime_vgem@fence-flip-hang
igt@prime_vgem@fence-read-hang

igt@prime_vgem@fence-write-hang


Anything stolen
---------------
igt@gem_pread@stolen-display
igt@gem_pread@stolen-normal
igt@gem_pread@stolen-snoop
igt@gem_pread@stolen-uncached
igt@gem_pwrite@stolen-display
igt@gem_pwrite@stolen-normal
igt@gem_pwrite@stolen-snoop
igt@gem_pwrite@stolen-uncached

igt@gem_stolen@large-object-alloc
igt@gem_stolen@stolen-clear
igt@gem_stolen@stolen-copy
igt@gem_stolen@stolen-fill-purge
igt@gem_stolen@stolen-no-mmap
igt@gem_stolen@stolen-pread
igt@gem_stolen@stolen-pwrite

anything gen3
-------------
igt@gen3_mixed_blits
igt@gen3_render_linear_blits
igt@gen3_render_mixed_blits
igt@gen3_render_tiledx_blits
igt@gen3_render_tiledy_blits


Anything 3x, 4x, 5x, 6x
-----------------------
and
igt@kms_frontbuffer_tracking@*psr-2p-*

d-e-f planes/pipe stuff
-------------------------
igt@kms_busy@*-[d|e|f]
igt@kms_ccs@pipe-[d|e|f]-*
igt@kms_color@pipe-[d|e|f]-*
igt@kms_concurrent@pipe-[d|e|f]
igt@kms_cursor_legacy@pipe-[d|e|f]-*
igt@kms_plane@*-pipe-[d|e|f]-planes
igt@kms_plane_lowres@pipe-[d|e|f]-*
igt@kms_plane_multiple@*-pipe-[d|e|f]-*
igt@kms_plane_scaling@pipe-[d|e|f]-*
igt@kms_universal_plane@*-[d|e|f]-*
igt@kms_vblank@pipe-[d|e|f]-*


Other unsupported stuff
-----------------------
igt@kms_chamelium@*
igt@kms_cursor_crc@cursor-512x*
igt@kms_draw_crc@draw-method-rgb565-render-*
igt@kms_force_connector_basic@*
igt@kms_frontbuffer_tracking@*drrs*
igt@kms_pipe_b_c_ivb@*
igt@kms_rotation_crc@*-reflect-*

---------------------------------------------------------------------------------------
Intel Finland OY 
Registered Office: Westendinkatu 7, 02160 Espoo, Finland 
Registration Number: 0357606-4


_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] Intel CI blacklist suggestion for tests that are skipped on all shards

[Jani Nikula]

On Tue, 06 Feb 2018, "Lofstedt, Marta" <marta.lofstedt@intel.com> wrote:
> After summarizing the skip time for our shards:
> GLK: 428s, APL: 371s, SNB:325s, KBL:283, HSW:182s

Why does it take so long on any of those platforms to skip the tests?

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] Intel CI blacklist suggestion for tests that are skipped on all shards

[Lofstedt, Marta]

> -----Original Message-----
> From: Jani Nikula [mailto:jani.nikula@linux.intel.com]
> Sent: Tuesday, February 6, 2018 8:42 PM
> To: Lofstedt, Marta <marta.lofstedt@intel.com>; igt-
> dev@lists.freedesktop.org
> Cc: Sarvela, Tomi P <tomi.p.sarvela@intel.com>; Peres, Martin
> <martin.peres@intel.com>
> Subject: Re: [igt-dev] Intel CI blacklist suggestion for tests that are skipped on
> all shards
> 
> On Tue, 06 Feb 2018, "Lofstedt, Marta" <marta.lofstedt@intel.com> wrote:
> > After summarizing the skip time for our shards:
> > GLK: 428s, APL: 371s, SNB:325s, KBL:283, HSW:182s
> 
> Why does it take so long on any of those platforms to skip the tests?

I believe it is the huge amount of skipped tests that is the main culprit, ~1500*0,1 seconds is real time. And the new DRRS test contribute a lot.

Also, I summarized some timings of the the kms_frontbuffer_tracking on GLK-shards:

Total time = 2153s, note this is more than half of the total GLK-shard runtime. 
Skipped time = 238s, this has more than doubled due to new DRRS tests
Fail time = 1365s, this is mostly for the PSR tests that have been failing for months, due to unsupported panels being shipped with GLK RVPs. Unfortunately we don't have a full set of supported PSR panels. And if we changed to supported PSR panels I assume that the runtime would only increase further.


> 
> BR,
> Jani.
> 
> 
> --
> Jani Nikula, Intel Open Source Technology Center
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] Intel CI blacklist suggestion for tests that are skipped on all shards

[Jani Nikula]

On Wed, 07 Feb 2018, "Lofstedt, Marta" <marta.lofstedt@intel.com> wrote:
>> -----Original Message-----
>> From: Jani Nikula [mailto:jani.nikula@linux.intel.com]
>> Sent: Tuesday, February 6, 2018 8:42 PM
>> To: Lofstedt, Marta <marta.lofstedt@intel.com>; igt-
>> dev@lists.freedesktop.org
>> Cc: Sarvela, Tomi P <tomi.p.sarvela@intel.com>; Peres, Martin
>> <martin.peres@intel.com>
>> Subject: Re: [igt-dev] Intel CI blacklist suggestion for tests that are skipped on
>> all shards
>> 
>> On Tue, 06 Feb 2018, "Lofstedt, Marta" <marta.lofstedt@intel.com> wrote:
>> > After summarizing the skip time for our shards:
>> > GLK: 428s, APL: 371s, SNB:325s, KBL:283, HSW:182s
>> 
>> Why does it take so long on any of those platforms to skip the tests?
>
> I believe it is the huge amount of skipped tests that is the main
> culprit, ~1500*0,1 seconds is real time. And the new DRRS test
> contribute a lot.
>
> Also, I summarized some timings of the the kms_frontbuffer_tracking on
> GLK-shards:
>
> Total time = 2153s, note this is more than half of the total GLK-shard
> runtime.  Skipped time = 238s, this has more than doubled due to new
> DRRS tests

> Fail time = 1365s, this is mostly for the PSR tests that have been
> failing for months, due to unsupported panels being shipped with GLK
> RVPs. Unfortunately we don't have a full set of supported PSR
> panels. And if we changed to supported PSR panels I assume that the
> runtime would only increase further.

Okay. I'm just wondering about the possibilities of optimizing the
skipping in the tests themselves, perhaps by bailing out earlier,
vs. maintainability of a single big blacklist which combines plenty of
features across plenty of platforms.

BR,
Jani.


>
>
>> 
>> BR,
>> Jani.
>> 
>> 
>> --
>> Jani Nikula, Intel Open Source Technology Center

-- 
Jani Nikula, Intel Open Source Technology Center
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] Intel CI blacklist suggestion for tests that are skipped on all shards

[Lofstedt, Marta]

> -----Original Message-----
> From: Jani Nikula [mailto:jani.nikula@linux.intel.com]
> Sent: Wednesday, February 7, 2018 1:06 PM
> To: Lofstedt, Marta <marta.lofstedt@intel.com>; igt-
> dev@lists.freedesktop.org
> Cc: Sarvela, Tomi P <tomi.p.sarvela@intel.com>; Peres, Martin
> <martin.peres@intel.com>
> Subject: RE: [igt-dev] Intel CI blacklist suggestion for tests that are skipped on
> all shards
> 
> On Wed, 07 Feb 2018, "Lofstedt, Marta" <marta.lofstedt@intel.com> wrote:
> >> -----Original Message-----
> >> From: Jani Nikula [mailto:jani.nikula@linux.intel.com]
> >> Sent: Tuesday, February 6, 2018 8:42 PM
> >> To: Lofstedt, Marta <marta.lofstedt@intel.com>; igt-
> >> dev@lists.freedesktop.org
> >> Cc: Sarvela, Tomi P <tomi.p.sarvela@intel.com>; Peres, Martin
> >> <martin.peres@intel.com>
> >> Subject: Re: [igt-dev] Intel CI blacklist suggestion for tests that
> >> are skipped on all shards
> >>
> >> On Tue, 06 Feb 2018, "Lofstedt, Marta" <marta.lofstedt@intel.com>
> wrote:
> >> > After summarizing the skip time for our shards:
> >> > GLK: 428s, APL: 371s, SNB:325s, KBL:283, HSW:182s
> >>
> >> Why does it take so long on any of those platforms to skip the tests?
> >
> > I believe it is the huge amount of skipped tests that is the main
> > culprit, ~1500*0,1 seconds is real time. And the new DRRS test
> > contribute a lot.
> >
> > Also, I summarized some timings of the the kms_frontbuffer_tracking on
> > GLK-shards:
> >
> > Total time = 2153s, note this is more than half of the total GLK-shard
> > runtime.  Skipped time = 238s, this has more than doubled due to new
> > DRRS tests
> 
> > Fail time = 1365s, this is mostly for the PSR tests that have been
> > failing for months, due to unsupported panels being shipped with GLK
> > RVPs. Unfortunately we don't have a full set of supported PSR panels.
> > And if we changed to supported PSR panels I assume that the runtime
> > would only increase further.
> 
> Okay. I'm just wondering about the possibilities of optimizing the skipping in
> the tests themselves, perhaps by bailing out earlier, vs. maintainability of a
> single big blacklist which combines plenty of features across plenty of
> platforms.

VIZ-8825 could do that, but I usually prefer picking low hanging fruit before redefining how CI run tests and how IGT test are structured.

> 
> BR,
> Jani.
> 
> 
> >
> >
> >>
> >> BR,
> >> Jani.
> >>
> >>
> >> --
> >> Jani Nikula, Intel Open Source Technology Center
> 
> --
> Jani Nikula, Intel Open Source Technology Center
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev