* [igt-dev] Intel CI blacklist suggestion for tests that are skipped on all shards @ 2018-02-06 11:17 Lofstedt, Marta 2018-02-06 18:41 ` Jani Nikula 0 siblings, 1 reply; 5+ messages in thread From: Lofstedt, Marta @ 2018-02-06 11:17 UTC (permalink / raw) To: igt-dev@lists.freedesktop.org; +Cc: Sarvela, Tomi P, Peres, Martin After summarizing the skip time for our shards: GLK: 428s, APL: 371s, SNB:325s, KBL:283, HSW:182s The idea came up to put tests that are skipped on all shards to the blacklist. Here is a list of patterns that should match all tests that are currently skipped on all shards. Some random or very small patterns ---------------------------------- igt@gem_busy@extended-semaphore-bsd igt@gem_busy@extended-semaphore-bsd1 igt@gem_create@stolen-invalid-flag igt@gem_exec_params@no-blt igt@gem_exec_params@no-bsd igt@gem_exec_params@rsvd2-dirt igt@gem_softpin@evict-snoop igt@gem_softpin@evict-snoop-interruptible igt@gvt_basic@invalid-placeholder-test igt@kms_flip@2x-flip-vs-bad-tiling igt@kms_flip@2x-flip-vs-bad-tiling-interruptible igt@kms_flip@flip-vs-bad-tiling igt@kms_flip@flip-vs-bad-tiling-interruptible igt@kms_tv_load_detect@load-detect igt@pm_lpsp@edp-native igt@pm_lpsp@edp-panel-fitter igt@pm_rpm@gem-execbuf-stress-pc8 igt@pm_rpm@modeset-pc8-residency-stress igt@pm_rpm@pc8-residency igt@prime_vgem@fence-flip-hang igt@prime_vgem@fence-read-hang igt@prime_vgem@fence-write-hang Anything stolen --------------- igt@gem_pread@stolen-display igt@gem_pread@stolen-normal igt@gem_pread@stolen-snoop igt@gem_pread@stolen-uncached igt@gem_pwrite@stolen-display igt@gem_pwrite@stolen-normal igt@gem_pwrite@stolen-snoop igt@gem_pwrite@stolen-uncached igt@gem_stolen@large-object-alloc igt@gem_stolen@stolen-clear igt@gem_stolen@stolen-copy igt@gem_stolen@stolen-fill-purge igt@gem_stolen@stolen-no-mmap igt@gem_stolen@stolen-pread igt@gem_stolen@stolen-pwrite anything gen3 ------------- igt@gen3_mixed_blits igt@gen3_render_linear_blits igt@gen3_render_mixed_blits igt@gen3_render_tiledx_blits igt@gen3_render_tiledy_blits Anything 3x, 4x, 5x, 6x ----------------------- and igt@kms_frontbuffer_tracking@*psr-2p-* d-e-f planes/pipe stuff ------------------------- igt@kms_busy@*-[d|e|f] igt@kms_ccs@pipe-[d|e|f]-* igt@kms_color@pipe-[d|e|f]-* igt@kms_concurrent@pipe-[d|e|f] igt@kms_cursor_legacy@pipe-[d|e|f]-* igt@kms_plane@*-pipe-[d|e|f]-planes igt@kms_plane_lowres@pipe-[d|e|f]-* igt@kms_plane_multiple@*-pipe-[d|e|f]-* igt@kms_plane_scaling@pipe-[d|e|f]-* igt@kms_universal_plane@*-[d|e|f]-* igt@kms_vblank@pipe-[d|e|f]-* Other unsupported stuff ----------------------- igt@kms_chamelium@* igt@kms_cursor_crc@cursor-512x* igt@kms_draw_crc@draw-method-rgb565-render-* igt@kms_force_connector_basic@* igt@kms_frontbuffer_tracking@*drrs* igt@kms_pipe_b_c_ivb@* igt@kms_rotation_crc@*-reflect-* --------------------------------------------------------------------------------------- Intel Finland OY Registered Office: Westendinkatu 7, 02160 Espoo, Finland Registration Number: 0357606-4 _______________________________________________ igt-dev mailing list igt-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/igt-dev ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [igt-dev] Intel CI blacklist suggestion for tests that are skipped on all shards 2018-02-06 11:17 [igt-dev] Intel CI blacklist suggestion for tests that are skipped on all shards Lofstedt, Marta @ 2018-02-06 18:41 ` Jani Nikula 2018-02-07 7:07 ` Lofstedt, Marta 0 siblings, 1 reply; 5+ messages in thread From: Jani Nikula @ 2018-02-06 18:41 UTC (permalink / raw) To: Lofstedt, Marta, igt-dev@lists.freedesktop.org Cc: Sarvela, Tomi P, Peres, Martin On Tue, 06 Feb 2018, "Lofstedt, Marta" <marta.lofstedt@intel.com> wrote: > After summarizing the skip time for our shards: > GLK: 428s, APL: 371s, SNB:325s, KBL:283, HSW:182s Why does it take so long on any of those platforms to skip the tests? BR, Jani. -- Jani Nikula, Intel Open Source Technology Center _______________________________________________ igt-dev mailing list igt-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/igt-dev ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [igt-dev] Intel CI blacklist suggestion for tests that are skipped on all shards 2018-02-06 18:41 ` Jani Nikula @ 2018-02-07 7:07 ` Lofstedt, Marta 2018-02-07 11:05 ` Jani Nikula 0 siblings, 1 reply; 5+ messages in thread From: Lofstedt, Marta @ 2018-02-07 7:07 UTC (permalink / raw) To: Jani Nikula, igt-dev@lists.freedesktop.org; +Cc: Sarvela, Tomi P, Peres, Martin > -----Original Message----- > From: Jani Nikula [mailto:jani.nikula@linux.intel.com] > Sent: Tuesday, February 6, 2018 8:42 PM > To: Lofstedt, Marta <marta.lofstedt@intel.com>; igt- > dev@lists.freedesktop.org > Cc: Sarvela, Tomi P <tomi.p.sarvela@intel.com>; Peres, Martin > <martin.peres@intel.com> > Subject: Re: [igt-dev] Intel CI blacklist suggestion for tests that are skipped on > all shards > > On Tue, 06 Feb 2018, "Lofstedt, Marta" <marta.lofstedt@intel.com> wrote: > > After summarizing the skip time for our shards: > > GLK: 428s, APL: 371s, SNB:325s, KBL:283, HSW:182s > > Why does it take so long on any of those platforms to skip the tests? I believe it is the huge amount of skipped tests that is the main culprit, ~1500*0,1 seconds is real time. And the new DRRS test contribute a lot. Also, I summarized some timings of the the kms_frontbuffer_tracking on GLK-shards: Total time = 2153s, note this is more than half of the total GLK-shard runtime. Skipped time = 238s, this has more than doubled due to new DRRS tests Fail time = 1365s, this is mostly for the PSR tests that have been failing for months, due to unsupported panels being shipped with GLK RVPs. Unfortunately we don't have a full set of supported PSR panels. And if we changed to supported PSR panels I assume that the runtime would only increase further. > > BR, > Jani. > > > -- > Jani Nikula, Intel Open Source Technology Center _______________________________________________ igt-dev mailing list igt-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/igt-dev ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [igt-dev] Intel CI blacklist suggestion for tests that are skipped on all shards 2018-02-07 7:07 ` Lofstedt, Marta @ 2018-02-07 11:05 ` Jani Nikula 2018-02-07 11:12 ` Lofstedt, Marta 0 siblings, 1 reply; 5+ messages in thread From: Jani Nikula @ 2018-02-07 11:05 UTC (permalink / raw) To: Lofstedt, Marta, igt-dev@lists.freedesktop.org Cc: Sarvela, Tomi P, Peres, Martin On Wed, 07 Feb 2018, "Lofstedt, Marta" <marta.lofstedt@intel.com> wrote: >> -----Original Message----- >> From: Jani Nikula [mailto:jani.nikula@linux.intel.com] >> Sent: Tuesday, February 6, 2018 8:42 PM >> To: Lofstedt, Marta <marta.lofstedt@intel.com>; igt- >> dev@lists.freedesktop.org >> Cc: Sarvela, Tomi P <tomi.p.sarvela@intel.com>; Peres, Martin >> <martin.peres@intel.com> >> Subject: Re: [igt-dev] Intel CI blacklist suggestion for tests that are skipped on >> all shards >> >> On Tue, 06 Feb 2018, "Lofstedt, Marta" <marta.lofstedt@intel.com> wrote: >> > After summarizing the skip time for our shards: >> > GLK: 428s, APL: 371s, SNB:325s, KBL:283, HSW:182s >> >> Why does it take so long on any of those platforms to skip the tests? > > I believe it is the huge amount of skipped tests that is the main > culprit, ~1500*0,1 seconds is real time. And the new DRRS test > contribute a lot. > > Also, I summarized some timings of the the kms_frontbuffer_tracking on > GLK-shards: > > Total time = 2153s, note this is more than half of the total GLK-shard > runtime. Skipped time = 238s, this has more than doubled due to new > DRRS tests > Fail time = 1365s, this is mostly for the PSR tests that have been > failing for months, due to unsupported panels being shipped with GLK > RVPs. Unfortunately we don't have a full set of supported PSR > panels. And if we changed to supported PSR panels I assume that the > runtime would only increase further. Okay. I'm just wondering about the possibilities of optimizing the skipping in the tests themselves, perhaps by bailing out earlier, vs. maintainability of a single big blacklist which combines plenty of features across plenty of platforms. BR, Jani. > > >> >> BR, >> Jani. >> >> >> -- >> Jani Nikula, Intel Open Source Technology Center -- Jani Nikula, Intel Open Source Technology Center _______________________________________________ igt-dev mailing list igt-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/igt-dev ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [igt-dev] Intel CI blacklist suggestion for tests that are skipped on all shards 2018-02-07 11:05 ` Jani Nikula @ 2018-02-07 11:12 ` Lofstedt, Marta 0 siblings, 0 replies; 5+ messages in thread From: Lofstedt, Marta @ 2018-02-07 11:12 UTC (permalink / raw) To: Jani Nikula, igt-dev@lists.freedesktop.org; +Cc: Sarvela, Tomi P, Peres, Martin > -----Original Message----- > From: Jani Nikula [mailto:jani.nikula@linux.intel.com] > Sent: Wednesday, February 7, 2018 1:06 PM > To: Lofstedt, Marta <marta.lofstedt@intel.com>; igt- > dev@lists.freedesktop.org > Cc: Sarvela, Tomi P <tomi.p.sarvela@intel.com>; Peres, Martin > <martin.peres@intel.com> > Subject: RE: [igt-dev] Intel CI blacklist suggestion for tests that are skipped on > all shards > > On Wed, 07 Feb 2018, "Lofstedt, Marta" <marta.lofstedt@intel.com> wrote: > >> -----Original Message----- > >> From: Jani Nikula [mailto:jani.nikula@linux.intel.com] > >> Sent: Tuesday, February 6, 2018 8:42 PM > >> To: Lofstedt, Marta <marta.lofstedt@intel.com>; igt- > >> dev@lists.freedesktop.org > >> Cc: Sarvela, Tomi P <tomi.p.sarvela@intel.com>; Peres, Martin > >> <martin.peres@intel.com> > >> Subject: Re: [igt-dev] Intel CI blacklist suggestion for tests that > >> are skipped on all shards > >> > >> On Tue, 06 Feb 2018, "Lofstedt, Marta" <marta.lofstedt@intel.com> > wrote: > >> > After summarizing the skip time for our shards: > >> > GLK: 428s, APL: 371s, SNB:325s, KBL:283, HSW:182s > >> > >> Why does it take so long on any of those platforms to skip the tests? > > > > I believe it is the huge amount of skipped tests that is the main > > culprit, ~1500*0,1 seconds is real time. And the new DRRS test > > contribute a lot. > > > > Also, I summarized some timings of the the kms_frontbuffer_tracking on > > GLK-shards: > > > > Total time = 2153s, note this is more than half of the total GLK-shard > > runtime. Skipped time = 238s, this has more than doubled due to new > > DRRS tests > > > Fail time = 1365s, this is mostly for the PSR tests that have been > > failing for months, due to unsupported panels being shipped with GLK > > RVPs. Unfortunately we don't have a full set of supported PSR panels. > > And if we changed to supported PSR panels I assume that the runtime > > would only increase further. > > Okay. I'm just wondering about the possibilities of optimizing the skipping in > the tests themselves, perhaps by bailing out earlier, vs. maintainability of a > single big blacklist which combines plenty of features across plenty of > platforms. VIZ-8825 could do that, but I usually prefer picking low hanging fruit before redefining how CI run tests and how IGT test are structured. > > BR, > Jani. > > > > > > > >> > >> BR, > >> Jani. > >> > >> > >> -- > >> Jani Nikula, Intel Open Source Technology Center > > -- > Jani Nikula, Intel Open Source Technology Center _______________________________________________ igt-dev mailing list igt-dev@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/igt-dev ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2018-02-07 11:12 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2018-02-06 11:17 [igt-dev] Intel CI blacklist suggestion for tests that are skipped on all shards Lofstedt, Marta 2018-02-06 18:41 ` Jani Nikula 2018-02-07 7:07 ` Lofstedt, Marta 2018-02-07 11:05 ` Jani Nikula 2018-02-07 11:12 ` Lofstedt, Marta
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox