Re: [PATCH i-g-t v2] tests: Add explicit test lists for CI use.

[Lionel Landwerlin <lionel.g.landwerlin@intel.com>]

Regardless of the list of tests we expect to pass, I think it's better 
to have an actual list rather than relying on the name of the tests (as 
we have been doing or are still doing, correct me if I'm wrong).

Thanks for doing this!

Acked-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>

On 06/10/16 09:54, Petri Latvala wrote:
> See tests/intel-ci/README for rationale and explanation.
>
> v2: Use the current BAT set for fast-feedback.testlist first
>
> Signed-off-by: Petri Latvala <petri.latvala@intel.com>
> ---
>
> Thanks for the feedback. This commit only contains the current test
> set, extended test suites are left for future discussion.
>
>
>   tests/intel-ci/README                 |  34 +++++
>   tests/intel-ci/fast-feedback.testlist | 248 ++++++++++++++++++++++++++++++++++
>   2 files changed, 282 insertions(+)
>   create mode 100644 tests/intel-ci/README
>   create mode 100644 tests/intel-ci/fast-feedback.testlist
>
> diff --git a/tests/intel-ci/README b/tests/intel-ci/README
> new file mode 100644
> index 0000000..dc7e5c8
> --- /dev/null
> +++ b/tests/intel-ci/README
> @@ -0,0 +1,34 @@
> +This directory contains test lists that are used by Intel's CI. The
> +files are passed to piglit with the --test-list parameter directly.
> +
> +The test lists are contained in the IGT repository for several
> +reasons:
> +
> +- The lists stay synchronized with the IGT codebase.
> +- Public availability. Kernel developers can see what tests are run,
> +  and can see what changes are done to the set, when, and why.
> +- Explicit test lists in general make it possible to implement a new
> +  test without having it run in CI (accidentally or on purpose) before
> +  CI is ready for it, and for CI to stop running a test without
> +  removing the test or changing its name. The previous method of test
> +  selection was based on name patterns.
> +
> +Changing the test lists should only happen with approval from the CI
> +people. If you make a new test that should be tested by CI regularly,
> +ask a maintainer (see MAINTAINERS).
> +
> +======================
> +fast-feedback.testlist
> +======================
> +
> +Fast-feedback is the testlist that is run for Patchwork and Trybot
> +testing. It is also run on various drm-intel-* branches.
> +
> +Time limit: Full run must complete in 15 minutes on all tested
> +platforms.
> +
> +Feature coverage: Should test all working features, with at least 1
> +test per feature.
> +
> +The string "basic" in a test name means the test probably belongs in
> +this list.
> diff --git a/tests/intel-ci/fast-feedback.testlist b/tests/intel-ci/fast-feedback.testlist
> new file mode 100644
> index 0000000..ae94381
> --- /dev/null
> +++ b/tests/intel-ci/fast-feedback.testlist
> @@ -0,0 +1,248 @@
> +igt@core_auth@basic-auth
> +igt@core_prop_blob@basic
> +igt@drv_getparams_basic@basic-eu-total
> +igt@drv_getparams_basic@basic-subslice-total
> +igt@drv_hangman@error-state-basic
> +igt@drv_module_reload_basic
> +igt@gem_basic@bad-close
> +igt@gem_basic@create-close
> +igt@gem_basic@create-fd-close
> +igt@gem_busy@basic-busy-default
> +igt@gem_busy@basic-hang-default
> +igt@gem_close_race@basic-process
> +igt@gem_close_race@basic-threads
> +igt@gem_cpu_reloc@basic
> +igt@gem_cs_tlb@basic-default
> +igt@gem_ctx_basic
> +igt@gem_ctx_create@basic
> +igt@gem_ctx_create@basic-files
> +igt@gem_ctx_exec@basic
> +igt@gem_ctx_param@basic
> +igt@gem_ctx_param@basic-default
> +igt@gem_ctx_switch@basic-default
> +igt@gem_ctx_switch@basic-default-heavy
> +igt@gem_exec_basic@basic-blt
> +igt@gem_exec_basic@basic-bsd
> +igt@gem_exec_basic@basic-bsd1
> +igt@gem_exec_basic@basic-bsd2
> +igt@gem_exec_basic@basic-default
> +igt@gem_exec_basic@basic-render
> +igt@gem_exec_basic@basic-vebox
> +igt@gem_exec_basic@gtt-blt
> +igt@gem_exec_basic@gtt-bsd
> +igt@gem_exec_basic@gtt-bsd1
> +igt@gem_exec_basic@gtt-bsd2
> +igt@gem_exec_basic@gtt-default
> +igt@gem_exec_basic@gtt-render
> +igt@gem_exec_basic@gtt-vebox
> +igt@gem_exec_basic@readonly-blt
> +igt@gem_exec_basic@readonly-bsd
> +igt@gem_exec_basic@readonly-bsd1
> +igt@gem_exec_basic@readonly-bsd2
> +igt@gem_exec_basic@readonly-default
> +igt@gem_exec_basic@readonly-render
> +igt@gem_exec_basic@readonly-vebox
> +igt@gem_exec_create@basic
> +igt@gem_exec_flush@basic-batch-kernel-default-cmd
> +igt@gem_exec_flush@basic-batch-kernel-default-uc
> +igt@gem_exec_flush@basic-batch-kernel-default-wb
> +igt@gem_exec_flush@basic-uc-pro-default
> +igt@gem_exec_flush@basic-uc-prw-default
> +igt@gem_exec_flush@basic-uc-ro-default
> +igt@gem_exec_flush@basic-uc-rw-default
> +igt@gem_exec_flush@basic-uc-set-default
> +igt@gem_exec_flush@basic-wb-pro-default
> +igt@gem_exec_flush@basic-wb-prw-default
> +igt@gem_exec_flush@basic-wb-ro-before-default
> +igt@gem_exec_flush@basic-wb-ro-default
> +igt@gem_exec_flush@basic-wb-rw-before-default
> +igt@gem_exec_flush@basic-wb-rw-default
> +igt@gem_exec_flush@basic-wb-set-default
> +igt@gem_exec_gttfill@basic
> +igt@gem_exec_nop@basic-parallel
> +igt@gem_exec_nop@basic-series
> +igt@gem_exec_parallel@basic
> +igt@gem_exec_parse@basic-allowed
> +igt@gem_exec_parse@basic-rejected
> +igt@gem_exec_store@basic-all
> +igt@gem_exec_store@basic-blt
> +igt@gem_exec_store@basic-bsd
> +igt@gem_exec_store@basic-bsd1
> +igt@gem_exec_store@basic-bsd2
> +igt@gem_exec_store@basic-default
> +igt@gem_exec_store@basic-render
> +igt@gem_exec_store@basic-vebox
> +igt@gem_exec_suspend@basic
> +igt@gem_exec_suspend@basic-s3
> +igt@gem_exec_suspend@basic-s4
> +igt@gem_flink_basic@bad-flink
> +igt@gem_flink_basic@bad-open
> +igt@gem_flink_basic@basic
> +igt@gem_flink_basic@double-flink
> +igt@gem_flink_basic@flink-lifetime
> +igt@gem_linear_blits@basic
> +igt@gem_mmap@basic
> +igt@gem_mmap@basic-small-bo
> +igt@gem_mmap_gtt@basic
> +igt@gem_mmap_gtt@basic-copy
> +igt@gem_mmap_gtt@basic-read
> +igt@gem_mmap_gtt@basic-read-no-prefault
> +igt@gem_mmap_gtt@basic-read-write
> +igt@gem_mmap_gtt@basic-read-write-distinct
> +igt@gem_mmap_gtt@basic-short
> +igt@gem_mmap_gtt@basic-small-bo
> +igt@gem_mmap_gtt@basic-small-bo-tiledx
> +igt@gem_mmap_gtt@basic-small-bo-tiledy
> +igt@gem_mmap_gtt@basic-small-copy
> +igt@gem_mmap_gtt@basic-small-copy-xy
> +igt@gem_mmap_gtt@basic-wc
> +igt@gem_mmap_gtt@basic-write
> +igt@gem_mmap_gtt@basic-write-cpu-read-gtt
> +igt@gem_mmap_gtt@basic-write-gtt
> +igt@gem_mmap_gtt@basic-write-gtt-no-prefault
> +igt@gem_mmap_gtt@basic-write-no-prefault
> +igt@gem_mmap_gtt@basic-write-read
> +igt@gem_mmap_gtt@basic-write-read-distinct
> +igt@gem_pread@basic
> +igt@gem_pwrite@basic
> +igt@gem_render_linear_blits@basic
> +igt@gem_render_tiled_blits@basic
> +igt@gem_ringfill@basic-default
> +igt@gem_ringfill@basic-default-forked
> +igt@gem_ringfill@basic-default-hang
> +igt@gem_ringfill@basic-default-interruptible
> +igt@gem_storedw_loop@basic-blt
> +igt@gem_storedw_loop@basic-bsd
> +igt@gem_storedw_loop@basic-bsd1
> +igt@gem_storedw_loop@basic-bsd2
> +igt@gem_storedw_loop@basic-default
> +igt@gem_storedw_loop@basic-render
> +igt@gem_storedw_loop@basic-vebox
> +igt@gem_sync@basic-all
> +igt@gem_sync@basic-each
> +igt@gem_sync@basic-many-each
> +igt@gem_sync@basic-store-all
> +igt@gem_sync@basic-store-each
> +igt@gem_tiled_blits@basic
> +igt@gem_tiled_fence_blits@basic
> +igt@gem_tiled_pread_basic
> +igt@gem_workarounds@basic-read
> +igt@gvt_basic@invalid-placeholder-test
> +igt@kms_addfb_basic@addfb25-bad-modifier
> +igt@kms_addfb_basic@addfb25-framebuffer-vs-set-tiling
> +igt@kms_addfb_basic@addfb25-modifier-no-flag
> +igt@kms_addfb_basic@addfb25-x-tiled
> +igt@kms_addfb_basic@addfb25-x-tiled-mismatch
> +igt@kms_addfb_basic@addfb25-yf-tiled
> +igt@kms_addfb_basic@addfb25-y-tiled
> +igt@kms_addfb_basic@addfb25-y-tiled-small
> +igt@kms_addfb_basic@bad-pitch-0
> +igt@kms_addfb_basic@bad-pitch-1024
> +igt@kms_addfb_basic@bad-pitch-128
> +igt@kms_addfb_basic@bad-pitch-256
> +igt@kms_addfb_basic@bad-pitch-32
> +igt@kms_addfb_basic@bad-pitch-63
> +igt@kms_addfb_basic@bad-pitch-65536
> +igt@kms_addfb_basic@bad-pitch-999
> +igt@kms_addfb_basic@basic
> +igt@kms_addfb_basic@basic-x-tiled
> +igt@kms_addfb_basic@basic-y-tiled
> +igt@kms_addfb_basic@bo-too-small
> +igt@kms_addfb_basic@bo-too-small-due-to-tiling
> +igt@kms_addfb_basic@clobberred-modifier
> +igt@kms_addfb_basic@framebuffer-vs-set-tiling
> +igt@kms_addfb_basic@invalid-get-prop
> +igt@kms_addfb_basic@invalid-get-prop-any
> +igt@kms_addfb_basic@invalid-set-prop
> +igt@kms_addfb_basic@invalid-set-prop-any
> +igt@kms_addfb_basic@no-handle
> +igt@kms_addfb_basic@size-max
> +igt@kms_addfb_basic@small-bo
> +igt@kms_addfb_basic@tile-pitch-mismatch
> +igt@kms_addfb_basic@too-high
> +igt@kms_addfb_basic@too-wide
> +igt@kms_addfb_basic@unused-handle
> +igt@kms_addfb_basic@unused-modifier
> +igt@kms_addfb_basic@unused-offsets
> +igt@kms_addfb_basic@unused-pitches
> +igt@kms_busy@basic-flip-default-a
> +igt@kms_busy@basic-flip-default-b
> +igt@kms_busy@basic-flip-default-c
> +igt@kms_cursor_legacy@basic-busy-flip-before-cursor-atomic
> +igt@kms_cursor_legacy@basic-busy-flip-before-cursor-legacy
> +igt@kms_cursor_legacy@basic-busy-flip-before-cursor-varying-size
> +igt@kms_cursor_legacy@basic-flip-after-cursor-atomic
> +igt@kms_cursor_legacy@basic-flip-after-cursor-legacy
> +igt@kms_cursor_legacy@basic-flip-after-cursor-varying-size
> +igt@kms_cursor_legacy@basic-flip-before-cursor-atomic
> +igt@kms_cursor_legacy@basic-flip-before-cursor-legacy
> +igt@kms_cursor_legacy@basic-flip-before-cursor-varying-size
> +igt@kms_flip@basic-flip-vs-dpms
> +igt@kms_flip@basic-flip-vs-modeset
> +igt@kms_flip@basic-flip-vs-wf_vblank
> +igt@kms_flip@basic-plain-flip
> +igt@kms_force_connector_basic@force-connector-state
> +igt@kms_force_connector_basic@force-edid
> +igt@kms_force_connector_basic@force-load-detect
> +igt@kms_force_connector_basic@prune-stale-modes
> +igt@kms_frontbuffer_tracking@basic
> +igt@kms_pipe_crc_basic@bad-nb-words-1
> +igt@kms_pipe_crc_basic@bad-nb-words-3
> +igt@kms_pipe_crc_basic@bad-pipe
> +igt@kms_pipe_crc_basic@bad-source
> +igt@kms_pipe_crc_basic@hang-read-crc-pipe-a
> +igt@kms_pipe_crc_basic@hang-read-crc-pipe-b
> +igt@kms_pipe_crc_basic@hang-read-crc-pipe-c
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a-frame-sequence
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-b-frame-sequence
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-c
> +igt@kms_pipe_crc_basic@nonblocking-crc-pipe-c-frame-sequence
> +igt@kms_pipe_crc_basic@read-crc-pipe-a
> +igt@kms_pipe_crc_basic@read-crc-pipe-a-frame-sequence
> +igt@kms_pipe_crc_basic@read-crc-pipe-b
> +igt@kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence
> +igt@kms_pipe_crc_basic@read-crc-pipe-c
> +igt@kms_pipe_crc_basic@read-crc-pipe-c-frame-sequence
> +igt@kms_pipe_crc_basic@suspend-read-crc-pipe-a
> +igt@kms_pipe_crc_basic@suspend-read-crc-pipe-b
> +igt@kms_pipe_crc_basic@suspend-read-crc-pipe-c
> +igt@kms_psr_sink_crc@psr_basic
> +igt@kms_setmode@basic-clone-single-crtc
> +igt@kms_sink_crc_basic
> +igt@pm_backlight@basic-brightness
> +igt@pm_rpm@basic-pci-d3-state
> +igt@pm_rpm@basic-rte
> +igt@pm_rps@basic-api
> +igt@prime_busy@basic-after-default
> +igt@prime_busy@basic-before-default
> +igt@prime_busy@basic-wait-after-default
> +igt@prime_busy@basic-wait-before-default
> +igt@prime_self_import@basic-llseek-bad
> +igt@prime_self_import@basic-llseek-size
> +igt@prime_self_import@basic-with_fd_dup
> +igt@prime_self_import@basic-with_one_bo
> +igt@prime_self_import@basic-with_one_bo_two_files
> +igt@prime_self_import@basic-with_two_bos
> +igt@prime_vgem@basic-busy-default
> +igt@prime_vgem@basic-fence-flip
> +igt@prime_vgem@basic-fence-mmap
> +igt@prime_vgem@basic-fence-read
> +igt@prime_vgem@basic-fence-wait-default
> +igt@prime_vgem@basic-gtt
> +igt@prime_vgem@basic-read
> +igt@prime_vgem@basic-sync-default
> +igt@prime_vgem@basic-wait-default
> +igt@prime_vgem@basic-write
> +igt@vgem_basic@create
> +igt@vgem_basic@debugfs
> +igt@vgem_basic@dmabuf-export
> +igt@vgem_basic@dmabuf-fence
> +igt@vgem_basic@dmabuf-fence-before
> +igt@vgem_basic@dmabuf-mmap
> +igt@vgem_basic@mmap
> +igt@vgem_basic@second-client
> +igt@vgem_basic@sysfs
> +igt@vgem_basic@unload
> +igt@vgem_reload_basic


_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx