Re: [igt-dev] [PATCH] RFC: tests/kms_setmode: rename subtest to simplify test combinations

[Imre Deak <imre.deak@intel.com>]

On Mon, Aug 14, 2023 at 01:35:27PM +0530, Jeevan B wrote:
> Both 'basic-clone-single-crtc' and 'invalid-clone-single-crtc' tests aim to
> assess a scenario where two connectors are attempted to be run with a single
> CRTC, which is not feasible.  Consequently, the  'basic-clone-single-crtc'
> test is marked as invalid and skipped due to the impractical scenario. On the
> other hand, the 'invalid-clone-single-crtc' test serves as a negative scenario,
> confirming the failure of the 'ioctl' and indicating a successful test.
> 
> similarly, the 'invalid-clone-exclusive-crtc' test attempts to run two connectors
> with two CRTCs, which is indeed possible, rendering the test nonsensical.
> Consequently, the 'clone-exclusive-crtc' test runs and passes as expected.
> 
> Signed-off-by: Jeevan B <jeevan.b@intel.com>
> Cc: Imre Deak <imre.deak@intel.com>
> ---
>  tests/kms_setmode.c | 6 +-----
>  1 file changed, 1 insertion(+), 5 deletions(-)
> 
> diff --git a/tests/kms_setmode.c b/tests/kms_setmode.c
> index 035bbd0b0..0c65e3937 100644
> --- a/tests/kms_setmode.c
> +++ b/tests/kms_setmode.c
> @@ -932,12 +932,8 @@ igt_main_args("det:", NULL, help_str, opt_handler, NULL)
>  		const char *name;
>  	} tests[] = {
>  		{ TEST_TIMINGS, "basic" },
> -		{ TEST_CLONE | TEST_SINGLE_CRTC_CLONE,
> -					"basic-clone-single-crtc" },

It is feasible to use a single CRTC with two connectors, for instance in
case of a VGA and an HDMI connector; so this subtest can't be removed.

>  		{ TEST_INVALID | TEST_CLONE | TEST_SINGLE_CRTC_CLONE,
> -					"invalid-clone-single-crtc" },
> -		{ TEST_INVALID | TEST_CLONE | TEST_EXCLUSIVE_CRTC_CLONE,

There are invalid configurations when each connector uses its own CRTC,
for instance when a connector uses a CRTC not compatible with the
connector. So this subtest can't be removed either.

I suppose you're fixing a problem where a subtest is failing if there
are no dynamic tests defined for it, on some platforms. In that case
the subtest should be made to pass explicitly, perhaps by ensuring that
there is at least one dynamic test defined for each subtest.

> -					"invalid-clone-exclusive-crtc" },
> +					"clone-single-crtc-negative" },
>  		{ TEST_CLONE | TEST_EXCLUSIVE_CRTC_CLONE,
>  					"clone-exclusive-crtc" },
>  		{ TEST_INVALID | TEST_CLONE | TEST_SINGLE_CRTC_CLONE | TEST_STEALING,
> -- 
> 2.25.1
>