Re: [PATCH] drm/doc: ci: Require more context for flaky tests

[Daniel Vetter <daniel@ffwll.ch>]

On Thu, Oct 19, 2023 at 11:46:09AM +0200, Maxime Ripard wrote:
> Flaky tests can be very difficult to reproduce after the facts, which
> will make it even harder to ever fix.
> 
> Let's document the metadata we agreed on to provide more context to
> anyone trying to address these fixes.
> 
> Link: https://lore.kernel.org/dri-devel/CAPj87rPbJ1V1-R7WMTHkDat2A4nwSd61Df9mdGH2PR=ZzxaU=Q@mail.gmail.com/
> Signed-off-by: Maxime Ripard <mripard@kernel.org>

Not that my opinion matters much since I'm really not involved in the
details, and no opinion on the specific format and all that, but this
sounds like a very good idea too me.

Acked-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Cheers, Sima
> ---
>  Documentation/gpu/automated_testing.rst | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/Documentation/gpu/automated_testing.rst b/Documentation/gpu/automated_testing.rst
> index 469b6fb65c30..2dd0e221c2c3 100644
> --- a/Documentation/gpu/automated_testing.rst
> +++ b/Documentation/gpu/automated_testing.rst
> @@ -67,6 +67,19 @@ Lists the tests that for a given driver on a specific hardware revision are
>  known to behave unreliably. These tests won't cause a job to fail regardless of
>  the result. They will still be run.
>  
> +Each new flake entry must be associated with a link to a bug report to
> +the author of the affected driver, the board name or Device Tree name of
> +the board, the first kernel version affected, and an approximation of
> +the failure rate.
> +
> +They should be provided under the following format::
> +
> +  # Bug Report: $LORE_OR_PATCHWORK_URL
> +  # Board Name: broken-board.dtb
> +  # Version: 6.6-rc1
> +  # Failure Rate: 100
> +  flaky-test
> +
>  drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-skips.txt
>  -----------------------------------------------------------
>  
> -- 
> 2.41.0
> 

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch