Re: [PATCH] drm/doc: ci: Require more context for flaky tests

[Helen Koike <helen.koike@collabora.com>]

On 19/10/2023 13:51, Helen Koike wrote:
> 
> 
> On 19/10/2023 06:46, Maxime Ripard wrote:
>> Flaky tests can be very difficult to reproduce after the facts, which
>> will make it even harder to ever fix.
>>
>> Let's document the metadata we agreed on to provide more context to
>> anyone trying to address these fixes.
>>
>> Link: 
>> https://lore.kernel.org/dri-devel/CAPj87rPbJ1V1-R7WMTHkDat2A4nwSd61Df9mdGH2PR=ZzxaU=Q@mail.gmail.com/
>> Signed-off-by: Maxime Ripard <mripard@kernel.org>
>> ---
>>   Documentation/gpu/automated_testing.rst | 13 +++++++++++++
>>   1 file changed, 13 insertions(+)
>>
>> diff --git a/Documentation/gpu/automated_testing.rst 
>> b/Documentation/gpu/automated_testing.rst
>> index 469b6fb65c30..2dd0e221c2c3 100644
>> --- a/Documentation/gpu/automated_testing.rst
>> +++ b/Documentation/gpu/automated_testing.rst
>> @@ -67,6 +67,19 @@ Lists the tests that for a given driver on a 
>> specific hardware revision are
>>   known to behave unreliably. These tests won't cause a job to fail 
>> regardless of
>>   the result. They will still be run.
>> +Each new flake entry must be associated with a link to a bug report to
> 
> What do you mean by but report? Just a link to an email to the mailing 
> list is enough?
> 
> Also, I had made a mistake to the first flakes lists, which I corrected 
> with https://www.spinics.net/lists/kernel/msg4959629.html (there was a 
> bug in my script which ended up erroneous adding a bunch of tests in the 
> flake list, so I cleaned them up), I would like to kind request to let 
> me add those documentation in a future patch to not block that patch 
> series.
> 
> Thanks
> Helen
> 
> 
>> +the author of the affected driver, the board name or Device Tree name of
>> +the board, the first kernel version affected, and an approximation of
>> +the failure rate.
>> +
>> +They should be provided under the following format::
>> +
>> +  # Bug Report: $LORE_OR_PATCHWORK_URL

I wonder if the commit adding the test into the flakes.txt file with and 
Acked-by from the device maintainer shouldn't be already considered the 
Bug Report.

>> +  # Board Name: broken-board.dtb

Maybe Board Name isn't required, since it is already in the name of the 
file.

>> +  # Version: 6.6-rc1
>> +  # Failure Rate: 100

Maybe also:

   # Pipeline url: 
https://gitlab.freedesktop.org/helen.fornazier/linux/-/pipelines/1014435

All this info will complicated a bit the update-xfails.py script, but 
well, we can handle...
(see 
https://patchwork.kernel.org/project/dri-devel/patch/20231020034124.136295-4-helen.koike@collabora.com/ 
)
We need to update that script to make life easier.
Vignesh sent a patch adding at least the pipeline url to the file
https://patchwork.kernel.org/project/linux-arm-msm/patch/20231019070650.61159-9-vignesh.raman@collabora.com/
but to meet this doc that needs to be updated too.

Regards,
Helen

>> +  flaky-test
>> +
>>   drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-skips.txt
>>   -----------------------------------------------------------