Re: [PATCH] drm/ci: add kms_cursor_legacy@torture-bo to apq8016 flakes

[Vignesh Raman <vignesh.raman@collabora.com>]

Hi Abhinav / Helen,

On 12/12/24 01:48, Abhinav Kumar wrote:
> Hi Helen / Vignesh
> 
> On 12/4/2024 12:33 PM, Helen Mae Koike Fornazier wrote:
>>
>>
>>
>>
>> ---- On Wed, 04 Dec 2024 16:21:26 -0300 Abhinav Kumar  wrote ---
>>
>>   > Hi Helen
>>   >
>>   > On 12/4/2024 11:14 AM, Helen Mae Koike Fornazier wrote:
>>   > > Hi Abhinav,
>>   > >
>>   > > Thanks for your patch.
>>   > >
>>   > >
>>   > >
>>   > > ---- On Wed, 04 Dec 2024 15:55:17 -0300 Abhinav Kumar  wrote ---
>>   > >
>>   > >   > From the jobs [1] and [2] of pipeline [3], its clear that
>>   > >   > kms_cursor_legacy@torture-bo is most certainly a flake and
>>   > >   > not a fail for apq8016. Mark the test accordingly to match 
>> the results.
>>   > >   >
>>   > >   > [1] : https://gitlab.freedesktop.org/drm/msm/-/jobs/67676481

The test passes - kms_cursor_legacy@torture-bo,UnexpectedImprovement(Pass)

>>   > >   > [2] : https://gitlab.freedesktop.org/drm/msm/-/jobs/67677430

There are no test failures

>>   > >   > [3]: https://gitlab.freedesktop.org/drm/msm/-/pipelines/1322770

The job is same as 2

In this case, the test passes and deqp-runner does not report it as 
flake. So we only need to remove it from fails file.


>>   > >   >
>>   > >   > Signed-off-by: Abhinav Kumar quic_abhinavk@quicinc.com>
>>   > >   > ---
>>   > >   >  drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt | 5 +++++
>>   > >   >  1 file changed, 5 insertions(+)
>>   > >   >
>>   > >   > diff --git 
>> a/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt 
>> b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
>>   > >   > new file mode 100644
>>   > >   > index 000000000000..18639853f18f
>>   > >   > --- /dev/null
>>   > >   > +++ b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
>>   > >   > @@ -0,0 +1,5 @@
>>   > >   > +# Board Name: msm-apq8016-db410c
>>   > >   > +# Failure Rate: 100
>>   > >
>>   > > Is failure rate is 100%, isn't it a fail than?
>>   > > (I know we have other cases with Failure Rate: 100, maybe we 
>> should fix them as well)
>>   > >
>>   >
>>   > Maybe I misunderstood the meaning of "Failure rate" for a flake.
>>   >
>>   > I interpreted this as this test being flaky 100% of the time :)
>>
>> Ah right, I see, inside deqp-runner (that auto-retries).
>>
>> I'd like to hear Vignesh's opinion on this.
>>
>> (In any case, we probably should document this better)

deqp-runner reports new (not present in flakes file) or known (present 
in flakes file) flakes

2024-12-11 07:25:44.709666: Some new flakes found:
2024-12-11 07:25:44.709676:   kms_lease@page-flip-implicit-plane

2024-12-11 13:15:16.482890: Some known flakes found:
2024-12-11 13:15:16.482898: 
kms_async_flips@async-flip-with-page-flip-events-atomic

we add it to flakes file if deqp runner reports new flakes. Another case 
where we update flake tests is when a test passes in one run but fails 
in another, but deqp-runner does not report it as flake.

Regards,
Vignesh

>>
>> Regards,
>> Helen
>>
> 
> Can you let me know which way we need to go?
> 
> Just in case I did post a v2 fixing this, 
> https://patchwork.freedesktop.org/patch/627276/
> 
> If thats the way to go, can you pls take a look?
> 
> Thanks
> 
> Abhinav
>>   >
>>   > Out of the 3 runs of the test, it passed 2/3 times and failed 1/3.
>>   >
>>   > So its fail % actually is 33.33% in that case.
>>   >
>>   > I think I saw a Failure rate of 100% on msm-sm8350-hdk-flakes.txt and
>>   > mistook that as the rate at which flakes are seen.
>>   >
>>   > Let me fix this up as 33%
>>   >
>>   > > Regards,
>>   > > Helen
>>   > >
>>   > >   > +# IGT Version: 1.28-ga73311079
>>   > >   > +# Linux Version: 6.12.0-rc2
>>   > >   > +kms_cursor_legacy@torture-bo
>>   > >   >
>>   > >   > ---
>>   > >   > base-commit: 798bb342e0416d846cf67f4725a3428f39bfb96b
>>   > >   > change-id: 20241204-cursor_tor_skip-9d128dd62c4f
>>   > >   >
>>   > >   > Best regards,
>>   > >   > --
>>   > >   > Abhinav Kumar quic_abhinavk@quicinc.com>
>>   > >   >
>>   > >   >
>>   > >
>>   >
>>