Re: [PATCH] drm/ci: add kms_cursor_legacy@torture-bo to apq8016 flakes

[Vignesh Raman <vignesh.raman@collabora.com>]

Hi Abhinav,

On 16/12/24 11:39, Abhinav Kumar wrote:
> 
> 
> On 12/15/2024 9:45 PM, Vignesh Raman wrote:
>> Hi Abhinav,
>>
>> On 14/12/24 01:09, Abhinav Kumar wrote:
>>> Hi Vignesh
>>>
>>> On 12/11/2024 9:10 PM, Vignesh Raman wrote:
>>>> Hi Abhinav / Helen,
>>>>
>>>> On 12/12/24 01:48, Abhinav Kumar wrote:
>>>>> Hi Helen / Vignesh
>>>>>
>>>>> On 12/4/2024 12:33 PM, Helen Mae Koike Fornazier wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---- On Wed, 04 Dec 2024 16:21:26 -0300 Abhinav Kumar  wrote ---
>>>>>>
>>>>>>   > Hi Helen
>>>>>>   >
>>>>>>   > On 12/4/2024 11:14 AM, Helen Mae Koike Fornazier wrote:
>>>>>>   > > Hi Abhinav,
>>>>>>   > >
>>>>>>   > > Thanks for your patch.
>>>>>>   > >
>>>>>>   > >
>>>>>>   > >
>>>>>>   > > ---- On Wed, 04 Dec 2024 15:55:17 -0300 Abhinav Kumar  wrote 
>>>>>> ---
>>>>>>   > >
>>>>>>   > >   > From the jobs [1] and [2] of pipeline [3], its clear that
>>>>>>   > >   > kms_cursor_legacy@torture-bo is most certainly a flake and
>>>>>>   > >   > not a fail for apq8016. Mark the test accordingly to 
>>>>>> match the results.
>>>>>>   > >   >
>>>>>>   > >   > [1] : 
>>>>>> https://gitlab.freedesktop.org/drm/msm/-/jobs/67676481
>>>>
>>>> The test passes - 
>>>> kms_cursor_legacy@torture-bo,UnexpectedImprovement(Pass)
>>>>
>>>
>>> Yes, thats the problem
>>>
>>> https://gitlab.freedesktop.org/drm/msm/-/jobs/67676481/viewer#L2696
>>>
>>> 24-12-04 03:51:55 R SERIAL> [  179.241309] [IGT] kms_cursor_legacy: 
>>> finished subtest all-pipes, SUCCESS
>>> 24-12-04 03:51:55 R SERIAL> [  179.241812] [IGT] kms_cursor_legacy: 
>>> finished subtest torture-bo, SUCCESS
>>>
>>> Here it passes whereas it was marked a failure. Hence pipeline fails.
>>
>> Yes it fails due to,
>>
>> Unexpected results:
>>   kms_cursor_legacy@torture-bo,UnexpectedImprovement(Pass)
>>
>> In this case, we need to remove this test from fails.txt
>>
>>>
>>>>>>   > >   > [2] : 
>>>>>> https://gitlab.freedesktop.org/drm/msm/-/jobs/67677430
>>>>
>>>> There are no test failures
>>>>
>>>
>>> No, thats not true
>>>
>>> https://gitlab.freedesktop.org/drm/msm/-/jobs/67677430/viewer#L2694
>>>
>>> 24-12-04 04:18:38 R SERIAL> [  170.379649] Console: switching to 
>>> colour dummy device 80x25
>>> 24-12-04 04:18:38 R SERIAL> [  170.379938] [IGT] kms_cursor_legacy: 
>>> executing
>>> 24-12-04 04:18:38 R SERIAL> [  170.393868] [IGT] kms_cursor_legacy: 
>>> starting subtest torture-bo
>>> 24-12-04 04:18:38 R SERIAL> [  170.394186] [IGT] kms_cursor_legacy: 
>>> starting dynamic subtest pipe-A
>>> 24-12-04 04:18:38 R SERIAL> [  170.661749] [IGT] kms_cursor_legacy: 
>>> finished subtest pipe-A, FAIL
>>> 24-12-04 04:18:38 R SERIAL> [  170.662060] [IGT] kms_cursor_legacy: 
>>> starting dynamic subtest all-pipes
>>> 24-12-04 04:18:38 R SERIAL> [  170.713237] [IGT] kms_cursor_legacy: 
>>> finished subtest all-pipes, FAIL
>>> 24-12-04 04:18:38 R SERIAL> [  170.713513] [IGT] kms_cursor_legacy: 
>>> finished subtest torture-bo, FAIL
>>> 24-12-04 04:18:38 R SERIAL> [  170.721263] [IGT] kms_cursor_legacy: 
>>> exiting, ret=98
>>> 24-12-04 04:18:38 R SERIAL> [  170.737857] Console: switching to 
>>> colour frame buffer device 128x48
>>>
>>> Please check these logs, the torture-bo test-case did fail. The 
>>> pipeline was marked pass because it was an expected fail.
>>>
>>> So we have two pipelines, where one failed and the other passed. So 
>>> thats a flake for me.
>>
>> Yes agree. So if we had removed the test from fails, deqp-runner would 
>> have reported this as flake.
>>
>> deqp-runner runs the test and if it fails, it retries. If the test 
>> passes on retry, it is reported as a flake.
>>
>>>
>>>>>>   > >   > [3]: 
>>>>>> https://gitlab.freedesktop.org/drm/msm/-/pipelines/1322770
>>>>
>>>> The job is same as 2
>>>>
>>>> In this case, the test passes and deqp-runner does not report it as 
>>>> flake. So we only need to remove it from fails file.
>>>>
>>>
>>> No, like I mentioned above we have a pass and a fail.
>>>
>>>>
>>>>>>   > >   >
>>>>>>   > >   > Signed-off-by: Abhinav Kumar quic_abhinavk@quicinc.com>
>>>>>>   > >   > ---
>>>>>>   > >   >  drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt | 5 +++++
>>>>>>   > >   >  1 file changed, 5 insertions(+)
>>>>>>   > >   >
>>>>>>   > >   > diff --git 
>>>>>> a/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt 
>>>>>> b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
>>>>>>   > >   > new file mode 100644
>>>>>>   > >   > index 000000000000..18639853f18f
>>>>>>   > >   > --- /dev/null
>>>>>>   > >   > +++ b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
>>>>>>   > >   > @@ -0,0 +1,5 @@
>>>>>>   > >   > +# Board Name: msm-apq8016-db410c
>>>>>>   > >   > +# Failure Rate: 100
>>>>>>   > >
>>>>>>   > > Is failure rate is 100%, isn't it a fail than?
>>>>>>   > > (I know we have other cases with Failure Rate: 100, maybe we 
>>>>>> should fix them as well)
>>>>>>   > >
>>>>>>   >
>>>>>>   > Maybe I misunderstood the meaning of "Failure rate" for a flake.
>>>>>>   >
>>>>>>   > I interpreted this as this test being flaky 100% of the time :)
>>>>>>
>>>>>> Ah right, I see, inside deqp-runner (that auto-retries).
>>>>>>
>>>>>> I'd like to hear Vignesh's opinion on this.
>>>>>>
>>>>>> (In any case, we probably should document this better)
>>>>
>>>> deqp-runner reports new (not present in flakes file) or known 
>>>> (present in flakes file) flakes
>>>>
>>>> 2024-12-11 07:25:44.709666: Some new flakes found:
>>>> 2024-12-11 07:25:44.709676:   kms_lease@page-flip-implicit-plane
>>>>
>>>> 2024-12-11 13:15:16.482890: Some known flakes found:
>>>> 2024-12-11 13:15:16.482898: 
>>>> kms_async_flips@async-flip-with-page-flip-events-atomic
>>>>
>>>> we add it to flakes file if deqp runner reports new flakes. Another 
>>>> case where we update flake tests is when a test passes in one run 
>>>> but fails in another, but deqp-runner does not report it as flake.
>>>>
>>>> Regards,
>>>> Vignesh
>>>>
>>>
>>> The confusion here i guess is about what to mention as a "Failure rate"
>>>
>>> Failure rate means how many times it fails (like normally) ? In that 
>>> case 100% which I used is wrong and I used 33% instead for which I 
>>> have pushed v2.
>>
>> Yes, 33% is correct and please remove this test from fails.txt
>>
>> Regards,
>> Vignesh
>>
> 
> Ack, let me remove this test from fails and keep it only in flakes.

Can you remove it from the fails without adding it to the flakes, and 
rerun the pipeline a few times to see if deqp-runner reports it as a flake?

Thanks.

Regards,
Vignesh

> 
> Thanks
> 
> Abhinav
>>>
>>>>>>
>>>>>> Regards,
>>>>>> Helen
>>>>>>
>>>>>
>>>>> Can you let me know which way we need to go?
>>>>>
>>>>> Just in case I did post a v2 fixing this, 
>>>>> https://patchwork.freedesktop.org/patch/627276/
>>>>>
>>>>> If thats the way to go, can you pls take a look?
>>>>>
>>>>> Thanks
>>>>>
>>>>> Abhinav
>>>>>>   >
>>>>>>   > Out of the 3 runs of the test, it passed 2/3 times and failed 
>>>>>> 1/3.
>>>>>>   >
>>>>>>   > So its fail % actually is 33.33% in that case.
>>>>>>   >
>>>>>>   > I think I saw a Failure rate of 100% on 
>>>>>> msm-sm8350-hdk-flakes.txt and
>>>>>>   > mistook that as the rate at which flakes are seen.
>>>>>>   >
>>>>>>   > Let me fix this up as 33%
>>>>>>   >
>>>>>>   > > Regards,
>>>>>>   > > Helen
>>>>>>   > >
>>>>>>   > >   > +# IGT Version: 1.28-ga73311079
>>>>>>   > >   > +# Linux Version: 6.12.0-rc2
>>>>>>   > >   > +kms_cursor_legacy@torture-bo
>>>>>>   > >   >
>>>>>>   > >   > ---
>>>>>>   > >   > base-commit: 798bb342e0416d846cf67f4725a3428f39bfb96b
>>>>>>   > >   > change-id: 20241204-cursor_tor_skip-9d128dd62c4f
>>>>>>   > >   >
>>>>>>   > >   > Best regards,
>>>>>>   > >   > --
>>>>>>   > >   > Abhinav Kumar quic_abhinavk@quicinc.com>
>>>>>>   > >   >
>>>>>>   > >   >
>>>>>>   > >
>>>>>>   >
>>>>>>