[PATCH] drm/ci: add kms_cursor_legacy@torture-bo to apq8016 flakes

[PATCH] drm/ci: add kms_cursor_legacy@torture-bo to apq8016 flakes

[Abhinav Kumar]

From the jobs [1] and [2] of pipeline [3], its clear that
kms_cursor_legacy@torture-bo is most certainly a flake and
not a fail for apq8016. Mark the test accordingly to match the results.

[1] : https://gitlab.freedesktop.org/drm/msm/-/jobs/67676481
[2] : https://gitlab.freedesktop.org/drm/msm/-/jobs/67677430
[3]: https://gitlab.freedesktop.org/drm/msm/-/pipelines/1322770

Signed-off-by: Abhinav Kumar <quic_abhinavk@quicinc.com>
---
 drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
new file mode 100644
index 000000000000..18639853f18f
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
@@ -0,0 +1,5 @@
+# Board Name: msm-apq8016-db410c
+# Failure Rate: 100
+# IGT Version: 1.28-ga73311079
+# Linux Version: 6.12.0-rc2
+kms_cursor_legacy@torture-bo

---
base-commit: 798bb342e0416d846cf67f4725a3428f39bfb96b
change-id: 20241204-cursor_tor_skip-9d128dd62c4f

Best regards,
-- 
Abhinav Kumar <quic_abhinavk@quicinc.com>

Re: [PATCH] drm/ci: add kms_cursor_legacy@torture-bo to apq8016 flakes

[Helen Mae Koike Fornazier]

Hi Abhinav,

Thanks for your patch.



---- On Wed, 04 Dec 2024 15:55:17 -0300 Abhinav Kumar  wrote ---

 > From the jobs [1] and [2] of pipeline [3], its clear that 
 > kms_cursor_legacy@torture-bo is most certainly a flake and 
 > not a fail for apq8016. Mark the test accordingly to match the results. 
 >  
 > [1] : https://gitlab.freedesktop.org/drm/msm/-/jobs/67676481 
 > [2] : https://gitlab.freedesktop.org/drm/msm/-/jobs/67677430 
 > [3]: https://gitlab.freedesktop.org/drm/msm/-/pipelines/1322770 
 >  
 > Signed-off-by: Abhinav Kumar quic_abhinavk@quicinc.com> 
 > --- 
 >  drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt | 5 +++++ 
 >  1 file changed, 5 insertions(+) 
 >  
 > diff --git a/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt 
 > new file mode 100644 
 > index 000000000000..18639853f18f 
 > --- /dev/null 
 > +++ b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt 
 > @@ -0,0 +1,5 @@ 
 > +# Board Name: msm-apq8016-db410c 
 > +# Failure Rate: 100 

Is failure rate is 100%, isn't it a fail than?
(I know we have other cases with Failure Rate: 100, maybe we should fix them as well)

Regards,
Helen

 > +# IGT Version: 1.28-ga73311079 
 > +# Linux Version: 6.12.0-rc2 
 > +kms_cursor_legacy@torture-bo 
 >  
 > --- 
 > base-commit: 798bb342e0416d846cf67f4725a3428f39bfb96b 
 > change-id: 20241204-cursor_tor_skip-9d128dd62c4f 
 >  
 > Best regards, 
 > -- 
 > Abhinav Kumar quic_abhinavk@quicinc.com> 
 >  
 > 

Re: [PATCH] drm/ci: add kms_cursor_legacy@torture-bo to apq8016 flakes

[Abhinav Kumar]

Hi Helen

On 12/4/2024 11:14 AM, Helen Mae Koike Fornazier wrote:
> Hi Abhinav,
> 
> Thanks for your patch.
> 
> 
> 
> ---- On Wed, 04 Dec 2024 15:55:17 -0300 Abhinav Kumar  wrote ---
> 
>   > From the jobs [1] and [2] of pipeline [3], its clear that
>   > kms_cursor_legacy@torture-bo is most certainly a flake and
>   > not a fail for apq8016. Mark the test accordingly to match the results.
>   >
>   > [1] : https://gitlab.freedesktop.org/drm/msm/-/jobs/67676481
>   > [2] : https://gitlab.freedesktop.org/drm/msm/-/jobs/67677430
>   > [3]: https://gitlab.freedesktop.org/drm/msm/-/pipelines/1322770
>   >
>   > Signed-off-by: Abhinav Kumar quic_abhinavk@quicinc.com>
>   > ---
>   >  drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt | 5 +++++
>   >  1 file changed, 5 insertions(+)
>   >
>   > diff --git a/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
>   > new file mode 100644
>   > index 000000000000..18639853f18f
>   > --- /dev/null
>   > +++ b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
>   > @@ -0,0 +1,5 @@
>   > +# Board Name: msm-apq8016-db410c
>   > +# Failure Rate: 100
> 
> Is failure rate is 100%, isn't it a fail than?
> (I know we have other cases with Failure Rate: 100, maybe we should fix them as well)
> 

Maybe I misunderstood the meaning of "Failure rate" for a flake.

I interpreted this as this test being flaky 100% of the time :)

Out of the 3 runs of the test, it passed 2/3 times and failed 1/3.

So its fail % actually is 33.33% in that case.

I think I saw a Failure rate of 100% on msm-sm8350-hdk-flakes.txt and 
mistook that as the rate at which flakes are seen.

Let me fix this up as 33%

> Regards,
> Helen
> 
>   > +# IGT Version: 1.28-ga73311079
>   > +# Linux Version: 6.12.0-rc2
>   > +kms_cursor_legacy@torture-bo
>   >
>   > ---
>   > base-commit: 798bb342e0416d846cf67f4725a3428f39bfb96b
>   > change-id: 20241204-cursor_tor_skip-9d128dd62c4f
>   >
>   > Best regards,
>   > --
>   > Abhinav Kumar quic_abhinavk@quicinc.com>
>   >
>   >
> 

Re: [PATCH] drm/ci: add kms_cursor_legacy@torture-bo to apq8016 flakes

[Helen Mae Koike Fornazier]

---- On Wed, 04 Dec 2024 16:21:26 -0300 Abhinav Kumar  wrote ---

 > Hi Helen 
 >  
 > On 12/4/2024 11:14 AM, Helen Mae Koike Fornazier wrote: 
 > > Hi Abhinav, 
 > > 
 > > Thanks for your patch. 
 > > 
 > > 
 > > 
 > > ---- On Wed, 04 Dec 2024 15:55:17 -0300 Abhinav Kumar  wrote --- 
 > > 
 > >   > From the jobs [1] and [2] of pipeline [3], its clear that 
 > >   > kms_cursor_legacy@torture-bo is most certainly a flake and 
 > >   > not a fail for apq8016. Mark the test accordingly to match the results. 
 > >   > 
 > >   > [1] : https://gitlab.freedesktop.org/drm/msm/-/jobs/67676481 
 > >   > [2] : https://gitlab.freedesktop.org/drm/msm/-/jobs/67677430 
 > >   > [3]: https://gitlab.freedesktop.org/drm/msm/-/pipelines/1322770 
 > >   > 
 > >   > Signed-off-by: Abhinav Kumar quic_abhinavk@quicinc.com> 
 > >   > --- 
 > >   >  drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt | 5 +++++ 
 > >   >  1 file changed, 5 insertions(+) 
 > >   > 
 > >   > diff --git a/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt 
 > >   > new file mode 100644 
 > >   > index 000000000000..18639853f18f 
 > >   > --- /dev/null 
 > >   > +++ b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt 
 > >   > @@ -0,0 +1,5 @@ 
 > >   > +# Board Name: msm-apq8016-db410c 
 > >   > +# Failure Rate: 100 
 > > 
 > > Is failure rate is 100%, isn't it a fail than? 
 > > (I know we have other cases with Failure Rate: 100, maybe we should fix them as well) 
 > > 
 >  
 > Maybe I misunderstood the meaning of "Failure rate" for a flake. 
 >  
 > I interpreted this as this test being flaky 100% of the time :) 

Ah right, I see, inside deqp-runner (that auto-retries).

I'd like to hear Vignesh's opinion on this.

(In any case, we probably should document this better)

Regards,
Helen

 >  
 > Out of the 3 runs of the test, it passed 2/3 times and failed 1/3. 
 >  
 > So its fail % actually is 33.33% in that case. 
 >  
 > I think I saw a Failure rate of 100% on msm-sm8350-hdk-flakes.txt and 
 > mistook that as the rate at which flakes are seen. 
 >  
 > Let me fix this up as 33% 
 >  
 > > Regards, 
 > > Helen 
 > > 
 > >   > +# IGT Version: 1.28-ga73311079 
 > >   > +# Linux Version: 6.12.0-rc2 
 > >   > +kms_cursor_legacy@torture-bo 
 > >   > 
 > >   > --- 
 > >   > base-commit: 798bb342e0416d846cf67f4725a3428f39bfb96b 
 > >   > change-id: 20241204-cursor_tor_skip-9d128dd62c4f 
 > >   > 
 > >   > Best regards, 
 > >   > -- 
 > >   > Abhinav Kumar quic_abhinavk@quicinc.com> 
 > >   > 
 > >   > 
 > > 
 > 

Re: [PATCH] drm/ci: add kms_cursor_legacy@torture-bo to apq8016 flakes

[Abhinav Kumar]

Hi Helen / Vignesh

On 12/4/2024 12:33 PM, Helen Mae Koike Fornazier wrote:
> 
> 
> 
> 
> ---- On Wed, 04 Dec 2024 16:21:26 -0300 Abhinav Kumar  wrote ---
> 
>   > Hi Helen
>   >
>   > On 12/4/2024 11:14 AM, Helen Mae Koike Fornazier wrote:
>   > > Hi Abhinav,
>   > >
>   > > Thanks for your patch.
>   > >
>   > >
>   > >
>   > > ---- On Wed, 04 Dec 2024 15:55:17 -0300 Abhinav Kumar  wrote ---
>   > >
>   > >   > From the jobs [1] and [2] of pipeline [3], its clear that
>   > >   > kms_cursor_legacy@torture-bo is most certainly a flake and
>   > >   > not a fail for apq8016. Mark the test accordingly to match the results.
>   > >   >
>   > >   > [1] : https://gitlab.freedesktop.org/drm/msm/-/jobs/67676481
>   > >   > [2] : https://gitlab.freedesktop.org/drm/msm/-/jobs/67677430
>   > >   > [3]: https://gitlab.freedesktop.org/drm/msm/-/pipelines/1322770
>   > >   >
>   > >   > Signed-off-by: Abhinav Kumar quic_abhinavk@quicinc.com>
>   > >   > ---
>   > >   >  drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt | 5 +++++
>   > >   >  1 file changed, 5 insertions(+)
>   > >   >
>   > >   > diff --git a/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
>   > >   > new file mode 100644
>   > >   > index 000000000000..18639853f18f
>   > >   > --- /dev/null
>   > >   > +++ b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
>   > >   > @@ -0,0 +1,5 @@
>   > >   > +# Board Name: msm-apq8016-db410c
>   > >   > +# Failure Rate: 100
>   > >
>   > > Is failure rate is 100%, isn't it a fail than?
>   > > (I know we have other cases with Failure Rate: 100, maybe we should fix them as well)
>   > >
>   >
>   > Maybe I misunderstood the meaning of "Failure rate" for a flake.
>   >
>   > I interpreted this as this test being flaky 100% of the time :)
> 
> Ah right, I see, inside deqp-runner (that auto-retries).
> 
> I'd like to hear Vignesh's opinion on this.
> 
> (In any case, we probably should document this better)
> 
> Regards,
> Helen
> 

Can you let me know which way we need to go?

Just in case I did post a v2 fixing this, 
https://patchwork.freedesktop.org/patch/627276/

If thats the way to go, can you pls take a look?

Thanks

Abhinav
>   >
>   > Out of the 3 runs of the test, it passed 2/3 times and failed 1/3.
>   >
>   > So its fail % actually is 33.33% in that case.
>   >
>   > I think I saw a Failure rate of 100% on msm-sm8350-hdk-flakes.txt and
>   > mistook that as the rate at which flakes are seen.
>   >
>   > Let me fix this up as 33%
>   >
>   > > Regards,
>   > > Helen
>   > >
>   > >   > +# IGT Version: 1.28-ga73311079
>   > >   > +# Linux Version: 6.12.0-rc2
>   > >   > +kms_cursor_legacy@torture-bo
>   > >   >
>   > >   > ---
>   > >   > base-commit: 798bb342e0416d846cf67f4725a3428f39bfb96b
>   > >   > change-id: 20241204-cursor_tor_skip-9d128dd62c4f
>   > >   >
>   > >   > Best regards,
>   > >   > --
>   > >   > Abhinav Kumar quic_abhinavk@quicinc.com>
>   > >   >
>   > >   >
>   > >
>   >
> 

Re: [PATCH] drm/ci: add kms_cursor_legacy@torture-bo to apq8016 flakes

[Vignesh Raman]

Hi Abhinav / Helen,

On 12/12/24 01:48, Abhinav Kumar wrote:
> Hi Helen / Vignesh
> 
> On 12/4/2024 12:33 PM, Helen Mae Koike Fornazier wrote:
>>
>>
>>
>>
>> ---- On Wed, 04 Dec 2024 16:21:26 -0300 Abhinav Kumar  wrote ---
>>
>>   > Hi Helen
>>   >
>>   > On 12/4/2024 11:14 AM, Helen Mae Koike Fornazier wrote:
>>   > > Hi Abhinav,
>>   > >
>>   > > Thanks for your patch.
>>   > >
>>   > >
>>   > >
>>   > > ---- On Wed, 04 Dec 2024 15:55:17 -0300 Abhinav Kumar  wrote ---
>>   > >
>>   > >   > From the jobs [1] and [2] of pipeline [3], its clear that
>>   > >   > kms_cursor_legacy@torture-bo is most certainly a flake and
>>   > >   > not a fail for apq8016. Mark the test accordingly to match 
>> the results.
>>   > >   >
>>   > >   > [1] : https://gitlab.freedesktop.org/drm/msm/-/jobs/67676481

The test passes - kms_cursor_legacy@torture-bo,UnexpectedImprovement(Pass)

>>   > >   > [2] : https://gitlab.freedesktop.org/drm/msm/-/jobs/67677430

There are no test failures

>>   > >   > [3]: https://gitlab.freedesktop.org/drm/msm/-/pipelines/1322770

The job is same as 2

In this case, the test passes and deqp-runner does not report it as 
flake. So we only need to remove it from fails file.


>>   > >   >
>>   > >   > Signed-off-by: Abhinav Kumar quic_abhinavk@quicinc.com>
>>   > >   > ---
>>   > >   >  drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt | 5 +++++
>>   > >   >  1 file changed, 5 insertions(+)
>>   > >   >
>>   > >   > diff --git 
>> a/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt 
>> b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
>>   > >   > new file mode 100644
>>   > >   > index 000000000000..18639853f18f
>>   > >   > --- /dev/null
>>   > >   > +++ b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
>>   > >   > @@ -0,0 +1,5 @@
>>   > >   > +# Board Name: msm-apq8016-db410c
>>   > >   > +# Failure Rate: 100
>>   > >
>>   > > Is failure rate is 100%, isn't it a fail than?
>>   > > (I know we have other cases with Failure Rate: 100, maybe we 
>> should fix them as well)
>>   > >
>>   >
>>   > Maybe I misunderstood the meaning of "Failure rate" for a flake.
>>   >
>>   > I interpreted this as this test being flaky 100% of the time :)
>>
>> Ah right, I see, inside deqp-runner (that auto-retries).
>>
>> I'd like to hear Vignesh's opinion on this.
>>
>> (In any case, we probably should document this better)

deqp-runner reports new (not present in flakes file) or known (present 
in flakes file) flakes

2024-12-11 07:25:44.709666: Some new flakes found:
2024-12-11 07:25:44.709676:   kms_lease@page-flip-implicit-plane

2024-12-11 13:15:16.482890: Some known flakes found:
2024-12-11 13:15:16.482898: 
kms_async_flips@async-flip-with-page-flip-events-atomic

we add it to flakes file if deqp runner reports new flakes. Another case 
where we update flake tests is when a test passes in one run but fails 
in another, but deqp-runner does not report it as flake.

Regards,
Vignesh

>>
>> Regards,
>> Helen
>>
> 
> Can you let me know which way we need to go?
> 
> Just in case I did post a v2 fixing this, 
> https://patchwork.freedesktop.org/patch/627276/
> 
> If thats the way to go, can you pls take a look?
> 
> Thanks
> 
> Abhinav
>>   >
>>   > Out of the 3 runs of the test, it passed 2/3 times and failed 1/3.
>>   >
>>   > So its fail % actually is 33.33% in that case.
>>   >
>>   > I think I saw a Failure rate of 100% on msm-sm8350-hdk-flakes.txt and
>>   > mistook that as the rate at which flakes are seen.
>>   >
>>   > Let me fix this up as 33%
>>   >
>>   > > Regards,
>>   > > Helen
>>   > >
>>   > >   > +# IGT Version: 1.28-ga73311079
>>   > >   > +# Linux Version: 6.12.0-rc2
>>   > >   > +kms_cursor_legacy@torture-bo
>>   > >   >
>>   > >   > ---
>>   > >   > base-commit: 798bb342e0416d846cf67f4725a3428f39bfb96b
>>   > >   > change-id: 20241204-cursor_tor_skip-9d128dd62c4f
>>   > >   >
>>   > >   > Best regards,
>>   > >   > --
>>   > >   > Abhinav Kumar quic_abhinavk@quicinc.com>
>>   > >   >
>>   > >   >
>>   > >
>>   >
>>

Re: [PATCH] drm/ci: add kms_cursor_legacy@torture-bo to apq8016 flakes

[Abhinav Kumar]

Hi Vignesh

On 12/11/2024 9:10 PM, Vignesh Raman wrote:
> Hi Abhinav / Helen,
> 
> On 12/12/24 01:48, Abhinav Kumar wrote:
>> Hi Helen / Vignesh
>>
>> On 12/4/2024 12:33 PM, Helen Mae Koike Fornazier wrote:
>>>
>>>
>>>
>>>
>>> ---- On Wed, 04 Dec 2024 16:21:26 -0300 Abhinav Kumar  wrote ---
>>>
>>>   > Hi Helen
>>>   >
>>>   > On 12/4/2024 11:14 AM, Helen Mae Koike Fornazier wrote:
>>>   > > Hi Abhinav,
>>>   > >
>>>   > > Thanks for your patch.
>>>   > >
>>>   > >
>>>   > >
>>>   > > ---- On Wed, 04 Dec 2024 15:55:17 -0300 Abhinav Kumar  wrote ---
>>>   > >
>>>   > >   > From the jobs [1] and [2] of pipeline [3], its clear that
>>>   > >   > kms_cursor_legacy@torture-bo is most certainly a flake and
>>>   > >   > not a fail for apq8016. Mark the test accordingly to match 
>>> the results.
>>>   > >   >
>>>   > >   > [1] : https://gitlab.freedesktop.org/drm/msm/-/jobs/67676481
> 
> The test passes - kms_cursor_legacy@torture-bo,UnexpectedImprovement(Pass)
> 

Yes, thats the problem

https://gitlab.freedesktop.org/drm/msm/-/jobs/67676481/viewer#L2696

24-12-04 03:51:55 R SERIAL> [  179.241309] [IGT] kms_cursor_legacy: 
finished subtest all-pipes, SUCCESS
24-12-04 03:51:55 R SERIAL> [  179.241812] [IGT] kms_cursor_legacy: 
finished subtest torture-bo, SUCCESS

Here it passes whereas it was marked a failure. Hence pipeline fails.

>>>   > >   > [2] : https://gitlab.freedesktop.org/drm/msm/-/jobs/67677430
> 
> There are no test failures
> 

No, thats not true

https://gitlab.freedesktop.org/drm/msm/-/jobs/67677430/viewer#L2694

24-12-04 04:18:38 R SERIAL> [  170.379649] Console: switching to colour 
dummy device 80x25
24-12-04 04:18:38 R SERIAL> [  170.379938] [IGT] kms_cursor_legacy: 
executing
24-12-04 04:18:38 R SERIAL> [  170.393868] [IGT] kms_cursor_legacy: 
starting subtest torture-bo
24-12-04 04:18:38 R SERIAL> [  170.394186] [IGT] kms_cursor_legacy: 
starting dynamic subtest pipe-A
24-12-04 04:18:38 R SERIAL> [  170.661749] [IGT] kms_cursor_legacy: 
finished subtest pipe-A, FAIL
24-12-04 04:18:38 R SERIAL> [  170.662060] [IGT] kms_cursor_legacy: 
starting dynamic subtest all-pipes
24-12-04 04:18:38 R SERIAL> [  170.713237] [IGT] kms_cursor_legacy: 
finished subtest all-pipes, FAIL
24-12-04 04:18:38 R SERIAL> [  170.713513] [IGT] kms_cursor_legacy: 
finished subtest torture-bo, FAIL
24-12-04 04:18:38 R SERIAL> [  170.721263] [IGT] kms_cursor_legacy: 
exiting, ret=98
24-12-04 04:18:38 R SERIAL> [  170.737857] Console: switching to colour 
frame buffer device 128x48

Please check these logs, the torture-bo test-case did fail. The pipeline 
was marked pass because it was an expected fail.

So we have two pipelines, where one failed and the other passed. So 
thats a flake for me.

>>>   > >   > [3]: 
>>> https://gitlab.freedesktop.org/drm/msm/-/pipelines/1322770
> 
> The job is same as 2
> 
> In this case, the test passes and deqp-runner does not report it as 
> flake. So we only need to remove it from fails file.
> 

No, like I mentioned above we have a pass and a fail.

> 
>>>   > >   >
>>>   > >   > Signed-off-by: Abhinav Kumar quic_abhinavk@quicinc.com>
>>>   > >   > ---
>>>   > >   >  drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt | 5 +++++
>>>   > >   >  1 file changed, 5 insertions(+)
>>>   > >   >
>>>   > >   > diff --git 
>>> a/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt 
>>> b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
>>>   > >   > new file mode 100644
>>>   > >   > index 000000000000..18639853f18f
>>>   > >   > --- /dev/null
>>>   > >   > +++ b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
>>>   > >   > @@ -0,0 +1,5 @@
>>>   > >   > +# Board Name: msm-apq8016-db410c
>>>   > >   > +# Failure Rate: 100
>>>   > >
>>>   > > Is failure rate is 100%, isn't it a fail than?
>>>   > > (I know we have other cases with Failure Rate: 100, maybe we 
>>> should fix them as well)
>>>   > >
>>>   >
>>>   > Maybe I misunderstood the meaning of "Failure rate" for a flake.
>>>   >
>>>   > I interpreted this as this test being flaky 100% of the time :)
>>>
>>> Ah right, I see, inside deqp-runner (that auto-retries).
>>>
>>> I'd like to hear Vignesh's opinion on this.
>>>
>>> (In any case, we probably should document this better)
> 
> deqp-runner reports new (not present in flakes file) or known (present 
> in flakes file) flakes
> 
> 2024-12-11 07:25:44.709666: Some new flakes found:
> 2024-12-11 07:25:44.709676:   kms_lease@page-flip-implicit-plane
> 
> 2024-12-11 13:15:16.482890: Some known flakes found:
> 2024-12-11 13:15:16.482898: 
> kms_async_flips@async-flip-with-page-flip-events-atomic
> 
> we add it to flakes file if deqp runner reports new flakes. Another case 
> where we update flake tests is when a test passes in one run but fails 
> in another, but deqp-runner does not report it as flake.
> 
> Regards,
> Vignesh
> 

The confusion here i guess is about what to mention as a "Failure rate"

Failure rate means how many times it fails (like normally) ? In that 
case 100% which I used is wrong and I used 33% instead for which I have 
pushed v2.

>>>
>>> Regards,
>>> Helen
>>>
>>
>> Can you let me know which way we need to go?
>>
>> Just in case I did post a v2 fixing this, 
>> https://patchwork.freedesktop.org/patch/627276/
>>
>> If thats the way to go, can you pls take a look?
>>
>> Thanks
>>
>> Abhinav
>>>   >
>>>   > Out of the 3 runs of the test, it passed 2/3 times and failed 1/3.
>>>   >
>>>   > So its fail % actually is 33.33% in that case.
>>>   >
>>>   > I think I saw a Failure rate of 100% on msm-sm8350-hdk-flakes.txt 
>>> and
>>>   > mistook that as the rate at which flakes are seen.
>>>   >
>>>   > Let me fix this up as 33%
>>>   >
>>>   > > Regards,
>>>   > > Helen
>>>   > >
>>>   > >   > +# IGT Version: 1.28-ga73311079
>>>   > >   > +# Linux Version: 6.12.0-rc2
>>>   > >   > +kms_cursor_legacy@torture-bo
>>>   > >   >
>>>   > >   > ---
>>>   > >   > base-commit: 798bb342e0416d846cf67f4725a3428f39bfb96b
>>>   > >   > change-id: 20241204-cursor_tor_skip-9d128dd62c4f
>>>   > >   >
>>>   > >   > Best regards,
>>>   > >   > --
>>>   > >   > Abhinav Kumar quic_abhinavk@quicinc.com>
>>>   > >   >
>>>   > >   >
>>>   > >
>>>   >
>>>

Re: [PATCH] drm/ci: add kms_cursor_legacy@torture-bo to apq8016 flakes

[Vignesh Raman]

Hi Abhinav,

On 14/12/24 01:09, Abhinav Kumar wrote:
> Hi Vignesh
> 
> On 12/11/2024 9:10 PM, Vignesh Raman wrote:
>> Hi Abhinav / Helen,
>>
>> On 12/12/24 01:48, Abhinav Kumar wrote:
>>> Hi Helen / Vignesh
>>>
>>> On 12/4/2024 12:33 PM, Helen Mae Koike Fornazier wrote:
>>>>
>>>>
>>>>
>>>>
>>>> ---- On Wed, 04 Dec 2024 16:21:26 -0300 Abhinav Kumar  wrote ---
>>>>
>>>>   > Hi Helen
>>>>   >
>>>>   > On 12/4/2024 11:14 AM, Helen Mae Koike Fornazier wrote:
>>>>   > > Hi Abhinav,
>>>>   > >
>>>>   > > Thanks for your patch.
>>>>   > >
>>>>   > >
>>>>   > >
>>>>   > > ---- On Wed, 04 Dec 2024 15:55:17 -0300 Abhinav Kumar  wrote ---
>>>>   > >
>>>>   > >   > From the jobs [1] and [2] of pipeline [3], its clear that
>>>>   > >   > kms_cursor_legacy@torture-bo is most certainly a flake and
>>>>   > >   > not a fail for apq8016. Mark the test accordingly to match 
>>>> the results.
>>>>   > >   >
>>>>   > >   > [1] : https://gitlab.freedesktop.org/drm/msm/-/jobs/67676481
>>
>> The test passes - 
>> kms_cursor_legacy@torture-bo,UnexpectedImprovement(Pass)
>>
> 
> Yes, thats the problem
> 
> https://gitlab.freedesktop.org/drm/msm/-/jobs/67676481/viewer#L2696
> 
> 24-12-04 03:51:55 R SERIAL> [  179.241309] [IGT] kms_cursor_legacy: 
> finished subtest all-pipes, SUCCESS
> 24-12-04 03:51:55 R SERIAL> [  179.241812] [IGT] kms_cursor_legacy: 
> finished subtest torture-bo, SUCCESS
> 
> Here it passes whereas it was marked a failure. Hence pipeline fails.

Yes it fails due to,

Unexpected results:
  kms_cursor_legacy@torture-bo,UnexpectedImprovement(Pass)

In this case, we need to remove this test from fails.txt

> 
>>>>   > >   > [2] : https://gitlab.freedesktop.org/drm/msm/-/jobs/67677430
>>
>> There are no test failures
>>
> 
> No, thats not true
> 
> https://gitlab.freedesktop.org/drm/msm/-/jobs/67677430/viewer#L2694
> 
> 24-12-04 04:18:38 R SERIAL> [  170.379649] Console: switching to colour 
> dummy device 80x25
> 24-12-04 04:18:38 R SERIAL> [  170.379938] [IGT] kms_cursor_legacy: 
> executing
> 24-12-04 04:18:38 R SERIAL> [  170.393868] [IGT] kms_cursor_legacy: 
> starting subtest torture-bo
> 24-12-04 04:18:38 R SERIAL> [  170.394186] [IGT] kms_cursor_legacy: 
> starting dynamic subtest pipe-A
> 24-12-04 04:18:38 R SERIAL> [  170.661749] [IGT] kms_cursor_legacy: 
> finished subtest pipe-A, FAIL
> 24-12-04 04:18:38 R SERIAL> [  170.662060] [IGT] kms_cursor_legacy: 
> starting dynamic subtest all-pipes
> 24-12-04 04:18:38 R SERIAL> [  170.713237] [IGT] kms_cursor_legacy: 
> finished subtest all-pipes, FAIL
> 24-12-04 04:18:38 R SERIAL> [  170.713513] [IGT] kms_cursor_legacy: 
> finished subtest torture-bo, FAIL
> 24-12-04 04:18:38 R SERIAL> [  170.721263] [IGT] kms_cursor_legacy: 
> exiting, ret=98
> 24-12-04 04:18:38 R SERIAL> [  170.737857] Console: switching to colour 
> frame buffer device 128x48
> 
> Please check these logs, the torture-bo test-case did fail. The pipeline 
> was marked pass because it was an expected fail.
> 
> So we have two pipelines, where one failed and the other passed. So 
> thats a flake for me.

Yes agree. So if we had removed the test from fails, deqp-runner would 
have reported this as flake.

deqp-runner runs the test and if it fails, it retries. If the test 
passes on retry, it is reported as a flake.

> 
>>>>   > >   > [3]: 
>>>> https://gitlab.freedesktop.org/drm/msm/-/pipelines/1322770
>>
>> The job is same as 2
>>
>> In this case, the test passes and deqp-runner does not report it as 
>> flake. So we only need to remove it from fails file.
>>
> 
> No, like I mentioned above we have a pass and a fail.
> 
>>
>>>>   > >   >
>>>>   > >   > Signed-off-by: Abhinav Kumar quic_abhinavk@quicinc.com>
>>>>   > >   > ---
>>>>   > >   >  drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt | 5 +++++
>>>>   > >   >  1 file changed, 5 insertions(+)
>>>>   > >   >
>>>>   > >   > diff --git 
>>>> a/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt 
>>>> b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
>>>>   > >   > new file mode 100644
>>>>   > >   > index 000000000000..18639853f18f
>>>>   > >   > --- /dev/null
>>>>   > >   > +++ b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
>>>>   > >   > @@ -0,0 +1,5 @@
>>>>   > >   > +# Board Name: msm-apq8016-db410c
>>>>   > >   > +# Failure Rate: 100
>>>>   > >
>>>>   > > Is failure rate is 100%, isn't it a fail than?
>>>>   > > (I know we have other cases with Failure Rate: 100, maybe we 
>>>> should fix them as well)
>>>>   > >
>>>>   >
>>>>   > Maybe I misunderstood the meaning of "Failure rate" for a flake.
>>>>   >
>>>>   > I interpreted this as this test being flaky 100% of the time :)
>>>>
>>>> Ah right, I see, inside deqp-runner (that auto-retries).
>>>>
>>>> I'd like to hear Vignesh's opinion on this.
>>>>
>>>> (In any case, we probably should document this better)
>>
>> deqp-runner reports new (not present in flakes file) or known (present 
>> in flakes file) flakes
>>
>> 2024-12-11 07:25:44.709666: Some new flakes found:
>> 2024-12-11 07:25:44.709676:   kms_lease@page-flip-implicit-plane
>>
>> 2024-12-11 13:15:16.482890: Some known flakes found:
>> 2024-12-11 13:15:16.482898: 
>> kms_async_flips@async-flip-with-page-flip-events-atomic
>>
>> we add it to flakes file if deqp runner reports new flakes. Another 
>> case where we update flake tests is when a test passes in one run but 
>> fails in another, but deqp-runner does not report it as flake.
>>
>> Regards,
>> Vignesh
>>
> 
> The confusion here i guess is about what to mention as a "Failure rate"
> 
> Failure rate means how many times it fails (like normally) ? In that 
> case 100% which I used is wrong and I used 33% instead for which I have 
> pushed v2.

Yes, 33% is correct and please remove this test from fails.txt

Regards,
Vignesh

> 
>>>>
>>>> Regards,
>>>> Helen
>>>>
>>>
>>> Can you let me know which way we need to go?
>>>
>>> Just in case I did post a v2 fixing this, 
>>> https://patchwork.freedesktop.org/patch/627276/
>>>
>>> If thats the way to go, can you pls take a look?
>>>
>>> Thanks
>>>
>>> Abhinav
>>>>   >
>>>>   > Out of the 3 runs of the test, it passed 2/3 times and failed 1/3.
>>>>   >
>>>>   > So its fail % actually is 33.33% in that case.
>>>>   >
>>>>   > I think I saw a Failure rate of 100% on 
>>>> msm-sm8350-hdk-flakes.txt and
>>>>   > mistook that as the rate at which flakes are seen.
>>>>   >
>>>>   > Let me fix this up as 33%
>>>>   >
>>>>   > > Regards,
>>>>   > > Helen
>>>>   > >
>>>>   > >   > +# IGT Version: 1.28-ga73311079
>>>>   > >   > +# Linux Version: 6.12.0-rc2
>>>>   > >   > +kms_cursor_legacy@torture-bo
>>>>   > >   >
>>>>   > >   > ---
>>>>   > >   > base-commit: 798bb342e0416d846cf67f4725a3428f39bfb96b
>>>>   > >   > change-id: 20241204-cursor_tor_skip-9d128dd62c4f
>>>>   > >   >
>>>>   > >   > Best regards,
>>>>   > >   > --
>>>>   > >   > Abhinav Kumar quic_abhinavk@quicinc.com>
>>>>   > >   >
>>>>   > >   >
>>>>   > >
>>>>   >
>>>>

Re: [PATCH] drm/ci: add kms_cursor_legacy@torture-bo to apq8016 flakes

[Abhinav Kumar]

On 12/15/2024 9:45 PM, Vignesh Raman wrote:
> Hi Abhinav,
> 
> On 14/12/24 01:09, Abhinav Kumar wrote:
>> Hi Vignesh
>>
>> On 12/11/2024 9:10 PM, Vignesh Raman wrote:
>>> Hi Abhinav / Helen,
>>>
>>> On 12/12/24 01:48, Abhinav Kumar wrote:
>>>> Hi Helen / Vignesh
>>>>
>>>> On 12/4/2024 12:33 PM, Helen Mae Koike Fornazier wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---- On Wed, 04 Dec 2024 16:21:26 -0300 Abhinav Kumar  wrote ---
>>>>>
>>>>>   > Hi Helen
>>>>>   >
>>>>>   > On 12/4/2024 11:14 AM, Helen Mae Koike Fornazier wrote:
>>>>>   > > Hi Abhinav,
>>>>>   > >
>>>>>   > > Thanks for your patch.
>>>>>   > >
>>>>>   > >
>>>>>   > >
>>>>>   > > ---- On Wed, 04 Dec 2024 15:55:17 -0300 Abhinav Kumar  wrote ---
>>>>>   > >
>>>>>   > >   > From the jobs [1] and [2] of pipeline [3], its clear that
>>>>>   > >   > kms_cursor_legacy@torture-bo is most certainly a flake and
>>>>>   > >   > not a fail for apq8016. Mark the test accordingly to 
>>>>> match the results.
>>>>>   > >   >
>>>>>   > >   > [1] : https://gitlab.freedesktop.org/drm/msm/-/jobs/67676481
>>>
>>> The test passes - 
>>> kms_cursor_legacy@torture-bo,UnexpectedImprovement(Pass)
>>>
>>
>> Yes, thats the problem
>>
>> https://gitlab.freedesktop.org/drm/msm/-/jobs/67676481/viewer#L2696
>>
>> 24-12-04 03:51:55 R SERIAL> [  179.241309] [IGT] kms_cursor_legacy: 
>> finished subtest all-pipes, SUCCESS
>> 24-12-04 03:51:55 R SERIAL> [  179.241812] [IGT] kms_cursor_legacy: 
>> finished subtest torture-bo, SUCCESS
>>
>> Here it passes whereas it was marked a failure. Hence pipeline fails.
> 
> Yes it fails due to,
> 
> Unexpected results:
>   kms_cursor_legacy@torture-bo,UnexpectedImprovement(Pass)
> 
> In this case, we need to remove this test from fails.txt
> 
>>
>>>>>   > >   > [2] : https://gitlab.freedesktop.org/drm/msm/-/jobs/67677430
>>>
>>> There are no test failures
>>>
>>
>> No, thats not true
>>
>> https://gitlab.freedesktop.org/drm/msm/-/jobs/67677430/viewer#L2694
>>
>> 24-12-04 04:18:38 R SERIAL> [  170.379649] Console: switching to 
>> colour dummy device 80x25
>> 24-12-04 04:18:38 R SERIAL> [  170.379938] [IGT] kms_cursor_legacy: 
>> executing
>> 24-12-04 04:18:38 R SERIAL> [  170.393868] [IGT] kms_cursor_legacy: 
>> starting subtest torture-bo
>> 24-12-04 04:18:38 R SERIAL> [  170.394186] [IGT] kms_cursor_legacy: 
>> starting dynamic subtest pipe-A
>> 24-12-04 04:18:38 R SERIAL> [  170.661749] [IGT] kms_cursor_legacy: 
>> finished subtest pipe-A, FAIL
>> 24-12-04 04:18:38 R SERIAL> [  170.662060] [IGT] kms_cursor_legacy: 
>> starting dynamic subtest all-pipes
>> 24-12-04 04:18:38 R SERIAL> [  170.713237] [IGT] kms_cursor_legacy: 
>> finished subtest all-pipes, FAIL
>> 24-12-04 04:18:38 R SERIAL> [  170.713513] [IGT] kms_cursor_legacy: 
>> finished subtest torture-bo, FAIL
>> 24-12-04 04:18:38 R SERIAL> [  170.721263] [IGT] kms_cursor_legacy: 
>> exiting, ret=98
>> 24-12-04 04:18:38 R SERIAL> [  170.737857] Console: switching to 
>> colour frame buffer device 128x48
>>
>> Please check these logs, the torture-bo test-case did fail. The 
>> pipeline was marked pass because it was an expected fail.
>>
>> So we have two pipelines, where one failed and the other passed. So 
>> thats a flake for me.
> 
> Yes agree. So if we had removed the test from fails, deqp-runner would 
> have reported this as flake.
> 
> deqp-runner runs the test and if it fails, it retries. If the test 
> passes on retry, it is reported as a flake.
> 
>>
>>>>>   > >   > [3]: 
>>>>> https://gitlab.freedesktop.org/drm/msm/-/pipelines/1322770
>>>
>>> The job is same as 2
>>>
>>> In this case, the test passes and deqp-runner does not report it as 
>>> flake. So we only need to remove it from fails file.
>>>
>>
>> No, like I mentioned above we have a pass and a fail.
>>
>>>
>>>>>   > >   >
>>>>>   > >   > Signed-off-by: Abhinav Kumar quic_abhinavk@quicinc.com>
>>>>>   > >   > ---
>>>>>   > >   >  drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt | 5 +++++
>>>>>   > >   >  1 file changed, 5 insertions(+)
>>>>>   > >   >
>>>>>   > >   > diff --git 
>>>>> a/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt 
>>>>> b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
>>>>>   > >   > new file mode 100644
>>>>>   > >   > index 000000000000..18639853f18f
>>>>>   > >   > --- /dev/null
>>>>>   > >   > +++ b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
>>>>>   > >   > @@ -0,0 +1,5 @@
>>>>>   > >   > +# Board Name: msm-apq8016-db410c
>>>>>   > >   > +# Failure Rate: 100
>>>>>   > >
>>>>>   > > Is failure rate is 100%, isn't it a fail than?
>>>>>   > > (I know we have other cases with Failure Rate: 100, maybe we 
>>>>> should fix them as well)
>>>>>   > >
>>>>>   >
>>>>>   > Maybe I misunderstood the meaning of "Failure rate" for a flake.
>>>>>   >
>>>>>   > I interpreted this as this test being flaky 100% of the time :)
>>>>>
>>>>> Ah right, I see, inside deqp-runner (that auto-retries).
>>>>>
>>>>> I'd like to hear Vignesh's opinion on this.
>>>>>
>>>>> (In any case, we probably should document this better)
>>>
>>> deqp-runner reports new (not present in flakes file) or known 
>>> (present in flakes file) flakes
>>>
>>> 2024-12-11 07:25:44.709666: Some new flakes found:
>>> 2024-12-11 07:25:44.709676:   kms_lease@page-flip-implicit-plane
>>>
>>> 2024-12-11 13:15:16.482890: Some known flakes found:
>>> 2024-12-11 13:15:16.482898: 
>>> kms_async_flips@async-flip-with-page-flip-events-atomic
>>>
>>> we add it to flakes file if deqp runner reports new flakes. Another 
>>> case where we update flake tests is when a test passes in one run but 
>>> fails in another, but deqp-runner does not report it as flake.
>>>
>>> Regards,
>>> Vignesh
>>>
>>
>> The confusion here i guess is about what to mention as a "Failure rate"
>>
>> Failure rate means how many times it fails (like normally) ? In that 
>> case 100% which I used is wrong and I used 33% instead for which I 
>> have pushed v2.
> 
> Yes, 33% is correct and please remove this test from fails.txt
> 
> Regards,
> Vignesh
> 

Ack, let me remove this test from fails and keep it only in flakes.

Thanks

Abhinav
>>
>>>>>
>>>>> Regards,
>>>>> Helen
>>>>>
>>>>
>>>> Can you let me know which way we need to go?
>>>>
>>>> Just in case I did post a v2 fixing this, 
>>>> https://patchwork.freedesktop.org/patch/627276/
>>>>
>>>> If thats the way to go, can you pls take a look?
>>>>
>>>> Thanks
>>>>
>>>> Abhinav
>>>>>   >
>>>>>   > Out of the 3 runs of the test, it passed 2/3 times and failed 1/3.
>>>>>   >
>>>>>   > So its fail % actually is 33.33% in that case.
>>>>>   >
>>>>>   > I think I saw a Failure rate of 100% on 
>>>>> msm-sm8350-hdk-flakes.txt and
>>>>>   > mistook that as the rate at which flakes are seen.
>>>>>   >
>>>>>   > Let me fix this up as 33%
>>>>>   >
>>>>>   > > Regards,
>>>>>   > > Helen
>>>>>   > >
>>>>>   > >   > +# IGT Version: 1.28-ga73311079
>>>>>   > >   > +# Linux Version: 6.12.0-rc2
>>>>>   > >   > +kms_cursor_legacy@torture-bo
>>>>>   > >   >
>>>>>   > >   > ---
>>>>>   > >   > base-commit: 798bb342e0416d846cf67f4725a3428f39bfb96b
>>>>>   > >   > change-id: 20241204-cursor_tor_skip-9d128dd62c4f
>>>>>   > >   >
>>>>>   > >   > Best regards,
>>>>>   > >   > --
>>>>>   > >   > Abhinav Kumar quic_abhinavk@quicinc.com>
>>>>>   > >   >
>>>>>   > >   >
>>>>>   > >
>>>>>   >
>>>>>

Re: [PATCH] drm/ci: add kms_cursor_legacy@torture-bo to apq8016 flakes

[Vignesh Raman]

Hi Abhinav,

On 16/12/24 11:39, Abhinav Kumar wrote:
> 
> 
> On 12/15/2024 9:45 PM, Vignesh Raman wrote:
>> Hi Abhinav,
>>
>> On 14/12/24 01:09, Abhinav Kumar wrote:
>>> Hi Vignesh
>>>
>>> On 12/11/2024 9:10 PM, Vignesh Raman wrote:
>>>> Hi Abhinav / Helen,
>>>>
>>>> On 12/12/24 01:48, Abhinav Kumar wrote:
>>>>> Hi Helen / Vignesh
>>>>>
>>>>> On 12/4/2024 12:33 PM, Helen Mae Koike Fornazier wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---- On Wed, 04 Dec 2024 16:21:26 -0300 Abhinav Kumar  wrote ---
>>>>>>
>>>>>>   > Hi Helen
>>>>>>   >
>>>>>>   > On 12/4/2024 11:14 AM, Helen Mae Koike Fornazier wrote:
>>>>>>   > > Hi Abhinav,
>>>>>>   > >
>>>>>>   > > Thanks for your patch.
>>>>>>   > >
>>>>>>   > >
>>>>>>   > >
>>>>>>   > > ---- On Wed, 04 Dec 2024 15:55:17 -0300 Abhinav Kumar  wrote 
>>>>>> ---
>>>>>>   > >
>>>>>>   > >   > From the jobs [1] and [2] of pipeline [3], its clear that
>>>>>>   > >   > kms_cursor_legacy@torture-bo is most certainly a flake and
>>>>>>   > >   > not a fail for apq8016. Mark the test accordingly to 
>>>>>> match the results.
>>>>>>   > >   >
>>>>>>   > >   > [1] : 
>>>>>> https://gitlab.freedesktop.org/drm/msm/-/jobs/67676481
>>>>
>>>> The test passes - 
>>>> kms_cursor_legacy@torture-bo,UnexpectedImprovement(Pass)
>>>>
>>>
>>> Yes, thats the problem
>>>
>>> https://gitlab.freedesktop.org/drm/msm/-/jobs/67676481/viewer#L2696
>>>
>>> 24-12-04 03:51:55 R SERIAL> [  179.241309] [IGT] kms_cursor_legacy: 
>>> finished subtest all-pipes, SUCCESS
>>> 24-12-04 03:51:55 R SERIAL> [  179.241812] [IGT] kms_cursor_legacy: 
>>> finished subtest torture-bo, SUCCESS
>>>
>>> Here it passes whereas it was marked a failure. Hence pipeline fails.
>>
>> Yes it fails due to,
>>
>> Unexpected results:
>>   kms_cursor_legacy@torture-bo,UnexpectedImprovement(Pass)
>>
>> In this case, we need to remove this test from fails.txt
>>
>>>
>>>>>>   > >   > [2] : 
>>>>>> https://gitlab.freedesktop.org/drm/msm/-/jobs/67677430
>>>>
>>>> There are no test failures
>>>>
>>>
>>> No, thats not true
>>>
>>> https://gitlab.freedesktop.org/drm/msm/-/jobs/67677430/viewer#L2694
>>>
>>> 24-12-04 04:18:38 R SERIAL> [  170.379649] Console: switching to 
>>> colour dummy device 80x25
>>> 24-12-04 04:18:38 R SERIAL> [  170.379938] [IGT] kms_cursor_legacy: 
>>> executing
>>> 24-12-04 04:18:38 R SERIAL> [  170.393868] [IGT] kms_cursor_legacy: 
>>> starting subtest torture-bo
>>> 24-12-04 04:18:38 R SERIAL> [  170.394186] [IGT] kms_cursor_legacy: 
>>> starting dynamic subtest pipe-A
>>> 24-12-04 04:18:38 R SERIAL> [  170.661749] [IGT] kms_cursor_legacy: 
>>> finished subtest pipe-A, FAIL
>>> 24-12-04 04:18:38 R SERIAL> [  170.662060] [IGT] kms_cursor_legacy: 
>>> starting dynamic subtest all-pipes
>>> 24-12-04 04:18:38 R SERIAL> [  170.713237] [IGT] kms_cursor_legacy: 
>>> finished subtest all-pipes, FAIL
>>> 24-12-04 04:18:38 R SERIAL> [  170.713513] [IGT] kms_cursor_legacy: 
>>> finished subtest torture-bo, FAIL
>>> 24-12-04 04:18:38 R SERIAL> [  170.721263] [IGT] kms_cursor_legacy: 
>>> exiting, ret=98
>>> 24-12-04 04:18:38 R SERIAL> [  170.737857] Console: switching to 
>>> colour frame buffer device 128x48
>>>
>>> Please check these logs, the torture-bo test-case did fail. The 
>>> pipeline was marked pass because it was an expected fail.
>>>
>>> So we have two pipelines, where one failed and the other passed. So 
>>> thats a flake for me.
>>
>> Yes agree. So if we had removed the test from fails, deqp-runner would 
>> have reported this as flake.
>>
>> deqp-runner runs the test and if it fails, it retries. If the test 
>> passes on retry, it is reported as a flake.
>>
>>>
>>>>>>   > >   > [3]: 
>>>>>> https://gitlab.freedesktop.org/drm/msm/-/pipelines/1322770
>>>>
>>>> The job is same as 2
>>>>
>>>> In this case, the test passes and deqp-runner does not report it as 
>>>> flake. So we only need to remove it from fails file.
>>>>
>>>
>>> No, like I mentioned above we have a pass and a fail.
>>>
>>>>
>>>>>>   > >   >
>>>>>>   > >   > Signed-off-by: Abhinav Kumar quic_abhinavk@quicinc.com>
>>>>>>   > >   > ---
>>>>>>   > >   >  drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt | 5 +++++
>>>>>>   > >   >  1 file changed, 5 insertions(+)
>>>>>>   > >   >
>>>>>>   > >   > diff --git 
>>>>>> a/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt 
>>>>>> b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
>>>>>>   > >   > new file mode 100644
>>>>>>   > >   > index 000000000000..18639853f18f
>>>>>>   > >   > --- /dev/null
>>>>>>   > >   > +++ b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
>>>>>>   > >   > @@ -0,0 +1,5 @@
>>>>>>   > >   > +# Board Name: msm-apq8016-db410c
>>>>>>   > >   > +# Failure Rate: 100
>>>>>>   > >
>>>>>>   > > Is failure rate is 100%, isn't it a fail than?
>>>>>>   > > (I know we have other cases with Failure Rate: 100, maybe we 
>>>>>> should fix them as well)
>>>>>>   > >
>>>>>>   >
>>>>>>   > Maybe I misunderstood the meaning of "Failure rate" for a flake.
>>>>>>   >
>>>>>>   > I interpreted this as this test being flaky 100% of the time :)
>>>>>>
>>>>>> Ah right, I see, inside deqp-runner (that auto-retries).
>>>>>>
>>>>>> I'd like to hear Vignesh's opinion on this.
>>>>>>
>>>>>> (In any case, we probably should document this better)
>>>>
>>>> deqp-runner reports new (not present in flakes file) or known 
>>>> (present in flakes file) flakes
>>>>
>>>> 2024-12-11 07:25:44.709666: Some new flakes found:
>>>> 2024-12-11 07:25:44.709676:   kms_lease@page-flip-implicit-plane
>>>>
>>>> 2024-12-11 13:15:16.482890: Some known flakes found:
>>>> 2024-12-11 13:15:16.482898: 
>>>> kms_async_flips@async-flip-with-page-flip-events-atomic
>>>>
>>>> we add it to flakes file if deqp runner reports new flakes. Another 
>>>> case where we update flake tests is when a test passes in one run 
>>>> but fails in another, but deqp-runner does not report it as flake.
>>>>
>>>> Regards,
>>>> Vignesh
>>>>
>>>
>>> The confusion here i guess is about what to mention as a "Failure rate"
>>>
>>> Failure rate means how many times it fails (like normally) ? In that 
>>> case 100% which I used is wrong and I used 33% instead for which I 
>>> have pushed v2.
>>
>> Yes, 33% is correct and please remove this test from fails.txt
>>
>> Regards,
>> Vignesh
>>
> 
> Ack, let me remove this test from fails and keep it only in flakes.

Can you remove it from the fails without adding it to the flakes, and 
rerun the pipeline a few times to see if deqp-runner reports it as a flake?

Thanks.

Regards,
Vignesh

> 
> Thanks
> 
> Abhinav
>>>
>>>>>>
>>>>>> Regards,
>>>>>> Helen
>>>>>>
>>>>>
>>>>> Can you let me know which way we need to go?
>>>>>
>>>>> Just in case I did post a v2 fixing this, 
>>>>> https://patchwork.freedesktop.org/patch/627276/
>>>>>
>>>>> If thats the way to go, can you pls take a look?
>>>>>
>>>>> Thanks
>>>>>
>>>>> Abhinav
>>>>>>   >
>>>>>>   > Out of the 3 runs of the test, it passed 2/3 times and failed 
>>>>>> 1/3.
>>>>>>   >
>>>>>>   > So its fail % actually is 33.33% in that case.
>>>>>>   >
>>>>>>   > I think I saw a Failure rate of 100% on 
>>>>>> msm-sm8350-hdk-flakes.txt and
>>>>>>   > mistook that as the rate at which flakes are seen.
>>>>>>   >
>>>>>>   > Let me fix this up as 33%
>>>>>>   >
>>>>>>   > > Regards,
>>>>>>   > > Helen
>>>>>>   > >
>>>>>>   > >   > +# IGT Version: 1.28-ga73311079
>>>>>>   > >   > +# Linux Version: 6.12.0-rc2
>>>>>>   > >   > +kms_cursor_legacy@torture-bo
>>>>>>   > >   >
>>>>>>   > >   > ---
>>>>>>   > >   > base-commit: 798bb342e0416d846cf67f4725a3428f39bfb96b
>>>>>>   > >   > change-id: 20241204-cursor_tor_skip-9d128dd62c4f
>>>>>>   > >   >
>>>>>>   > >   > Best regards,
>>>>>>   > >   > --
>>>>>>   > >   > Abhinav Kumar quic_abhinavk@quicinc.com>
>>>>>>   > >   >
>>>>>>   > >   >
>>>>>>   > >
>>>>>>   >
>>>>>>