From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from bali.collaboradmins.com (bali.collaboradmins.com [148.251.105.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9FB9D158853; Wed, 18 Dec 2024 13:57:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.251.105.195 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734530248; cv=none; b=UcrUOeS3otVCE/fzH53Etk2xfmxbIbpHTTv7Qd3r0KPu5y+Gq3nQQ3GGJKVGBwNsC85exOZaxPrueOcFECAwBimtnvcyLD7yuJLXxCcjBh0qFPvsofinaq7+JtEWvpaZDxeQIcnC4R3eNAA67rcGe9g8zwZ02A7MrE95hNgyiDM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734530248; c=relaxed/simple; bh=Vc/ZM9fAvz09aWlUkt1tgjwmGq/TmY7xSsGfE0yV9pQ=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=qWYk6+G60Uq9bqS2aE0+M9e7dIQ4Jb8S1QibyxoQ2EOMu2yU66y+mtU1/b4SWwba/K2r5xbLP5vDGCHu/rQmbu6xx3qkNY0sufBXIq9RmqvatsKqhl4KFjbg1X6tWsASGOnRfQDBv41gUlAtEk/1X14nhrwlt8fkYSSIv5sHJN0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=collabora.com; spf=pass smtp.mailfrom=collabora.com; dkim=pass (2048-bit key) header.d=collabora.com header.i=@collabora.com header.b=X6q9nV5K; arc=none smtp.client-ip=148.251.105.195 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=collabora.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=collabora.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=collabora.com header.i=@collabora.com header.b="X6q9nV5K" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=collabora.com; s=mail; t=1734530243; bh=Vc/ZM9fAvz09aWlUkt1tgjwmGq/TmY7xSsGfE0yV9pQ=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=X6q9nV5KSijo9GZAWpqJ5ytDPQ9aWfV7WRw5Mnhew/vWo/qEnHr0KrDfjP3VEuud5 F8Bh8VLz+uK4BefvbhHZLcWd2sxoIzHDh0zhvKp7cuVkCNpQSqsoiXtaTv+TZnyqyE yD40DmXrgZwl02e45GfmVqWSwIv6rcgzUnx6K1AFWi/AZuK/2wkscnhVlpuxK8E8gA CEGWSDGDC4YJH4zB9p+HUtW2mQmZ9NeUIm41d5tfDLIMRnegU5YyvPvgxFu39TzU6s jTdoEP4CrW3aEFROSFeNBGUWhDbOFVLq+mkGx+E05mOJk86KZN4hwGhPlnutDt0lOL ayyn+GWoaTvOg== Received: from [192.168.50.250] (unknown [171.76.86.135]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: vignesh) by bali.collaboradmins.com (Postfix) with ESMTPSA id A989B17E376D; Wed, 18 Dec 2024 14:57:20 +0100 (CET) Message-ID: <58005033-5708-4a3a-a5b1-58898ce3fbac@collabora.com> Date: Wed, 18 Dec 2024 19:27:12 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] drm/ci: add kms_cursor_legacy@torture-bo to apq8016 flakes To: Abhinav Kumar , Helen Mae Koike Fornazier Cc: Rob Clark , Dmitry Baryshkov , Sean Paul , Marijn Suijten , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter , linux-kernel , linux-arm-msm , dri-devel , freedreno References: <20241204-cursor_tor_skip-v1-1-f5f0bba5df7b@quicinc.com> <193931869a5.f923adf2270026.8321075661083367617@collabora.com> <20a3955e-3d10-47c5-8e68-d70342805010@quicinc.com> <19393604e18.f9b6fe7d298023.1937039548910081216@collabora.com> <8edba6c9-cf7d-4616-8791-65abd108f538@collabora.com> Content-Language: en-US From: Vignesh Raman In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Hi Abhinav, On 16/12/24 11:39, Abhinav Kumar wrote: > > > On 12/15/2024 9:45 PM, Vignesh Raman wrote: >> Hi Abhinav, >> >> On 14/12/24 01:09, Abhinav Kumar wrote: >>> Hi Vignesh >>> >>> On 12/11/2024 9:10 PM, Vignesh Raman wrote: >>>> Hi Abhinav / Helen, >>>> >>>> On 12/12/24 01:48, Abhinav Kumar wrote: >>>>> Hi Helen / Vignesh >>>>> >>>>> On 12/4/2024 12:33 PM, Helen Mae Koike Fornazier wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> ---- On Wed, 04 Dec 2024 16:21:26 -0300 Abhinav Kumar  wrote --- >>>>>> >>>>>>   > Hi Helen >>>>>>   > >>>>>>   > On 12/4/2024 11:14 AM, Helen Mae Koike Fornazier wrote: >>>>>>   > > Hi Abhinav, >>>>>>   > > >>>>>>   > > Thanks for your patch. >>>>>>   > > >>>>>>   > > >>>>>>   > > >>>>>>   > > ---- On Wed, 04 Dec 2024 15:55:17 -0300 Abhinav Kumar  wrote >>>>>> --- >>>>>>   > > >>>>>>   > >   > From the jobs [1] and [2] of pipeline [3], its clear that >>>>>>   > >   > kms_cursor_legacy@torture-bo is most certainly a flake and >>>>>>   > >   > not a fail for apq8016. Mark the test accordingly to >>>>>> match the results. >>>>>>   > >   > >>>>>>   > >   > [1] : >>>>>> https://gitlab.freedesktop.org/drm/msm/-/jobs/67676481 >>>> >>>> The test passes - >>>> kms_cursor_legacy@torture-bo,UnexpectedImprovement(Pass) >>>> >>> >>> Yes, thats the problem >>> >>> https://gitlab.freedesktop.org/drm/msm/-/jobs/67676481/viewer#L2696 >>> >>> 24-12-04 03:51:55 R SERIAL> [  179.241309] [IGT] kms_cursor_legacy: >>> finished subtest all-pipes, SUCCESS >>> 24-12-04 03:51:55 R SERIAL> [  179.241812] [IGT] kms_cursor_legacy: >>> finished subtest torture-bo, SUCCESS >>> >>> Here it passes whereas it was marked a failure. Hence pipeline fails. >> >> Yes it fails due to, >> >> Unexpected results: >>   kms_cursor_legacy@torture-bo,UnexpectedImprovement(Pass) >> >> In this case, we need to remove this test from fails.txt >> >>> >>>>>>   > >   > [2] : >>>>>> https://gitlab.freedesktop.org/drm/msm/-/jobs/67677430 >>>> >>>> There are no test failures >>>> >>> >>> No, thats not true >>> >>> https://gitlab.freedesktop.org/drm/msm/-/jobs/67677430/viewer#L2694 >>> >>> 24-12-04 04:18:38 R SERIAL> [  170.379649] Console: switching to >>> colour dummy device 80x25 >>> 24-12-04 04:18:38 R SERIAL> [  170.379938] [IGT] kms_cursor_legacy: >>> executing >>> 24-12-04 04:18:38 R SERIAL> [  170.393868] [IGT] kms_cursor_legacy: >>> starting subtest torture-bo >>> 24-12-04 04:18:38 R SERIAL> [  170.394186] [IGT] kms_cursor_legacy: >>> starting dynamic subtest pipe-A >>> 24-12-04 04:18:38 R SERIAL> [  170.661749] [IGT] kms_cursor_legacy: >>> finished subtest pipe-A, FAIL >>> 24-12-04 04:18:38 R SERIAL> [  170.662060] [IGT] kms_cursor_legacy: >>> starting dynamic subtest all-pipes >>> 24-12-04 04:18:38 R SERIAL> [  170.713237] [IGT] kms_cursor_legacy: >>> finished subtest all-pipes, FAIL >>> 24-12-04 04:18:38 R SERIAL> [  170.713513] [IGT] kms_cursor_legacy: >>> finished subtest torture-bo, FAIL >>> 24-12-04 04:18:38 R SERIAL> [  170.721263] [IGT] kms_cursor_legacy: >>> exiting, ret=98 >>> 24-12-04 04:18:38 R SERIAL> [  170.737857] Console: switching to >>> colour frame buffer device 128x48 >>> >>> Please check these logs, the torture-bo test-case did fail. The >>> pipeline was marked pass because it was an expected fail. >>> >>> So we have two pipelines, where one failed and the other passed. So >>> thats a flake for me. >> >> Yes agree. So if we had removed the test from fails, deqp-runner would >> have reported this as flake. >> >> deqp-runner runs the test and if it fails, it retries. If the test >> passes on retry, it is reported as a flake. >> >>> >>>>>>   > >   > [3]: >>>>>> https://gitlab.freedesktop.org/drm/msm/-/pipelines/1322770 >>>> >>>> The job is same as 2 >>>> >>>> In this case, the test passes and deqp-runner does not report it as >>>> flake. So we only need to remove it from fails file. >>>> >>> >>> No, like I mentioned above we have a pass and a fail. >>> >>>> >>>>>>   > >   > >>>>>>   > >   > Signed-off-by: Abhinav Kumar quic_abhinavk@quicinc.com> >>>>>>   > >   > --- >>>>>>   > >   >  drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt | 5 +++++ >>>>>>   > >   >  1 file changed, 5 insertions(+) >>>>>>   > >   > >>>>>>   > >   > diff --git >>>>>> a/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt >>>>>> b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt >>>>>>   > >   > new file mode 100644 >>>>>>   > >   > index 000000000000..18639853f18f >>>>>>   > >   > --- /dev/null >>>>>>   > >   > +++ b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt >>>>>>   > >   > @@ -0,0 +1,5 @@ >>>>>>   > >   > +# Board Name: msm-apq8016-db410c >>>>>>   > >   > +# Failure Rate: 100 >>>>>>   > > >>>>>>   > > Is failure rate is 100%, isn't it a fail than? >>>>>>   > > (I know we have other cases with Failure Rate: 100, maybe we >>>>>> should fix them as well) >>>>>>   > > >>>>>>   > >>>>>>   > Maybe I misunderstood the meaning of "Failure rate" for a flake. >>>>>>   > >>>>>>   > I interpreted this as this test being flaky 100% of the time :) >>>>>> >>>>>> Ah right, I see, inside deqp-runner (that auto-retries). >>>>>> >>>>>> I'd like to hear Vignesh's opinion on this. >>>>>> >>>>>> (In any case, we probably should document this better) >>>> >>>> deqp-runner reports new (not present in flakes file) or known >>>> (present in flakes file) flakes >>>> >>>> 2024-12-11 07:25:44.709666: Some new flakes found: >>>> 2024-12-11 07:25:44.709676:   kms_lease@page-flip-implicit-plane >>>> >>>> 2024-12-11 13:15:16.482890: Some known flakes found: >>>> 2024-12-11 13:15:16.482898: >>>> kms_async_flips@async-flip-with-page-flip-events-atomic >>>> >>>> we add it to flakes file if deqp runner reports new flakes. Another >>>> case where we update flake tests is when a test passes in one run >>>> but fails in another, but deqp-runner does not report it as flake. >>>> >>>> Regards, >>>> Vignesh >>>> >>> >>> The confusion here i guess is about what to mention as a "Failure rate" >>> >>> Failure rate means how many times it fails (like normally) ? In that >>> case 100% which I used is wrong and I used 33% instead for which I >>> have pushed v2. >> >> Yes, 33% is correct and please remove this test from fails.txt >> >> Regards, >> Vignesh >> > > Ack, let me remove this test from fails and keep it only in flakes. Can you remove it from the fails without adding it to the flakes, and rerun the pipeline a few times to see if deqp-runner reports it as a flake? Thanks. Regards, Vignesh > > Thanks > > Abhinav >>> >>>>>> >>>>>> Regards, >>>>>> Helen >>>>>> >>>>> >>>>> Can you let me know which way we need to go? >>>>> >>>>> Just in case I did post a v2 fixing this, >>>>> https://patchwork.freedesktop.org/patch/627276/ >>>>> >>>>> If thats the way to go, can you pls take a look? >>>>> >>>>> Thanks >>>>> >>>>> Abhinav >>>>>>   > >>>>>>   > Out of the 3 runs of the test, it passed 2/3 times and failed >>>>>> 1/3. >>>>>>   > >>>>>>   > So its fail % actually is 33.33% in that case. >>>>>>   > >>>>>>   > I think I saw a Failure rate of 100% on >>>>>> msm-sm8350-hdk-flakes.txt and >>>>>>   > mistook that as the rate at which flakes are seen. >>>>>>   > >>>>>>   > Let me fix this up as 33% >>>>>>   > >>>>>>   > > Regards, >>>>>>   > > Helen >>>>>>   > > >>>>>>   > >   > +# IGT Version: 1.28-ga73311079 >>>>>>   > >   > +# Linux Version: 6.12.0-rc2 >>>>>>   > >   > +kms_cursor_legacy@torture-bo >>>>>>   > >   > >>>>>>   > >   > --- >>>>>>   > >   > base-commit: 798bb342e0416d846cf67f4725a3428f39bfb96b >>>>>>   > >   > change-id: 20241204-cursor_tor_skip-9d128dd62c4f >>>>>>   > >   > >>>>>>   > >   > Best regards, >>>>>>   > >   > -- >>>>>>   > >   > Abhinav Kumar quic_abhinavk@quicinc.com> >>>>>>   > >   > >>>>>>   > >   > >>>>>>   > > >>>>>>   > >>>>>>