From: Alexandre Belloni <alexandre.belloni@bootlin.com>
To: alexis.lothore@bootlin.com
Cc: Openembedded-core@lists.openembedded.org,
Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Subject: Re: [OE-Core][PATCH 0/2] Add a display limit for regression report generation
Date: Fri, 20 Oct 2023 08:05:13 +0200 [thread overview]
Message-ID: <20231020060513c625e441@mail.local> (raw)
In-Reply-To: <20231019095352.25923-1-alexis.lothore@bootlin.com>
Hello Alexis,
On 19/10/2023 11:53:50+0200, Alexis Lothor� via lists.openembedded.org wrote:
> It has been observed that useful information in regression report can be
> drowned in huge regression lists which are often false-positives (for
> example, a whole set of tests has been temporarily disabled).
>
> This series brings a default limit to how many changes are displayed per
> base/target comparison. This default can still be overriden on commandline,
> for example to have a better look at the whole regression list when trying
> to debug an issue (i.e. by disabling the limit)
>
> First commit implement the limit, its default value and the corresponding
> commandline option in resulttool.
> Second commit allow yocto_testresults_query.py to drive this value.
>
> As a result, one can for example do the following:
> - yocto_testresults_query 4.3_M1 4.3_M2
> -> will display at most 50 regressions per test
> - yocto_testresults_query -l 10 4.3_M1 4.3_M2
> -> override the display limit and reduce it to 10 regressions per pair.
> - yocto_testresults_query -l 0 4.3_M1 4.3_M2
> -> disable the display limit, print all regressions
>
> An example of regression report with display limit can be found here:
> https://pastebin.com/6QbfGstR
>
> Alexis Lothor� (2):
> scripts/resulttool: limit the number of changes displayed per test
> scripts/yocto_testresults_query: add option to change display limit
>
This causes:
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5886/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5935/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5952/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5936/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2296/steps/14/logs/stdio
2023-10-19 07:28:44,229 - oe-selftest - INFO - resulttooltests.ResultToolTests.test_can_match_non_static_ptest_names (subunit.RemotedTestCase)
2023-10-19 07:28:44,229 - oe-selftest - INFO - ... ERROR
Stderr:
2023-10-19 07:05:57,080 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-271681/conf/local.conf
2023-10-19 07:05:57,081 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2023-10-19 07:28:44,229 - oe-selftest - INFO - 6: 24/42 185/543 (0.01s) (0 failed) (resulttooltests.ResultToolTests.test_can_match_non_static_ptest_names)
2023-10-19 07:28:44,230 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/resulttooltests.py", line 372, in test_can_match_non_static_ptest_names
result, resultstring = regression.compare_result(
^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: compare_result() missing 1 required positional argument: 'display_limit'
2023-10-19 07:28:44,311 - oe-selftest - INFO - resulttooltests.ResultToolTests.test_regression_can_get_regression_result (subunit.RemotedTestCase)
2023-10-19 07:28:44,311 - oe-selftest - INFO - ... ERROR
Stderr:
2023-10-19 07:05:57,080 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-271681/conf/local.conf
2023-10-19 07:05:57,081 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2023-10-19 07:28:44,312 - oe-selftest - INFO - 6: 31/42 192/543 (0.01s) (2 failed) (resulttooltests.ResultToolTests.test_regression_can_get_regression_result)
2023-10-19 07:28:44,312 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/resulttooltests.py", line 85, in test_regression_can_get_regression_result
result, text = regression.compare_result(self.logger, "BaseTestRunName", "TargetTestRunName", base_result_data, target_result_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: compare_result() missing 1 required positional argument: 'display_limit'
> scripts/lib/resulttool/regression.py | 23 +++++++++++++++++++----
> scripts/yocto_testresults_query.py | 13 ++++++++++---
> 2 files changed, 29 insertions(+), 7 deletions(-)
>
> --
> 2.42.0
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#189440): https://lists.openembedded.org/g/openembedded-core/message/189440
> Mute This Topic: https://lists.openembedded.org/mt/102057050/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
next prev parent reply other threads:[~2023-10-20 6:05 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-10-19 9:53 [OE-Core][PATCH 0/2] Add a display limit for regression report generation Alexis Lothoré
2023-10-19 9:53 ` [OE-Core][PATCH 1/2] scripts/resulttool: limit the number of changes displayed per test Alexis Lothoré
2023-10-19 9:53 ` [OE-Core][PATCH 2/2] scripts/yocto_testresults_query: add option to change display limit Alexis Lothoré
2023-10-20 6:05 ` Alexandre Belloni [this message]
2023-10-20 6:32 ` [OE-Core][PATCH 0/2] Add a display limit for regression report generation Alexis Lothoré
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20231020060513c625e441@mail.local \
--to=alexandre.belloni@bootlin.com \
--cc=Openembedded-core@lists.openembedded.org \
--cc=alexis.lothore@bootlin.com \
--cc=thomas.petazzoni@bootlin.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox