From: Alexandre Belloni <alexandre.belloni@bootlin.com>
To: alexis.lothore@bootlin.com
Cc: Openembedded-core@lists.openembedded.org,
Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Subject: Re: [OE-Core][PATCH 0/2] Add a display limit for regression report generation
Date: Fri, 20 Oct 2023 08:05:13 +0200 [thread overview]
Message-ID: <20231020060513c625e441@mail.local> (raw)
In-Reply-To: <20231019095352.25923-1-alexis.lothore@bootlin.com>
Hello Alexis,
On 19/10/2023 11:53:50+0200, Alexis Lothor� via lists.openembedded.org wrote:
> It has been observed that useful information in regression report can be
> drowned in huge regression lists which are often false-positives (for
> example, a whole set of tests has been temporarily disabled).
>
> This series brings a default limit to how many changes are displayed per
> base/target comparison. This default can still be overriden on commandline,
> for example to have a better look at the whole regression list when trying
> to debug an issue (i.e. by disabling the limit)
>
> First commit implement the limit, its default value and the corresponding
> commandline option in resulttool.
> Second commit allow yocto_testresults_query.py to drive this value.
>
> As a result, one can for example do the following:
> - yocto_testresults_query 4.3_M1 4.3_M2
> -> will display at most 50 regressions per test
> - yocto_testresults_query -l 10 4.3_M1 4.3_M2
> -> override the display limit and reduce it to 10 regressions per pair.
> - yocto_testresults_query -l 0 4.3_M1 4.3_M2
> -> disable the display limit, print all regressions
>
> An example of regression report with display limit can be found here:
> https://pastebin.com/6QbfGstR
>
> Alexis Lothor� (2):
> scripts/resulttool: limit the number of changes displayed per test
> scripts/yocto_testresults_query: add option to change display limit
>
This causes:
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5886/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5935/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5952/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5936/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2296/steps/14/logs/stdio
2023-10-19 07:28:44,229 - oe-selftest - INFO - resulttooltests.ResultToolTests.test_can_match_non_static_ptest_names (subunit.RemotedTestCase)
2023-10-19 07:28:44,229 - oe-selftest - INFO - ... ERROR
Stderr:
2023-10-19 07:05:57,080 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-271681/conf/local.conf
2023-10-19 07:05:57,081 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2023-10-19 07:28:44,229 - oe-selftest - INFO - 6: 24/42 185/543 (0.01s) (0 failed) (resulttooltests.ResultToolTests.test_can_match_non_static_ptest_names)
2023-10-19 07:28:44,230 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/resulttooltests.py", line 372, in test_can_match_non_static_ptest_names
result, resultstring = regression.compare_result(
^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: compare_result() missing 1 required positional argument: 'display_limit'
2023-10-19 07:28:44,311 - oe-selftest - INFO - resulttooltests.ResultToolTests.test_regression_can_get_regression_result (subunit.RemotedTestCase)
2023-10-19 07:28:44,311 - oe-selftest - INFO - ... ERROR
Stderr:
2023-10-19 07:05:57,080 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-271681/conf/local.conf
2023-10-19 07:05:57,081 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2023-10-19 07:28:44,312 - oe-selftest - INFO - 6: 31/42 192/543 (0.01s) (2 failed) (resulttooltests.ResultToolTests.test_regression_can_get_regression_result)
2023-10-19 07:28:44,312 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/resulttooltests.py", line 85, in test_regression_can_get_regression_result
result, text = regression.compare_result(self.logger, "BaseTestRunName", "TargetTestRunName", base_result_data, target_result_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: compare_result() missing 1 required positional argument: 'display_limit'
> scripts/lib/resulttool/regression.py | 23 +++++++++++++++++++----
> scripts/yocto_testresults_query.py | 13 ++++++++++---
> 2 files changed, 29 insertions(+), 7 deletions(-)
>
> --
> 2.42.0
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#189440): https://lists.openembedded.org/g/openembedded-core/message/189440
> Mute This Topic: https://lists.openembedded.org/mt/102057050/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
next prev parent reply other threads:[~2023-10-20 6:05 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-10-19 9:53 [OE-Core][PATCH 0/2] Add a display limit for regression report generation Alexis Lothoré
2023-10-19 9:53 ` [OE-Core][PATCH 1/2] scripts/resulttool: limit the number of changes displayed per test Alexis Lothoré
2023-10-19 9:53 ` [OE-Core][PATCH 2/2] scripts/yocto_testresults_query: add option to change display limit Alexis Lothoré
2023-10-20 6:05 ` Alexandre Belloni [this message]
2023-10-20 6:32 ` [OE-Core][PATCH 0/2] Add a display limit for regression report generation Alexis Lothoré
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20231020060513c625e441@mail.local \
--to=alexandre.belloni@bootlin.com \
--cc=Openembedded-core@lists.openembedded.org \
--cc=alexis.lothore@bootlin.com \
--cc=thomas.petazzoni@bootlin.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.