* [PATCH v3 0/6] scripts/resulttool/regression: add metadata filtering
@ 2023-02-24 16:45 alexis.lothore
2023-02-24 16:45 ` [PATCH v3 1/6] scripts/oe-selftest: append metadata to tests results alexis.lothore
` (7 more replies)
0 siblings, 8 replies; 17+ messages in thread
From: alexis.lothore @ 2023-02-24 16:45 UTC (permalink / raw)
To: openembedded-core; +Cc: alexandre.belloni, thomas.petazzoni
From: Alexis Lothoré <alexis.lothore@bootlin.com>
Hello,
this new series is the follow-up of [1] to make regression reports more
meaningful, by reducing noise and false positives.
Change since v2:
- add filtering on MACHINE field from test results configuration: the MACHINE
should always match
- add "metadata guessing" mechanism based on Richard proposal ([2]). Up to the
point where this series will be merged, tests results stored in git are not
enriched with OESELFTEST_METADATA. To allow proper test comparison even with
those tests, try to guess what oeselftest command line has been used to run
the corresponding tests, and generate OESELFTEST_METADATA accordingly
- add new tool to ease test results usage: yocto_testresults_query. For now the
tool only manages regression report and is a thin layer between send-qa-email
(in yocto-autobuilder-helper) and resulttool. Its main role is to translate
regression reports arguments (which are tags or branches) to fixed revisions
and to call resulttool accordingly. Most of its code is a transfer from
send-qa-email (another series for the autobuilder will follow this one to make
send-qa-email use this new helper, but this current series works
independently)
Example: "yocto_testresults_query.py regression-report 4.2_M1 4.2_M2" will
replay the regression report generated when the 4.2_M2 has been generated.
Change since v1:
- properly configure "From" field in series
With those improvements, the regression report is significantly reduced and some
useful data start to emerge from the removed noise:
- with the MACHINE filtering, the 4.2_M2 report goes from 5.5GB to 627MB
- with the OESELFTEST_METADATA enrichment + metadata guessing for older tests,
the report goes from 627MB to 1.5MB
After manual inspection on some entries, the remaining oeselftest regression
raised in the report seems valid. There are still some issues to tackle:
- it seems that now one major remaining source of noise is on the "runtime"
tests (comparison to tests not run on "target" results)
- when a ptest managed by oe-selftest fails, I guess the remaining tests are not
run, so when 1 failure is logged, we have many "PASSED->None" transitions in
regression report, we should probably silence it.
- some transitions appear as regression while those are in fact improvements
(e.g: "UNRESOLVED->PASSED")
[1] https://lore.kernel.org/openembedded-core/20230214165309.63527-1-alexis.lothore@bootlin.com/
[2] https://lore.kernel.org/openembedded-core/124b9c9667b038b8502f6457ba7d894fc4ef3c58.camel@linuxfoundation.org/
Alexis Lothoré (6):
scripts/oe-selftest: append metadata to tests results
scripts/resulttool/regression: remove unused import
scripts/resulttool/regression: add metadata filtering for oeselftest
oeqa/selftest/resulttool: add test for metadata filtering on
regression
scripts: add new helper for regression report generation
oeqa/selftest: add test for yocto_testresults_query.py
.../oeqa/selftest/cases/resulttooltests.py | 137 +++++++++++++++
.../cases/yoctotestresultsquerytests.py | 39 +++++
meta/lib/oeqa/selftest/context.py | 15 +-
scripts/lib/resulttool/regression.py | 163 +++++++++++++++++-
scripts/yocto_testresults_query.py | 106 ++++++++++++
5 files changed, 458 insertions(+), 2 deletions(-)
create mode 100644 meta/lib/oeqa/selftest/cases/yoctotestresultsquerytests.py
create mode 100755 scripts/yocto_testresults_query.py
--
2.39.1
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH v3 1/6] scripts/oe-selftest: append metadata to tests results
2023-02-24 16:45 [PATCH v3 0/6] scripts/resulttool/regression: add metadata filtering alexis.lothore
@ 2023-02-24 16:45 ` alexis.lothore
2023-02-24 16:45 ` [PATCH v3 2/6] scripts/resulttool/regression: remove unused import alexis.lothore
` (6 subsequent siblings)
7 siblings, 0 replies; 17+ messages in thread
From: alexis.lothore @ 2023-02-24 16:45 UTC (permalink / raw)
To: openembedded-core; +Cc: alexandre.belloni, thomas.petazzoni
From: Alexis Lothoré <alexis.lothore@bootlin.com>
Many stored results TEST_TYPE are set to "oeselftest", however all those
tests are not run with the same sets of parameters, so those tests results may
not be comparable.
Attach relevant parameters as tests metadata to allow identifying tests
configuration so we can compare tests only when they are run with the same
parameters.
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
---
meta/lib/oeqa/selftest/context.py | 15 ++++++++++++++-
1 file changed, 14 insertions(+), 1 deletion(-)
diff --git a/meta/lib/oeqa/selftest/context.py b/meta/lib/oeqa/selftest/context.py
index c7dd03ce378..8cc46283ed0 100644
--- a/meta/lib/oeqa/selftest/context.py
+++ b/meta/lib/oeqa/selftest/context.py
@@ -22,6 +22,17 @@ from oeqa.core.exception import OEQAPreRun, OEQATestNotFound
from oeqa.utils.commands import runCmd, get_bb_vars, get_test_layer
+OESELFTEST_METADATA=["run_all_tests", "run_tests", "skips", "machine", "select_tags", "exclude_tags"]
+
+def get_oeselftest_metadata(args):
+ result = {}
+ raw_args = vars(args)
+ for metadata in OESELFTEST_METADATA:
+ if metadata in raw_args:
+ result[metadata] = raw_args[metadata]
+
+ return result
+
class NonConcurrentTestSuite(unittest.TestSuite):
def __init__(self, suite, processes, setupfunc, removefunc):
super().__init__([suite])
@@ -334,12 +345,14 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
import platform
from oeqa.utils.metadata import metadata_from_bb
metadata = metadata_from_bb()
+ oeselftest_metadata = get_oeselftest_metadata(args)
configuration = {'TEST_TYPE': 'oeselftest',
'STARTTIME': args.test_start_time,
'MACHINE': self.tc.td["MACHINE"],
'HOST_DISTRO': oe.lsb.distro_identifier().replace(' ', '-'),
'HOST_NAME': metadata['hostname'],
- 'LAYERS': metadata['layers']}
+ 'LAYERS': metadata['layers'],
+ 'OESELFTEST_METADATA':oeselftest_metadata}
return configuration
def get_result_id(self, configuration):
--
2.39.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v3 2/6] scripts/resulttool/regression: remove unused import
2023-02-24 16:45 [PATCH v3 0/6] scripts/resulttool/regression: add metadata filtering alexis.lothore
2023-02-24 16:45 ` [PATCH v3 1/6] scripts/oe-selftest: append metadata to tests results alexis.lothore
@ 2023-02-24 16:45 ` alexis.lothore
2023-02-24 16:45 ` [PATCH v3 3/6] scripts/resulttool/regression: add metadata filtering for oeselftest alexis.lothore
` (5 subsequent siblings)
7 siblings, 0 replies; 17+ messages in thread
From: alexis.lothore @ 2023-02-24 16:45 UTC (permalink / raw)
To: openembedded-core; +Cc: alexandre.belloni, thomas.petazzoni
From: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
---
scripts/lib/resulttool/regression.py | 1 -
1 file changed, 1 deletion(-)
diff --git a/scripts/lib/resulttool/regression.py b/scripts/lib/resulttool/regression.py
index 9f952951b3f..d0b0c318051 100644
--- a/scripts/lib/resulttool/regression.py
+++ b/scripts/lib/resulttool/regression.py
@@ -7,7 +7,6 @@
#
import resulttool.resultutils as resultutils
-import json
from oeqa.utils.git import GitRepo
import oeqa.utils.gitarchive as gitarchive
--
2.39.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v3 3/6] scripts/resulttool/regression: add metadata filtering for oeselftest
2023-02-24 16:45 [PATCH v3 0/6] scripts/resulttool/regression: add metadata filtering alexis.lothore
2023-02-24 16:45 ` [PATCH v3 1/6] scripts/oe-selftest: append metadata to tests results alexis.lothore
2023-02-24 16:45 ` [PATCH v3 2/6] scripts/resulttool/regression: remove unused import alexis.lothore
@ 2023-02-24 16:45 ` alexis.lothore
2023-02-24 16:45 ` [PATCH v3 4/6] oeqa/selftest/resulttool: add test for metadata filtering on regression alexis.lothore
` (4 subsequent siblings)
7 siblings, 0 replies; 17+ messages in thread
From: alexis.lothore @ 2023-02-24 16:45 UTC (permalink / raw)
To: openembedded-core; +Cc: alexandre.belloni, thomas.petazzoni
From: Alexis Lothoré <alexis.lothore@bootlin.com>
When generating regression reports, many false positive can be observed since
some tests results are compared while the corresponding tests sets are not the
same, as it can be seen for example for oeselftest tests (oeselftest is run
multiple time but with different parameters, resulting in different tests sets)
Add a filtering mechanism in resulttool regression module to enable a better
matching between tests. The METADATA_MATCH_TABLE defines that when the TEST_TYPE
is "oeselftest", then resulttool should filter pairs based on
OESELFTEST_METADATA appended to test configuration. If metadata is absent from
test results, in order to keep compatibility with older results, add a
"guessing" mechanism to generate the missing OESELFTEST_METADATA. The guessed
data is tightly coupled to the autobuilder configuration, where all oe-selftest
executions are described
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
---
scripts/lib/resulttool/regression.py | 162 +++++++++++++++++++++++++++
1 file changed, 162 insertions(+)
diff --git a/scripts/lib/resulttool/regression.py b/scripts/lib/resulttool/regression.py
index d0b0c318051..1b0c8335a39 100644
--- a/scripts/lib/resulttool/regression.py
+++ b/scripts/lib/resulttool/regression.py
@@ -11,6 +11,164 @@ import resulttool.resultutils as resultutils
from oeqa.utils.git import GitRepo
import oeqa.utils.gitarchive as gitarchive
+METADATA_MATCH_TABLE = {
+ "oeselftest": "OESELFTEST_METADATA"
+}
+
+OESELFTEST_METADATA_GUESS_TABLE={
+ "trigger-build-posttrigger": {
+ "run_all_tests": False,
+ "run_tests":["buildoptions.SourceMirroring.test_yocto_source_mirror"],
+ "skips": None,
+ "machine": None,
+ "select_tags":None,
+ "exclude_tags": None
+ },
+ "reproducible": {
+ "run_all_tests": False,
+ "run_tests":["reproducible"],
+ "skips": None,
+ "machine": None,
+ "select_tags":None,
+ "exclude_tags": None
+ },
+ "arch-qemu-quick": {
+ "run_all_tests": True,
+ "run_tests":None,
+ "skips": None,
+ "machine": None,
+ "select_tags":["machine"],
+ "exclude_tags": None
+ },
+ "arch-qemu-full-x86-or-x86_64": {
+ "run_all_tests": True,
+ "run_tests":None,
+ "skips": None,
+ "machine": None,
+ "select_tags":["machine", "toolchain-system"],
+ "exclude_tags": None
+ },
+ "arch-qemu-full-others": {
+ "run_all_tests": True,
+ "run_tests":None,
+ "skips": None,
+ "machine": None,
+ "select_tags":["machine", "toolchain-user"],
+ "exclude_tags": None
+ },
+ "selftest": {
+ "run_all_tests": True,
+ "run_tests":None,
+ "skips": ["distrodata.Distrodata.test_checkpkg", "buildoptions.SourceMirroring.test_yocto_source_mirror", "reproducible"],
+ "machine": None,
+ "select_tags":None,
+ "exclude_tags": ["machine", "toolchain-system", "toolchain-user"]
+ },
+ "bringup": {
+ "run_all_tests": True,
+ "run_tests":None,
+ "skips": ["distrodata.Distrodata.test_checkpkg", "buildoptions.SourceMirroring.test_yocto_source_mirror"],
+ "machine": None,
+ "select_tags":None,
+ "exclude_tags": ["machine", "toolchain-system", "toolchain-user"]
+ }
+}
+
+def test_has_at_least_one_matching_tag(test, tag_list):
+ return "oetags" in test and any(oetag in tag_list for oetag in test["oetags"])
+
+def all_tests_have_at_least_one_matching_tag(results, tag_list):
+ return all(test_has_at_least_one_matching_tag(test_result, tag_list) or test_name.startswith("ptestresult") for (test_name, test_result) in results.items())
+
+def any_test_have_any_matching_tag(results, tag_list):
+ return any(test_has_at_least_one_matching_tag(test, tag_list) for test in results.values())
+
+def have_skipped_test(result, test_prefix):
+ return all( result[test]['status'] == "SKIPPED" for test in result if test.startswith(test_prefix))
+
+def have_all_tests_skipped(result, test_prefixes_list):
+ return all(have_skipped_test(result, test_prefix) for test_prefix in test_prefixes_list)
+
+def guess_oeselftest_metadata(results):
+ """
+ When an oeselftest test result is lacking OESELFTEST_METADATA, we can try to guess it based on results content.
+ Check results for specific values (absence/presence of oetags, number and name of executed tests...),
+ and if it matches one of known configuration from autobuilder configuration, apply guessed OSELFTEST_METADATA
+ to it to allow proper test filtering.
+ This guessing process is tightly coupled to config.json in autobuilder. It should trigger less and less,
+ as new tests will have OESELFTEST_METADATA properly appended at test reporting time
+ """
+
+ if len(results) == 1 and "buildoptions.SourceMirroring.test_yocto_source_mirror" in results:
+ return OESELFTEST_METADATA_GUESS_TABLE['trigger-build-posttrigger']
+ elif all(result.startswith("reproducible") for result in results):
+ return OESELFTEST_METADATA_GUESS_TABLE['reproducible']
+ elif all_tests_have_at_least_one_matching_tag(results, ["machine"]):
+ return OESELFTEST_METADATA_GUESS_TABLE['arch-qemu-quick']
+ elif all_tests_have_at_least_one_matching_tag(results, ["machine", "toolchain-system"]):
+ return OESELFTEST_METADATA_GUESS_TABLE['arch-qemu-full-x86-or-x86_64']
+ elif all_tests_have_at_least_one_matching_tag(results, ["machine", "toolchain-user"]):
+ return OESELFTEST_METADATA_GUESS_TABLE['arch-qemu-full-others']
+ elif not any_test_have_any_matching_tag(results, ["machine", "toolchain-user", "toolchain-system"]):
+ if have_all_tests_skipped(results, ["distrodata.Distrodata.test_checkpkg", "buildoptions.SourceMirroring.test_yocto_source_mirror", "reproducible"]):
+ return OESELFTEST_METADATA_GUESS_TABLE['selftest']
+ elif have_all_tests_skipped(results, ["distrodata.Distrodata.test_checkpkg", "buildoptions.SourceMirroring.test_yocto_source_mirror"]):
+ return OESELFTEST_METADATA_GUESS_TABLE['bringup']
+
+ return None
+
+
+def metadata_matches(base_configuration, target_configuration):
+ """
+ For passed base and target, check test type. If test type matches one of
+ properties described in METADATA_MATCH_TABLE, compare metadata if it is
+ present in base. Return true if metadata matches, or if base lacks some
+ data (either TEST_TYPE or the corresponding metadata)
+ """
+ test_type = base_configuration.get('TEST_TYPE')
+ if test_type not in METADATA_MATCH_TABLE:
+ return True
+
+ metadata_key = METADATA_MATCH_TABLE.get(test_type)
+ if target_configuration.get(metadata_key) != base_configuration.get(metadata_key):
+ return False
+
+ return True
+
+
+def machine_matches(base_configuration, target_configuration):
+ return base_configuration.get('MACHINE') == target_configuration.get('MACHINE')
+
+
+def can_be_compared(logger, base, target):
+ """
+ Some tests are not relevant to be compared, for example some oeselftest
+ run with different tests sets or parameters. Return true if tests can be
+ compared
+ """
+ base_configuration = base['configuration']
+ target_configuration = target['configuration']
+
+ # Older test results lack proper OESELFTEST_METADATA: if not present, try to guess it based on tests results.
+ if base_configuration.get('TEST_TYPE') == 'oeselftest' and 'OESELFTEST_METADATA' not in base_configuration:
+ guess = guess_oeselftest_metadata(base['result'])
+ if guess is None:
+ logger.error(f"ERROR: did not manage to guess oeselftest metadata for {base_configuration['STARTTIME']}")
+ else:
+ logger.debug(f"Enriching {base_configuration['STARTTIME']} with {guess}")
+ base_configuration['OESELFTEST_METADATA'] = guess
+ if target_configuration.get('TEST_TYPE') == 'oeselftest' and 'OESELFTEST_METADATA' not in target_configuration:
+ guess = guess_oeselftest_metadata(target['result'])
+ if guess is None:
+ logger.error(f"ERROR: did not manage to guess oeselftest metadata for {target_configuration['STARTTIME']}")
+ else:
+ logger.debug(f"Enriching {target_configuration['STARTTIME']} with {guess}")
+ target_configuration['OESELFTEST_METADATA'] = guess
+
+ return metadata_matches(base_configuration, target_configuration) \
+ and machine_matches(base_configuration, target_configuration)
+
+
def compare_result(logger, base_name, target_name, base_result, target_result):
base_result = base_result.get('result')
target_result = target_result.get('result')
@@ -61,6 +219,8 @@ def regression_common(args, logger, base_results, target_results):
# removing any pairs which match
for c in base.copy():
for b in target.copy():
+ if not can_be_compared(logger, base_results[a][c], target_results[a][b]):
+ continue
res, resstr = compare_result(logger, c, b, base_results[a][c], target_results[a][b])
if not res:
matches.append(resstr)
@@ -70,6 +230,8 @@ def regression_common(args, logger, base_results, target_results):
# Should only now see regressions, we may not be able to match multiple pairs directly
for c in base:
for b in target:
+ if not can_be_compared(logger, base_results[a][c], target_results[a][b]):
+ continue
res, resstr = compare_result(logger, c, b, base_results[a][c], target_results[a][b])
if res:
regressions.append(resstr)
--
2.39.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v3 4/6] oeqa/selftest/resulttool: add test for metadata filtering on regression
2023-02-24 16:45 [PATCH v3 0/6] scripts/resulttool/regression: add metadata filtering alexis.lothore
` (2 preceding siblings ...)
2023-02-24 16:45 ` [PATCH v3 3/6] scripts/resulttool/regression: add metadata filtering for oeselftest alexis.lothore
@ 2023-02-24 16:45 ` alexis.lothore
2023-02-26 0:03 ` [OE-core] " Richard Purdie
2023-02-24 16:45 ` [PATCH v3 5/6] scripts: add new helper for regression report generation alexis.lothore
` (3 subsequent siblings)
7 siblings, 1 reply; 17+ messages in thread
From: alexis.lothore @ 2023-02-24 16:45 UTC (permalink / raw)
To: openembedded-core; +Cc: alexandre.belloni, thomas.petazzoni
From: Alexis Lothoré <alexis.lothore@bootlin.com>
Introduce new tests for the metadata-based filtering added for oeselftest
results
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
---
.../oeqa/selftest/cases/resulttooltests.py | 137 ++++++++++++++++++
1 file changed, 137 insertions(+)
diff --git a/meta/lib/oeqa/selftest/cases/resulttooltests.py b/meta/lib/oeqa/selftest/cases/resulttooltests.py
index efdfd98af3c..75d406c122d 100644
--- a/meta/lib/oeqa/selftest/cases/resulttooltests.py
+++ b/meta/lib/oeqa/selftest/cases/resulttooltests.py
@@ -98,3 +98,140 @@ class ResultToolTests(OESelftestTestCase):
resultutils.append_resultsdata(results, ResultToolTests.target_results_data, configmap=resultutils.flatten_map)
self.assertEqual(len(results[''].keys()), 5, msg="Flattened results not correct %s" % str(results))
+ def test_results_without_metadata_can_be_compared(self):
+ base_configuration = {"TEST_TYPE": "oeselftest",
+ "TESTSERIES": "series1",
+ "IMAGE_BASENAME": "image",
+ "IMAGE_PKGTYPE": "ipk",
+ "DISTRO": "mydistro",
+ "MACHINE": "qemux86"}
+ target_configuration = {"TEST_TYPE": "oeselftest",
+ "TESTSERIES": "series1",
+ "IMAGE_BASENAME": "image",
+ "IMAGE_PKGTYPE": "ipk",
+ "DISTRO": "mydistro",
+ "MACHINE": "qemux86"}
+ self.assertTrue(regression.can_be_compared(base_configuration, target_configuration),
+ msg="incorrect metadata filtering, tests without metadata should be compared")
+
+ def test_target_result_with_missing_metadata_can_not_be_compared(self):
+ base_configuration = {"TEST_TYPE": "oeselftest",
+ "TESTSERIES": "series1",
+ "IMAGE_BASENAME": "image",
+ "IMAGE_PKGTYPE": "ipk",
+ "DISTRO": "mydistro",
+ "MACHINE": "qemux86",
+ "OESELFTEST_METADATA": {"run_all_tests": True,
+ "run_tests": None,
+ "skips": None,
+ "machine": None,
+ "select_tags": ["toolchain-user", "toolchain-system"],
+ "exclude_tags": None}}
+ target_configuration = {"TEST_TYPE": "oeselftest",
+ "TESTSERIES": "series1",
+ "IMAGE_BASENAME": "image",
+ "IMAGE_PKGTYPE": "ipk",
+ "DISTRO": "mydistro",
+ "MACHINE": "qemux86"}
+ self.assertFalse(regression.can_be_compared(base_configuration, target_configuration),
+ msg="incorrect metadata filtering, tests should not be compared")
+
+ def test_results_with_matching_metadata_can_be_compared(self):
+ base_configuration = {"TEST_TYPE": "oeselftest",
+ "TESTSERIES": "series1",
+ "IMAGE_BASENAME": "image",
+ "IMAGE_PKGTYPE": "ipk",
+ "DISTRO": "mydistro",
+ "MACHINE": "qemux86",
+ "OESELFTEST_METADATA": {"run_all_tests": True,
+ "run_tests": None,
+ "skips": None,
+ "machine": None,
+ "select_tags": ["toolchain-user", "toolchain-system"],
+ "exclude_tags": None}}
+ target_configuration = {"TEST_TYPE": "oeselftest",
+ "TESTSERIES": "series1",
+ "IMAGE_BASENAME": "image",
+ "IMAGE_PKGTYPE": "ipk",
+ "DISTRO": "mydistro",
+ "MACHINE": "qemux86",
+ "OESELFTEST_METADATA": {"run_all_tests": True,
+ "run_tests": None,
+ "skips": None,
+ "machine": None,
+ "select_tags": ["toolchain-user", "toolchain-system"],
+ "exclude_tags": None}}
+ self.assertTrue(regression.can_be_compared(base_configuration, target_configuration),
+ msg="incorrect metadata filtering, tests with matching metadata should be compared")
+
+ def test_results_with_mismatching_metadata_can_not_be_compared(self):
+ base_configuration = {"TEST_TYPE": "oeselftest",
+ "TESTSERIES": "series1",
+ "IMAGE_BASENAME": "image",
+ "IMAGE_PKGTYPE": "ipk",
+ "DISTRO": "mydistro",
+ "MACHINE": "qemux86",
+ "OESELFTEST_METADATA": {"run_all_tests": True,
+ "run_tests": None,
+ "skips": None,
+ "machine": None,
+ "select_tags": ["toolchain-user", "toolchain-system"],
+ "exclude_tags": None}}
+ target_configuration = {"TEST_TYPE": "oeselftest",
+ "TESTSERIES": "series1",
+ "IMAGE_BASENAME": "image",
+ "IMAGE_PKGTYPE": "ipk",
+ "DISTRO": "mydistro",
+ "MACHINE": "qemux86",
+ "OESELFTEST_METADATA": {"run_all_tests": True,
+ "run_tests": None,
+ "skips": None,
+ "machine": None,
+ "select_tags": ["machine"],
+ "exclude_tags": None}}
+ self.assertFalse(regression.can_be_compared(base_configuration, target_configuration),
+ msg="incorrect metadata filtering, tests with mismatching metadata should not be compared")
+
+ def test_metadata_matching_is_only_checked_for_relevant_test_type(self):
+ base_configuration = {"TEST_TYPE": "runtime",
+ "TESTSERIES": "series1",
+ "IMAGE_BASENAME": "image",
+ "IMAGE_PKGTYPE": "ipk",
+ "DISTRO": "mydistro",
+ "MACHINE": "qemux86",
+ "OESELFTEST_METADATA": {"run_all_tests": True,
+ "run_tests": None,
+ "skips": None,
+ "machine": None,
+ "select_tags": ["toolchain-user", "toolchain-system"],
+ "exclude_tags": None}}
+ target_configuration = {"TEST_TYPE": "runtime",
+ "TESTSERIES": "series1",
+ "IMAGE_BASENAME": "image",
+ "IMAGE_PKGTYPE": "ipk",
+ "DISTRO": "mydistro",
+ "MACHINE": "qemux86",
+ "OESELFTEST_METADATA": {"run_all_tests": True,
+ "run_tests": None,
+ "skips": None,
+ "machine": None,
+ "select_tags": ["machine"],
+ "exclude_tags": None}}
+ self.assertTrue(regression.can_be_compared(base_configuration, target_configuration),
+ msg="incorrect metadata filtering, %s tests should be compared" % base_configuration['TEST_TYPE'])
+
+ def test_machine_matches(self):
+ base_configuration = {"TEST_TYPE": "runtime",
+ "MACHINE": "qemux86"}
+ target_configuration = {"TEST_TYPE": "runtime",
+ "MACHINE": "qemux86"}
+ self.assertTrue(regression.can_be_compared(base_configuration, target_configuration),
+ msg="incorrect machine filtering, identical machine tests should be compared")
+
+ def test_machine_mismatches(self):
+ base_configuration = {"TEST_TYPE": "runtime",
+ "MACHINE": "qemux86"}
+ target_configuration = {"TEST_TYPE": "runtime",
+ "MACHINE": "qemux86_64"}
+ self.assertFalse(regression.can_be_compared(base_configuration, target_configuration),
+ msg="incorrect machine filtering, mismatching machine tests should not be compared")
--
2.39.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v3 5/6] scripts: add new helper for regression report generation
2023-02-24 16:45 [PATCH v3 0/6] scripts/resulttool/regression: add metadata filtering alexis.lothore
` (3 preceding siblings ...)
2023-02-24 16:45 ` [PATCH v3 4/6] oeqa/selftest/resulttool: add test for metadata filtering on regression alexis.lothore
@ 2023-02-24 16:45 ` alexis.lothore
2023-02-24 16:45 ` [PATCH v3 6/6] oeqa/selftest: add test for yocto_testresults_query.py alexis.lothore
` (2 subsequent siblings)
7 siblings, 0 replies; 17+ messages in thread
From: alexis.lothore @ 2023-02-24 16:45 UTC (permalink / raw)
To: openembedded-core; +Cc: alexandre.belloni, thomas.petazzoni
From: Alexis Lothoré <alexis.lothore@bootlin.com>
Add yocto-testresults-query script. This is a thin wrapper over resulttool which
is able to translate tags or branch name to specific revisions, and then to work
with those "guessed" revisions with resulttool
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
---
scripts/yocto_testresults_query.py | 106 +++++++++++++++++++++++++++++
1 file changed, 106 insertions(+)
create mode 100755 scripts/yocto_testresults_query.py
diff --git a/scripts/yocto_testresults_query.py b/scripts/yocto_testresults_query.py
new file mode 100755
index 00000000000..fee3855c6d8
--- /dev/null
+++ b/scripts/yocto_testresults_query.py
@@ -0,0 +1,106 @@
+#!/usr/bin/env python3
+
+# Yocto Project test results management tool
+# This script is an thin layer over resulttool to manage tes results and regression reports.
+# Its main feature is to translate tags or branch names to revisions SHA1, and then to run resulttool
+# with those computed revisions
+#
+# Copyright (C) 2023 OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+import sys
+import os
+import argparse
+import subprocess
+import tempfile
+import lib.scriptutils as scriptutils
+
+script_path = os.path.dirname(os.path.realpath(__file__))
+poky_path = os.path.abspath(os.path.join(script_path, ".."))
+resulttool = os.path.abspath(os.path.join(script_path, "resulttool"))
+logger = scriptutils.logger_create(sys.argv[0])
+testresults_default_url="git://git.yoctoproject.org/yocto-testresults"
+
+def create_workdir():
+ workdir = tempfile.mkdtemp(prefix='yocto-testresults-query.')
+ logger.info(f"Shallow-cloning testresults in {workdir}")
+ subprocess.check_call(["git", "clone", testresults_default_url, workdir, "--depth", "1"])
+ return workdir
+
+def get_sha1(pokydir, revision):
+ rev = subprocess.check_output(["git", "rev-list", "-n", "1", revision], cwd=pokydir).decode('utf-8').strip()
+ logger.info(f"SHA-1 revision for {revision} in {pokydir} is {rev}")
+ return rev
+
+def fetch_testresults(workdir, sha1):
+ logger.info(f"Fetching test results for {sha1} in {workdir}")
+ rawtags = subprocess.check_output(["git", "ls-remote", "--refs", "--tags", "origin", f"*{sha1}*"], cwd=workdir).decode('utf-8').strip()
+ if not rawtags:
+ raise Exception(f"No reference found for commit {sha1} in {workdir}")
+ for rev in [rawtag.split()[1] for rawtag in rawtags.splitlines()]:
+ logger.info(f"Fetching matching revisions: {rev}")
+ subprocess.check_call(["git", "fetch", "--depth", "1", "origin", f"{rev}:{rev}"], cwd=workdir)
+
+def compute_regression_report(workdir, baserevision, targetrevision):
+ logger.info(f"Running resulttool regression between SHA1 {baserevision} and {targetrevision}")
+ report = subprocess.check_output([resulttool, "regression-git", "--commit", baserevision, "--commit2", targetrevision, workdir]).decode("utf-8")
+ return report
+
+def print_report_with_header(report, baseversion, baserevision, targetversion, targetrevision):
+ print("========================== Regression report ==============================")
+ print(f'{"=> Target:": <16}{targetversion: <16}({targetrevision})')
+ print(f'{"=> Base:": <16}{baseversion: <16}({baserevision})')
+ print("===========================================================================\n")
+ print(report, end='')
+
+def regression(args):
+ logger.info(f"Compute regression report between {args.base} and {args.target}")
+ if args.testresultsdir:
+ workdir = args.testresultsdir
+ else:
+ workdir = create_workdir()
+
+ try:
+ baserevision = get_sha1(poky_path, args.base)
+ targetrevision = get_sha1(poky_path, args.target)
+ fetch_testresults(workdir, baserevision)
+ fetch_testresults(workdir, targetrevision)
+ report = compute_regression_report(workdir, baserevision, targetrevision)
+ print_report_with_header(report, args.base, baserevision, args.target, targetrevision)
+ finally:
+ if not args.testresultsdir:
+ subprocess.check_call(["rm", "-rf", workdir])
+
+def main():
+ parser = argparse.ArgumentParser(description="Yocto Project test results helper")
+ subparsers = parser.add_subparsers(
+ help="Supported commands for test results helper",
+ required=True)
+ parser_regression_report = subparsers.add_parser(
+ "regression-report",
+ help="Generate regression report between two fixed revisions. Revisions can be branch name or tag")
+ parser_regression_report.add_argument(
+ 'base',
+ help="Revision or tag against which to compare results (i.e: the older)")
+ parser_regression_report.add_argument(
+ 'target',
+ help="Revision or tag to compare against the base (i.e: the newer)")
+ parser_regression_report.add_argument(
+ '-t',
+ '--testresultsdir',
+ help=f"An existing test results directory. {sys.argv[0]} will automatically clone it and use default branch if not provided")
+ parser_regression_report.set_defaults(func=regression)
+
+ args = parser.parse_args()
+ args.func(args)
+
+if __name__ == '__main__':
+ try:
+ ret = main()
+ except Exception:
+ ret = 1
+ import traceback
+ traceback.print_exc()
+ sys.exit(ret)
\ No newline at end of file
--
2.39.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v3 6/6] oeqa/selftest: add test for yocto_testresults_query.py
2023-02-24 16:45 [PATCH v3 0/6] scripts/resulttool/regression: add metadata filtering alexis.lothore
` (4 preceding siblings ...)
2023-02-24 16:45 ` [PATCH v3 5/6] scripts: add new helper for regression report generation alexis.lothore
@ 2023-02-24 16:45 ` alexis.lothore
2023-02-24 18:06 ` [OE-core] [PATCH v3 0/6] scripts/resulttool/regression: add metadata filtering Richard Purdie
[not found] ` <1746D4E8592324E9.29542@lists.openembedded.org>
7 siblings, 0 replies; 17+ messages in thread
From: alexis.lothore @ 2023-02-24 16:45 UTC (permalink / raw)
To: openembedded-core; +Cc: alexandre.belloni, thomas.petazzoni
From: Alexis Lothoré <alexis.lothore@bootlin.com>
Add some tests for new yocto_testresults_query.py helper. First test is taken
from yocto-autobuilder-helper feature which has moved in yocto_testresults_query
Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
---
.../cases/yoctotestresultsquerytests.py | 39 +++++++++++++++++++
1 file changed, 39 insertions(+)
create mode 100644 meta/lib/oeqa/selftest/cases/yoctotestresultsquerytests.py
diff --git a/meta/lib/oeqa/selftest/cases/yoctotestresultsquerytests.py b/meta/lib/oeqa/selftest/cases/yoctotestresultsquerytests.py
new file mode 100644
index 00000000000..312edb64319
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/yoctotestresultsquerytests.py
@@ -0,0 +1,39 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+import os
+import sys
+import subprocess
+import shutil
+from oeqa.selftest.case import OESelftestTestCase
+from yocto_testresults_query import get_sha1, create_workdir
+basepath = os.path.abspath(os.path.dirname(__file__) + '/../../../../../')
+lib_path = basepath + '/scripts/lib'
+sys.path = sys.path + [lib_path]
+
+
+class TestResultsQueryTests(OESelftestTestCase):
+ def test_get_sha1(self):
+ test_data_get_sha1 = [
+ {"input": "yocto-4.0", "expected": "00cfdde791a0176c134f31e5a09eff725e75b905"},
+ {"input": "4.1_M1", "expected": "95066dde6861ee08fdb505ab3e0422156cc24fae"},
+ ]
+ for data in test_data_get_sha1:
+ test_name = data["input"]
+ with self.subTest(f"Test SHA1 from {test_name}"):
+ self.assertEqual(
+ get_sha1(basepath, data["input"]), data["expected"])
+
+ def test_create_workdir(self):
+ workdir = create_workdir()
+ try:
+ url = subprocess.check_output(
+ ["git", "-C", workdir, "remote", "get-url", "origin"]).strip().decode("utf-8")
+ except:
+ shutil.rmtree(workdir, ignore_errors=True)
+ self.fail(f"Can not execute git commands in {workdir}")
+ shutil.rmtree(workdir)
+ self.assertEqual(url, "git://git.yoctoproject.org/yocto-testresults")
--
2.39.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [OE-core] [PATCH v3 0/6] scripts/resulttool/regression: add metadata filtering
2023-02-24 16:45 [PATCH v3 0/6] scripts/resulttool/regression: add metadata filtering alexis.lothore
` (5 preceding siblings ...)
2023-02-24 16:45 ` [PATCH v3 6/6] oeqa/selftest: add test for yocto_testresults_query.py alexis.lothore
@ 2023-02-24 18:06 ` Richard Purdie
[not found] ` <1746D4E8592324E9.29542@lists.openembedded.org>
7 siblings, 0 replies; 17+ messages in thread
From: Richard Purdie @ 2023-02-24 18:06 UTC (permalink / raw)
To: alexis.lothore, openembedded-core; +Cc: alexandre.belloni, thomas.petazzoni
Hi Alexis,
Firstly, this looks very much improved, thanks. It is great to start to
see some meaningful data from this.
On Fri, 2023-02-24 at 17:45 +0100, Alexis Lothoré via
lists.openembedded.org wrote:
> From: Alexis Lothoré <alexis.lothore@bootlin.com>
>
> Hello,
> this new series is the follow-up of [1] to make regression reports more
> meaningful, by reducing noise and false positives.
>
> Change since v2:
> - add filtering on MACHINE field from test results configuration: the MACHINE
> should always match
> - add "metadata guessing" mechanism based on Richard proposal ([2]). Up to the
> point where this series will be merged, tests results stored in git are not
> enriched with OESELFTEST_METADATA. To allow proper test comparison even with
> those tests, try to guess what oeselftest command line has been used to run
> the corresponding tests, and generate OESELFTEST_METADATA accordingly
> - add new tool to ease test results usage: yocto_testresults_query. For now the
> tool only manages regression report and is a thin layer between send-qa-email
> (in yocto-autobuilder-helper) and resulttool. Its main role is to translate
> regression reports arguments (which are tags or branches) to fixed revisions
> and to call resulttool accordingly. Most of its code is a transfer from
> send-qa-email (another series for the autobuilder will follow this one to make
> send-qa-email use this new helper, but this current series works
> independently)
> Example: "yocto_testresults_query.py regression-report 4.2_M1 4.2_M2" will
> replay the regression report generated when the 4.2_M2 has been generated.
>
> Change since v1:
> - properly configure "From" field in series
>
> With those improvements, the regression report is significantly reduced and some
> useful data start to emerge from the removed noise:
> - with the MACHINE filtering, the 4.2_M2 report goes from 5.5GB to 627MB
> - with the OESELFTEST_METADATA enrichment + metadata guessing for older tests,
> the report goes from 627MB to 1.5MB
That is just a bit more readable!
>
> After manual inspection on some entries, the remaining oeselftest regression
> raised in the report seems valid. There are still some issues to tackle:
> - it seems that now one major remaining source of noise is on the "runtime"
> tests (comparison to tests not run on "target" results)
> - when a ptest managed by oe-selftest fails, I guess the remaining tests are not
> run, so when 1 failure is logged, we have many "PASSED->None" transitions in
> regression report, we should probably silence it.
> - some transitions appear as regression while those are in fact improvements
> (e.g: "UNRESOLVED->PASSED")
I had quick play. Firstly, if I try "yocto_testresults_query.py
regression-report 4.2_M1 4.2_M2" in an openembedded-core repository
instead of poky, it breaks. That isn't surprising but we should either
make it work or show a sensible error.
I also took a look the report and wondered why the matching isn't quite
right and why we have these "regressions". If we could remove that
noise, I think we'd get down to the real issues. I ended up doing:
resulttool report --commit 4d19594b8bdacde6d809d3f2a25cff7c5a42295e . > /tmp/repa
resulttool report --commit 5e249ec855517765f4b99e8039cb888ffa09c211 . > /tmp/repb
meld /tmp/rep*
which was interesting as gave lots of warnings like:
"Warning duplicate ptest result 'acl.test/cp.test' for qemuarm64"
so it looks like we had a couple of different test runs for qemuarm64
ptests which is confusing your new code. I suspect this happened due to
some autobuilder glitch during the release build which restarted some
of the build pieces. Not sure how to handle that yet, I'll give it some
further thought but I wanted to share what I think is the source of
some of the issues. Basically we need to get the regression report
looking more like that meld output!
Cheers,
Richard
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [OE-core] [PATCH v3 0/6] scripts/resulttool/regression: add metadata filtering
[not found] ` <1746D4E8592324E9.29542@lists.openembedded.org>
@ 2023-02-25 9:15 ` Richard Purdie
[not found] ` <1747067DAE80068A.29542@lists.openembedded.org>
1 sibling, 0 replies; 17+ messages in thread
From: Richard Purdie @ 2023-02-25 9:15 UTC (permalink / raw)
To: alexis.lothore, openembedded-core; +Cc: alexandre.belloni, thomas.petazzoni
On Fri, 2023-02-24 at 18:06 +0000, Richard Purdie via
lists.openembedded.org wrote:
> Hi Alexis,
>
> Firstly, this looks very much improved, thanks. It is great to start to
> see some meaningful data from this.
>
> On Fri, 2023-02-24 at 17:45 +0100, Alexis Lothoré via
> lists.openembedded.org wrote:
> > From: Alexis Lothoré <alexis.lothore@bootlin.com>
> >
> > Hello,
> > this new series is the follow-up of [1] to make regression reports more
> > meaningful, by reducing noise and false positives.
> >
> > Change since v2:
> > - add filtering on MACHINE field from test results configuration: the MACHINE
> > should always match
> > - add "metadata guessing" mechanism based on Richard proposal ([2]). Up to the
> > point where this series will be merged, tests results stored in git are not
> > enriched with OESELFTEST_METADATA. To allow proper test comparison even with
> > those tests, try to guess what oeselftest command line has been used to run
> > the corresponding tests, and generate OESELFTEST_METADATA accordingly
> > - add new tool to ease test results usage: yocto_testresults_query. For now the
> > tool only manages regression report and is a thin layer between send-qa-email
> > (in yocto-autobuilder-helper) and resulttool. Its main role is to translate
> > regression reports arguments (which are tags or branches) to fixed revisions
> > and to call resulttool accordingly. Most of its code is a transfer from
> > send-qa-email (another series for the autobuilder will follow this one to make
> > send-qa-email use this new helper, but this current series works
> > independently)
> > Example: "yocto_testresults_query.py regression-report 4.2_M1 4.2_M2" will
> > replay the regression report generated when the 4.2_M2 has been generated.
> >
> > Change since v1:
> > - properly configure "From" field in series
> >
> > With those improvements, the regression report is significantly reduced and some
> > useful data start to emerge from the removed noise:
> > - with the MACHINE filtering, the 4.2_M2 report goes from 5.5GB to 627MB
> > - with the OESELFTEST_METADATA enrichment + metadata guessing for older tests,
> > the report goes from 627MB to 1.5MB
>
> That is just a bit more readable!
>
> >
> > After manual inspection on some entries, the remaining oeselftest regression
> > raised in the report seems valid. There are still some issues to tackle:
> > - it seems that now one major remaining source of noise is on the "runtime"
> > tests (comparison to tests not run on "target" results)
> > - when a ptest managed by oe-selftest fails, I guess the remaining tests are not
> > run, so when 1 failure is logged, we have many "PASSED->None" transitions in
> > regression report, we should probably silence it.
> > - some transitions appear as regression while those are in fact improvements
> > (e.g: "UNRESOLVED->PASSED")
>
> I had quick play. Firstly, if I try "yocto_testresults_query.py
> regression-report 4.2_M1 4.2_M2" in an openembedded-core repository
> instead of poky, it breaks. That isn't surprising but we should either
> make it work or show a sensible error.
>
> I also took a look the report and wondered why the matching isn't quite
> right and why we have these "regressions". If we could remove that
> noise, I think we'd get down to the real issues. I ended up doing:
>
> resulttool report --commit 4d19594b8bdacde6d809d3f2a25cff7c5a42295e . > /tmp/repa
> resulttool report --commit 5e249ec855517765f4b99e8039cb888ffa09c211 . > /tmp/repb
> meld /tmp/rep*
>
> which was interesting as gave lots of warnings like:
>
> "Warning duplicate ptest result 'acl.test/cp.test' for qemuarm64"
>
> so it looks like we had a couple of different test runs for qemuarm64
> ptests which is confusing your new code. I suspect this happened due to
> some autobuilder glitch during the release build which restarted some
> of the build pieces. Not sure how to handle that yet, I'll give it some
> further thought but I wanted to share what I think is the source of
> some of the issues. Basically we need to get the regression report
> looking more like that meld output!
I was wrong about the duplication, that isn't the issue, or at least I
found some other more pressing ones. For the ltp issue, I found an easy
fix:
diff --git a/scripts/lib/resulttool/regression.py b/scripts/lib/resulttool/regression.py
index 1b0c8335a39..9d7c35942a6 100644
--- a/scripts/lib/resulttool/regression.py
+++ b/scripts/lib/resulttool/regression.py
@@ -146,6 +146,7 @@ def can_be_compared(logger, base, target):
run with different tests sets or parameters. Return true if tests can be
compared
"""
+ ret = True
base_configuration = base['configuration']
target_configuration = target['configuration']
@@ -165,7 +166,10 @@ def can_be_compared(logger, base, target):
logger.debug(f"Enriching {target_configuration['STARTTIME']} with {guess}")
target_configuration['OESELFTEST_METADATA'] = guess
- return metadata_matches(base_configuration, target_configuration) \
+ if base_configuration.get('TEST_TYPE') == 'runtime' and any(result.startswith("ltpresult") for result in base['result']):
+ ret = target_configuration.get('TEST_TYPE') == 'runtime' and any(result.startswith("ltpresult") for result in target['result'])
+
+ return ret and metadata_matches(base_configuration, target_configuration) \
and machine_matches(base_configuration, target_configuration)
i.e. only compare ltp to ltp. The issue is we don't use a special image
name for the ltp test runs, we just extend a standard one so it was
comparing ltp to non-ltp.
We should also perhaps consider a clause in there which only compares
runs with ptests with other runs with ptests? Our test matrix won't
trigger that but other usage might in future and it is a safe check?
A lot of the rest of the noise is poor test naming for ptests, e.g.:
ptestresult.lttng-tools.ust/buffers-pid/test_buffers_pid_10_-_Create_session_buffers-pid_in_-o_/tmp/tmp.XXXXXXXXXXrs_pid_trace_path.XTnDY5
which has a random string at the end. I'm wondering if we should pre-
filter ptest result names and truncate a known list of them at the "-"
(lttng-tools, babeltrace, babeltrace2). Curl could also be truncated at
the ",":
ptestresult.curl.test_0010__10_out_of_1506,_remaining:_06:44,_took_1.075s,_duration:_00:02_
We can adjust the ptest generation code to do this at source (we should
perhaps file a bug for that for the four above?) but that won't fix the
older results so we'll probably need some filtering in the code too.
There is something more going on with the ptest results too, I don't
understand why quilt/python3 changed but I suspect we just have to go
through the issues step by step now.
I did look into the:
ptestresult.glibc-user.debug/tst-fortify-c-default-1
'regression' and it is because the test was renamed in the new glibc. I
was therefore thinking a summary of added/removed would be useful in
but only in these cases. Something along the line of if only tests
added, just summarise X new added and call it a match. If tests removed
and added, list and show a count summary (X removed, Y added) and call
it a regression.
I think I might be tempted to merge this series and then we can change
the code to improve from here as this is clearly a vast improvement on
where we were! Improvements can be incremental on top of these changes.
Cheers,
Richard
^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [OE-core] [PATCH v3 0/6] scripts/resulttool/regression: add metadata filtering
[not found] ` <1747067DAE80068A.29542@lists.openembedded.org>
@ 2023-02-25 12:32 ` Richard Purdie
2023-02-25 15:59 ` Alexis Lothoré
[not found] ` <1747113C8A4DBAD6.29542@lists.openembedded.org>
1 sibling, 1 reply; 17+ messages in thread
From: Richard Purdie @ 2023-02-25 12:32 UTC (permalink / raw)
To: alexis.lothore, openembedded-core; +Cc: alexandre.belloni, thomas.petazzoni
On Sat, 2023-02-25 at 09:15 +0000, Richard Purdie via
lists.openembedded.org wrote:
> On Fri, 2023-02-24 at 18:06 +0000, Richard Purdie via
> lists.openembedded.org wrote:
> > Hi Alexis,
> >
> > Firstly, this looks very much improved, thanks. It is great to start to
> > see some meaningful data from this.
> >
> > On Fri, 2023-02-24 at 17:45 +0100, Alexis Lothoré via
> > lists.openembedded.org wrote:
> > > From: Alexis Lothoré <alexis.lothore@bootlin.com>
> > >
> > > Hello,
> > > this new series is the follow-up of [1] to make regression reports more
> > > meaningful, by reducing noise and false positives.
> > >
> > > Change since v2:
> > > - add filtering on MACHINE field from test results configuration: the MACHINE
> > > should always match
> > > - add "metadata guessing" mechanism based on Richard proposal ([2]). Up to the
> > > point where this series will be merged, tests results stored in git are not
> > > enriched with OESELFTEST_METADATA. To allow proper test comparison even with
> > > those tests, try to guess what oeselftest command line has been used to run
> > > the corresponding tests, and generate OESELFTEST_METADATA accordingly
> > > - add new tool to ease test results usage: yocto_testresults_query. For now the
> > > tool only manages regression report and is a thin layer between send-qa-email
> > > (in yocto-autobuilder-helper) and resulttool. Its main role is to translate
> > > regression reports arguments (which are tags or branches) to fixed revisions
> > > and to call resulttool accordingly. Most of its code is a transfer from
> > > send-qa-email (another series for the autobuilder will follow this one to make
> > > send-qa-email use this new helper, but this current series works
> > > independently)
> > > Example: "yocto_testresults_query.py regression-report 4.2_M1 4.2_M2" will
> > > replay the regression report generated when the 4.2_M2 has been generated.
> > >
> > > Change since v1:
> > > - properly configure "From" field in series
> > >
> > > With those improvements, the regression report is significantly reduced and some
> > > useful data start to emerge from the removed noise:
> > > - with the MACHINE filtering, the 4.2_M2 report goes from 5.5GB to 627MB
> > > - with the OESELFTEST_METADATA enrichment + metadata guessing for older tests,
> > > the report goes from 627MB to 1.5MB
> >
> > That is just a bit more readable!
> >
> > >
> > > After manual inspection on some entries, the remaining oeselftest regression
> > > raised in the report seems valid. There are still some issues to tackle:
> > > - it seems that now one major remaining source of noise is on the "runtime"
> > > tests (comparison to tests not run on "target" results)
> > > - when a ptest managed by oe-selftest fails, I guess the remaining tests are not
> > > run, so when 1 failure is logged, we have many "PASSED->None" transitions in
> > > regression report, we should probably silence it.
> > > - some transitions appear as regression while those are in fact improvements
> > > (e.g: "UNRESOLVED->PASSED")
> >
> > I had quick play. Firstly, if I try "yocto_testresults_query.py
> > regression-report 4.2_M1 4.2_M2" in an openembedded-core repository
> > instead of poky, it breaks. That isn't surprising but we should either
> > make it work or show a sensible error.
> >
> > I also took a look the report and wondered why the matching isn't quite
> > right and why we have these "regressions". If we could remove that
> > noise, I think we'd get down to the real issues. I ended up doing:
> >
> > resulttool report --commit 4d19594b8bdacde6d809d3f2a25cff7c5a42295e . > /tmp/repa
> > resulttool report --commit 5e249ec855517765f4b99e8039cb888ffa09c211 . > /tmp/repb
> > meld /tmp/rep*
> >
> > which was interesting as gave lots of warnings like:
> >
> > "Warning duplicate ptest result 'acl.test/cp.test' for qemuarm64"
> >
> > so it looks like we had a couple of different test runs for qemuarm64
> > ptests which is confusing your new code. I suspect this happened due to
> > some autobuilder glitch during the release build which restarted some
> > of the build pieces. Not sure how to handle that yet, I'll give it some
> > further thought but I wanted to share what I think is the source of
> > some of the issues. Basically we need to get the regression report
> > looking more like that meld output!
>
> I was wrong about the duplication, that isn't the issue, or at least I
> found some other more pressing ones. For the ltp issue, I found an easy
> fix:
>
> diff --git a/scripts/lib/resulttool/regression.py b/scripts/lib/resulttool/regression.py
> index 1b0c8335a39..9d7c35942a6 100644
> --- a/scripts/lib/resulttool/regression.py
> +++ b/scripts/lib/resulttool/regression.py
> @@ -146,6 +146,7 @@ def can_be_compared(logger, base, target):
> run with different tests sets or parameters. Return true if tests can be
> compared
> """
> + ret = True
> base_configuration = base['configuration']
> target_configuration = target['configuration']
>
> @@ -165,7 +166,10 @@ def can_be_compared(logger, base, target):
> logger.debug(f"Enriching {target_configuration['STARTTIME']} with {guess}")
> target_configuration['OESELFTEST_METADATA'] = guess
>
> - return metadata_matches(base_configuration, target_configuration) \
> + if base_configuration.get('TEST_TYPE') == 'runtime' and any(result.startswith("ltpresult") for result in base['result']):
> + ret = target_configuration.get('TEST_TYPE') == 'runtime' and any(result.startswith("ltpresult") for result in target['result'])
> +
> + return ret and metadata_matches(base_configuration, target_configuration) \
> and machine_matches(base_configuration, target_configuration)
>
>
> i.e. only compare ltp to ltp. The issue is we don't use a special image
> name for the ltp test runs, we just extend a standard one so it was
> comparing ltp to non-ltp.
>
> We should also perhaps consider a clause in there which only compares
> runs with ptests with other runs with ptests? Our test matrix won't
> trigger that but other usage might in future and it is a safe check?
>
> A lot of the rest of the noise is poor test naming for ptests, e.g.:
>
> ptestresult.lttng-tools.ust/buffers-pid/test_buffers_pid_10_-_Create_session_buffers-pid_in_-o_/tmp/tmp.XXXXXXXXXXrs_pid_trace_path.XTnDY5
>
> which has a random string at the end. I'm wondering if we should pre-
> filter ptest result names and truncate a known list of them at the "-"
> (lttng-tools, babeltrace, babeltrace2). Curl could also be truncated at
> the ",":
>
> ptestresult.curl.test_0010__10_out_of_1506,_remaining:_06:44,_took_1.075s,_duration:_00:02_
>
> We can adjust the ptest generation code to do this at source (we should
> perhaps file a bug for that for the four above?) but that won't fix the
> older results so we'll probably need some filtering in the code too.
>
> There is something more going on with the ptest results too, I don't
> understand why quilt/python3 changed but I suspect we just have to go
> through the issues step by step now.
>
> I did look into the:
>
> ptestresult.glibc-user.debug/tst-fortify-c-default-1
>
> 'regression' and it is because the test was renamed in the new glibc. I
> was therefore thinking a summary of added/removed would be useful in
> but only in these cases. Something along the line of if only tests
> added, just summarise X new added and call it a match. If tests removed
> and added, list and show a count summary (X removed, Y added) and call
> it a regression.
>
> I think I might be tempted to merge this series and then we can change
> the code to improve from here as this is clearly a vast improvement on
> where we were! Improvements can be incremental on top of these changes.
This goes a long way to shrinking the report even further. Looks like
the curl test reporting needs some work as the IDs look like they
change but this at least makes the issue clearer and the real deltas
are becoming much easier to see outside the noise.
diff --git a/scripts/lib/resulttool/regression.py b/scripts/lib/resulttool/regression.py
index 1b0c8335a39..0d8948f012f 100644
--- a/scripts/lib/resulttool/regression.py
+++ b/scripts/lib/resulttool/regression.py
@@ -243,6 +247,21 @@ def regression_common(args, logger, base_results, target_results):
return 0
+def fixup_ptest_names(results, logger):
+ for r in results:
+ for i in results[r]:
+ tests = list(results[r][i]['result'].keys())
+ for test in tests:
+ new = None
+ if test.startswith(("ptestresult.lttng-tools.", "ptestresult.babeltrace.", "ptestresult.babeltrace2")) and "_-_" in test:
+ new = test.split("_-_")[0]
+ elif test.startswith(("ptestresult.curl.")) and "__" in test:
+ new = test.split("__")[0]
+ if new:
+ results[r][i]['result'][new] = results[r][i]['result'][test]
+ del results[r][i]['result'][test]
+
+
def regression_git(args, logger):
base_results = {}
target_results = {}
@@ -304,6 +323,9 @@ def regression_git(args, logger):
base_results = resultutils.git_get_result(repo, revs[index1][2])
target_results = resultutils.git_get_result(repo, revs[index2][2])
+ fixup_ptest_names(base_results, logger)
+ fixup_ptest_names(target_results, logger)
+
regression_common(args, logger, base_results, target_results)
return 0
^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [OE-core] [PATCH v3 0/6] scripts/resulttool/regression: add metadata filtering
[not found] ` <1747113C8A4DBAD6.29542@lists.openembedded.org>
@ 2023-02-25 12:44 ` Richard Purdie
2023-02-27 13:14 ` Ross Burton
0 siblings, 1 reply; 17+ messages in thread
From: Richard Purdie @ 2023-02-25 12:44 UTC (permalink / raw)
To: alexis.lothore, openembedded-core
Cc: alexandre.belloni, thomas.petazzoni, Ross Burton
I'll try and stop poking at this but it is all rather interesting and I
think we have spotted our first nasty regression. The quilt ptests did
really stop running properly and reporting test results!
Looking at a recent master report:
https://autobuilder.yocto.io/pub/non-release/20230224-14/testresults/testresult-report.txt
you can see the quilt ptest count is still zero as it was in M2 but not
in M1.
I'm thinking Ross might have been responsible with:
https://git.yoctoproject.org/poky/commit/?id=61bb4d8e75dfaaf980c32fbe992d34f794b7c537
!
The reason there are python3 and python3-cryptography changes are that
there are 35,000 python3 ptests and about 250 changes so that isn't
really unexpected for what was probably a version change. I'd guess
python3-cryptography made changes to their test suite. Overall the
counts of pass rates increased and no failures so those overall counts
may be good to have in the report too.
We might want to log the version of the recipe in the data somewhere so
we can show if this was a python3 version change.
Cheers,
Richard
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [OE-core] [PATCH v3 0/6] scripts/resulttool/regression: add metadata filtering
2023-02-25 12:32 ` Richard Purdie
@ 2023-02-25 15:59 ` Alexis Lothoré
2023-02-26 12:15 ` Richard Purdie
0 siblings, 1 reply; 17+ messages in thread
From: Alexis Lothoré @ 2023-02-25 15:59 UTC (permalink / raw)
To: Richard Purdie, openembedded-core; +Cc: alexandre.belloni, thomas.petazzoni
Hello Richard,
as usual, thanks for the prompt feedback !
On 2/25/23 13:32, Richard Purdie wrote:
> On Sat, 2023-02-25 at 09:15 +0000, Richard Purdie via
> lists.openembedded.org wrote:
>> On Fri, 2023-02-24 at 18:06 +0000, Richard Purdie via
>> lists.openembedded.org wrote:
>>> Hi Alexis,
>>>
>>> Firstly, this looks very much improved, thanks. It is great to start to
>>> see some meaningful data from this.
>>>
>>> On Fri, 2023-02-24 at 17:45 +0100, Alexis Lothoré via
>>> lists.openembedded.org wrote:
>>>> After manual inspection on some entries, the remaining oeselftest regression
>>>> raised in the report seems valid. There are still some issues to tackle:
>>>> - it seems that now one major remaining source of noise is on the "runtime"
>>>> tests (comparison to tests not run on "target" results)
>>>> - when a ptest managed by oe-selftest fails, I guess the remaining tests are not
>>>> run, so when 1 failure is logged, we have many "PASSED->None" transitions in
>>>> regression report, we should probably silence it.
>>>> - some transitions appear as regression while those are in fact improvements
>>>> (e.g: "UNRESOLVED->PASSED")
>>>
>>> I had quick play. Firstly, if I try "yocto_testresults_query.py
>>> regression-report 4.2_M1 4.2_M2" in an openembedded-core repository
>>> instead of poky, it breaks. That isn't surprising but we should either
>>> make it work or show a sensible error.
Oh right, I am working in a Poky build configuration, so I have assumed that this
would be the unique use case.
Since the test results commits are tightly coupled to revisions in poky (so not
oecore), I plan to merely log an error about not found revision (and suggesting
the user to check that the repository is poky and not oecore).
But please let me know if I miss a major use case here and that a smarter
fallback plan (shallow-clone poky if we are running in oecore ?) is needed
>> I think I might be tempted to merge this series and then we can change
>> the code to improve from here as this is clearly a vast improvement on
>> where we were! Improvements can be incremental on top of these changes.
I am in favor of this :) If it is OK for you, I will just re-submit a series with
the fix for the proper error logging when running the tool from oecore and not poky.
Next we could introduce all the suggestions you have suggested, but I feel that
with the quick increase of "hotfixes" count to support issues with older test
results, and for the sake of maintainability of resulttool and its submodules,
those specific hotfixes need to be properly isolated (and documented), like in a
"regression_quirks.py" or something like that. What do you think ?
Alexis
--
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [OE-core] [PATCH v3 4/6] oeqa/selftest/resulttool: add test for metadata filtering on regression
2023-02-24 16:45 ` [PATCH v3 4/6] oeqa/selftest/resulttool: add test for metadata filtering on regression alexis.lothore
@ 2023-02-26 0:03 ` Richard Purdie
0 siblings, 0 replies; 17+ messages in thread
From: Richard Purdie @ 2023-02-26 0:03 UTC (permalink / raw)
To: alexis.lothore, openembedded-core; +Cc: alexandre.belloni, thomas.petazzoni
On Fri, 2023-02-24 at 17:45 +0100, Alexis Lothoré via
lists.openembedded.org wrote:
> From: Alexis Lothoré <alexis.lothore@bootlin.com>
>
> Introduce new tests for the metadata-based filtering added for oeselftest
> results
>
> Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
> ---
> .../oeqa/selftest/cases/resulttooltests.py | 137 ++++++++++++++++++
> 1 file changed, 137 insertions(+)
>
> diff --git a/meta/lib/oeqa/selftest/cases/resulttooltests.py b/meta/lib/oeqa/selftest/cases/resulttooltests.py
> index efdfd98af3c..75d406c122d 100644
> --- a/meta/lib/oeqa/selftest/cases/resulttooltests.py
> +++ b/meta/lib/oeqa/selftest/cases/resulttooltests.py
> @@ -98,3 +98,140 @@ class ResultToolTests(OESelftestTestCase):
> resultutils.append_resultsdata(results, ResultToolTests.target_results_data, configmap=resultutils.flatten_map)
> self.assertEqual(len(results[''].keys()), 5, msg="Flattened results not correct %s" % str(results))
>
> + def test_results_without_metadata_can_be_compared(self):
> + base_configuration = {"TEST_TYPE": "oeselftest",
> + "TESTSERIES": "series1",
> + "IMAGE_BASENAME": "image",
> + "IMAGE_PKGTYPE": "ipk",
> + "DISTRO": "mydistro",
> + "MACHINE": "qemux86"}
> + target_configuration = {"TEST_TYPE": "oeselftest",
> + "TESTSERIES": "series1",
> + "IMAGE_BASENAME": "image",
> + "IMAGE_PKGTYPE": "ipk",
> + "DISTRO": "mydistro",
> + "MACHINE": "qemux86"}
> + self.assertTrue(regression.can_be_compared(base_configuration, target_configuration),
> + msg="incorrect metadata filtering, tests without metadata should be compared")
> +
> + def test_target_result_with_missing_metadata_can_not_be_compared(self):
> + base_configuration = {"TEST_TYPE": "oeselftest",
> + "TESTSERIES": "series1",
> + "IMAGE_BASENAME": "image",
> + "IMAGE_PKGTYPE": "ipk",
> + "DISTRO": "mydistro",
> + "MACHINE": "qemux86",
> + "OESELFTEST_METADATA": {"run_all_tests": True,
> + "run_tests": None,
> + "skips": None,
> + "machine": None,
> + "select_tags": ["toolchain-user", "toolchain-system"],
> + "exclude_tags": None}}
> + target_configuration = {"TEST_TYPE": "oeselftest",
> + "TESTSERIES": "series1",
> + "IMAGE_BASENAME": "image",
> + "IMAGE_PKGTYPE": "ipk",
> + "DISTRO": "mydistro",
> + "MACHINE": "qemux86"}
> + self.assertFalse(regression.can_be_compared(base_configuration, target_configuration),
> + msg="incorrect metadata filtering, tests should not be compared")
> +
> + def test_results_with_matching_metadata_can_be_compared(self):
> + base_configuration = {"TEST_TYPE": "oeselftest",
> + "TESTSERIES": "series1",
> + "IMAGE_BASENAME": "image",
> + "IMAGE_PKGTYPE": "ipk",
> + "DISTRO": "mydistro",
> + "MACHINE": "qemux86",
> + "OESELFTEST_METADATA": {"run_all_tests": True,
> + "run_tests": None,
> + "skips": None,
> + "machine": None,
> + "select_tags": ["toolchain-user", "toolchain-system"],
> + "exclude_tags": None}}
> + target_configuration = {"TEST_TYPE": "oeselftest",
> + "TESTSERIES": "series1",
> + "IMAGE_BASENAME": "image",
> + "IMAGE_PKGTYPE": "ipk",
> + "DISTRO": "mydistro",
> + "MACHINE": "qemux86",
> + "OESELFTEST_METADATA": {"run_all_tests": True,
> + "run_tests": None,
> + "skips": None,
> + "machine": None,
> + "select_tags": ["toolchain-user", "toolchain-system"],
> + "exclude_tags": None}}
> + self.assertTrue(regression.can_be_compared(base_configuration, target_configuration),
> + msg="incorrect metadata filtering, tests with matching metadata should be compared")
> +
> + def test_results_with_mismatching_metadata_can_not_be_compared(self):
> + base_configuration = {"TEST_TYPE": "oeselftest",
> + "TESTSERIES": "series1",
> + "IMAGE_BASENAME": "image",
> + "IMAGE_PKGTYPE": "ipk",
> + "DISTRO": "mydistro",
> + "MACHINE": "qemux86",
> + "OESELFTEST_METADATA": {"run_all_tests": True,
> + "run_tests": None,
> + "skips": None,
> + "machine": None,
> + "select_tags": ["toolchain-user", "toolchain-system"],
> + "exclude_tags": None}}
> + target_configuration = {"TEST_TYPE": "oeselftest",
> + "TESTSERIES": "series1",
> + "IMAGE_BASENAME": "image",
> + "IMAGE_PKGTYPE": "ipk",
> + "DISTRO": "mydistro",
> + "MACHINE": "qemux86",
> + "OESELFTEST_METADATA": {"run_all_tests": True,
> + "run_tests": None,
> + "skips": None,
> + "machine": None,
> + "select_tags": ["machine"],
> + "exclude_tags": None}}
> + self.assertFalse(regression.can_be_compared(base_configuration, target_configuration),
> + msg="incorrect metadata filtering, tests with mismatching metadata should not be compared")
> +
> + def test_metadata_matching_is_only_checked_for_relevant_test_type(self):
> + base_configuration = {"TEST_TYPE": "runtime",
> + "TESTSERIES": "series1",
> + "IMAGE_BASENAME": "image",
> + "IMAGE_PKGTYPE": "ipk",
> + "DISTRO": "mydistro",
> + "MACHINE": "qemux86",
> + "OESELFTEST_METADATA": {"run_all_tests": True,
> + "run_tests": None,
> + "skips": None,
> + "machine": None,
> + "select_tags": ["toolchain-user", "toolchain-system"],
> + "exclude_tags": None}}
> + target_configuration = {"TEST_TYPE": "runtime",
> + "TESTSERIES": "series1",
> + "IMAGE_BASENAME": "image",
> + "IMAGE_PKGTYPE": "ipk",
> + "DISTRO": "mydistro",
> + "MACHINE": "qemux86",
> + "OESELFTEST_METADATA": {"run_all_tests": True,
> + "run_tests": None,
> + "skips": None,
> + "machine": None,
> + "select_tags": ["machine"],
> + "exclude_tags": None}}
> + self.assertTrue(regression.can_be_compared(base_configuration, target_configuration),
> + msg="incorrect metadata filtering, %s tests should be compared" % base_configuration['TEST_TYPE'])
> +
> + def test_machine_matches(self):
> + base_configuration = {"TEST_TYPE": "runtime",
> + "MACHINE": "qemux86"}
> + target_configuration = {"TEST_TYPE": "runtime",
> + "MACHINE": "qemux86"}
> + self.assertTrue(regression.can_be_compared(base_configuration, target_configuration),
> + msg="incorrect machine filtering, identical machine tests should be compared")
> +
> + def test_machine_mismatches(self):
> + base_configuration = {"TEST_TYPE": "runtime",
> + "MACHINE": "qemux86"}
> + target_configuration = {"TEST_TYPE": "runtime",
> + "MACHINE": "qemux86_64"}
> + self.assertFalse(regression.can_be_compared(base_configuration, target_configuration),
> + msg="incorrect machine filtering, mismatching machine tests should not be compared")
I love the fact this has tests but they don't work:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/4854
then with the obvious error fixed to add self.logger:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/4858/steps/14/logs/stdio
Cheers,
Richard
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [OE-core] [PATCH v3 0/6] scripts/resulttool/regression: add metadata filtering
2023-02-25 15:59 ` Alexis Lothoré
@ 2023-02-26 12:15 ` Richard Purdie
2023-02-26 15:42 ` Alexis Lothoré
0 siblings, 1 reply; 17+ messages in thread
From: Richard Purdie @ 2023-02-26 12:15 UTC (permalink / raw)
To: Alexis Lothoré, openembedded-core
Cc: alexandre.belloni, thomas.petazzoni
On Sat, 2023-02-25 at 16:59 +0100, Alexis Lothoré wrote:
> Hello Richard,
> as usual, thanks for the prompt feedback !
>
> On 2/25/23 13:32, Richard Purdie wrote:
> > On Sat, 2023-02-25 at 09:15 +0000, Richard Purdie via
> > lists.openembedded.org wrote:
> > > On Fri, 2023-02-24 at 18:06 +0000, Richard Purdie via
> > > lists.openembedded.org wrote:
> > > > Hi Alexis,
> > > >
> > > > Firstly, this looks very much improved, thanks. It is great to start to
> > > > see some meaningful data from this.
> > > >
> > > > On Fri, 2023-02-24 at 17:45 +0100, Alexis Lothoré via
> > > > lists.openembedded.org wrote:
> > > > > After manual inspection on some entries, the remaining oeselftest regression
> > > > > raised in the report seems valid. There are still some issues to tackle:
> > > > > - it seems that now one major remaining source of noise is on the "runtime"
> > > > > tests (comparison to tests not run on "target" results)
> > > > > - when a ptest managed by oe-selftest fails, I guess the remaining tests are not
> > > > > run, so when 1 failure is logged, we have many "PASSED->None" transitions in
> > > > > regression report, we should probably silence it.
> > > > > - some transitions appear as regression while those are in fact improvements
> > > > > (e.g: "UNRESOLVED->PASSED")
> > > >
> > > > I had quick play. Firstly, if I try "yocto_testresults_query.py
> > > > regression-report 4.2_M1 4.2_M2" in an openembedded-core repository
> > > > instead of poky, it breaks. That isn't surprising but we should either
> > > > make it work or show a sensible error.
>
> Oh right, I am working in a Poky build configuration, so I have assumed that this
> would be the unique use case.
> Since the test results commits are tightly coupled to revisions in poky (so not
> oecore), I plan to merely log an error about not found revision (and suggesting
> the user to check that the repository is poky and not oecore).
> But please let me know if I miss a major use case here and that a smarter
> fallback plan (shallow-clone poky if we are running in oecore ?) is needed
I'm happy to for it just to give an human readable error, someone can
add this functionality if they need/want it.
> > > I think I might be tempted to merge this series and then we can change
> > > the code to improve from here as this is clearly a vast improvement on
> > > where we were! Improvements can be incremental on top of these changes.
>
> I am in favor of this :) If it is OK for you, I will just re-submit a series with
> the fix for the proper error logging when running the tool from oecore and not poky.
>
> Next we could introduce all the suggestions you have suggested, but I feel that
> with the quick increase of "hotfixes" count to support issues with older test
> results, and for the sake of maintainability of resulttool and its submodules,
> those specific hotfixes need to be properly isolated (and documented), like in a
> "regression_quirks.py" or something like that. What do you think ?
I'm hoping we don't have many of these quirks. We have a huge history
at this point so it would be sad if the tool can't work with it. From
what I've seen so far, we can manage with the code in the regression
module itself. I've tried to add some comments.
I wondered what to do with this series since I needed to get M3 built.
Since this series was available and mostly usable, it would be better
to have a nicer report this time, it is a good test of the code.
In the end I've merged most of it, along with my two tweaks to handle
LTP and the bigger ptest results issue. I couldn't take one set of the
selftests since they simply don't work. This will give us a useful
realworld test of the M3 report.
I'm working on the assumption you'll send a follow up series with the
tests, the oe-core check and some of the other issues I've mentioned in
other emails?
Cheers,
Richard
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [OE-core] [PATCH v3 0/6] scripts/resulttool/regression: add metadata filtering
2023-02-26 12:15 ` Richard Purdie
@ 2023-02-26 15:42 ` Alexis Lothoré
2023-02-27 13:41 ` Richard Purdie
0 siblings, 1 reply; 17+ messages in thread
From: Alexis Lothoré @ 2023-02-26 15:42 UTC (permalink / raw)
To: Richard Purdie, openembedded-core; +Cc: alexandre.belloni, thomas.petazzoni
Hello Richard,
On 2/26/23 13:15, Richard Purdie wrote:
> On Sat, 2023-02-25 at 16:59 +0100, Alexis Lothoré wrote:
>> Hello Richard,
>> as usual, thanks for the prompt feedback !
>>
>> On 2/25/23 13:32, Richard Purdie wrote:
>>> On Sat, 2023-02-25 at 09:15 +0000, Richard Purdie via
>>> lists.openembedded.org wrote:
>>>> On Fri, 2023-02-24 at 18:06 +0000, Richard Purdie via
>>>> lists.openembedded.org wrote:
>>>>> Hi Alexis,
>>>>>
>>>>> Firstly, this looks very much improved, thanks. It is great to start to
>>>>> see some meaningful data from this.
>>>>>
>>>>> On Fri, 2023-02-24 at 17:45 +0100, Alexis Lothoré via
>>>>> lists.openembedded.org wrote:
>>>>>> After manual inspection on some entries, the remaining oeselftest regression
>>>>>> raised in the report seems valid. There are still some issues to tackle:
>>>>>> - it seems that now one major remaining source of noise is on the "runtime"
>>>>>> tests (comparison to tests not run on "target" results)
>>>>>> - when a ptest managed by oe-selftest fails, I guess the remaining tests are not
>>>>>> run, so when 1 failure is logged, we have many "PASSED->None" transitions in
>>>>>> regression report, we should probably silence it.
>>>>>> - some transitions appear as regression while those are in fact improvements
>>>>>> (e.g: "UNRESOLVED->PASSED")
>>>>>
>>>>> I had quick play. Firstly, if I try "yocto_testresults_query.py
>>>>> regression-report 4.2_M1 4.2_M2" in an openembedded-core repository
>>>>> instead of poky, it breaks. That isn't surprising but we should either
>>>>> make it work or show a sensible error.
>>
>> Oh right, I am working in a Poky build configuration, so I have assumed that this
>> would be the unique use case.
>> Since the test results commits are tightly coupled to revisions in poky (so not
>> oecore), I plan to merely log an error about not found revision (and suggesting
>> the user to check that the repository is poky and not oecore).
>> But please let me know if I miss a major use case here and that a smarter
>> fallback plan (shallow-clone poky if we are running in oecore ?) is needed
>
> I'm happy to for it just to give an human readable error, someone can
> add this functionality if they need/want it.
ACK
>>>> I think I might be tempted to merge this series and then we can change
>>>> the code to improve from here as this is clearly a vast improvement on
>>>> where we were! Improvements can be incremental on top of these changes.
>>
>> I am in favor of this :) If it is OK for you, I will just re-submit a series with
>> the fix for the proper error logging when running the tool from oecore and not poky.
>>
>> Next we could introduce all the suggestions you have suggested, but I feel that
>> with the quick increase of "hotfixes" count to support issues with older test
>> results, and for the sake of maintainability of resulttool and its submodules,
>> those specific hotfixes need to be properly isolated (and documented), like in a
>> "regression_quirks.py" or something like that. What do you think ?
>
> I'm hoping we don't have many of these quirks. We have a huge history
> at this point so it would be sad if the tool can't work with it. From
> what I've seen so far, we can manage with the code in the regression
> module itself. I've tried to add some comments.
>
> I wondered what to do with this series since I needed to get M3 built.
> Since this series was available and mostly usable, it would be better
> to have a nicer report this time, it is a good test of the code.
>
> In the end I've merged most of it, along with my two tweaks to handle
> LTP and the bigger ptest results issue. I couldn't take one set of the
> selftests since they simply don't work. This will give us a useful
> realworld test of the M3 report.
Ok, great. For the selftest failing, my bad, adding proper logging was one of
those "one last change before sending", and obviously I did forget to re-run the
tests before sending.
>
> I'm working on the assumption you'll send a follow up series with the
> tests, the oe-core check and some of the other issues I've mentioned in
> other emails?
Absolutely. Besides the tests, oecore check and the improvements mentioned in
this mail thread, the next thing I was keeping in mind is fixing the report
generation against "master-next" branches you have mentioned a few weeks ago.
Regards,
--
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [OE-core] [PATCH v3 0/6] scripts/resulttool/regression: add metadata filtering
2023-02-25 12:44 ` Richard Purdie
@ 2023-02-27 13:14 ` Ross Burton
0 siblings, 0 replies; 17+ messages in thread
From: Ross Burton @ 2023-02-27 13:14 UTC (permalink / raw)
To: Richard Purdie
Cc: alexis.lothore@bootlin.com, openembedded-core, Alexandre Belloni,
thomas.petazzoni@bootlin.com
On 25 Feb 2023, at 12:44, Richard Purdie via lists.openembedded.org <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
>
> I'll try and stop poking at this but it is all rather interesting and I
> think we have spotted our first nasty regression. The quilt ptests did
> really stop running properly and reporting test results!
>
> Looking at a recent master report:
>
> https://autobuilder.yocto.io/pub/non-release/20230224-14/testresults/testresult-report.txt
>
> you can see the quilt ptest count is still zero as it was in M2 but not
> in M1.
>
> I'm thinking Ross might have been responsible with:
>
> https://git.yoctoproject.org/poky/commit/?id=61bb4d8e75dfaaf980c32fbe992d34f794b7c537
>
> !
Grumble. Yes, I was.
Fixes sent.
Cheers,
Ross
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [OE-core] [PATCH v3 0/6] scripts/resulttool/regression: add metadata filtering
2023-02-26 15:42 ` Alexis Lothoré
@ 2023-02-27 13:41 ` Richard Purdie
0 siblings, 0 replies; 17+ messages in thread
From: Richard Purdie @ 2023-02-27 13:41 UTC (permalink / raw)
To: Alexis Lothoré, openembedded-core
Cc: alexandre.belloni, thomas.petazzoni
On Sun, 2023-02-26 at 16:42 +0100, Alexis Lothoré wrote:
> > I'm hoping we don't have many of these quirks. We have a huge history
> > at this point so it would be sad if the tool can't work with it. From
> > what I've seen so far, we can manage with the code in the regression
> > module itself. I've tried to add some comments.
> >
> > I wondered what to do with this series since I needed to get M3 built.
> > Since this series was available and mostly usable, it would be better
> > to have a nicer report this time, it is a good test of the code.
> >
> > In the end I've merged most of it, along with my two tweaks to handle
> > LTP and the bigger ptest results issue. I couldn't take one set of the
> > selftests since they simply don't work. This will give us a useful
> > realworld test of the M3 report.
>
> Ok, great. For the selftest failing, my bad, adding proper logging was one of
> those "one last change before sending", and obviously I did forget to re-run the
> tests before sending.
> >
> > I'm working on the assumption you'll send a follow up series with the
> > tests, the oe-core check and some of the other issues I've mentioned in
> > other emails?
>
> Absolutely. Besides the tests, oecore check and the improvements mentioned in
> this mail thread, the next thing I was keeping in mind is fixing the report
> generation against "master-next" branches you have mentioned a few weeks ago.
FWIW, here is the result from M3 rc1:
https://autobuilder.yocto.io/pub/releases/yocto-4.2_M3.rc1/testresults/testresult-regressions-report.txt
It looks like a couple of runs were mismatched so the ltp issue isn't
quite resolved with my tweak. All in all it is definitely a lot more
readable than it was though so progress :)
Looks like dbus may need the same fix as curl.
Cheers,
Richard
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2023-02-27 13:41 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-02-24 16:45 [PATCH v3 0/6] scripts/resulttool/regression: add metadata filtering alexis.lothore
2023-02-24 16:45 ` [PATCH v3 1/6] scripts/oe-selftest: append metadata to tests results alexis.lothore
2023-02-24 16:45 ` [PATCH v3 2/6] scripts/resulttool/regression: remove unused import alexis.lothore
2023-02-24 16:45 ` [PATCH v3 3/6] scripts/resulttool/regression: add metadata filtering for oeselftest alexis.lothore
2023-02-24 16:45 ` [PATCH v3 4/6] oeqa/selftest/resulttool: add test for metadata filtering on regression alexis.lothore
2023-02-26 0:03 ` [OE-core] " Richard Purdie
2023-02-24 16:45 ` [PATCH v3 5/6] scripts: add new helper for regression report generation alexis.lothore
2023-02-24 16:45 ` [PATCH v3 6/6] oeqa/selftest: add test for yocto_testresults_query.py alexis.lothore
2023-02-24 18:06 ` [OE-core] [PATCH v3 0/6] scripts/resulttool/regression: add metadata filtering Richard Purdie
[not found] ` <1746D4E8592324E9.29542@lists.openembedded.org>
2023-02-25 9:15 ` Richard Purdie
[not found] ` <1747067DAE80068A.29542@lists.openembedded.org>
2023-02-25 12:32 ` Richard Purdie
2023-02-25 15:59 ` Alexis Lothoré
2023-02-26 12:15 ` Richard Purdie
2023-02-26 15:42 ` Alexis Lothoré
2023-02-27 13:41 ` Richard Purdie
[not found] ` <1747113C8A4DBAD6.29542@lists.openembedded.org>
2023-02-25 12:44 ` Richard Purdie
2023-02-27 13:14 ` Ross Burton
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.