[OE-core][PATCH 1/3] patchtest: add --error-on-failure option

[OE-core][PATCH 1/3] patchtest: add --error-on-failure option

[naftaly.ralamboarivony]

From: Naftaly RALAMBOARIVONY <naftaly.ralamboarivony@smile.fr>

Add a command line option to make patchtest return a non-zero exit
status when a test fails.

This keeps the current default behavior unchanged while allowing calling
scripts and shells to detect test failures through the process exit
status when explicitly requested.

Signed-off-by: Naftaly RALAMBOARIVONY <naftaly.ralamboarivony@smile.fr>
---
 meta/lib/patchtest/patchtest_parser.py | 3 +++
 scripts/patchtest                      | 8 ++++++--
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/meta/lib/patchtest/patchtest_parser.py b/meta/lib/patchtest/patchtest_parser.py
index 2a11cb76c2..ba5a171e27 100644
--- a/meta/lib/patchtest/patchtest_parser.py
+++ b/meta/lib/patchtest/patchtest_parser.py
@@ -73,6 +73,9 @@ class PatchtestParser(object):
                             action='store_true', 
                             help='Enable logging to a file matching the target patch name with ".testresult" appended')
 
+        parser.add_argument('--error-on-failure',
+                            action='store_true',
+                            help='Return non-zero exit status if a test fails')
 
         return parser
 
diff --git a/scripts/patchtest b/scripts/patchtest
index 9218db232a..653f2b217a 100755
--- a/scripts/patchtest
+++ b/scripts/patchtest
@@ -183,6 +183,7 @@ def print_result_message(preresult, postresult):
     print("----------------------------------------------------------------------\n")
 
 def main():
+    ret = 0
     tmp_patch = False
     patch_path = PatchtestParser.patch_path
     log_results = PatchtestParser.log_results
@@ -214,13 +215,16 @@ def main():
 
         try:
             if log_path:
-                run(patch, log_path)
+                ret = run(patch, log_path)
             else:
-                run(patch)
+                ret = run(patch)
         finally:
             if tmp_patch:
                 os.remove(patch)
 
+        if PatchtestParser.error_on_failure and ret != 0:
+            return ret
+
 if __name__ == '__main__':
     ret = 1
 

[OE-core][PATCH 2/3] patchtest/selftest: allow passing extra arguments to patchtest

[naftaly.ralamboarivony]

From: Naftaly RALAMBOARIVONY <naftaly.ralamboarivony@smile.fr>

Add an optional extra_args parameter to test() and append it to the
patchtest command when provided.

Signed-off-by: Naftaly RALAMBOARIVONY <naftaly.ralamboarivony@smile.fr>
---
 meta/lib/patchtest/selftest/selftest | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/lib/patchtest/selftest/selftest b/meta/lib/patchtest/selftest/selftest
index 43cccf4c85..51681bec38 100755
--- a/meta/lib/patchtest/selftest/selftest
+++ b/meta/lib/patchtest/selftest/selftest
@@ -130,11 +130,14 @@ def git_detach_head():
     return get_git_state()
 
 # Once the tests are in oe-core, we can remove the testdir param and use os.path.dirname to get relative paths
-def test(root, patch):
+def test(root, patch, extra_args=None):
     res = True
     patchpath = os.path.abspath(os.path.join(root, patch))
     
     cmd     = 'patchtest --base-commit HEAD --repodir %s --testdir %s/tests --patch %s' % (repodir, topdir, patchpath)
+    if extra_args:
+        cmd += " " + extra_args
+
     results = subprocess.check_output(cmd, stderr=subprocess.STDOUT, universal_newlines=True, shell=True)
 
     return results

[OE-core][PATCH 3/3] patchtest/selftest: add test case for --error-on-failure return code

[naftaly.ralamboarivony]

From: Naftaly RALAMBOARIVONY <naftaly.ralamboarivony@smile.fr>

Add a test case to verify that patchtest --error-on-failure
returns the correct exit code for PASS,FAIL and SKIP scenarios.

Signed-off-by: Naftaly RALAMBOARIVONY <naftaly.ralamboarivony@smile.fr>
---
 meta/lib/patchtest/selftest/selftest | 37 ++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/meta/lib/patchtest/selftest/selftest b/meta/lib/patchtest/selftest/selftest
index 51681bec38..39114a45a0 100755
--- a/meta/lib/patchtest/selftest/selftest
+++ b/meta/lib/patchtest/selftest/selftest
@@ -142,6 +142,42 @@ def test(root, patch, extra_args=None):
 
     return results
 
+def test_error_on_failure(root, patch):
+    try:
+        output = test(root, patch, "--error-on-failure")
+        return output, 0
+    except subprocess.CalledProcessError as e:
+        return e.output, e.returncode
+
+def test_returncode(patches, counts):
+
+    for target in ("FAIL", "PASS", "SKIP"):
+        for patch_info in patches:
+
+            testid = patch_info["testid"]
+            expected = str(patch_info["expected"]).upper()
+
+            if expected == target:
+                results, returncode = test_error_on_failure(
+                    patch_info["root"], patch_info["patch"]
+                )
+
+                if target == "FAIL" and returncode != 0:
+                    print("XFAIL: test_returncode.%s (file: %s)" % (testid.strip("."), os.path.basename(patch_info["patch"])))
+                    counts["xfail"] = counts["xfail"] + 1
+                elif target == "PASS" and returncode == 0:
+                    counts["xpass"] = counts["xpass"] + 1
+                    print("XPASS: test_returncode.%s (file: %s)" % (testid.strip("."), os.path.basename(patch_info["patch"])))
+                elif target == "SKIP" and returncode == 0:
+                    counts["xskip"] = counts["xskip"] + 1
+                    print("XSKIP: test_returncode.%s (file: %s)" % (testid.strip("."), os.path.basename(patch_info["patch"])))
+                else:
+                    print(f"Test failed: target '{target}', expected return code '{expected}', got '{returncode}'")
+
+                break
+
+    return counts
+
 def test_head_attached(patches, counts, branch):
 
     git_attach_head(branch)
@@ -178,6 +214,7 @@ def run_tests(patches, counts):
     counts = test_head_detached(patches, counts)
     restore_git_state(git_state)
     run_sh(f"git branch -D {temp_branch}")
+    counts = test_returncode(patches, counts)
 
     return counts
 

Re: [OE-core][PATCH 1/3] patchtest: add --error-on-failure option

[Richard Purdie]

On Wed, 2026-03-18 at 17:12 +0100, naftaly.ralamboarivony via lists.openembedded.org wrote:
> From: Naftaly RALAMBOARIVONY <naftaly.ralamboarivony@smile.fr>
> 
> Add a command line option to make patchtest return a non-zero exit
> status when a test fails.
> 
> This keeps the current default behavior unchanged while allowing calling
> scripts and shells to detect test failures through the process exit
> status when explicitly requested.

Is there anywhere we rely on the current behaviour?

I'm wondering if we shouldn't just change the existing behaviour?

I'm not against an option but adding complexity if we don't need it and
can just change the behaviour might be preferable...

Cheers,

Richard