[PATCH v1] dts: update overall result pass/skip logic

[PATCH v1] dts: update overall result pass/skip logic

[Dean Marx]

Update overall result of test suites such that
when some cases skip and at least one passes,
the result is a pass instead of a skip. Only
when all cases skip is the result a skip.

Bugzilla ID: 1899

Signed-off-by: Dean Marx <dmarx@iol.unh.edu>
---
 dts/framework/test_result.py | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index c6bddc55a9..0a43c23c04 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -195,10 +195,11 @@ def extract_result(value: ResultNode | ResultLeaf) -> ResultLeaf:
                 case ResultLeaf():
                     return value
 
-        return max(
-            (extract_result(child) for child in self.children),
-            default=ResultLeaf(result=Result.PASS),
-        )
+        results = [extract_result(child) for child in self.children]
+        max_result = max(results, default=ResultLeaf(result=Result.PASS))
+        if max_result.result == Result.SKIP and any(r.result == Result.PASS for r in results):
+            return ResultLeaf(result=Result.PASS)
+        return max_result
 
     def make_summary(self) -> Counter[Result]:
         """Make the summary of the underlying results while ignoring special nodes."""
-- 
2.52.0

Re: [PATCH v1] dts: update overall result pass/skip logic

[Andrew Bailey]

[-- Attachment #1: Type: text/plain, Size: 96 bytes --]

Tested-by: Andrew Bailey <abailey@iol.unh.edu>
Reviewed-by: Andrew Bailey <abailey@iol.unh.edu>

[-- Attachment #2: Type: text/html, Size: 235 bytes --]

[PATCH v2] dts: update overall result pass/skip logic

[Dean Marx]

Update overall result of test suites such that
when some cases skip and at least one passes,
the result is a pass instead of a skip. Only
when all cases skip is the result a skip.

Bugzilla ID: 1899

Signed-off-by: Dean Marx <dmarx@iol.unh.edu>
Tested-by: Andrew Bailey <abailey@iol.unh.edu>
Reviewed-by: Andrew Bailey <abailey@iol.unh.edu>
---
 dts/framework/test_result.py | 29 +++++++++++++++++------------
 1 file changed, 17 insertions(+), 12 deletions(-)

diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index c6bddc55a9..e05663f90e 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -187,18 +187,23 @@ def serialize_model(self) -> dict[str, Any]:
 
     def get_overall_result(self) -> ResultLeaf:
         """The overall result of the underlying results."""
-
-        def extract_result(value: ResultNode | ResultLeaf) -> ResultLeaf:
-            match value:
-                case ResultNode():
-                    return value.get_overall_result()
-                case ResultLeaf():
-                    return value
-
-        return max(
-            (extract_result(child) for child in self.children),
-            default=ResultLeaf(result=Result.PASS),
-        )
+        results = [
+            child.get_overall_result() if isinstance(child, ResultNode) else child
+            for child in self.children
+        ]
+        max_result = max(results, default=ResultLeaf(result=Result.PASS))
+
+        if max_result.result != Result.SKIP:
+            return max_result
+
+        if any(
+            r.result == Result.PASS
+            for child, r in zip(self.children, results)
+            if not (isinstance(child, ResultNode) and child.label in self.__ignore_steps)
+        ):
+            return ResultLeaf(result=Result.PASS)
+
+        return max_result
 
     def make_summary(self) -> Counter[Result]:
         """Make the summary of the underlying results while ignoring special nodes."""
-- 
2.52.0

Re: [PATCH v2] dts: update overall result pass/skip logic

[Patrick Robb]

Recheck-request: github-robot

Re: [PATCH v2] dts: update overall result pass/skip logic

[Patrick Robb]

[-- Attachment #1: Type: text/plain, Size: 3017 bytes --]

Okay, the implementation makes sense and I see that it works.

However, isn't the whole issue stemming from the fact that the currently
existing get_overall_result() function calls max() on self.children, and
self.children (the result objects) are IntEnum enums which are ordered in
ascending order, starting with PASS, then SKIP, then BLOCK, then FAIL, then
ERROR? If we just swap PASS and SKIP, doesn't that make the existing logic
(using the max() function call) work correctly?

I.e. re-order this class:

class Result(IntEnum):
    """The possible states that a setup, a teardown or a test case may end
up in."""

    #:
    PASS = auto()
    #:
    SKIP = auto()
    #:
    BLOCK = auto()
    #:
    FAIL = auto()
    #:
    ERROR = auto()



On Fri, Mar 13, 2026 at 2:22 PM Dean Marx <dmarx@iol.unh.edu> wrote:

> Update overall result of test suites such that
> when some cases skip and at least one passes,
> the result is a pass instead of a skip. Only
> when all cases skip is the result a skip.
>
> Bugzilla ID: 1899
>
> Signed-off-by: Dean Marx <dmarx@iol.unh.edu>
> Tested-by: Andrew Bailey <abailey@iol.unh.edu>
> Reviewed-by: Andrew Bailey <abailey@iol.unh.edu>
> ---
>  dts/framework/test_result.py | 29 +++++++++++++++++------------
>  1 file changed, 17 insertions(+), 12 deletions(-)
>
> diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
> index c6bddc55a9..e05663f90e 100644
> --- a/dts/framework/test_result.py
> +++ b/dts/framework/test_result.py
> @@ -187,18 +187,23 @@ def serialize_model(self) -> dict[str, Any]:
>
>      def get_overall_result(self) -> ResultLeaf:
>          """The overall result of the underlying results."""
> -
> -        def extract_result(value: ResultNode | ResultLeaf) -> ResultLeaf:
> -            match value:
> -                case ResultNode():
> -                    return value.get_overall_result()
> -                case ResultLeaf():
> -                    return value
> -
> -        return max(
> -            (extract_result(child) for child in self.children),
> -            default=ResultLeaf(result=Result.PASS),
> -        )
> +        results = [
> +            child.get_overall_result() if isinstance(child, ResultNode)
> else child
> +            for child in self.children
> +        ]
> +        max_result = max(results, default=ResultLeaf(result=Result.PASS))
> +
> +        if max_result.result != Result.SKIP:
> +            return max_result
> +
> +        if any(
> +            r.result == Result.PASS
> +            for child, r in zip(self.children, results)
> +            if not (isinstance(child, ResultNode) and child.label in
> self.__ignore_steps)
> +        ):
> +            return ResultLeaf(result=Result.PASS)
> +
> +        return max_result
>
>      def make_summary(self) -> Counter[Result]:
>          """Make the summary of the underlying results while ignoring
> special nodes."""
> --
> 2.52.0
>
>

[-- Attachment #2: Type: text/html, Size: 3986 bytes --]

Re: [PATCH v2] dts: update overall result pass/skip logic

[Luca Vizzarro]

Hi Dean,

Thank you for your patch. I agree with Patrick on this one. The overall 
result is indeed obtained by doing max on the results, which would yield 
the maximum severity found. Technically speaking we could indeed say 
that SKIP is less severe or important than PASS.

If we have a test suite with only SKIP the max will be SKIP, if we have 
a test suite with SKIP and PASS, the max will be PASS.

The rest of the logic would remain unchanged, meaning that if we have 
SKIP, PASS, FAIL, then the max will still be FAIL.

Therefore, I agree that the solution to this is as simple as re-ordering 
the IntEnum, by swapping SKIP and PASS.

Best,
Luca

Re: [PATCH v2] dts: update overall result pass/skip logic

[Dean Marx]

Fair enough, I wasn't sure if reordering the enum would impact
something outside of the scope of this patch, but since it doesn't
that makes it easy. I'll submit a new version

[PATCH v3] dts: update overall result pass/skip logic

[Dean Marx]

Update overall result of test suites such that
when some cases skip and at least one passes,
the result is a pass instead of a skip. Only
when all cases skip is the result a skip.

Bugzilla ID: 1899

Signed-off-by: Dean Marx <dmarx@iol.unh.edu>
Tested-by: Andrew Bailey <abailey@iol.unh.edu>
Reviewed-by: Andrew Bailey <abailey@iol.unh.edu>
---
 dts/framework/test_result.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index c6bddc55a9..66cd67fd48 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -45,11 +45,11 @@
 class Result(IntEnum):
     """The possible states that a setup, a teardown or a test case may end up in."""
 
-    #:
-    PASS = auto()
     #:
     SKIP = auto()
     #:
+    PASS = auto()
+    #:
     BLOCK = auto()
     #:
     FAIL = auto()
-- 
2.52.0

[PATCH v4] dts: update overall result pass/skip logic

[Dean Marx]

Update overall result of test suites such that
when some cases skip and at least one passes,
the result is a pass instead of a skip. Only
when all cases skip is the result a skip.

Bugzilla ID: 1899

Signed-off-by: Dean Marx <dmarx@iol.unh.edu>
Tested-by: Andrew Bailey <abailey@iol.unh.edu>
Reviewed-by: Andrew Bailey <abailey@iol.unh.edu>
---
 dts/framework/test_result.py | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/dts/framework/test_result.py b/dts/framework/test_result.py
index c6bddc55a9..21faa55dc1 100644
--- a/dts/framework/test_result.py
+++ b/dts/framework/test_result.py
@@ -45,11 +45,11 @@
 class Result(IntEnum):
     """The possible states that a setup, a teardown or a test case may end up in."""
 
-    #:
-    PASS = auto()
     #:
     SKIP = auto()
     #:
+    PASS = auto()
+    #:
     BLOCK = auto()
     #:
     FAIL = auto()
@@ -195,8 +195,15 @@ def extract_result(value: ResultNode | ResultLeaf) -> ResultLeaf:
                 case ResultLeaf():
                     return value
 
+        # Filter out setup/teardown steps
+        results = [
+            extract_result(child)
+            for child in self.children
+            if not (isinstance(child, ResultNode) and child.label in self.__ignore_steps)
+        ]
+
         return max(
-            (extract_result(child) for child in self.children),
+            results,
             default=ResultLeaf(result=Result.PASS),
         )
 
-- 
2.52.0

Re: [PATCH v4] dts: update overall result pass/skip logic

[Patrick Robb]

[-- Attachment #1: Type: text/plain, Size: 46 bytes --]

Reviewed-by: Patrick Robb <probb@iol.unh.edu>

[-- Attachment #2: Type: text/html, Size: 112 bytes --]

Re: [PATCH v4] dts: update overall result pass/skip logic

[Luca Vizzarro]

This is ok by me, but let's make sure the new behaviour is documented 
(in the commit description)

Reviewed-by: Luca Vizzarro <luca.vizzarro@arm.com>