[PATCH] oeqa/sdk/sdktest: Add test discovery across layers

[PATCH] oeqa/sdk/sdktest: Add test discovery across layers

[Thune Tran]

The testsdk task only loads SDK tests from the openembedded-core
layer's meta/lib/oeqa/sdk/cases/.

This patch implements test discovery to include SDK tests from all
configured layers that mirror <LAYER_DIR>/lib/oeqa/sdk/cases/. The
implementation mirrors get_runtime_paths() in meta/classes-recipe/testimage.

Tested by:

1. Adding SDK test cases to <LAYER_DIR>/lib/oeqa/sdk/cases/ and
   verifying they are discovered and executed with no errors.
2. Verifying existing SDK test cases in openembedded-core layer
   are discovered and executed with no errors.

Signed-off-by: Thune Tran <thune.a.tran@boeing.com>
Signed-off-by: Chuck Wolber <chuck.wolber@boeing.com>
---
 meta/lib/oeqa/sdk/testsdk.py | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/sdk/testsdk.py b/meta/lib/oeqa/sdk/testsdk.py
index 518b09febb..505add8f8c 100644
--- a/meta/lib/oeqa/sdk/testsdk.py
+++ b/meta/lib/oeqa/sdk/testsdk.py
@@ -31,6 +31,20 @@ class TestSDK(TestSDKBase):
     context_class = OESDKTestContext
     test_type = 'sdk'
 
+    def get_sdk_paths(self, d):
+        """
+        Returns a list of paths where SDK test must reside.
+
+        SDK tests are expected in <LAYER_DIR>/lib/oeqa/sdk/cases/
+        """
+        paths = []
+
+        for layer in d.getVar('BBLAYERS').split():
+            path = os.path.join(layer, 'lib/oeqa/sdk/cases')
+            if os.path.isdir(path):
+                paths.append(path)
+        return paths
+
     def get_tcname(self, d):
         """
         Get the name of the SDK file
@@ -114,7 +128,7 @@ class TestSDK(TestSDKBase):
                 host_pkg_manifest=host_pkg_manifest, **context_args)
 
             try:
-                tc.loadTests(self.context_executor_class.default_cases)
+                tc.loadTests(self.get_sdk_paths(d))
             except Exception as e:
                 import traceback
                 bb.fatal("Loading tests failed:\n%s" % traceback.format_exc())
-- 
2.34.1

Re: [OE-core] [PATCH] oeqa/sdk/sdktest: Add test discovery across layers

[Gyorgy Sarvari]

The same issue is also present for sdkext - since you are already on
this, do you think you could make that work too?

On 4/16/25 18:41, Thune Tran via lists.openembedded.org wrote:
> The testsdk task only loads SDK tests from the openembedded-core
> layer's meta/lib/oeqa/sdk/cases/.
>
> This patch implements test discovery to include SDK tests from all
> configured layers that mirror <LAYER_DIR>/lib/oeqa/sdk/cases/. The
> implementation mirrors get_runtime_paths() in meta/classes-recipe/testimage.
>
> Tested by:
>
> 1. Adding SDK test cases to <LAYER_DIR>/lib/oeqa/sdk/cases/ and
>    verifying they are discovered and executed with no errors.
> 2. Verifying existing SDK test cases in openembedded-core layer
>    are discovered and executed with no errors.
>
> Signed-off-by: Thune Tran <thune.a.tran@boeing.com>
> Signed-off-by: Chuck Wolber <chuck.wolber@boeing.com>
> ---
>  meta/lib/oeqa/sdk/testsdk.py | 16 +++++++++++++++-
>  1 file changed, 15 insertions(+), 1 deletion(-)
>
> diff --git a/meta/lib/oeqa/sdk/testsdk.py b/meta/lib/oeqa/sdk/testsdk.py
> index 518b09febb..505add8f8c 100644
> --- a/meta/lib/oeqa/sdk/testsdk.py
> +++ b/meta/lib/oeqa/sdk/testsdk.py
> @@ -31,6 +31,20 @@ class TestSDK(TestSDKBase):
>      context_class = OESDKTestContext
>      test_type = 'sdk'
>  
> +    def get_sdk_paths(self, d):
> +        """
> +        Returns a list of paths where SDK test must reside.
> +
> +        SDK tests are expected in <LAYER_DIR>/lib/oeqa/sdk/cases/
> +        """
> +        paths = []
> +
> +        for layer in d.getVar('BBLAYERS').split():
> +            path = os.path.join(layer, 'lib/oeqa/sdk/cases')
> +            if os.path.isdir(path):
> +                paths.append(path)
> +        return paths
> +
>      def get_tcname(self, d):
>          """
>          Get the name of the SDK file
> @@ -114,7 +128,7 @@ class TestSDK(TestSDKBase):
>                  host_pkg_manifest=host_pkg_manifest, **context_args)
>  
>              try:
> -                tc.loadTests(self.context_executor_class.default_cases)
> +                tc.loadTests(self.get_sdk_paths(d))
>              except Exception as e:
>                  import traceback
>                  bb.fatal("Loading tests failed:\n%s" % traceback.format_exc())
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#215001): https://lists.openembedded.org/g/openembedded-core/message/215001
> Mute This Topic: https://lists.openembedded.org/mt/112298220/6084445
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [skandigraun@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH] oeqa/sdk/sdktest: Add test discovery across layers

[Tran (US), Thune A]

> The same issue is also present for sdkext - since you are already on
> this, do you think you could make that work too?

Thanks for pointing that out. A quick look at sdkext. It's possible to make 
it work for it. I'm open to submitting a similar patch for sdkext. I'm not 
immediately set up to test it. Would you be able to help verify the patch 
for sdkext and also sign off on it?
> On 4/16/25 18:41, Thune Tran via lists.openembedded.org wrote:
>> The testsdk task only loads SDK tests from the openembedded-core
>> layer's meta/lib/oeqa/sdk/cases/.
>> 
>> This patch implements test discovery to include SDK tests from all
>> configured layers that mirror <LAYER_DIR>/lib/oeqa/sdk/cases/. The
>> implementation mirrors get_runtime_paths() in meta/classes-
>> recipe/testimage.
>> 
>> Tested by:
>> 
>> 1. Adding SDK test cases to <LAYER_DIR>/lib/oeqa/sdk/cases/ and
>>    verifying they are discovered and executed with no errors. 2.
>>    Verifying existing SDK test cases in openembedded-core layer are
>>    discovered and executed with no errors.
>> Signed-off-by: Thune Tran <thune.a.tran@boeing.com>
>> Signed-off-by: Chuck Wolber <chuck.wolber@boeing.com>
>> ---
>>  meta/lib/oeqa/sdk/testsdk.py | 16 +++++++++++++++-
>>  1 file changed, 15 insertions(+), 1 deletion(-)
>> diff --git a/meta/lib/oeqa/sdk/testsdk.py b/meta/lib/oeqa/sdk/testsdk.py
>> index 518b09febb..505add8f8c 100644
>> --- a/meta/lib/oeqa/sdk/testsdk.py
>> +++ b/meta/lib/oeqa/sdk/testsdk.py
>> @@ -31,6 +31,20 @@ class TestSDK(TestSDKBase):
>>      context_class = OESDKTestContext
>>      test_type = 'sdk'
>> +    def get_sdk_paths(self, d):
>> +        """
>> +        Returns a list of paths where SDK test must reside.
>> +
>> +        SDK tests are expected in <LAYER_DIR>/lib/oeqa/sdk/cases/
>> +        """
>> +        paths = []
>> +
>> +        for layer in d.getVar('BBLAYERS').split():
>> +            path = os.path.join(layer, 'lib/oeqa/sdk/cases')
>> +            if os.path.isdir(path):
>> +                paths.append(path)
>> +        return paths
>> +
>>      def get_tcname(self, d):
>>          """
>>          Get the name of the SDK file
>> @@ -114,7 +128,7 @@ class TestSDK(TestSDKBase):
>>                  host_pkg_manifest=host_pkg_manifest, **context_args)
>>              try:
>> -                tc.loadTests(self.context_executor_class.default_cases)
>> +                tc.loadTests(self.get_sdk_paths(d))
>>              except Exception as e:
>>                  import traceback
>>                  bb.fatal("Loading tests failed:\n%s" % traceback.format_exc())
>> 
>>

Re: [OE-core] [PATCH] oeqa/sdk/sdktest: Add test discovery across layers

[Gyorgy Sarvari]

On 4/16/25 22:54, Tran (US), Thune A wrote:
>> The same issue is also present for sdkext - since you are already on
>> this, do you think you could make that work too?
> Thanks for pointing that out. A quick look at sdkext. It's possible to make 
> it work for it. I'm open to submitting a similar patch for sdkext. I'm not 
> immediately set up to test it. Would you be able to help verify the patch 
> for sdkext and also sign off on it?

[Just to have it here also, not only in private messages]

Yeah, I think I should be able to do that - I have an extensible sdk set
up, and some classes also that I can use for testing.


>> On 4/16/25 18:41, Thune Tran via lists.openembedded.org wrote:
>>> The testsdk task only loads SDK tests from the openembedded-core
>>> layer's meta/lib/oeqa/sdk/cases/.
>>>
>>> This patch implements test discovery to include SDK tests from all
>>> configured layers that mirror <LAYER_DIR>/lib/oeqa/sdk/cases/. The
>>> implementation mirrors get_runtime_paths() in meta/classes-
>>> recipe/testimage.
>>>
>>> Tested by:
>>>
>>> 1. Adding SDK test cases to <LAYER_DIR>/lib/oeqa/sdk/cases/ and
>>>    verifying they are discovered and executed with no errors. 2.
>>>    Verifying existing SDK test cases in openembedded-core layer are
>>>    discovered and executed with no errors.
>>> Signed-off-by: Thune Tran <thune.a.tran@boeing.com>
>>> Signed-off-by: Chuck Wolber <chuck.wolber@boeing.com>
>>> ---
>>>  meta/lib/oeqa/sdk/testsdk.py | 16 +++++++++++++++-
>>>  1 file changed, 15 insertions(+), 1 deletion(-)
>>> diff --git a/meta/lib/oeqa/sdk/testsdk.py b/meta/lib/oeqa/sdk/testsdk.py
>>> index 518b09febb..505add8f8c 100644
>>> --- a/meta/lib/oeqa/sdk/testsdk.py
>>> +++ b/meta/lib/oeqa/sdk/testsdk.py
>>> @@ -31,6 +31,20 @@ class TestSDK(TestSDKBase):
>>>      context_class = OESDKTestContext
>>>      test_type = 'sdk'
>>> +    def get_sdk_paths(self, d):
>>> +        """
>>> +        Returns a list of paths where SDK test must reside.
>>> +
>>> +        SDK tests are expected in <LAYER_DIR>/lib/oeqa/sdk/cases/
>>> +        """
>>> +        paths = []
>>> +
>>> +        for layer in d.getVar('BBLAYERS').split():
>>> +            path = os.path.join(layer, 'lib/oeqa/sdk/cases')
>>> +            if os.path.isdir(path):
>>> +                paths.append(path)
>>> +        return paths
>>> +
>>>      def get_tcname(self, d):
>>>          """
>>>          Get the name of the SDK file
>>> @@ -114,7 +128,7 @@ class TestSDK(TestSDKBase):
>>>                  host_pkg_manifest=host_pkg_manifest, **context_args)
>>>              try:
>>> -                tc.loadTests(self.context_executor_class.default_cases)
>>> +                tc.loadTests(self.get_sdk_paths(d))
>>>              except Exception as e:
>>>                  import traceback
>>>                  bb.fatal("Loading tests failed:\n%s" % traceback.format_exc())
>>>
>>>
>

Re: [OE-core] [PATCH] oeqa/sdk/sdktest: Add test discovery across layers

[Richard Purdie]

On Wed, 2025-04-16 at 16:41 +0000, Thune Tran via lists.openembedded.org wrote:
> The testsdk task only loads SDK tests from the openembedded-core
> layer's meta/lib/oeqa/sdk/cases/.
> 
> This patch implements test discovery to include SDK tests from all
> configured layers that mirror <LAYER_DIR>/lib/oeqa/sdk/cases/. The
> implementation mirrors get_runtime_paths() in meta/classes-recipe/testimage.
> 
> Tested by:
> 
> 1. Adding SDK test cases to <LAYER_DIR>/lib/oeqa/sdk/cases/ and
>    verifying they are discovered and executed with no errors.
> 2. Verifying existing SDK test cases in openembedded-core layer
>    are discovered and executed with no errors.
> 
> Signed-off-by: Thune Tran <thune.a.tran@boeing.com>
> Signed-off-by: Chuck Wolber <chuck.wolber@boeing.com>
> ---
>  meta/lib/oeqa/sdk/testsdk.py | 16 +++++++++++++++-
>  1 file changed, 15 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/lib/oeqa/sdk/testsdk.py b/meta/lib/oeqa/sdk/testsdk.py
> index 518b09febb..505add8f8c 100644
> --- a/meta/lib/oeqa/sdk/testsdk.py
> +++ b/meta/lib/oeqa/sdk/testsdk.py
> @@ -31,6 +31,20 @@ class TestSDK(TestSDKBase):
>      context_class = OESDKTestContext
>      test_type = 'sdk'
>  
> +    def get_sdk_paths(self, d):
> +        """
> +        Returns a list of paths where SDK test must reside.
> +
> +        SDK tests are expected in <LAYER_DIR>/lib/oeqa/sdk/cases/
> +        """
> +        paths = []
> +
> +        for layer in d.getVar('BBLAYERS').split():
> +            path = os.path.join(layer, 'lib/oeqa/sdk/cases')
> +            if os.path.isdir(path):
> +                paths.append(path)
> +        return paths
> +
>      def get_tcname(self, d):
>          """
>          Get the name of the SDK file
> @@ -114,7 +128,7 @@ class TestSDK(TestSDKBase):
>                  host_pkg_manifest=host_pkg_manifest, **context_args)
>  
>              try:
> -                tc.loadTests(self.context_executor_class.default_cases)
> +                tc.loadTests(self.get_sdk_paths(d))
>              except Exception as e:
>                  import traceback
>                  bb.fatal("Loading tests failed:\n%s" % traceback.format_exc())
> 

I suspect this causes some regressions in our automated testing unfortunately:

buildtools:
https://autobuilder.yoctoproject.org/valkyrie/#/builders/43/builds/1396

meta-mingw:
https://autobuilder.yoctoproject.org/valkyrie/#/builders/7/builds/1430

Cheers,

Richard

Re: [OE-core] [PATCH] oeqa/sdk/sdktest: Add test discovery across layers

[Chuck Wolber]

[-- Attachment #1: Type: text/plain, Size: 808 bytes --]

On Wed, Apr 16, 2025 at 3:07 PM Richard Purdie via lists.openembedded.org
<richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:

%< SNIP $<

>
> I suspect this causes some regressions in our automated testing
> unfortunately:
>
> buildtools:
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/43/builds/1396
>
> meta-mingw:
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/7/builds/1430


Would it make more sense to add self.context_executor_class.default_cases
to the list of paths returned by get_sdk_paths?

..Ch:W..


-- 



*  Opinions expressed are my own. | --------------------------------|------
"Perfection must be reached by degrees              she requires the slow
hand of time."                                 |       -Voltaire*

[-- Attachment #2: Type: text/html, Size: 2116 bytes --]

Re: [OE-core] [PATCH] oeqa/sdk/sdktest: Add test discovery across layers

[Gyorgy Sarvari]

On 4/17/25 06:39, Chuck Wolber via lists.openembedded.org wrote:
> On Wed, Apr 16, 2025 at 3:07 PM Richard Purdie via
> lists.openembedded.org <http://lists.openembedded.org>
> <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
>
> %< SNIP $<
>
>
>     I suspect this causes some regressions in our automated testing
>     unfortunately:
>
>     buildtools:
>     https://autobuilder.yoctoproject.org/valkyrie/#/builders/43/builds/1396
>
>     meta-mingw:
>     https://autobuilder.yoctoproject.org/valkyrie/#/builders/7/builds/1430
>
>
> Would it make more sense to add self.context_executor_class.default_cases
> to the list of paths returned by get_sdk_paths?
>
Well, the missing tests is one thing, but the other one is that the
errors are thrown by tests that were not meant to be executed (at least
when it comes to buildtools - haven't checked meta-mingw yet).

> ..Ch:W..
>
>  
> -- 
> *  Opinions expressed are my own. |
>  --------------------------------|------ "Perfection must be reached
> by degrees
>               she requires the slow hand of time."
>                                  |       -Voltaire*

Re: [OE-core] [PATCH] oeqa/sdk/sdktest: Add test discovery across layers

[Richard Purdie]

On Wed, 2025-04-16 at 21:39 -0700, Chuck Wolber wrote:
> On Wed, Apr 16, 2025 at 3:07 PM Richard Purdie via
> lists.openembedded.org
> <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
> 
> %< SNIP $< 
> > 
> > I suspect this causes some regressions in our automated testing
> > unfortunately:
> > 
> > buildtools:
> > https://autobuilder.yoctoproject.org/valkyrie/#/builders/43/builds/1396
> > 
> > meta-mingw:
> > https://autobuilder.yoctoproject.org/valkyrie/#/builders/7/builds/1430
> > 
> 
> 
> Would it make more sense to add
> self.context_executor_class.default_cases
> to the list of paths returned by get_sdk_paths?

meta/recipes-core/meta/buildtools-tarball.bb does:

    cases_path = os.path.join(os.path.abspath(os.path.dirname(oeqa.sdk.testsdk.__file__)), d.getVar("TESTSDK_CASES"))
    testsdk.context_executor_class.default_cases = cases_path

mingw does things in a different place, setting:

    default_cases = [os.path.join(os.path.abspath(os.path.dirname(__file__)), 'cases')]

in meta-mingw/lib/oeqa/sdkmingw/contex.py

I think we need to add something which represents the name of the
directory for cases, i.e. "sdk", "sdkming", "sdk/buildtools-cases" and
then allow this to be passed into your function so regardless, it
allows customisation within a layer.

Just adding default cases to anything in the layer is going to always
add the layer's test cases which isn't what we want (i.e. your layer
would add to buildtools, mingw and sdk).

Cheers,

Richard

Re: [OE-core] [PATCH] oeqa/sdk/sdktest: Add test discovery across layers

[Tran (US), Thune A]

> On Wed, 2025-04-16 at 21:39 -0700, Chuck Wolber wrote:
>> On Wed, Apr 16, 2025 at 3:07 PM Richard Purdie via
>> lists.openembedded.org
>> <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
>> 
>> %< SNIP $<
>>> 
>>> I suspect this causes some regressions in our automated testing
>>> unfortunately:
>>> 
>>> buildtools:
>>> https://autobuilder.yoctoproject.org/valkyrie/#/builders/43/builds/1
>>> 396
>>> 
>>> meta-mingw:
>>> https://autobuilder.yoctoproject.org/valkyrie/#/builders/7/builds/14
>>> 30
>>> 
>> 
>> 
>> Would it make more sense to add
>> self.context_executor_class.default_cases
>> to the list of paths returned by get_sdk_paths?
> 
> meta/recipes-core/meta/buildtools-tarball.bb does:
> 
>     cases_path =
> os.path.join(os.path.abspath(os.path.dirname(oeqa.sdk.testsdk.__file__)),
> d.getVar("TESTSDK_CASES"))
>     testsdk.context_executor_class.default_cases = cases_path
> mingw does things in a different place, setting:
> 
>     default_cases = [os.path.join(os.path.abspath(os.path.dirname(__file__)),
> 'cases')]
> 
> in meta-mingw/lib/oeqa/sdkmingw/contex.py
> 
> I think we need to add something which represents the name of the
> directory for cases, i.e. "sdk", "sdkming", "sdk/buildtools-cases" and
> then allow this to be passed into your function so regardless, it allows
> customisation within a layer.
We can match the test directory pattern (sdk, sdkming, sdk/buildtools-cases)  
found in default_cases. The test discovery will then be based on the matched pattern.
> 
> Just adding default cases to anything in the layer is going to always add the
> layer's test cases which isn't what we want (i.e. your layer would add to
> buildtools, mingw and sdk).
> 
> Cheers,
> 
> Richard

Re: [OE-core] [PATCH] oeqa/sdk/sdktest: Add test discovery across layers

[Richard Purdie]

On Thu, 2025-04-17 at 12:54 +0000, Thune Tran via lists.openembedded.org wrote:
> > On Wed, 2025-04-16 at 21:39 -0700, Chuck Wolber wrote:
> > > On Wed, Apr 16, 2025 at 3:07 PM Richard Purdie via
> > > lists.openembedded.org
> > > <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
> > > 
> > > %< SNIP $<
> > > > 
> > > > I suspect this causes some regressions in our automated testing
> > > > unfortunately:
> > > > 
> > > > buildtools:
> > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/43/builds/1
> > > > 396
> > > > 
> > > > meta-mingw:
> > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/7/builds/14
> > > > 30
> > > > 
> > > 
> > > 
> > > Would it make more sense to add
> > > self.context_executor_class.default_cases
> > > to the list of paths returned by get_sdk_paths?
> > 
> > meta/recipes-core/meta/buildtools-tarball.bb does:
> > 
> >     cases_path =
> > os.path.join(os.path.abspath(os.path.dirname(oeqa.sdk.testsdk.__file__)),
> > d.getVar("TESTSDK_CASES"))
> >     testsdk.context_executor_class.default_cases = cases_path
> > mingw does things in a different place, setting:
> > 
> >     default_cases = [os.path.join(os.path.abspath(os.path.dirname(__file__)),
> > 'cases')]
> > 
> > in meta-mingw/lib/oeqa/sdkmingw/contex.py
> > 
> > I think we need to add something which represents the name of the
> > directory for cases, i.e. "sdk", "sdkming", "sdk/buildtools-cases" and
> > then allow this to be passed into your function so regardless, it allows
> > customisation within a layer.
> We can match the test directory pattern (sdk, sdkming, sdk/buildtools-cases)  
> found in default_cases. The test discovery will then be based on the matched pattern.
> 

We could do that but in this case I think it might be better to improve
the code and make it a bit cleaner if we can, even if that is a little
more invasive.

This area of the code could do with simplification/cleanup...

Cheers,

Richard

Re: [OE-core] [PATCH] oeqa/sdk/sdktest: Add test discovery across layers

[Tran (US), Thune A]

> On Thu, 2025-04-17 at 12:54 +0000, Thune Tran via lists.openembedded.org
> wrote:
>>> On Wed, 2025-04-16 at 21:39 -0700, Chuck Wolber wrote:
>>>> On Wed, Apr 16, 2025 at 3:07 PM Richard Purdie via
>>>> lists.openembedded.org
>>>> <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
>>>> 
>>>> %< SNIP $<
>>>>> 
>>>>> I suspect this causes some regressions in our automated testing
>>>>> unfortunately:
>>>>> 
>>>>> buildtools:
>>>>> https://autobuilder.yoctoproject.org/valkyrie/#/builders/43/buil
>>>>> ds/1
>>>>> 396
>>>>> 
>>>>> meta-mingw:
>>>>> https://autobuilder.yoctoproject.org/valkyrie/#/builders/7/build
>>>>> s/14
>>>>> 30
>>>>> 
>>>> 
>>>> 
>>>> Would it make more sense to add
>>>> self.context_executor_class.default_cases
>>>> to the list of paths returned by get_sdk_paths?
>>> 
>>> meta/recipes-core/meta/buildtools-tarball.bb does:
>>> 
>>>     cases_path =
>>> os.path.join(os.path.abspath(os.path.dirname(oeqa.sdk.testsdk.__file
>>> __)),
>>> d.getVar("TESTSDK_CASES"))
>>>     testsdk.context_executor_class.default_cases = cases_path mingw
>>> does things in a different place, setting:
>>> 
>>>     default_cases =
>>> [os.path.join(os.path.abspath(os.path.dirname(__file__)),
>>> 'cases')]
>>> 
>>> in meta-mingw/lib/oeqa/sdkmingw/contex.py
>>> 
>>> I think we need to add something which represents the name of the
>>> directory for cases, i.e. "sdk", "sdkming", "sdk/buildtools-cases"
>>> and then allow this to be passed into your function so regardless,
>>> it allows customisation within a layer.
>> We can match the test directory pattern (sdk, sdkming,
>> sdk/buildtools-cases) found in default_cases. The test discovery will
>> then be based on the matched pattern.
>> 
> 
> We could do that but in this case I think it might be better to improve
> the code and make it a bit cleaner if we can, even if that is a little
> more invasive.
> 
> This area of the code could do with simplification/cleanup...

Makes sense. We'll look into how the code can be improved as we continue 
to work in and become more familiar with this area of the code.
> Cheers,
> 
> Richard

RE: [OE-core] [PATCH] oeqa/sdk/sdktest: Add test discovery across layers

[Tran (US), Thune A]

>> On Thu, 2025-04-17 at 12:54 +0000, Thune Tran via 
>> lists.openembedded.org
>> wrote:
>>>> On Wed, 2025-04-16 at 21:39 -0700, Chuck Wolber wrote:
>>>>> On Wed, Apr 16, 2025 at 3:07 PM Richard Purdie via 
>>>>> lists.openembedded.org 
>>>>> <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
>>>>> 
>>>>> %< SNIP $<
>>>>>> 
>>>>>> I suspect this causes some regressions in our automated testing
>>>>>> unfortunately:
>>>>>> 
>>>>>> buildtools:
>>>>>> https://autobuilder.yoctoproject.org/valkyrie/#/builders/43/buil
>>>>>> ds/1
>>>>>> 396
>>>>>> 
>>>>>> meta-mingw:
>>>>>> https://autobuilder.yoctoproject.org/valkyrie/#/builders/7/build
>>>>>> s/14
>>>>>> 30
>>>>>> 
>>>>> 
>>>>> 
>>>>> Would it make more sense to add
>>>>> self.context_executor_class.default_cases
>>>>> to the list of paths returned by get_sdk_paths?
>>>> 
>>>> meta/recipes-core/meta/buildtools-tarball.bb does:
>>>> 
>>>>     cases_path =
>>>> os.path.join(os.path.abspath(os.path.dirname(oeqa.sdk.testsdk.__fil
>>>> e
>>>> __)),
>>>> d.getVar("TESTSDK_CASES"))
>>>>     testsdk.context_executor_class.default_cases = cases_path mingw 
>>>> does things in a different place, setting:
>>>> 
>>>>     default_cases =
>>>> [os.path.join(os.path.abspath(os.path.dirname(__file__)),
>>>> 'cases')]
>>>> 
>>>> in meta-mingw/lib/oeqa/sdkmingw/contex.py
>>>> 
>>>> I think we need to add something which represents the name of the 
>>>> directory for cases, i.e. "sdk", "sdkming", "sdk/buildtools-cases"
>>>> and then allow this to be passed into your function so regardless, 
>>>> it allows customisation within a layer.
>>> We can match the test directory pattern (sdk, sdkming,
>>> sdk/buildtools-cases) found in default_cases. The test discovery 
>>> will then be based on the matched pattern.
>>> 
>> 
>> We could do that but in this case I think it might be better to 
>> improve the code and make it a bit cleaner if we can, even if that is 
>> a little more invasive.
>> 
>> This area of the code could do with simplification/cleanup...
> 
> Makes sense. We'll look into how the code can be improved as we 
> continue to work in and become more familiar with this area of the code.

Looked into it and found some places to improve on, so planning a  PATCHv2 oeqa/sdk to simplify the way tests are specified for the SDK configurations.
>> Cheers,
>> 
>> Richard
>