[PATCH v3 1/5] oeqa runtime: set self.runner and handle None

[PATCH v3 1/5] oeqa runtime: set self.runner and handle None

[Mikko Rapeli]

Set default self.runner to None. qemu target sets
the runner to qemu. Then handle self.runner None in
run_network_serialdebug(). This way ssh runner
and failing ping or ssh tests handle the error cases.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/lib/oeqa/core/target/__init__.py | 1 +
 meta/lib/oeqa/runtime/case.py         | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/meta/lib/oeqa/core/target/__init__.py b/meta/lib/oeqa/core/target/__init__.py
index 1382aa9b5239..177f648fe358 100644
--- a/meta/lib/oeqa/core/target/__init__.py
+++ b/meta/lib/oeqa/core/target/__init__.py
@@ -10,6 +10,7 @@ class OETarget(object):
 
     def __init__(self, logger, *args, **kwargs):
         self.logger = logger
+        self.runner = None
 
     @abstractmethod
     def start(self):
diff --git a/meta/lib/oeqa/runtime/case.py b/meta/lib/oeqa/runtime/case.py
index 9515ca2f3d62..2a47771a3d47 100644
--- a/meta/lib/oeqa/runtime/case.py
+++ b/meta/lib/oeqa/runtime/case.py
@@ -23,6 +23,8 @@ class OERuntimeTestCase(OETestCase):
         uninstall_package(self)
 
 def run_network_serialdebug(runner):
+    if not runner:
+        return
     status, output = runner.run_serial("ip addr")
     print("ip addr on target: %s %s" % (output, status))
     status, output = runner.run_serial("ping -c 1 %s" % self.target.server_ip)
-- 
2.43.0

[PATCH v3 2/5] oeqa context.py: use TEST_SUITES if set

[Mikko Rapeli]

If build target has set TEST_SUITES, then that should
be the default test modules to execute. Fixes testexport.bbclass
to run same tests as testimage.bbclass which already
uses TEST_SUITES.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/lib/oeqa/core/context.py | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/meta/lib/oeqa/core/context.py b/meta/lib/oeqa/core/context.py
index 9313271f5865..46de9135c179 100644
--- a/meta/lib/oeqa/core/context.py
+++ b/meta/lib/oeqa/core/context.py
@@ -179,9 +179,16 @@ class OETestContextExecutor(object):
         else:
             self.tc_kwargs['init']['td'] = {}
 
+        # Run image specific TEST_SUITE like testimage.bbclass by default
+        test_suites = self.tc_kwargs['init']['td'].get("TEST_SUITES")
+        if test_suites:
+            test_suites = test_suites.split()
+
         if args.run_tests:
             self.tc_kwargs['load']['modules'] = args.run_tests
             self.tc_kwargs['load']['modules_required'] = args.run_tests
+        elif test_suites:
+            self.tc_kwargs['load']['modules'] = test_suites
         else:
             self.tc_kwargs['load']['modules'] = []
 
-- 
2.43.0

[PATCH v3 3/5] testexport.bbclass oe-test: capture all tests and data from all layers

[Mikko Rapeli]

testexport.bbclass only copied files from core layer to
the testexport.tar.gz to run tests. Then it filtered
out tests and files which were not specified in
TEST_SUITES variable.

Remove filtering of files to include parselogs.py
test data files which are machine and/or layer specific.
TEST_SUITES variable is now read from build time exported
data store when running tests so there is no need to remove
files from exported tests in testexport.bbclass.

Adapt oe-test script to find "lib" directories from
the new structure with layer specific paths which are
used to find tests and test data files.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/classes-recipe/testexport.bbclass | 43 +++++++++++++-------------
 scripts/oe-test                        | 12 ++++---
 2 files changed, 30 insertions(+), 25 deletions(-)

diff --git a/meta/classes-recipe/testexport.bbclass b/meta/classes-recipe/testexport.bbclass
index cc4088c71a13..843d777e3bb9 100644
--- a/meta/classes-recipe/testexport.bbclass
+++ b/meta/classes-recipe/testexport.bbclass
@@ -85,6 +85,7 @@ def copy_needed_files(d, tc):
 
     export_path = d.getVar('TEST_EXPORT_DIR')
     corebase_path = d.getVar('COREBASE')
+    bblayers = d.getVar('BBLAYERS').split()
 
     # Clean everything before starting
     oe.path.remove(export_path)
@@ -92,17 +93,11 @@ def copy_needed_files(d, tc):
 
     # The source of files to copy are relative to 'COREBASE' directory
     # The destination is relative to 'TEST_EXPORT_DIR'
-    # Because we are squashing the libraries, we need to remove
-    # the layer/script directory
-    files_to_copy = [ os.path.join('meta', 'lib', 'oeqa', 'core'),
-                      os.path.join('meta', 'lib', 'oeqa', 'runtime'),
-                      os.path.join('meta', 'lib', 'oeqa', 'files'),
-                      os.path.join('meta', 'lib', 'oeqa', 'utils'),
-                      os.path.join('scripts', 'oe-test'),
+    # core files/dirs first
+    core_files_to_copy = [ os.path.join('scripts', 'oe-test'),
                       os.path.join('scripts', 'lib', 'argparse_oe.py'),
                       os.path.join('scripts', 'lib', 'scriptutils.py'), ]
-
-    for f in files_to_copy:
+    for f in core_files_to_copy:
         src = os.path.join(corebase_path, f)
         dst = os.path.join(export_path, f.split('/', 1)[-1])
         if os.path.isdir(src):
@@ -110,18 +105,21 @@ def copy_needed_files(d, tc):
         else:
             shutil.copy2(src, dst)
 
-    # Remove cases and just copy the ones specified
-    cases_path = os.path.join(export_path, 'lib', 'oeqa', 'runtime', 'cases')
-    oe.path.remove(cases_path)
-    bb.utils.mkdirhier(cases_path)
-    test_paths = get_runtime_paths(d)
-    test_modules = d.getVar('TEST_SUITES').split()
-    tc.loadTests(test_paths, modules=test_modules)
-    for f in getSuiteCasesFiles(tc.suites):
-        shutil.copy2(f, cases_path)
-        json_file = _get_json_file(f)
-        if json_file:
-            shutil.copy2(json_file, cases_path)
+    # layer specific files/dirs
+    layer_files_to_copy = [ os.path.join('lib', 'oeqa', 'core'),
+                      os.path.join('lib', 'oeqa', 'runtime'),
+                      os.path.join('lib', 'oeqa', 'files'),
+                      os.path.join('lib', 'oeqa', 'utils'),]
+    for layer in bblayers:
+        meta = os.path.basename(layer)
+        for f in layer_files_to_copy:
+            src = os.path.join(layer, f)
+            dst = os.path.join(export_path, meta, f)
+            if os.path.exists(src):
+                if os.path.isdir(src):
+                    oe.path.copytree(src, dst)
+                else:
+                    shutil.copy2(src, dst)
 
     # Copy test data
     image_name = ("%s/%s" % (d.getVar('DEPLOY_DIR_IMAGE'),
@@ -142,6 +140,9 @@ def copy_needed_files(d, tc):
     testexport_create_tarball(d, "testexport.tar.gz", d.getVar("TEST_EXPORT_DIR"))
 
     # Copy packages needed for runtime testing
+    test_paths = get_runtime_paths(d)
+    test_modules = d.getVar('TEST_SUITES').split()
+    tc.loadTests(test_paths, modules=test_modules)
     package_extraction(d, tc.suites)
     test_pkg_dir = d.getVar("TEST_NEEDED_PACKAGES_DIR")
     if os.path.isdir(test_pkg_dir) and os.listdir(test_pkg_dir):
diff --git a/scripts/oe-test b/scripts/oe-test
index 55985b0b2453..efb83c3e7809 100755
--- a/scripts/oe-test
+++ b/scripts/oe-test
@@ -7,14 +7,18 @@
 # SPDX-License-Identifier: MIT
 #
 
-import os
-import sys
 import argparse
+import glob
 import logging
+import os
+import sys
 
 scripts_path = os.path.dirname(os.path.realpath(__file__))
-lib_path = scripts_path + '/lib'
-sys.path = sys.path + [lib_path]
+lib_path = os.path.join(scripts_path, 'lib')
+sys.path.append(lib_path)
+meta_lib_paths = glob.glob(scripts_path + '/*/lib', root_dir=scripts_path, recursive=True)
+for p in meta_lib_paths:
+    sys.path.append(p)
 import argparse_oe
 import scriptutils
 
-- 
2.43.0

[PATCH v3 4/5] testexport.bbclass: use image suffix in testexport tar ball name

[Mikko Rapeli]

testexport.tar.gz is image specific. Thus add same image name
suffix to the tar ball name. For example genericarm64 machine
core-image-minimal image tar ball changes name from
testexport.tar.gz to testexport-core-image-minimal-genericarm64.tar.gz

This way testexport from multiple images can exists in the same
directory.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/classes-recipe/testexport.bbclass | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/meta/classes-recipe/testexport.bbclass b/meta/classes-recipe/testexport.bbclass
index 843d777e3bb9..3005fc0dfa52 100644
--- a/meta/classes-recipe/testexport.bbclass
+++ b/meta/classes-recipe/testexport.bbclass
@@ -136,8 +136,12 @@ def copy_needed_files(d, tc):
             if dir == '__pycache__':
                 shutil.rmtree(os.path.join(subdir, dir))
 
+    image_basename = d.getVar('IMAGE_BASENAME')
+    image_machine_suffix = d.getVar('IMAGE_MACHINE_SUFFIX')
+
     # Create tar file for common parts of testexport
-    testexport_create_tarball(d, "testexport.tar.gz", d.getVar("TEST_EXPORT_DIR"))
+    testexport_create_tarball(d, "testexport-%s%s.tar.gz" %
+        (image_basename, image_machine_suffix), d.getVar("TEST_EXPORT_DIR"))
 
     # Copy packages needed for runtime testing
     test_paths = get_runtime_paths(d)
@@ -149,7 +153,7 @@ def copy_needed_files(d, tc):
         export_pkg_dir = os.path.join(d.getVar("TEST_EXPORT_DIR"), "packages")
         oe.path.copytree(test_pkg_dir, export_pkg_dir)
         # Create tar file for packages needed by the DUT
-        testexport_create_tarball(d, "testexport_packages_%s.tar.gz" % d.getVar("MACHINE"), export_pkg_dir)
+        testexport_create_tarball(d, "testexport_packages%s.tar.gz" % image_machine_suffix, export_pkg_dir)
 
     # Copy SDK
     if d.getVar("TEST_EXPORT_SDK_ENABLED") == "1":
-- 
2.43.0

Re: [OE-core] [PATCH v3 3/5] testexport.bbclass oe-test: capture all tests and data from all layers

[Richard Purdie]

On Wed, 2025-07-02 at 10:25 +0300, Mikko Rapeli via
lists.openembedded.org wrote:
> 
> -import os
> -import sys
>  import argparse
> +import glob
>  import logging
> +import os
> +import sys
>  
>  scripts_path = os.path.dirname(os.path.realpath(__file__))
> -lib_path = scripts_path + '/lib'
> -sys.path = sys.path + [lib_path]
> +lib_path = os.path.join(scripts_path, 'lib')
> +sys.path.append(lib_path)
> +meta_lib_paths = glob.glob(scripts_path + '/*/lib',
> root_dir=scripts_path, recursive=True)
> +for p in meta_lib_paths:
> +    sys.path.append(p)
>  import argparse_oe
>  import scriptutils
>  
> 

This assumes python 3.10 which we can't quite do yet:

https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/1850

Cheers,

Richard

Re: [OE-core] [PATCH v3 3/5] testexport.bbclass oe-test: capture all tests and data from all layers

[Mikko Rapeli]

Hi,

On Sat, Jul 05, 2025 at 05:07:57PM +0100, Richard Purdie wrote:
> On Wed, 2025-07-02 at 10:25 +0300, Mikko Rapeli via
> lists.openembedded.org wrote:
> > 
> > -import os
> > -import sys
> > �import argparse
> > +import glob
> > �import logging
> > +import os
> > +import sys
> > �
> > �scripts_path = os.path.dirname(os.path.realpath(__file__))
> > -lib_path = scripts_path + '/lib'
> > -sys.path = sys.path + [lib_path]
> > +lib_path = os.path.join(scripts_path, 'lib')
> > +sys.path.append(lib_path)
> > +meta_lib_paths = glob.glob(scripts_path + '/*/lib',
> > root_dir=scripts_path, recursive=True)
> > +for p in meta_lib_paths:
> > +��� sys.path.append(p)
> > �import argparse_oe
> > �import scriptutils
> > �
> > 
> 
> This assumes python 3.10 which we can't quite do yet:
> 
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/1850

Ok, looks like it's safe to remove root_dir=scripts_path completely.
I'll send a v4 for this.

Cheers,

-Mikko

Re: [OE-core] [PATCH v3 4/5] testexport.bbclass: use image suffix in testexport tar ball name

[Richard Purdie]

On Wed, 2025-07-02 at 10:25 +0300, Mikko Rapeli via lists.openembedded.org wrote:
> testexport.tar.gz is image specific. Thus add same image name
> suffix to the tar ball name. For example genericarm64 machine
> core-image-minimal image tar ball changes name from
> testexport.tar.gz to testexport-core-image-minimal-genericarm64.tar.gz
> 
> This way testexport from multiple images can exists in the same
> directory.
> 
> Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> ---
>  meta/classes-recipe/testexport.bbclass | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes-recipe/testexport.bbclass b/meta/classes-recipe/testexport.bbclass
> index 843d777e3bb9..3005fc0dfa52 100644
> --- a/meta/classes-recipe/testexport.bbclass
> +++ b/meta/classes-recipe/testexport.bbclass
> @@ -136,8 +136,12 @@ def copy_needed_files(d, tc):
>              if dir == '__pycache__':
>                  shutil.rmtree(os.path.join(subdir, dir))
>  
> +    image_basename = d.getVar('IMAGE_BASENAME')
> +    image_machine_suffix = d.getVar('IMAGE_MACHINE_SUFFIX')
> +
>      # Create tar file for common parts of testexport
> -    testexport_create_tarball(d, "testexport.tar.gz", d.getVar("TEST_EXPORT_DIR"))
> +    testexport_create_tarball(d, "testexport-%s%s.tar.gz" %
> +        (image_basename, image_machine_suffix), d.getVar("TEST_EXPORT_DIR"))
>  
>      # Copy packages needed for runtime testing
>      test_paths = get_runtime_paths(d)
> @@ -149,7 +153,7 @@ def copy_needed_files(d, tc):
>          export_pkg_dir = os.path.join(d.getVar("TEST_EXPORT_DIR"), "packages")
>          oe.path.copytree(test_pkg_dir, export_pkg_dir)
>          # Create tar file for packages needed by the DUT
> -        testexport_create_tarball(d, "testexport_packages_%s.tar.gz" % d.getVar("MACHINE"), export_pkg_dir)
> +        testexport_create_tarball(d, "testexport_packages%s.tar.gz" % image_machine_suffix, export_pkg_dir)
>  
>      # Copy SDK
>      if d.getVar("TEST_EXPORT_SDK_ENABLED") == "1":

Since most of the machine and image specific elements of this have been
removed, does it not make sense just to have one testexport tarball
generated by an external script?

The testdata json file is already exported separately.

I've held off this patch for that reason, making it machine specific
seems a backwards step?

Cheers,

Richard

Re: [OE-core] [PATCH v3 4/5] testexport.bbclass: use image suffix in testexport tar ball name

[Mikko Rapeli]

Hi,

On Mon, Jul 14, 2025 at 01:08:42PM +0100, Richard Purdie wrote:
> On Wed, 2025-07-02 at 10:25 +0300, Mikko Rapeli via lists.openembedded.org wrote:
> > testexport.tar.gz is image specific. Thus add same image name
> > suffix to the tar ball name. For example genericarm64 machine
> > core-image-minimal image tar ball changes name from
> > testexport.tar.gz to testexport-core-image-minimal-genericarm64.tar.gz
> > 
> > This way testexport from multiple images can exists in the same
> > directory.
> > 
> > Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> > ---
> > �meta/classes-recipe/testexport.bbclass | 8 ++++++--
> > �1 file changed, 6 insertions(+), 2 deletions(-)
> > 
> > diff --git a/meta/classes-recipe/testexport.bbclass b/meta/classes-recipe/testexport.bbclass
> > index 843d777e3bb9..3005fc0dfa52 100644
> > --- a/meta/classes-recipe/testexport.bbclass
> > +++ b/meta/classes-recipe/testexport.bbclass
> > @@ -136,8 +136,12 @@ def copy_needed_files(d, tc):
> > ������������ if dir == '__pycache__':
> > ���������������� shutil.rmtree(os.path.join(subdir, dir))
> > �
> > +��� image_basename = d.getVar('IMAGE_BASENAME')
> > +��� image_machine_suffix = d.getVar('IMAGE_MACHINE_SUFFIX')
> > +
> > ���� # Create tar file for common parts of testexport
> > -��� testexport_create_tarball(d, "testexport.tar.gz", d.getVar("TEST_EXPORT_DIR"))
> > +��� testexport_create_tarball(d, "testexport-%s%s.tar.gz" %
> > +������� (image_basename, image_machine_suffix), d.getVar("TEST_EXPORT_DIR"))
> > �
> > ���� # Copy packages needed for runtime testing
> > ���� test_paths = get_runtime_paths(d)
> > @@ -149,7 +153,7 @@ def copy_needed_files(d, tc):
> > �������� export_pkg_dir = os.path.join(d.getVar("TEST_EXPORT_DIR"), "packages")
> > �������� oe.path.copytree(test_pkg_dir, export_pkg_dir)
> > �������� # Create tar file for packages needed by the DUT
> > -������� testexport_create_tarball(d, "testexport_packages_%s.tar.gz" % d.getVar("MACHINE"), export_pkg_dir)
> > +������� testexport_create_tarball(d, "testexport_packages%s.tar.gz" % image_machine_suffix, export_pkg_dir)
> > �
> > ���� # Copy SDK
> > ���� if d.getVar("TEST_EXPORT_SDK_ENABLED") == "1":
> 
> Since most of the machine and image specific elements of this have been
> removed, does it not make sense just to have one testexport tarball
> generated by an external script?
> 
> The testdata json file is already exported separately.
> 
> I've held off this patch for that reason, making it machine specific
> seems a backwards step?

Currently the image specific testdata json is inside the testexport
tar ball as core-image-sato/data/testdata.json. Same file is available
after build in
tmp/deploy/images/genericarm64/core-image-sato-genericarm64.rootfs.testdata.json
so I could switch the exported tests over to that.

I'll try to set this up with a different patch. Thanks for the feedback!

The new approach needs some standardisation between what is in build
output and what is in the test environment. There is a similar issue
with opkg/ipk runtime test and the need for just a few files around
80kb from the ipk archive, not the full archive which is possibly
gigabytes in size. I've now set these up with a separate script
outside of poky. It would be nice if these small scripts could live
in poky, possibly scripts/contrib/ ? I could capture the image specific
testdata.json file in the same way too, for example a single tar ball.

Cheers,

-Mikko