Re: [OE-core] [PATCH] devtool: Add test-image plugin for testing packages via devtool via images.

[Paul Barker <paul@pbarker.dev>]

[-- Attachment #1: Type: text/plain, Size: 7407 bytes --]

On Mon, 2026-02-23 at 00:24 +0100, Tom Geelen via lists.openembedded.org
wrote:
> Based of a feature of AUH this is a plugin to run a testimage directly with packages installed that you are working on via devtool.
> 
> Inputs would be a:
> - target image
> - target packages
> 
> The tool will take care to make sure it also installs the minimal necessary dependencies to be able to run ptest on the target image.
> Logs will be captured and stored in the devtool workspace for easy access.
> 
> An oe-selftest is added to test the plugin.
> 
> Signed-off-by: Tom Geelen <t.f.g.geelen@gmail.com>
> ---
>  meta/lib/oeqa/selftest/cases/devtool.py |  61 +++++++++++
>  scripts/lib/devtool/test_image.py       | 130 ++++++++++++++++++++++++
>  2 files changed, 191 insertions(+)
>  create mode 100644 scripts/lib/devtool/test_image.py
> 
> diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py
> index d1209dd94e..c27934a3ca 100644
> --- a/meta/lib/oeqa/selftest/cases/devtool.py
> +++ b/meta/lib/oeqa/selftest/cases/devtool.py
> @@ -1932,6 +1932,67 @@ class DevtoolBuildImageTests(DevtoolBase):
>          if reqpkgs:
>              self.fail('The following packages were not present in the image as expected: %s' % ', '.join(reqpkgs))
>  
> +
> +class DevtoolTestImageTests(DevtoolBase):
> +
> +    @OETestTag("runqemu")
> +    def test_devtool_test_image(self):
> +        """Test devtool test-image plugin."""
> +
> +        machine = get_bb_var('MACHINE')
> +        if not machine or not machine.startswith('qemu'):
> +            self.skipTest('This test only works with qemu machines')
> +
> +        self.assertTrue(not os.path.exists(self.workspacedir),
> +                        'This test cannot be run with a workspace directory under the build directory')
> +
> +        image = 'oe-selftest-image'
> +        recipe = 'python3-atomicwrites'

I'm not sure that this is the best image recipe to use. Why not build
core-image-ptest-<packagename>?

> +
> +        # Ensure selected test package is ptest-capable.
> +        ptest_path = get_bb_var('PTEST_PATH', recipe)
> +        self.assertTrue(ptest_path,
> +                'Selected package %s does not appear to inherit ptest' % recipe)
> +
> +        self.track_for_cleanup(self.workspacedir)
> +        # self.add_command_to_tearDown('bitbake -c clean %s' % image)
> +        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
> +
> +        # Ensure we're starting from a clean state
> +        bitbake('%s -c clean' % image)
> +
> +        result = runCmd('devtool test-image %s -p %s' % (image, recipe), ignore_status=True)
> +        if result.status != 0 and 'runqemu - ERROR - Unknown path arg' in result.output:
> +            self.skipTest('runqemu in this environment does not accept testimage rootfs path args')
> +        self.assertEqual(result.status, 0,
> +                         'devtool test-image failed unexpectedly:\n%s' % result.output)
> +
> +        # Check that requested package and its ptest package were installed
> +        deploy_dir_image = get_bb_var('DEPLOY_DIR_IMAGE')
> +        self.assertTrue(deploy_dir_image, 'Unable to get DEPLOY_DIR_IMAGE')
> +
> +        manifests = sorted(glob.glob(os.path.join(deploy_dir_image, '%s*.manifest' % image)))
> +        self.assertTrue(manifests, 'Image manifest not found for %s in %s' % (image, deploy_dir_image))
> +        manifest = manifests[-1]
> +        self.assertExists(manifest, 'Image manifest not found: %s' % manifest)
> +
> +        pkgs = set()
> +        with open(manifest, 'r') as f:
> +            for line in f:
> +                splitval = line.split()
> +                if splitval:
> +                    pkgs.add(splitval[0])
> +
> +        self.assertIn(recipe, pkgs)
> +        self.assertIn(recipe + '-ptest', pkgs)
> +
> +        match = re.search(r'Logs are in (\S+)', result.output)
> +        if match:
> +            logdir = match.group(1)
> +        else:
> +            logdir = os.path.join(self.workspacedir, 'testimage-logs')
> +        self.assertTrue(os.path.isdir(logdir), 'Expected logs directory not found: %s' % logdir)
> +

This test case took 10+ hours to run on the autobuilder [1]. Any ideas
why it's so slow?

[1]: https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3280

> +def test_image(args, config, basepath, workspace):
> +    """Entry point for the devtool 'test-image' subcommand."""
> +
> +    if not args.imagename:
> +        raise DevtoolError('Image recipe to test must be specified')
> +    if not args.package:
> +        raise DevtoolError('Package(s) to install must be specified via -p/--package')
> +
> +    package_names = [p.strip() for p in args.package.split(',') if p.strip()]
> +    if not package_names:
> +        raise DevtoolError('No valid package name(s) provided')
> +
> +    install_pkgs = package_names
> +
> +    logdir = os.path.join(config.workspace_path, 'testimage-logs')
> +    try:
> +        os.makedirs(logdir, exist_ok=True)
> +    except Exception as exc:
> +        raise DevtoolError(f'Failed to create test logs directory {logdir}: {exc}')
> +
> +    pkg_append = ' '.join(sorted(set(install_pkgs)))
> +    extra_append = [
> +        f'TEST_LOG_DIR = "{logdir}"',
> +        # Ensure runtime test framework is enabled even if image/distro omitted it
> +        'IMAGE_CLASSES += " testimage"',
> +        # Ensure the testimage task has the correct IMAGE_FEATURES set in case the TEST_TARGET is qemu
> +        'IMAGE_FEATURES += "allow-empty-password empty-root-password allow-root-login"',
> +        'TEST_SUITES = " ping ssh ptest"',
> +        'TEST_RUNQEMUPARAMS += "slirp"',

This should not be set by default as different users may have different
needs. We need to rely on users setting the runqemuparams that the need
in local.conf/site.conf.

> +        # Ensure image artifacts are built before do_testimage reads them.
> +        'do_testimage[depends] += " ${PN}:do_image_complete virtual/kernel:do_deploy"',

The do_testimage task should already have appropriate dependencies.

> +        # Ensure rootfs link naming is runqemu-compatible for image names that
> +        # otherwise end with '-image' (without a trailing '-').
> +        'IMAGE_LINK_NAME = "${IMAGE_BASENAME}-${MACHINE}"',
> +        # Ensure a qemu-supported rootfs type is built/selected
> +        'IMAGE_FSTYPES:append = " ext4"',
> +        'QB_DEFAULT_FSTYPE = "ext4"',

Why do we need to force ext4 here?

> +        # Enable ptests and include available -ptest packages for installed content
> +        'DISTRO_FEATURES:append = " ptest"',
> +        'IMAGE_FEATURES += " ptest-pkgs"',
> +        'IMAGE_INSTALL:append = " ptest-runner dropbear"',

We shouldn't assume dropbear here. If an ssh server is needed, using
"sshd" here should do the job and allow openssh or dropbear to be
chosen.

> +        # Ensure requested packages (and -ptest where available) are installed
> +        f'IMAGE_INSTALL:append = " {pkg_append}"',
> +    ]

The extra_append argument to build_image_task is written to a bbappend
file for the image recipe, so it can't set global configuration like
DISTRO_FEATURES.

Best regards,

-- 
Paul Barker


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 252 bytes --]