[PATCH 0/5] devtool: Support multiconfig workflows and fix kernel extraction

[PATCH 0/5] devtool: Support multiconfig workflows and fix kernel extraction

[Anusmita Dutta Mazumder]

This series adds multiconfig support to devtool workflows,
isolates workspace changes between configurations, and fixes
kernel configure failure handling and source revision baselines.

It also adds selftests for multiconfig workflows and isolation.

This series addresses [YOCTO #13933].

Anusmita Dutta Mazumder (5):
  devtool: Support multiconfig recipe workflows
  devtool: Check kernel configure result
  devtool: Isolate multiconfig workspace entries
  devtool: Record kernel baseline after revision validation
  oe-selftest: devtool: Add multiconfig workflow tests

 meta/classes/devtool-source.bbclass     |   3 +-
 meta/lib/oeqa/selftest/cases/devtool.py | 133 ++++++++++++++++++++++++
 scripts/devtool                         |  24 ++++-
 scripts/lib/devtool/__init__.py         |   8 +-
 scripts/lib/devtool/standard.py         |  88 ++++++++++++----
 5 files changed, 230 insertions(+), 26 deletions(-)


base-commit: ae5cafacec8e99c5a27fa5b3b378eefc595910ef

[PATCH 1/5] devtool: Support multiconfig recipe workflows

[Anusmita Dutta Mazumder]

Preserve multiconfig-qualified targets when extracting and
configuring sources.

Use the recipe name without the multiconfig prefix for
workspace lookups. Keep the prefix when calling BitBake
from reset, update-recipe, and finish.

Fixes [YOCTO #13933]

Signed-off-by: Anusmita Dutta Mazumder <anusmita.duttamazumder@arm.com>
---
 scripts/lib/devtool/standard.py | 78 +++++++++++++++++++++++++--------
 1 file changed, 60 insertions(+), 18 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 0d933e68b1..84fa19c7aa 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -23,6 +23,7 @@ from collections import OrderedDict
 from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, setup_git_repo, recipe_to_append, get_bbclassextend_targets, update_unlockedsigs, check_prerelease_version, check_git_repo_dirty, check_git_repo_op, DevtoolError
 from devtool import parse_recipe
 
+import bb.runqueue
 import bb.utils
 
 logger = logging.getLogger('devtool')
@@ -30,6 +31,21 @@ logger = logging.getLogger('devtool')
 override_branch_prefix = 'devtool-override-'
 
 
+def _prefix_multiconfig_targets(mc, targets):
+    """Add a multiconfig prefix to build targets."""
+    if not mc:
+        return targets
+
+    return ' '.join(target if target.startswith('mc:') else
+                    'mc:%s:%s' % (mc, target)
+                    for target in targets.split())
+
+
+def _get_multiconfig_targets(d, targets):
+    """Add the recipe datastore's multiconfig prefix to build targets."""
+    return _prefix_multiconfig_targets(d.getVar('BB_CURRENT_MC'), targets)
+
+
 def add(args, config, basepath, workspace):
     """Entry point for the devtool 'add' subcommand"""
     import bb.data
@@ -596,7 +612,7 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
                         task = None
 
             # Run the fetch + unpack tasks
-            res = tinfoil.build_targets(pn,
+            res = tinfoil.build_targets(_get_multiconfig_targets(d, pn),
                                         task,
                                         handle_events=True)
         finally:
@@ -829,7 +845,8 @@ def modify(args, config, basepath, workspace):
                             f.write('%s\n' % fname)
 
                 task = 'do_configure'
-                res = tinfoil.build_targets(pn, task, handle_events=True)
+                target = _get_multiconfig_targets(rd, pn)
+                res = tinfoil.build_targets(target, task, handle_events=True)
 
                 # Copy .config to workspace
                 kconfpath = rd.getVar('B')
@@ -1919,7 +1936,7 @@ def _update_recipe(recipename, workspace, rd, mode, appendlayerdir, wildcard_ver
 
 def update_recipe(args, config, basepath, workspace):
     """Entry point for the devtool 'update-recipe' subcommand"""
-    check_workspace_recipe(workspace, args.recipename)
+    workspacepn = check_workspace_recipe(workspace, args.recipename)
 
     if args.append:
         if not os.path.exists(args.append):
@@ -1941,7 +1958,11 @@ def update_recipe(args, config, basepath, workspace):
         if args.dry_run:
             dry_run_output = tempfile.TemporaryDirectory(prefix='devtool')
             dry_run_outdir = dry_run_output.name
-        updated, _, _ = _update_recipe(args.recipename, workspace, rd, args.mode, args.append, args.wildcard_version, args.no_remove, args.initial_rev, dry_run_outdir=dry_run_outdir, no_overrides=args.no_overrides, force_patch_refresh=args.force_patch_refresh)
+        updated, _, _ = _update_recipe(
+            workspacepn, workspace, rd, args.mode, args.append,
+            args.wildcard_version, args.no_remove, args.initial_rev,
+            dry_run_outdir=dry_run_outdir, no_overrides=args.no_overrides,
+            force_patch_refresh=args.force_patch_refresh)
 
         if updated:
             rf = rd.getVar('FILE')
@@ -1968,11 +1989,17 @@ def status(args, config, basepath, workspace):
     return 0
 
 
-def _reset(recipes, no_clean, remove_work, config, basepath, workspace):
+def _reset(recipes, no_clean, remove_work, config, basepath, workspace,
+           bitbake_targets=None):
     """Reset one or more recipes"""
     import bb.process
     import oe.path
 
+    if bitbake_targets is None:
+        bitbake_targets = recipes
+    if len(bitbake_targets) != len(recipes):
+        raise DevtoolError('Reset recipes and BitBake targets do not match')
+
     def clean_preferred_provider(pn, layerconf_path):
         """Remove PREFERRED_PROVIDER from layer.conf'"""
         import re
@@ -2002,11 +2029,15 @@ def _reset(recipes, no_clean, remove_work, config, basepath, workspace):
         # it uses BBCLASSEXTEND, then we need to also clean the other
         # variants
         targets = []
-        for recipe in recipes:
-            targets.append(recipe)
+        for recipe, bitbake_target in zip(recipes, bitbake_targets):
+            targets.append(bitbake_target)
             recipefile = workspace[recipe]['recipefile']
             if recipefile and os.path.exists(recipefile):
-                targets.extend(get_bbclassextend_targets(recipefile, recipe))
+                variants = get_bbclassextend_targets(recipefile, recipe)
+                mc, _ = bb.runqueue.split_mc(bitbake_target)
+                variants = _prefix_multiconfig_targets(mc,
+                                                       ' '.join(variants))
+                targets.extend(variants.split())
         try:
             exec_build_env_command(config.init_path, basepath, 'bitbake -c clean %s' % ' '.join(targets))
         except bb.process.ExecutionError as e:
@@ -2086,23 +2117,27 @@ def _reset(recipes, no_clean, remove_work, config, basepath, workspace):
 def reset(args, config, basepath, workspace):
     """Entry point for the devtool 'reset' subcommand"""
 
-    recipes = ""
+    recipes = []
+    bitbake_targets = []
 
     if args.recipename:
         if args.all:
             raise DevtoolError("Recipe cannot be specified if -a/--all is used")
         else:
             for recipe in args.recipename:
-                check_workspace_recipe(workspace, recipe, checksrc=False)
+                workspacepn = check_workspace_recipe(workspace, recipe,
+                                                     checksrc=False)
+                recipes.append(workspacepn)
+                bitbake_targets.append(recipe)
     elif not args.all:
         raise DevtoolError("Recipe must be specified, or specify -a/--all to "
                            "reset all recipes")
     if args.all:
         recipes = list(workspace.keys())
-    else:
-        recipes = args.recipename
+        bitbake_targets = recipes
 
-    _reset(recipes, args.no_clean, args.remove_work, config, basepath, workspace)
+    _reset(recipes, args.no_clean, args.remove_work, config, basepath, workspace,
+           bitbake_targets=bitbake_targets)
 
     return 0
 
@@ -2137,14 +2172,14 @@ def finish(args, config, basepath, workspace):
     import bb
     import oe.recipeutils
 
-    check_workspace_recipe(workspace, args.recipename)
+    workspacepn = check_workspace_recipe(workspace, args.recipename)
 
     dry_run_suffix = ' (dry-run)' if args.dry_run else ''
 
     # Grab the equivalent of COREBASE without having to initialise tinfoil
     corebasedir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..', '..'))
 
-    srctree = workspace[args.recipename]['srctree']
+    srctree = workspace[workspacepn]['srctree']
     check_git_repo_op(srctree, [corebasedir])
     dirty = check_git_repo_dirty(srctree)
     if dirty:
@@ -2175,7 +2210,7 @@ def finish(args, config, basepath, workspace):
         # If it's an upgrade, grab the original path
         origpath = None
         origfilelist = None
-        append = workspace[args.recipename]['bbappend']
+        append = workspace[workspacepn]['bbappend']
         with open(append, 'r') as f:
             for line in f:
                 if line.startswith('# original_path:'):
@@ -2217,7 +2252,12 @@ def finish(args, config, basepath, workspace):
         if args.dry_run:
             dry_run_output = tempfile.TemporaryDirectory(prefix='devtool')
             dry_run_outdir = dry_run_output.name
-        updated, appendfile, removed = _update_recipe(args.recipename, workspace, rd, args.mode, appendlayerdir, wildcard_version=True, no_remove=False, no_report_remove=removing_original, initial_rev=args.initial_rev, dry_run_outdir=dry_run_outdir, no_overrides=args.no_overrides, force_patch_refresh=args.force_patch_refresh)
+        updated, appendfile, removed = _update_recipe(
+            workspacepn, workspace, rd, args.mode, appendlayerdir,
+            wildcard_version=True, no_remove=False,
+            no_report_remove=removing_original, initial_rev=args.initial_rev,
+            dry_run_outdir=dry_run_outdir, no_overrides=args.no_overrides,
+            force_patch_refresh=args.force_patch_refresh)
         removed = [os.path.relpath(pth, recipedir) for pth in removed]
 
         # Remove any old files in the case of an upgrade
@@ -2306,7 +2346,9 @@ def finish(args, config, basepath, workspace):
     if args.dry_run:
         logger.info('Resetting recipe (dry-run)')
     else:
-        _reset([args.recipename], no_clean=no_clean, remove_work=remove_work, config=config, basepath=basepath, workspace=workspace)
+        _reset([workspacepn], no_clean=no_clean, remove_work=remove_work,
+               config=config, basepath=basepath, workspace=workspace,
+               bitbake_targets=[args.recipename])
 
     return 0
 

[PATCH 2/5] devtool: Check kernel configure result

[Anusmita Dutta Mazumder]

Stop source modification when the kernel do_configure task
fails instead of continuing with a potentially stale
configuration.

Signed-off-by: Anusmita Dutta Mazumder <anusmita.duttamazumder@arm.com>
---
 scripts/lib/devtool/standard.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 84fa19c7aa..1f6ae1d931 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -847,6 +847,9 @@ def modify(args, config, basepath, workspace):
                 task = 'do_configure'
                 target = _get_multiconfig_targets(rd, pn)
                 res = tinfoil.build_targets(target, task, handle_events=True)
+                if not res:
+                    raise DevtoolError('Configuring source for %s failed' %
+                                       target)
 
                 # Copy .config to workspace
                 kconfpath = rd.getVar('B')

[PATCH 3/5] devtool: Isolate multiconfig workspace entries

[Anusmita Dutta Mazumder]

Store workspace appends in multiconfig-specific directories
and select them using BB_CURRENT_MC to prevent changes from
affecting same-named recipes in other configurations.

Fixes [YOCTO #13933]

Signed-off-by: Anusmita Dutta Mazumder <anusmita.duttamazumder@arm.com>
---
 scripts/devtool                 | 24 ++++++++++++++++++++++--
 scripts/lib/devtool/__init__.py |  8 ++++++--
 scripts/lib/devtool/standard.py |  7 ++++---
 3 files changed, 32 insertions(+), 7 deletions(-)

diff --git a/scripts/devtool b/scripts/devtool
index 39cebec0d8..bae7c8bbe4 100755
--- a/scripts/devtool
+++ b/scripts/devtool
@@ -28,6 +28,11 @@ import scriptutils
 import argparse_oe
 logger = scriptutils.logger_create('devtool')
 
+# Only expose workspace appends belonging to the current configuration.
+workspace_append_pattern = ("${LAYERDIR}/appends/"
+                            "${@'multiconfig/%s/' % d.getVar('BB_CURRENT_MC') "
+                            "if d.getVar('BB_CURRENT_MC') else ''}*.bbappend")
+
 
 class ConfigHandler:
     basepath = None
@@ -99,9 +104,21 @@ def read_workspace(basepath, context):
     if not context.fixed_setup:
         _enable_workspace_layer(context.config.workspace_path, context.config, basepath)
 
+    # Upgrade existing workspace layers as well as newly created ones.
+    layerconf = os.path.join(context.config.workspace_path, 'conf', 'layer.conf')
+    with open(layerconf) as f:
+        contents = f.read()
+    updated = contents.replace('${LAYERDIR}/appends/*.bbappend', workspace_append_pattern)
+    if updated != contents:
+        with open(layerconf, 'w') as f:
+            f.write(updated)
+
     logger.debug('Reading workspace in %s' % context.config.workspace_path)
     externalsrc_re = re.compile(r'^EXTERNALSRC(:pn-([^ =]+))? *= *"([^"]*)"$')
-    for fn in glob.glob(os.path.join(context.config.workspace_path, 'appends', '*.bbappend')):
+    appenddir = os.path.join(context.config.workspace_path, 'appends')
+    appendfiles = glob.glob(os.path.join(appenddir, '*.bbappend'))
+    appendfiles += glob.glob(os.path.join(appenddir, 'multiconfig', '*', '*.bbappend'))
+    for fn in appendfiles:
         with open(fn, 'r') as f:
             pnvalues = {}
             pn = None
@@ -130,6 +147,9 @@ def read_workspace(basepath, context):
                 if not pnvalues.get('srctreebase', None):
                     pnvalues['srctreebase'] = pnvalues['srctree']
                 logger.debug('Found recipe %s' % pnvalues)
+                if pn in workspace:
+                    raise DevtoolError("Multiple workspace appends found for recipe '%s'; "
+                                       "only one configuration can be modified at a time" % pn)
                 workspace[pn] = pnvalues
 
     return workspace
@@ -166,7 +186,7 @@ def _create_workspace(workspacedir, basepath, layerseries=None):
             f.write('# ### workspace layer auto-generated by devtool ###\n')
             f.write('BBPATH =. "$' + '{LAYERDIR}:"\n')
             f.write('BBFILES += "$' + '{LAYERDIR}/recipes/*/*.bb \\\n')
-            f.write('            $' + '{LAYERDIR}/appends/*.bbappend"\n')
+            f.write('            %s"\n' % workspace_append_pattern)
             f.write('BBFILE_COLLECTIONS += "workspacelayer"\n')
             f.write('BBFILE_PATTERN_workspacelayer = "^$' + '{LAYERDIR}/"\n')
             f.write('BBFILE_PATTERN_IGNORE_EMPTY_workspacelayer = "1"\n')
diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index 9d227e2032..f728f7ada0 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -133,13 +133,15 @@ def setup_tinfoil(config_only=False, basepath=None, tracking=False):
 def parse_recipe(config, tinfoil, pn, appends, filter_workspace=True):
     """Parse the specified recipe"""
     import bb.providers
+    import bb.runqueue
     try:
         recipefile = tinfoil.get_recipe_file(pn)
     except bb.providers.NoProvider as e:
         logger.error(str(e))
         return None
     if appends:
-        append_files = tinfoil.get_file_appends(recipefile)
+        mc, _ = bb.runqueue.split_mc(pn)
+        append_files = tinfoil.get_file_appends(recipefile, mc)
         if filter_workspace:
             # Filter out appends from the workspace
             append_files = [path for path in append_files if
@@ -276,7 +278,7 @@ def setup_git_repo(repodir, version, devbranch, basetag='devtool-base', d=None):
     if os.path.exists(os.path.join(repodir, '.gitmodules')):
         bb.process.run('git submodule foreach --recursive  "git tag -f --no-sign %s"' % basetag, cwd=repodir)
 
-def recipe_to_append(recipefile, config, wildcard=False):
+def recipe_to_append(recipefile, config, wildcard=False, mc=''):
     """
     Convert a recipe file to a bbappend file path within the workspace.
     NOTE: if the bbappend already exists, you should be using
@@ -287,6 +289,8 @@ def recipe_to_append(recipefile, config, wildcard=False):
     if wildcard:
         appendname = re.sub(r'_.*', '_%', appendname)
     appendpath = os.path.join(config.workspace_path, 'appends')
+    if mc:
+        appendpath = os.path.join(appendpath, 'multiconfig', mc)
     appendfile = os.path.join(appendpath, appendname + '.bbappend')
     return appendfile
 
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 1f6ae1d931..0f7cfe132f 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -540,7 +540,7 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
     initial_rev = None
 
     recipefile = d.getVar('FILE')
-    appendfile = recipe_to_append(recipefile, config)
+    appendfile = recipe_to_append(recipefile, config, mc=d.getVar('BB_CURRENT_MC'))
     is_kernel_yocto = bb.data.inherits_class('kernel-yocto', d)
 
     # We need to redirect WORKDIR, STAMPS_DIR etc. under a temporary
@@ -804,7 +804,8 @@ def modify(args, config, basepath, workspace):
                             srctree)
 
         recipefile = rd.getVar('FILE')
-        appendfile = recipe_to_append(recipefile, config, args.wildcard)
+        appendfile = recipe_to_append(recipefile, config, args.wildcard,
+                                      mc=rd.getVar('BB_CURRENT_MC'))
         if os.path.exists(appendfile):
             raise DevtoolError("Another variant of recipe %s is already in your "
                             "workspace (only one variant of a recipe can "
@@ -2076,7 +2077,7 @@ def _reset(recipes, no_clean, remove_work, config, basepath, workspace,
             # This should always be true if recipefile is set, but just in case
             preservedir(os.path.dirname(recipefile))
         # We don't automatically create this dir next to appends, but the user can
-        preservedir(os.path.join(config.workspace_path, 'appends', pn))
+        preservedir(os.path.join(os.path.dirname(appendfile), pn))
 
         srctreebase = workspace[pn]['srctreebase']
         if os.path.isdir(srctreebase):

[PATCH 4/5] devtool: Record kernel baseline after revision validation

[Anusmita Dutta Mazumder]

Devtool creates its kernel branch before the source is reset
to the commit specified by the recipe. This can leave the
workspace on the wrong kernel version and fail the build.

Create the devtool branch after revision validation so it
uses the commit specified by the recipe.

Signed-off-by: Anusmita Dutta Mazumder <anusmita.duttamazumder@arm.com>
---
 meta/classes/devtool-source.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes/devtool-source.bbclass b/meta/classes/devtool-source.bbclass
index f29f40588f..24f8a3b616 100644
--- a/meta/classes/devtool-source.bbclass
+++ b/meta/classes/devtool-source.bbclass
@@ -52,7 +52,8 @@ python() {
     # Hook in pre/postfuncs
     is_kernel_yocto = bb.data.inherits_class('kernel-yocto', d)
     if is_kernel_yocto:
-        unpacktask = 'do_kernel_checkout'
+        # Record the devtool baseline after the kernel SRCREV is enforced.
+        unpacktask = 'do_validate_branches'
         d.appendVarFlag('do_configure', 'postfuncs', ' devtool_post_configure')
     else:
         unpacktask = 'do_unpack'

[PATCH 5/5] oe-selftest: devtool: Add multiconfig workflow tests

[Anusmita Dutta Mazumder]

Cover modify, build, reset, update-recipe and finish with a
recipe available only in the selected multiconfig. Check
workspace isolation between configurations.

[YOCTO #13933]

Signed-off-by: Anusmita Dutta Mazumder <anusmita.duttamazumder@arm.com>
---
 meta/lib/oeqa/selftest/cases/devtool.py | 133 ++++++++++++++++++++++++
 1 file changed, 133 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py
index b848623a6e..1b83ee16d4 100644
--- a/meta/lib/oeqa/selftest/cases/devtool.py
+++ b/meta/lib/oeqa/selftest/cases/devtool.py
@@ -1327,6 +1327,139 @@ class DevtoolModifyTests(DevtoolBase):
         self.assertExists(os.path.join(source_repo_gitsm_gitmodules, 'bitbake'), 'Submodule not found')
         self.assertExists(os.path.join(source_repo_gitsm_gitmodules, 'bitbake-gitsm-test1'), 'Submodule not found')
 
+class DevtoolMulticonfigTests(DevtoolBase):
+
+    def setUp(self):
+        super(DevtoolMulticonfigTests, self).setUp()
+        self.recipe = 'devtool-multiconfig-test'
+        self.target = 'mc:devtooltest:%s' % self.recipe
+        self.tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(self.tempdir)
+        self.track_for_cleanup(self.workspacedir)
+        self.srctree = os.path.join(self.tempdir, 'source')
+        self.layerdir = os.path.join(self.tempdir, 'meta-devtool-multiconfig')
+
+        self.append_config('BBMULTICONFIG = "devtooltest"\n')
+        # Only this configuration satisfies the test recipe's compatibility
+        # restriction, regardless of the machine used by the selftest runner.
+        self.write_config('MACHINEOVERRIDES .= ":devtool-mc"\n'
+                          'TMPDIR = "${TOPDIR}/tmp-devtool-mc"\n', 'devtooltest')
+        self.add_command_to_tearDown('bitbake -c clean %s' % self.target)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        self._create_temp_layer(self.layerdir, True, 'devtoolmulticonfig')
+
+        self.recipedir = os.path.join(self.layerdir, 'recipes-test', self.recipe)
+        filesdir = os.path.join(self.recipedir, self.recipe)
+        os.makedirs(filesdir)
+        archive = 'devtool-test-subdir.tar.gz'
+        shutil.copyfile(os.path.join(get_test_layer(), 'recipes-test', 'devtool',
+                                     'devtool-test-subdir', archive),
+                        os.path.join(filesdir, archive))
+        with open(os.path.join(self.recipedir, self.recipe + '_1.0.bb'), 'w') as f:
+            f.write('LICENSE = "MIT"\n'
+                    'LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"\n'
+                    'inherit allarch\n'
+                    'INHIBIT_DEFAULT_DEPS = "1"\n'
+                    'COMPATIBLE_MACHINE = "devtool-mc"\n'
+                    'SRC_URI = "file://devtool-test-subdir.tar.gz"\n'
+                    'S = "${UNPACKDIR}/devtool-test-subdir"\n'
+                    'B = "${WORKDIR}/build"\n'
+                    'do_compile() {\n'
+                    '    cp ${S}/testfile ${B}/testfile\n'
+                    '}\n'
+                    'do_install() {\n'
+                    '    install -Dm0644 ${B}/testfile ${D}${datadir}/${BPN}/testfile\n'
+                    '}\n'
+                    'FILES:${PN} = "${datadir}/${BPN}"\n')
+
+    def _modify(self, wildcard=False):
+        runCmd('devtool modify %s %s %s' %
+               ('--wildcard' if wildcard else '', self.target, self.srctree))
+        self._check_src_repo(self.srctree)
+        append = os.path.join(self.workspacedir, 'appends', 'multiconfig',
+                              'devtooltest', self.recipe + ('_%.bbappend' if wildcard else '_1.0.bbappend'))
+        self.assertExists(append)
+        self.assertEqual(get_bb_var('EXTERNALSRC', self.target), self.srctree)
+        self.assertIn(self.recipe, runCmd('devtool status').output)
+        return append
+
+    def test_devtool_multiconfig_modify_build_reset(self):
+        """Extract, build and clean a recipe unavailable in the default config."""
+        result = bitbake('%s -n' % self.recipe, ignore_status=True)
+        self.assertNotEqual(result.status, 0)
+        self.assertIn('incompatible with machine', result.output)
+
+        append = self._modify()
+        with open(os.path.join(self.srctree, 'testfile'), 'a') as f:
+            f.write('\nmulticonfig workspace change\n')
+        runCmd('devtool build %s' % self.target)
+        builddir = get_bb_var('B', self.target)
+        with open(os.path.join(builddir, 'testfile')) as f:
+            self.assertIn('multiconfig workspace change', f.read())
+
+        # Do not use --no-clean: cleaning must retain the multiconfig prefix.
+        runCmd('devtool reset %s' % self.target)
+        self.assertNotExists(append)
+        self.assertNotIn(self.recipe, runCmd('devtool status').output)
+        self.assertFalse(get_bb_var('EXTERNALSRC', self.target))
+
+    def test_devtool_multiconfig_update_finish(self):
+        """Update and finish using qualified targets and bare workspace keys."""
+        append = self._modify()
+        for command, marker in [('update-recipe', 'update'), ('finish', 'finish')]:
+            with open(os.path.join(self.srctree, 'testfile'), 'a') as f:
+                f.write('\nmulticonfig %s change\n' % marker)
+            runCmd('git commit -am "Multiconfig %s change" '
+                   '-m "Upstream-Status: Inappropriate [selftest]"' % marker,
+                   cwd=self.srctree)
+            destination = self.layerdir if command == 'finish' else ''
+            runCmd('devtool %s %s %s' % (command, self.target, destination))
+            patches = glob.glob(os.path.join(self.recipedir, self.recipe, '*.patch'))
+            self.assertTrue(patches, 'No patches written by %s' % command)
+            contents = ''
+            for patch in patches:
+                with open(patch) as f:
+                    contents += f.read()
+            self.assertIn('multiconfig %s change' % marker, contents)
+            if command == 'update-recipe':
+                self.assertExists(append)
+                self.assertIn(self.recipe, runCmd('devtool status').output)
+
+        self.assertNotExists(append)
+        self.assertNotIn(self.recipe, runCmd('devtool status').output)
+        self.assertFalse(get_bb_var('EXTERNALSRC', self.target))
+        bitbake('%s -c patch' % self.target)
+        with open(os.path.join(get_bb_var('S', self.target), 'testfile')) as f:
+            contents = f.read()
+        self.assertIn('multiconfig update change', contents)
+        self.assertIn('multiconfig finish change', contents)
+
+    def test_devtool_multiconfig_workspace_isolation(self):
+        """A wildcard workspace append must affect only its configuration."""
+        self.append_config('COMPATIBLE_MACHINE:pn-%s = ".*"\n'
+                           'BBMULTICONFIG += "devtoolother"\n' % self.recipe)
+        self.write_config('TMPDIR = "${TOPDIR}/tmp-devtool-other"\n', 'devtoolother')
+        other_target = 'mc:devtoolother:%s' % self.recipe
+        append = self._modify(wildcard=True)
+        self.assertFalse(get_bb_var('EXTERNALSRC', self.recipe))
+        self.assertFalse(get_bb_var('EXTERNALSRC', other_target))
+
+        result = runCmd('devtool modify -n %s %s' % (self.recipe, self.srctree),
+                        ignore_status=True)
+        self.assertNotEqual(result.status, 0)
+        self.assertIn('already in your workspace', result.output)
+        self.assertExists(append)
+        runCmd('devtool reset %s' % self.target)
+        self.assertNotExists(append)
+
+        # The default configuration's append must likewise stay out of MCs.
+        runCmd('devtool modify -n %s %s' % (self.recipe, self.srctree))
+        self.assertEqual(get_bb_var('EXTERNALSRC', self.recipe), self.srctree)
+        self.assertFalse(get_bb_var('EXTERNALSRC', self.target))
+        self.assertFalse(get_bb_var('EXTERNALSRC', other_target))
+        runCmd('devtool reset %s' % self.recipe)
+
+
 class DevtoolUpdateTests(DevtoolBase):
 
     def test_devtool_update_recipe(self):

Re: [OE-core] [PATCH 0/5] devtool: Support multiconfig workflows and fix kernel extraction

[adrian.freihofer]

Hi Anusmita,

Thanks for working on this, multiconfig support in devtool is long
overdue (YOCTO #13933). Handling the mc: prefix when devtool calls
bitbake is clearly needed. Before going further, I'd like to agree on
the long term design of the workspace, as my series works on the same
problem from another angle.
https://git.openembedded.org/openembedded-core-contrib/log/?h=adrianf/ide-sdk-improvements

In most cases there is one recipe and one source tree, and what
differs is only what that source is built for: another multiconfig,
-native, nativesdk or a multilib variant such as lib32. Build output
is already separate for all of them: each multiconfig has its own
TMPDIR, and the BBCLASSEXTEND variants have their own WORKDIR. So the
question is mostly how the workspace records which variants are
modified, and which source tree they build.

Your series puts the appends in a directory per multiconfig
(appends/multiconfig/<mc>/) and selects them through BBFILES with
BB_CURRENT_MC. This is small and makes it explicit which
configuration is modified. The workspace is still keyed by the bare
PN though, so a recipe can only be modified in one configuration at a
time. This matches the current restriction to one BBCLASSEXTEND
variant per recipe.

My series keeps one bbappend per recipe file and scopes all settings
with pn-<pn>, so that e.g. foo, foo-native and lib32-foo can be in the
workspace at the same time. By default they share one source tree. A
variant gets its own tree only if its SRC_URI, SRCREV or S differ, or
if a variant builds in the source tree (B = S). The oe-workdir and
oe-logs symlinks in the shared tree are named per variant, e.g.
oe-workdir-native or oe-workdir-mc1. This does not cover BBMULTICONFIG
yet: PN is the same in every configuration, so pn- cannot tell them
apart, and something like a multiconfig specific override would be
needed in the shared bbappend.

The two approaches overlap but don't fit together well. With both
merged, devtool would have one mechanism for multiconfigs (separate
append files) and another one for BBCLASSEXTEND variants (scoped lines
in a shared append). read_workspace, reset, update-recipe/finish and
ide-sdk would have to handle both. Modifying e.g. foo-native and
mc:mc1:foo at the same time would combine both mechanisms.

So my question: what is the better long term design, also with
-native, nativesdk and multilib in mind? Separate workspace appends
per configuration or variant, or one append per recipe file with the
modified variants scoped inside it and one shared source tree?

Regards,
Adrian

Re: [PATCH 0/5] devtool: Support multiconfig workflows and fix kernel extraction

[Anusmita Dutta Mazumder]

[-- Attachment #1: Type: text/plain, Size: 576 bytes --]

Hi Adrian,

Thank you for the review. I agree that a common workspace mechanism for multiconfig and BBCLASSEXTEND variants makes sense. The shared-bbappend approach looks sensible to me, with settings scoped by both recipe variant and multiconfig.

I’d be happy to work on the multiconfig extension, building on the relevant shared-workspace changes already in your series and keeping the changes separately reviewable.

Would that be a suitable way forward? If so, which patches or version of your series would you recommend using as the base?

Regards,
Anusmita

[-- Attachment #2: Type: text/html, Size: 627 bytes --]